What is a Gold Standard?
A Gold Standard (GS) is an assessment used to verify that an annotator can annotate to the required quality before they are allowed to work on a live batch of annotations.
You can create a test made up of sequences that have already been annotated correctly. Those correct annotations become the ground truth. An annotator is then given a blank copy of the same sequences and annotates them from scratch. Their work is automatically compared with the ground truth and scored for precision and recall. The annotator passes or fails based on the pass marks you configure.
A Gold Standard is always tied to a Profile (the label/annotation schema). This is what links a test to real work: a project that requires Gold Standard will look for a Gold Standard project using the same profile, so annotators are tested on exactly the schema they will be annotating with.
Key terms
Term | Meaning |
Gold Standard project | A container for tests, tied to one root profile, holding the pass mark configuration. |
Gold Standard test (batch) | An individual test containing a set of sequences that an annotator must complete. The test is tied to a specific Profile ID. |
Profile | The annotation schema; the link between a GS test and live work. |
Ground truth | The pre-existing correct annotations the annotator is scored against. |
Access granted | The pass/fail outcome of an attempt. |
Create a Gold Standard project
In the main navigation, select Training.
Select Create Gold Standard Project.
Complete Step 1: Project setup:
Project Name: Enter a name for the Gold Standard project.
Profile: Select the profile that the assessment will use. This must match the profile used by the live project whose annotators you want to assess.
Select Next.
Complete Step 2: Configuration by setting the required pass marks. For more information, see Configure pass marks.
Select Create.
Note: The Create button remains unavailable until you enter a project name, select a profile, and provide a valid configuration.
You can filter the Gold Standards list by:
Project Name
Profile ID
Batch Description
Batch ID
Video ID
Create a Gold Standard test
A Gold Standard test is created using previously annotated sequences from a live batch.
Open the Gold Standard project.
Select Add Batch.
Select a project: Select the live project that contains the sequences you want to use. The list is filtered to projects that match the profile of the Gold Standard project.
Select a batch: Select the live batch. Available batches are also filtered by profile to ensure that the schemas match.
Select sequences: Select the sequences to include in the test.
The table displays:
Sequence No.
Step
Step Type
Frames
You can filter the table by:
Sequence No.
Root ID
Video ID
Status
Confirm your selections to create the test.
Choose appropriate sequences
Use sequences that are fully annotated and complete.
The existing annotations become the ground truth for the assessment. Errors in the source annotations may therefore result in an annotator being scored incorrectly.
The Gold Standard project page displays information about each test, including:
Project Name
Gold Standard Test
Batch Type
Profile Name
Profile Version
Annotation Step
Completed
Date Created
Add sequences to an existing test
You can add sequences after a Gold Standard test has been created. This can be useful if you need to make the assessment longer or cover additional scenarios.
Open the Gold Standard test.
Select Add Sequences.
Complete the following steps:
Select project
Select batch
Select sequences
Confirm your selections.
The available projects and batches are filtered to match the profile used by the existing test. This ensures that all sequences within a test use the same annotation schema.
You can add sequences from more than one source batch.
Caution: Adding sequences changes the assessment for users who take it afterward. Existing results are not recalculated. If you make significant changes to a test, consider treating it as a new version and retesting affected annotators.
Assign users to a test
Gold Standard tests use a multi-assignment workflow. Instead of dividing sequences among users, every assigned user receives every sequence in the test.
This ensures that each user completes the same assessment.
To assign users:
Open the Gold Standard test.
Select Assign Users.
In the Assign Users to all Gold Standard Sequences dialog:
Select the status/step.
Select one or more users from the Assignees field.
Select Save.
The Assignees field becomes available after you select a status/step and displays only users who have access to that step.
Assignment requirements and limitations
Users must be members of the Gold Standard project before you can assign them. Add users through Members / Teams in the project settings.
You cannot reassign an individual sequence within a Gold Standard test.
If you attempt to reassign an individual sequence, SAVANT displays the message: "Cannot reassign sequences in a gold standard batch."
Assignment applies to the complete test so that every assigned user receives the same sequences.
Annotators can find their assigned assessment in their work list and complete it using the video or frame editor.
Configure pass marks
The pass-mark configuration is set at the Gold Standard project level. All tests within the project therefore use the same configuration.
You can set the configuration when you create the project or update it later from:
Project Settings > Gold Standard Configuration
Overall pass marks
Setting | Range | Default | Description |
Overall Pass Mark (0 to 100) | 0–100 | 80 | The percentage of sequences that must meet the configured precision or recall pass mark for the annotator to pass the assessment. |
Overall Precision (0 to 100) | 0–100 | 50 | The percentage of predicted positives that are true positives. |
Overall Recall (0 to 100) | 0–100 | 0 | The percentage of actual positives that the annotator correctly identifies. |
Months Valid For | Minimum 1 | 12 | The number of months that a successful assessment remains valid before it expires. |
Validation: At least one of Overall Precision or Overall Recall must be greater than 0. They cannot both be 0 or blank.
How the pass marks work together
Precision and recall are assessed for each sequence. The Overall Pass Mark determines how many sequences must meet those requirements for the annotator to pass the complete assessment.
For example, with:
Overall Pass Mark: 80
Overall Precision: 50
the annotator must achieve at least 50% precision on at least 80% of the sequences in the test.
Optional pass marks
Optional pass marks are enabled individually and apply only when their corresponding checkbox is selected.
Frame pass mark
Enable Frame Pass Mark for frame- or shape-based annotations.
Frame IOU Pass Mark (0 to 100): Default 80. Defines the minimum Intersection over Union between the annotator's shape and the ground-truth shape for the annotation to count as a match.
Frame Euclidean Distance (px): Default 80. Defines the maximum allowed pixel distance between the annotator's annotation and the ground truth.
Tag pass mark
Enable Tag Pass Mark for tagging or classification assessments.
Tag Similarity Pass Mark (0 to 100): Default 80. Defines the minimum tag-similarity score required.
Video pass mark
Enable Video Pass Mark for video or temporal annotations.
For Video Pass Parameter, select one of:
Start Difference
End Difference
Total Difference
Enter a value greater than 0 for the selected parameter. Only one of the three parameters can be set.
Update the Gold Standard configuration
To change an existing configuration:
Go to Project Settings > Gold Standard Configuration.
Update the required fields.
Select Update Configuration.
Confirm the change in the warning dialog.
SAVANT warns that updating the configuration may affect existing assessments and user results.
Where possible, change pass marks between assessment cycles rather than during an active cycle.
The Update Configuration button is unavailable when:
You do not have permission to update the configuration.
The configuration is invalid.
You have not made any changes.
How Gold Standards link to live projects
Gold Standard assessments are linked to live projects through their profiles rather than by linking directly to a project.
A Gold Standard project stores a Profile ID, which identifies a specific version of a profile.
A live batch contains:
A Profile ID, which identifies the specific profile version.
A Profile Root ID, which identifies the profile lineage across versions.
SAVANT identifies the appropriate Gold Standard project by matching its Profile Root ID with the live batch's Profile Root ID.
Within that project, SAVANT then identifies the Gold Standard test whose Profile ID matches the live batch's Profile ID.
What this means in practice
One Gold Standard project per profile: You can create only one Gold Standard project for each profile.
When Gold Standard Assessment required is enabled for a live project, annotators see a "Gold Standard Assessment required" notice with a link directly to the relevant assessment.
Sequence selection is filtered by profile throughout the workflow, preventing a test from containing sequences that use different schemas.
Each result records the Profile ID, Project ID, and Batch ID used for the assessment. This identifies the schema version associated with each result.
Find Profile and Project IDs
Go to:
Gold Standard Project Settings > Project Information
This page displays:
Name
Project ID
Profile
Profile ID
Each value includes a copy button.
You can use these IDs when confirming how a Gold Standard is linked or when raising a support request.
View assessment results
Open the Gold Standard test.
Select the Results tab.
Users with permission to read all sequences can view results for every candidate. Annotators can view only their own results.
The user list displays the number of tests each user has passed and failed.
Select a user to view their assessment attempts. Attempts are displayed newest first.
Each attempt includes:
Version No.: The attempt number.
Overall Achieved: The overall score as a percentage.
Time Taken At: The timestamp in UTC.
Expires: The Time Taken At date plus the configured Months Valid For.
Access granted: A green check mark indicates a pass and a red X indicates a fail.
Expand an attempt to view results for each sequence, including:
Group ID
Precision Result
Recall Result
Config Results
Select a sequence to open it in the annotation viewer. You can then review the annotator's submission and identify where marks were lost.
Retest an annotator
Multiple assessment attempts are supported.
Each attempt is stored separately with its own:
Version number
Timestamp
Pass or fail result
Previous attempts are not overwritten, providing a record of the annotator's assessment history.
When to retest
Consider retesting when:
The annotator failed the assessment and has since received additional coaching.
The annotator's successful assessment has expired.
The profile has been updated to a new version and the annotator needs to complete an assessment using the new schema.
The pass-mark configuration has been increased.
Retest a user
Reassign the user to the Gold Standard test by following the steps in Assign users to a test.
A new assessment attempt is created. Previous results remain available on the Results tab.
Good practice
Use sequences that have passed QA. Errors in the ground truth can result in incorrect assessment outcomes.
Include enough sequences to make the assessment meaningful. Because the Overall Pass Mark is based on the percentage of sequences passed, each sequence in a three-sequence test represents approximately 33% of the overall result.
The default configuration is Overall Pass Mark: 80, Overall Precision: 50, Overall Recall: 0, and Months Valid For: 12 months. Adjust these values as appropriate when you have sufficient assessment results to evaluate the configuration.
Enable only the frame, tag, and video pass marks that apply to the annotation type being assessed.
Set Months Valid For to align with your quality-assurance cycle.
Review failed assessments in the annotation viewer and use the results to inform coaching before retesting the annotator.