Download sequence data

For either video or frame annotations, you can download all the data you will need by navigating into the batch that you want to work with and clicking on the “Download” button at the top right of your screen. This will download a JSON file that you can use within your downstream scripts which need to use the annotations that you have done in the platform.

The JSON file has the same schema for both video and frame annotation batches, but to use each of them you will need to access specific fields. In this guide we will not cover all the fields that are available in the JSON file, only those that are necessary for the most basic functionality. You can explore the JSON schema yourself, or request help from our support team if you would like to learn more about the fields that are available.

Working with video annotations

Assuming that you have done timestamp annotations, the JSON file will contain a dictionary with the following fields:

{
  ...
  "sequences": [
    # First sequence
    {
      ...
      "videoAnnotations": [
        # First annotation
        {
          "timestampStart": 1000,
          "timestampEnd": 2000,
          "label": {
            "code": "label_code",
            "displayName": "Label display name",
            ...
          }
        },
        # Second annotation
        {...},
        ...
      ]
    },
    # Second sequence
    {...},
    ...
  ]
}

Assuming that you have used python to read the JSON file:

import json

with open("file_path.json", "r") as json_file:
  annotation_data = json.load(json_file)

annotation_jobs = annotation_data[“sequences”]
n  = 0
nth_sequence = annotation_jobs[n]
video_annotations = nth_sequence["videoAnnotations"]

Then the explanation for the fields you will need is:

annotation_data[“sequences”] is a list where each item contains all the data from a given sequence (annotation job).
annotation_jobs[n] is a dictionary which contains all the data from the nth annotation sequence.
nth_sequence["videoAnnotations"] is a list of dictionaries where each entry which will contain all the data for each of the annotations that have been done on that particular sequence.
Within video_annotations[n] the fields that you will be most interested in are:
- video_annotations[n][“timestampStart”] contains the start time of the annotation in milliseconds
- video_annotations[n][“timestampEnd”] contains the end time of the annotation in milliseconds
- video_annotations[n][“label”][“code”] contains the name of the label

With these you should be able to iterate over all the annotations that have been done on every sequence of the video batch in question.

Working with frame annotations

For frame annotations, the JSON file will be very similar to that of the video annotations, except that instead of “videoAnnotations” each sequence will have a field called “featureSet” which will contain all the frames in the sequence and all the annotations in each frame:

{
  ...
  "sequences": [
    # First sequence
    {
      ...
      "featureSet": [
        # First frame
        {
          "framePath": "s3_path_to_frame.png",
          "labels": [
            #Label 1
            {
              "code": "label_code",
              "shapes": [
                # Annotation 1
                {
                  "type": "shape_type",
                  "points": [
                    {
                      "x": 0.031,
                      "y": 0.088,
                    },
                    {...},
                    ...
                  ]
                  ...
                },
                # Annotation 2
                {...},
                ...
              ],
            },
            #Label 2
            {...},
            ...
          ]
          ...
        },
        # Second frame
        {...},
        ...
      ],
    },
    # Second sequence
    {...},
    ...
  ]
}

To read the JSON file, you can use the following boilerplate code:

import json

with open("file_path.json", "r") as json_file:
  annotation_data = json.load(json_file)

annotation_jobs = annotation_data[“sequences”]
# Sequence index
n  = 0
# Frame index
f = 0
# Label index
l = 0
# Shape index
s = 0
nth_sequence = annotation_jobs[n]
frame_annotations = nth_sequence["featureSet"][f]
lth_annotation = frame_annotations["labels"][l]
shape_type = lth_annotation["shapes"][s]["shape_type"]
xy_coordinates = [(point["x"], point["y"]) for point in lth_annotation["shapes"][s]["points"]]

Then the explanation for the fields you will need is:

annotation_data["sequences"] is a list where each item contains all the data from a given sequence (annotation job).
annotation_jobs[n] is a dictionary which contains all the data from the nth annotation sequence.
nth_sequence["featureSet"] is a list of dictionaries where each entry in the list which will contain all the data for all the annotations done in each frame of this particular sequence.
frame_annotations["labels"] is a list of dictionaries where each entry in the list is one annotation done in the frame in question.
lth_annotation["shapes"] is a list of dictionaries where each entry will be one shape associated with the annotation in question. Each shape will have a type and a list of “points”, the way to interpret the points depends on the shape type:
- For "close_polygon" the xy coordinates are the outline of the polygon that was annotated
- For "rectangle" the xy coordinates will be two corners which define the rectangle
- For "open_polygon" the xy coordinates will be the points that join the lines
- For "keypoint" the xy coordinates will be the coordinates of the point that was annotated

How to interpret XY coordinates

The x and y coordinates for each point in an annotation are a value between 0 and 1 where:

(x,y) = (0,0) is the top left corner of the image
(x,y) = (1,0) is the top right corner of the image
(x,y) = (0,1) is the bottom left corner of the image
(x,y) = (1,1) is the bottom right corner of the image

The coordinates are normalised so that you can find the annotations regardless of changes in the resolution of the frames. The image below shows some examples of where some points can be located.

Download frames

To work with frame annotations you will also need to download the frames that you can see in the SAVANT UI. These frames are securely stored in an S3 location and you will need to use SAVANT’s API to download them. To do so, you will need the same JSON file as before, you can then make a request using the following script:

"""Boiler plate code to read a single frame from an annotated sequence in SAVANT

NOTE: This script reads a single frame, you will need to adapt it to read all the
frames that you need to download. You will also need to add logic to catch API error
responses and manage them appropriately.
"""

import json
import requests
import urllib

SAVANT_USERNAME = ""  # Your email login for SAVANT
SAVANT_PASSWORD = ""  # Your password for SAVANT

SEQUENCES_JSON_FILE = ".json"  # The file you downloaded from SAVANT
FRAME_OUTPUT_FILE = (
    ".png"  # Where to save the frame that will be read (frames are in PNG format)
)

SAVANT_DOMAIN = "https://ext-production-savant.mdtsurgicalai.com"


def authenticate(domain: str, username: str, password: str) -> str:
    """Authenticate with the SAVANT API"""
    admin_token_res = requests.post(
        f"{domain}/auth/jwt/login",
        data={
            "username": username,
            "password": password,
        },
    )

    return admin_token_res.json()["access_token"]

def get_frame_from_uuid(
    sequence_root_id: str,
    frame_id: str,
    workspace_id: str,
    token: str,
    domain: str,
):
    """Return a frame dictionary given a workspace, sequence and frame IDs"""
    frames_response = requests.get(
        f"{domain}/sequences/{sequence_root_id}/frames/{frame_id}",
        params={
            "workspace_id": workspace_id,
            "presigned_url": True,
        },
        headers={"Authorization": f"bearer {token}"},
    )
    frame = frames_response.json()
    return frame

with open(SEQUENCES_JSON_FILE, "r") as json_file:
    sequences_json = json.load(json_file)

# Read the first sequence in the list. You can iterate over sequences_json["sequences"]
# if you want to get all the sequences.
sample_sequence = sequences_json["sequences"][0]
sequence_id = sample_sequence["rootId"]
workspace_id = sample_sequence["workspaceId"]

# Read the first frame in the sequence. You can iterate over
# sample_sequence["featureSet"] to get all the frames in the sequence.
frame_id = sample_sequence["featureSet"][0]["id"]

token = authenticate(
    domain=SAVANT_DOMAIN,
    username=SAVANT_USERNAME,
    password=SAVANT_PASSWORD,
)

frame = get_frame_from_uuid(
    sequence_root_id=sequence_id,
    frame_id=frame_id,
    workspace_id=workspace_id,
    token=token,
    domain=SAVANT_DOMAIN,
)

# You will need to put some logic here to check that the API response (frame) was OK
# and retry if it was not

# Get the pre-signed URL from the response
presigned_url = frame["frame"]["frame_path"]

# Save the frame into your local storage
urllib.request.urlretrieve(presigned_url, FRAME_OUTPUT_FILE)

What next?

With this you have now completed the basic usage of SAVANT. You should now have all you need to complete most of your annotation projects. If you are interested in more advanced functionality, we recommend that you move to SAVANT’s advanced functionality guides. If there is something you would like to do but can’t find a guide for it feel free to contact us.

Welcome to SAVANT

Annotation profiles

Management of annotation projects

How to create frame annotation jobs

Frame annotation interface