Surgical Annotation Schema for Robotics: Fields That Matter

A surgical annotation schema for robotics is the shared labeling structure that turns raw procedure footage into a dataset a robotics team can train on, evaluate against, or license without rework. The fields that matter are the ones that map directly to a downstream task: phase, instrument, hand activity, object interaction, error and recovery, and a small set of frame- and clip-level context fields that preserve provenance. A schema that is too generic forces the buyer to re-label; a schema that is too dense creates label drift and expensive cleanup. The right level is task-aligned, operationally documented, and stable across the dataset.

For Simovian Intelligence, the schema is where structured surgical egocentric data and surgical dexterity datasets earn their keep. The capture modality is the easy part; the schema is what makes the data usable for imitation learning, phase recognition, instrument tracking, and workflow modeling on a physical AI or embodied AI team.

Why the schema, not the video, is the leverage point

Robotics and embodied AI teams do not buy video. They buy a labeled representation of behavior that can be sliced, sampled, and routed into a training or evaluation loop. A schema is the contract between the capture side and the model side. If the contract is loose, the team ends up re-labeling internally before any model sees the data. If the contract is too rigid, the vendor cannot adapt it to a new procedure or a new instrument, and the buyer is stuck with a frozen dataset that ages out as their task definition evolves.

The most useful schemas are stable at the field level and flexible at the value level. Field names like phase, instrument_id, hand_role, grip_type, object_state, and event do not need to change when the procedure does; the controlled vocabularies underneath them do.

Fields a buyer should expect

The list below is not a universal standard. It is a working set we use at Simovian when packaging structured surgical egocentric data and surgical dexterity datasets for robotics and physical AI buyers. Buyers should treat it as a reference contract: the field names, the level of granularity, and the controlled vocabularies are negotiable; the absence of these categories is not.

Clip- and session-level fields. procedure_id, capture_session_id, operator_role, viewpoint (egocentric, overhead, room), device_class (headset model, glove, room camera), consent_scope, and deidentification_profile. These are the fields that make a dataset auditable. Without them, procurement and legal cannot sign off, and the dataset gets stuck in review even if the labels are excellent.

Frame- and span-level fields. phase, step, and substep give the team a segmentation backbone. instrument_id and instrument_state (inserted, engaged, withdrawn, exchanged) track what tool is doing what. hand_role (dominant, assisting, off-field) and grip_type (precision, power, pinch, tripodal) capture the dexterity signal that surgical egocentric data is uniquely positioned to provide.

Interaction fields. object_state and contact_event capture what the instruments are touching and when contact begins and ends. event with a controlled vocabulary — for example, knot_tied, suture_cut, instrument_drop, camera_adjust, conversion_to_open — is where procedural exceptions live. This is the layer that most schema designs under-invest in, and it is the layer that determines whether a dataset supports realistic evaluation rather than only average-case training.

Quality and provenance fields. annotator_id, annotation_confidence, double_pass (a boolean for clips that were labeled twice for agreement), and notes. The buyer does not need to see annotator identity, but the team does need to be able to audit disagreement and route low-confidence clips to a second pass. Inter-annotator agreement on a small gold set is a stronger signal of dataset quality than any aggregate metric.

What the schema does not do

A schema is not ground truth. It is a structured representation that has been validated against the data, but it can still encode the wrong convention, the wrong phase boundaries, or the wrong instrument taxonomy. Buyers should plan for a short validation pass on a held-out subset before signing off on a large package.

A schema is also not a model. Field design choices — for example, whether event is a flat vocabulary or a small ontology — change the kinds of models a buyer can train. The schema should be selected with the target model family in mind, not chosen for aesthetics.

Finally, a schema is not a compliance artifact on its own. De-identification, consent, and licensing live in provenance fields and in the dataset's separate governance documentation. Conflating annotation structure with governance is one of the most common reasons surgical datasets stall in procurement review.

How to evaluate a vendor's schema in practice

When we work with a robotics or physical AI team on a structured surgical egocentric dataset or a surgical dexterity dataset, the first conversation is about the schema, not the video. Three questions tend to surface the real state of the dataset:

1. Which fields are derived from the procedure itself, and which are derived from a separate annotation pass? The answer tells you where the labeling budget actually went.
2. What is the unit of annotation — frame, span, clip, or session — for each field? Mixed units are a smell. A clean schema is consistent about the unit it expects.
3. What does disagreement look like? A vendor that can show inter-annotator agreement on a small gold set, and can describe the adjudication process, is a vendor you can scale with. A vendor that can only show aggregate label counts is not.

Where this fits in a buyer's data strategy

For a team building a manipulation policy, a planning model, or a phase-recognition benchmark, the schema determines how much of the dataset is reusable across model iterations. A good schema makes the data composable: phase labels can be used for segmentation, instrument labels for tracking, event labels for evaluation, and the clip-level provenance fields for governance. A bad schema forces the team to relabel the same video three times for three different projects.

Simovian's wedge is structured surgical egocentric data and surgical dexterity datasets for robotics, embodied AI, and physical AI teams. The schema is the part of the product a buyer can read before the first evaluation clip plays. Getting it right is the difference between a dataset the team uses for two experiments and a dataset the team builds on for a year.

If you are evaluating a surgical annotation schema against your own task definition, our working-with-us page is the fastest way to compare notes.

← Back to Resources