Can a VLM annotate a robot run? Results on a revised WGO-Bench
We re-annotated all 100 videos of WGO-Bench, human and robot, then measured how well a VLM pipeline segments and labels the actions in them. VideoDB reaches 28.93% semantic F1 at IoU 0.5, ahead of the public baseline at 19.92%, and the remaining errors show where the work goes next.
Consider a robot moving a cup from a counter to a shelf. The cup slips before it reaches the shelf. A researcher reviewing the run wants to find that attempt, look at the grasp, and compare it with transfers that worked. Human demonstrations of the same move would help too. Every one of those comparisons starts the same way: something has to find the action inside continuous video and describe it correctly.
That is the task measured here: action segmentation and labeling on human and robot video. Given a recording, produce the list of manipulation events in it, each with a start time, an end time, and a description a researcher could search for. If the boundary cuts away the setup, or the label names the wrong object, the record is less useful for understanding the run, even when it looks plausible.
For VideoDB this is one part of a larger direction. We want training runs and test runs to become searchable memory, so that a failure can be traced to the experience that explains it, and that experience can be used to improve the next model. Annotation is where that memory begins.
The result. On our revised WGO-Bench reference, VideoDB reaches 28.93% semantic F1 at IoU 0.5. The public baseline, MacroData Refiner, reaches 19.92% on the same 100 videos, a gap of 9.01 percentage points. The lead holds in all three domains of the benchmark. The errors that remain also say where the work needs to go next: more reliable descriptions, better boundaries, and a way to flag the records that need a human look.
Why annotation quality matters for robot learning
In the cup example, the final drop is only part of the evidence. How the cup was approached and held may explain what happened later. A useful annotation keeps that context while naming the object and the operation precisely enough to find comparable attempts.
This makes annotation a research problem in its own right. A boundary can cut away the setup. A label can name the wrong object. Either error makes the record less useful, even when the rest of the annotation looks fine.
The connection to robot learning is already concrete. Physical Intelligence’s π0.5 uses high-level behavior annotations in training and predicts a textual subtask before generating motor commands. Physical Intelligence, π0.5 SARM derives reward labels from natural-language subtask annotations to estimate stages and progress through demonstrations. SARM paper
Both uses give annotation quality a practical consequence. Our focus here is narrower: measure how accurate the descriptions and their time windows are, as a foundation for making recorded experience easier to inspect and reuse.
The benchmark: 100 videos, three viewpoints
We chose WGO-Bench because it covers both human egocentric activity and robot manipulation. The benchmark was introduced by MacroData with 100 episodes and 743 annotated segments. WGO-Bench
We reviewed all 100 videos across its three domains and produced a revised reference of 534 action segments. Task descriptions or segment annotations changed in 90 videos; 10 kept their original annotations.
| Domain | Viewpoint | Videos | Revised actions |
|---|---|---|---|
| DROID | External robot camera | 50 | 83 |
| Galaxea | Robot head camera | 25 | 54 |
| HomER | Human egocentric camera | 25 | 397 |
| Total | 100 | 534 |
These views test different things. DROID includes small or partly occluded manipulated objects. HomER has many more annotated events per episode: 15.88 on average in our revised reference, compared with 1.66 for DROID and 2.16 for Galaxea.
What counts as one action
An action can begin before contact. A gripper reaching for a power plug belongs to the pick-up event when the next frames confirm the connection. During re-annotation we used the surrounding video to decide which movement belonged to each manipulation. The principle was one completed manipulation event per segment, with preparatory movement included when the video supports the connection.
Reaching and grip adjustments stay inside the associated event; independent manipulations stay separate. Idle time, walking and camera movement can remain uncovered. The exported segments never overlap. Labels describe visible actions and objects in self-contained terms: looking ahead helped set the boundary, while the description stayed grounded in what actually occurred.
The revised reference still has mixed granularity. Some segments combine related operations, such as opening a pitcher, pouring, and putting it down. It also keeps some movement-only segments, such as moving toward a kitchen counter. The scores therefore measure agreement with this reference, including these annotation choices.
The benchmark’s original annotation discussion highlights ambiguous boundaries and limited visual evidence, and it also emphasizes completed manipulation events and object-state changes. MacroData’s annotation discussion
These choices decide what the score rewards. Including the approach preserves evidence about how an interaction began. Grouping several operations changes the level of detail a prediction must match. The reference defines the kind of event being evaluated, as well as its accepted timestamps.
How an annotation is scored
Every predicted segment goes through two checks. First, temporal matching: predictions are matched one-to-one with reference segments, in order, and a pair counts when its intersection over union reaches the threshold, 0.5 in the main setting and 0.75 in the strict one. IoU is the shared duration divided by the total duration the two segments cover. Segment F1 summarizes this step. Second, a label judge: GPT-5.5 reads the task description, the reference label and the predicted label, and decides whether the prediction describes the same action. It does not see the video. Label accuracy is the share of temporal matches whose label is accepted. Semantic F1 requires both a qualifying window and an accepted label, and it is our main measure.
Two details of the protocol matter when reading the numbers. Scores pool action counts across episodes rather than averaging per episode. Where eligible, the first predicted action’s start and the last predicted action’s end are aligned to the corresponding reference boundaries before scoring. The judge’s verdicts are a proxy for semantic correctness and can themselves be wrong.
Results on all 100 videos
Both systems were scored against the same revised reference. We compare the selected VideoDB configuration with the public MacroData Refiner 0.3.7 baseline, both running on Gemini 3.7 Flash. The 100-video set includes 15 videos used for configuration selection; the other 85 were evaluated with the VideoDB configuration fixed. The full set is therefore not a fully held-out test.
VideoDB leads by 9.01 percentage points in semantic F1 and by 14.17 points in label accuracy at IoU 0.5.
All 100 videos · IoU ≥ 0.5 · Higher is better
| Metric | MacroData Refiner | VideoDB |
|---|---|---|
| Semantic F1 | 19.92 | 28.93 (+9.01) |
| Label accuracy | 34.84 | 49.01 (+14.17) |
| Segment F1 | 57.17 | 59.03 (+1.86) |
All 100 videos · IoU ≥ 0.75 · Higher is better
| Metric | MacroData Refiner | VideoDB |
|---|---|---|
| Semantic F1 | 12.75 | 18.06 (+5.31) |
| Label accuracy | 42.67 | 52.84 (+10.17) |
| Segment F1 | 29.88 | 34.17 (+4.29) |
Scores are percentages. Parentheses show VideoDB’s absolute difference from the baseline in percentage points, calculated before rounding. All tables compare the selected results on the same 100 videos.
The counts behind the scores explain the lead. VideoDB produced 304 temporal matches against 287 for the baseline. Of those, 149 VideoDB labels passed the judge, compared with 100. The larger gap is in matches that also carry an accepted description, which is the combination a researcher needs when using an annotation to find and inspect an event.
A stricter IoU threshold changes which predictions qualify. The higher label accuracy at 0.75 reflects a smaller matched set with a larger share of correct labels.
By domain: a lead everywhere, and a timing gap on HomER
VideoDB has the higher semantic F1 on DROID, Galaxea and HomER. The largest gain is on Galaxea, at 11.76 percentage points.
Semantic F1 by domain · IoU ≥ 0.5
| Domain | MacroData Refiner | VideoDB |
|---|---|---|
| DROID | 14.88 | 25.58 (+10.70) |
| Galaxea | 39.06 | 50.82 (+11.76) |
| HomER | 17.85 | 26.09 (+8.24) |
The supporting metrics show different strengths. On Galaxea, VideoDB’s segment F1 is 81.97 against 64.06 (+17.90 points). On DROID it is 77.91 against 62.33 (+15.58 points). HomER is a temporal weakness: segment F1 is 50.82 against 54.16 (−3.35 points). Label accuracy on HomER is higher, at 51.34 against 32.96 (+18.38 points), and that is what produces the stronger semantic F1.
The domain results set the next priorities. HomER needs better temporal matching, even though its labels are ahead. DROID needs more reliable descriptions: its temporal scores are high and its semantic scores are not.
Where the errors are
The best results still leave a lot of room. At IoU 0.5, 155 of VideoDB’s 304 temporal matches failed the label check. The baseline had 187 such failures among 287 matches.
DROID makes the gap clearest. VideoDB’s segment F1 reaches 77.91, while label accuracy among temporal matches is 32.84. The scores establish a semantic weakness; the specific causes need individual clips.
For a researcher using these records, confidence matters at the level of a single event. An aggregate score cannot say whether one description is reliable. Flagging the annotations that need review is therefore a practical next goal, with the source video at hand to check the description and the boundary together.
The step after that is to connect events across recordings. Attempts involving the same object and operation should be retrievable despite changes in camera position or execution, so that the conditions around a failure can be compared with the conditions around a success.
These are directions, not results. This benchmark measures annotation quality; downstream gains in robot learning remain to be tested.
Annotation has to work across a full shift
Real recordings often span a worker’s whole shift or hours of a robot in a warehouse. Finding a failed grasp or a repeated mistake means locating brief events inside that continuous history.
The public baseline sends all sampled contact sheets from an episode in one model request. As recordings grow, more visual evidence has to fit into that one request, so input size becomes a scaling constraint. Its authors note the same limit: their experiments with separate calls introduced artificial action boundaries, and they identify decomposition as important for longer recordings. MacroData’s decomposition experiments VideoDB annotates videos of any duration, so full recordings from real work and robot deployments are processed whole, and an event can be examined in the context of the work that led up to it.
Every run should make the next run better
The longer-term goal is to connect annotation, retrieval and model development. In the cup example, a researcher should be able to find similar failed transfers and compare them with successful ones. The relevant evidence might span robot test runs and human demonstrations recorded under different conditions.
Those comparisons should show what is missing. A pattern of failures with one cup shape might point to a gap in object coverage. Failed recovery attempts might point to a need for more recovery demonstrations. These are hypotheses to check against the data, with each retrieved event linked to the video that supports it.
Eventually we want the system to recommend what experience to collect next. Researchers could use those recommendations to gather data, train the next model, and test whether the same failures become less frequent. The new runs would then join the memory available for the next investigation.
WGO-Bench measures one early part of this process: producing action records that agree with a human reference. Building a memory that improves models will need tests of retrieval, data selection, and the effect of new experience on later runs. That is the direction this work is meant to support.
Every run should make the next run better.