A query engine for robot video

Most robotics labs we work with keep more video than they can look at. Every teleoperation session, every test run on the robot and, increasingly, every rollout from a world model gets recorded. The recordings go into a bucket and stay there. When a researcher wants to know how often the gripper slips on a particular part, or what the new policy does on a cluttered bench, someone has to go and find out, and finding out means writing a pipeline.

This post is about the tool we built to change that. It is a query engine that sits on top of a lab’s existing video archive, keeps every episode whole, and answers questions in plain English with the moments themselves. We describe the problem as we found it, what we built, and how the pieces connect into a training loop.

The pipeline problem

The usual way to learn something from an archive of robot video is to build a pipeline for the question. Collect the recordings, cut them into clips, embed or label the clips, then train or measure. The cut is decided up front, before anyone knows what the next question will be. When the next experiment asks something different, the pipeline is rebuilt and the whole archive is processed again. Compute is spent per experiment rather than per hour of video, and every question waits for engineering before it gets an answer.

The archives are large. Dyna Robotics recently described training Dyna-2 on more than a million hours of first-person video, and that is roughly one model iteration. Finding one moment in that much video means locating the right files, decoding them, cutting and re-encoding. It is a ticket, not a query.

Test runs make this worse. In most labs they land in a separate bucket, unsegmented, and never join the training data. What the model failed on yesterday is not searchable today, and the researcher who could learn the most from those failures is the one least likely to see them.

One archive, kept whole

The first decision was to stop cutting. We put every recording in one place: first-person capture from human demonstrations, real test runs on the robot, and dream runs from a world model. Nothing is pre-cut. Each episode is stored whole, with what we know about it: the task, the environment, the outcome, the model version, and whether it is real, simulated or generated.

Then we index it. Any model can run as an analyzer over the archive: ours, an open-weight model, a proprietary one, or the model you trained last week. It runs over every episode on distributed pipelines and writes down what happens, when, and with what result. The index is versioned. When a better model arrives, we re-read the archive instead of re-collecting it.

The files never leave your bucket. VideoDB sits on top as a query engine over your object store, the way a SQL engine sits over files in a data lake. It runs in our cloud paired with your storage, or entirely inside your own account.

Asking a question

A question is written in plain English: “show every failed insertion on a cluttered bench.” The answer is not a list of file names. It is the moments themselves, playing, about half a second after you ask.

Because nothing was cut in advance, the question decides the window. A slip needs about five seconds of context. A regrasp needs ninety. A question about a whole shift needs twenty minutes. The window is chosen at query time, not at ingestion, and the same episode can answer all three.

Every result keeps its links: the episode id, the timestamp, the camera, the model version, and the actions and states in your own MCAP or LeRobot files. Those files remain the source of truth for what the robot did. We take the video part, which is the part nobody could search.

In practice, six kinds of question cover most of what researchers ask us, and all six run on the same index.

01FindA moment, described in plain English. The evidence plays.

02CountHow often it happens, across the archive or a slice of it.

03FilterBy task, outcome, camera, environment or model version.

04SampleA balanced set across a condition, ready to export.

05Look closerWiden the window, switch the camera, step through frames.

06WatchThe same question on a schedule, as new runs land.

You can ask these yourself, or hand them to an agent. An agent searches, looks at the frames, widens the window when the evidence is incomplete, and comes back with the patterns it found and the clips behind them. We supply the search and the access to the frames. You choose the agent’s reasoning model and its workflow.

From an answer to a training set

The hits from one question are a cohort: every episode where the model failed in the same way. We filter the cohort, balance it across a condition, and export a manifest. Each row carries the source id, the time range, the episode, the reason it was selected, and the version of the index it came from. The manifest is reproducible, so the same query on the same index version gives the same training set. The next model is trained on it.

The loop closes

Then we test the new model. Its runs, real and dream, land in the same archive with their outcomes, and they are searchable the moment they arrive. The new model’s failures become the first question of the next iteration. The archive gets more useful with every run, because every run can be asked about as soon as it lands.

The same archive also answers a question labs rarely get to ask: what is missing? A collection campaign can be drafted from the real gaps in the archive, so a team records or buys only what it does not already have.

What we need from you

Three things.

• Where the video lives: the bucket, the path and the format. We read MP4, MCAP, LeRobot, RLDS and RTSP.
• The metadata you already have: episode id, camera, timestamp, task, environment, outcome, model version and source type.
• Which of your own models, if any, should run as analyzers.

How to start

The quickest way to try it is to upload a few sample videos, give your VideoDB API key to Claude or Codex, and ask it to annotate them.

For a lab, the fastest way to see whether this fits your pipeline is thirty minutes on a live demo with your own recordings, or with an open first-person dataset that looks like yours. From there, a pilot runs on about a hundred hours of your recordings.

We have run this at 100,000 hours for one of the largest video data providers, and we can show you how it improves your model iterations.