# Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence

> VideoDB Technical Report · July 2026 · Sankalp Nagaonkar, Rohit Garg, Ankit Raj, Ashish Choithani, Ashutosh Trivedi

[Read the PDF](https://labs.videodb.io/papers/search-over-the-visual-world.pdf)

---

## Abstract

Most video-retrieval systems assume a bounded corpus and return ranked files or timestamps. Agents operating over cameras, screens, streams, and archives face a different systems problem: observations arrive continuously; different models interpret them at different temporal granularities; useful context must be selected without replaying the complete visual record; and a result must remain connected to inspectable source evidence. We argue that search over such a corpus is an infrastructure problem that cannot be reduced to ranking video files. We develop a conceptual and formal model of search over the visual world built on analyzer-defined scenes, persistent understanding artifacts, visual memory as coexisting scene spaces over shared source time, and capability-declared indexes; we draw a strict distinction between memory (everything retained), context (what is selected for a task), and evidence (the source intervals that ground it). The VideoDB data format (VDB) realizes this conceptual model in production, and a typed search surface exposes it: planned retrieval, a bounded stateful investigation mode, direct semantic, structured, and aggregate access, and grounded synthesis. We contrast this model-agnostic infrastructure (segmentation, sampling, model choice, embeddings, and ranking all exposed as system decisions, with live streams as first-class sources) with video-native foundation models offered as fixed APIs. In a complete-system semantic-retrieval comparison against a commercial video-native retrieval engine spanning 9,800+ natural-language queries over four public datasets, a pipeline of general-purpose components, none trained end-to-end for video retrieval, achieves higher macro-averaged Recall@1/@3/@10 (73.09/83.39/91.20 versus 65.75/77.13/89.10), while the baseline is higher at Recall@50 (96.42 versus 96.07). The results suggest that, today, retrieval quality over the visual world is governed more by system design than by video-specific pretraining, and that visual-memory infrastructure can deliver it while keeping source-grounded, playable evidence a first-class system object.

## Headline results (Table 9, complete-system comparison, 9,834 queries)

| Dataset | System | R@1 | R@3 | R@10 | R@50 |
|---|---|---|---|---|---|
| MSVD | VideoDB | 70.10 | 80.73 | 89.17 | 96.39 |
| MSVD | TwelveLabs | 67.86 | 78.22 | 89.48 | 97.10 |
| YouCook2 | VideoDB | 65.96 | 80.98 | 93.48 | 97.87 |
| YouCook2 | TwelveLabs | 47.07 | 65.56 | 84.18 | 96.41 |
| VATEX | VideoDB | 83.46 | 90.28 | 95.24 | 97.77 |
| VATEX | TwelveLabs | 85.43 | 92.40 | 97.30 | 99.43 |
| MSR-VTT | VideoDB | 72.82 | 81.55 | 86.89 | 92.23 |
| MSR-VTT | TwelveLabs | 62.62 | 72.33 | 85.44 | 92.72 |
| **Macro-average** | **VideoDB** | **73.09** | **83.39** | **91.20** | 96.07 |
| **Macro-average** | **TwelveLabs** | 65.75 | 77.13 | 89.10 | **96.42** |

Honest reading: the general-purpose pipeline leads at the early ranks that matter for agents (R@1/@3/@10); the video-native baseline is slightly ahead at R@50 and stronger on VATEX.

## Cite

```bibtex
@techreport{videodb2026searchvisualworld,
  title  = {Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence},
  author = {Nagaonkar, Sankalp and Garg, Rohit and Raj, Ankit and Choithani, Ashish and Trivedi, Ashutosh},
  institution = {VideoDB},
  year   = {2026},
  month  = {July},
  url    = {https://videodb.io/research/search-over-the-visual-world}
}
```