# Conference Slide Extraction: Search What Was On the Screen

> Combine spoken-word search with visual scene indexing to pull slide content out of any conference talk, retrieved by what the speaker was saying at the time.

Category: Tutorials
Published: 2026-08-25

---

When you try to recall a specific part of a talk, usually only a few keywords come to mind — and often what caught your attention was on the *slide*, not in the speech. This tutorial builds a pipeline that stores talks in VideoDB and returns the on-screen slide content, in text form, from a spoken-word query.

Runnable notebook: https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/conference_slide_scraper.ipynb

Example question: what was on the screen when the speaker discussed the "hard and fast rule" in https://www.youtube.com/watch?v=IEe-5VOv0Js

## The approach

Index the video on two modalities, then join them on time:

1. **Spoken content** — transcribe and index the speech, making it keyword-searchable.
2. **Visual content** — shot-based scene extraction, each scene described by a prompt that reads slide text.

At query time, keyword-search the transcript, take the returned time ranges, and keep the scenes that overlap them. The transcript locates the moment; the scene index supplies what was on screen.

## Setup

```bash
!pip install videodb
```

Get an API key from https://console.videodb.io (free for the first 50 uploads, no credit card required).

### Step 1: Connect to VideoDB

```python
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

### Step 2: Upload the video

```python
# Upload a video by URL
video = coll.upload(url="https://www.youtube.com/watch?v=IEe-5VOv0Js")
```

### Step 3: Understand and index both modalities

Run spoken-word and slide analyzers together. Conference videos need a lower shot threshold to capture subtle slide changes, while one representative frame per shot is enough to read a static slide.

```python
slide_prompt = (
    "Extract all text on the presentation slide. "
    "Return None if no slide is visible."
)

understanding = video.understand(
    segmentation={"type": "shot", "threshold": 10},
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {
            "type": "vlm",
            "name": "slides",
            "sampling": {"strategy": "uniform", "frame_count": 1},
            "config": {
                "prompt": slide_prompt,
                "schema": {"description": "string"},
            },
        },
    ],
)
understanding.wait_until_complete()

transcript_analyzer = understanding.get_analyzer("transcript")
slides_analyzer = understanding.get_analyzer("slides")
transcript_output = transcript_analyzer.get_output()
slides_output = slides_analyzer.get_output()
```

Build a sentence-level transcript index so exact phrase matches remain tied to tight time ranges. Index the slide artifact separately for direct visual retrieval.

```python
import re

sentence_records = []
sentence_words = []

for scene in transcript_output["scenes"]:
    for word in scene["data"]["words"]:
        sentence_words.append(word)
        if re.search(r'''[.!?]+["'”’)\]]*$''', word["text"]):
            sentence_records.append(
                {
                    "start": sentence_words[0]["start"],
                    "end": sentence_words[-1]["end"],
                    "text": " ".join(item["text"] for item in sentence_words),
                }
            )
            sentence_words = []

if sentence_words:
    sentence_records.append(
        {
            "start": sentence_words[0]["start"],
            "end": sentence_words[-1]["end"],
            "text": " ".join(item["text"] for item in sentence_words),
        }
    )

transcript_index = video.index(
    name="conference_transcript_sentences",
    source=sentence_records,
    use_for=["query"],
    fields={"filter": ["text"]},
)
slides_index = video.index(
    name="conference_slides",
    source=slides_analyzer,
    use_for=["semantic", "query"],
    fields={"semantic": ["description"], "filter": ["description"]},
)

transcript_index.wait_until_complete()
slides_index.wait_until_complete()
print(transcript_index.status, slides_index.status)
```

### Step 4: Search pipeline implementation

Query the spoken index, extract time ranges, then filter the timed slide artifact by overlap.

```python
scene_index = [
    {
        "start": float(scene["start"]),
        "end": float(scene["end"]),
        "description": scene["data"]["description"],
    }
    for scene in slides_output["scenes"]
]
scene_index.sort(key=lambda scene: (scene["start"], scene["end"]))


def filter_overlapping_scenes(time_ranges, scenes):
    def overlaps(scene, range_start, range_end):
        return scene["start"] < range_end and range_start < scene["end"]

    filtered_scenes = []
    for start, end in time_ranges:
        filtered_scenes.extend(scene for scene in scenes if overlaps(scene, start, end))

    # Remove duplicates while preserving order
    seen = set()
    return [
        scene
        for scene in filtered_scenes
        if not (
            (scene["start"], scene["end"]) in seen
            or seen.add((scene["start"], scene["end"]))
        )
    ]
```

```python
def search_pipeline(query, video):
    transcript_result = video.query(
        index_id=transcript_index.index_id,
        filter=[{"field": "text", "op": "contains", "value": query}],
        limit=100,
        sort=[("start", "asc")],
    )
    time_ranges = [
        (shot.start, shot.end) for shot in transcript_result.get_shots()
    ]

    final_result = filter_overlapping_scenes(time_ranges, scene_index)

    result_text = "\n\n".join(
        result_entry["description"]
        for result_entry in final_result
        if result_entry.get("description", "").lower().strip() != "none"
    )
    result_timeline = [
        (result_entry.get("start"), result_entry.get("end"))
        for result_entry in final_result
    ]

    return result_text, result_timeline
```

### Step 5: Viewing the search results

```python
from videodb import play_stream

query = "hard and fast rule"

result_text, result_timeline = search_pipeline(query, video)

stream_link = video.generate_stream(result_timeline)
play_stream(stream_link)

print(result_text)
```

It returns scenes where the spoken words match the query, plus the content of any slides visible in those scenes, plus a playable stream of only the matching moments.

## Results

Query: **"hard and fast rule"** — the slide reads:

```text
IT'S ALL IN THE DETAILS

- Prefer American English for naming
- Avoid payment-industry jargon
- Timestamp fields should use <verbed_at>
- Amount properties should also provide a currency
- API resources with IDs are top-level
- New API resources should be retrieved and listed one way
- API resource mutations should be reflected in API responses
- Use nested structures for future extensibility
- Prefer enums to booleans for new properties
- Use a type field for polymorphic objects
- Use verbs for properties with side effects
- Use top-level namespaces for product APIs
- Evaluate new features in the Dashboard before building an API
- Use simple, unambiguous language
- Always paginate unbounded lists
- Iterate on designs with beta users with the feature behind a gate
```

Query: **"stripe api review"** — the slide is an API review checklist (title, gavel block for pinging PMs, and a change summary section).

Query: **"Friction Log"** — the slide is a set of internal Terminal dogfooding instructions, including how to order hardware and set up the iOS/Android SDK environment.

## Conclusion

The technique adapts to any case where visual information needs to be retrieved from audio content:

- Finding product demonstrations in long-form video content
- Identifying key moments in educational videos
- Searching for specific visual elements in recorded meetings or presentations

## FAQ

**How do you search for what was on a slide, not just what was said?** Understand and index both channels. Query the transcript to locate the moment, then pull the slide artifact records that overlap those time ranges.

**Why keyword search on the transcript instead of semantic search?** The transcript only locates the moment, it doesn't answer the question. Speakers say the phrase you half-remember, so exact matching gives tight, high-precision ranges.

**What scene extraction threshold works for conference talks?** Lower than the default. Slide decks change gradually, so the default shot detection skips transitions between similar slides. Shot-based extraction with a threshold of 10 captured every slide change.

**How do I keep indexing costs down while tuning the prompt?** Use a short representative test video, inspect `slides_analyzer.get_output()`, and only run the finalized analyzer configuration over the complete talk.

**Can I get a playable clip of the results, not just text?** Yes. Pass the returned time ranges to `generate_stream()` for a single stream of only the matching moments.

**Does this work on live talks and streams?** Yes, with streaming ingestion — the same two-channel pattern runs continuously and the index is queryable while the talk is happening. See https://videodb.io/blogs/rtsp-ai-analysis

---

Understanding Artifacts: https://docs.videodb.io/pages/understand/indexing-pipelines/understanding-artifacts · Create an Index: https://docs.videodb.io/pages/understand/indexing-pipelines/create-an-index · Search and Retrieval: https://docs.videodb.io/pages/understand/search-and-retrieval/natural-language-query · Architecture: https://videodb.io/blogs/video-rag · Discord: https://discord.gg/py9P639jGz