Conference slide extraction: search what was on the screen.

When watching a talk or presentation, it’s common to take notes or share interesting points with others. Often, the content on the slides is what caught your attention. At VideoDB we follow many top engineering processes and regularly take notes from talks and conferences, which we share internally on Slack. But when you try to recall a specific part of a talk, usually only a few keywords come to mind. So we built an internal tool that stores all these talks in VideoDB and lets us find and share what was on screen, in text form, from a search query.

Open in Google Colab

Let’s look at the problem. What was on the screen when the speaker discussed the “hard and fast rule” in the following video?

The source talk. Somewhere in here, a slide answers the question.

Introduction

In this tutorial we’ll walk through an advanced but accessible technique for retrieving visual information from video based on what the speaker was discussing. Specifically, finding information on slides in a recording of a talk.

As video content grows in volume and importance, being able to quickly find specific information inside it becomes crucial. Imagine locating a particular statistic mentioned in an hour-long presentation without watching the entire video. That’s the power of multimodal video search.

This approach combines VideoDB’s reusable understanding artifacts with exact transcript retrieval to create a robust, multimodal search pipeline. Don’t worry if these terms sound complex, we’ll break everything down step by step.

Setup

Installing packages

!pip install videodb

API keys

Before proceeding, ensure access to VideoDB. If you don’t have it, sign up for API access first.

Get your API key from the VideoDB Console. It’s free for the first 50 uploads, and no credit card is required.

Step 1: Connect to VideoDB

Gear up by establishing a connection to VideoDB.

import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()

Step 2: Upload the video

Next, let’s upload our sample video:

# Upload a video by URL
video = coll.upload(url="https://www.youtube.com/watch?v=IEe-5VOv0Js")

Step 3: Understand and index both modalities

Run spoken-word and slide analyzers together. Conference videos need a lower shot threshold to capture subtle slide changes, while one representative frame per shot is enough to read a static slide.

slide_prompt = (
    "Extract all text on the presentation slide. "
    "Return None if no slide is visible."
)

understanding = video.understand(
    segmentation={"type": "shot", "threshold": 10},
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {
            "type": "vlm",
            "name": "slides",
            "sampling": {"strategy": "uniform", "frame_count": 1},
            "config": {
                "prompt": slide_prompt,
                "schema": {"description": "string"},
            },
        },
    ],
)
understanding.wait_until_complete()

transcript_analyzer = understanding.get_analyzer("transcript")
slides_analyzer = understanding.get_analyzer("slides")
transcript_output = transcript_analyzer.get_output()
slides_output = slides_analyzer.get_output()

Build a sentence-level transcript index so exact phrase matches remain tied to tight time ranges. Index the slide artifact separately for direct visual retrieval.

import re

sentence_records = []
sentence_words = []

for scene in transcript_output["scenes"]:
    for word in scene["data"]["words"]:
        sentence_words.append(word)
        if re.search(r'''[.!?]+["'”’)\]]*$''', word["text"]):
            sentence_records.append(
                {
                    "start": sentence_words[0]["start"],
                    "end": sentence_words[-1]["end"],
                    "text": " ".join(item["text"] for item in sentence_words),
                }
            )
            sentence_words = []

if sentence_words:
    sentence_records.append(
        {
            "start": sentence_words[0]["start"],
            "end": sentence_words[-1]["end"],
            "text": " ".join(item["text"] for item in sentence_words),
        }
    )

transcript_index = video.index(
    name="conference_transcript_sentences",
    source=sentence_records,
    use_for=["query"],
    fields={"filter": ["text"]},
)
slides_index = video.index(
    name="conference_slides",
    source=slides_analyzer,
    use_for=["semantic", "query"],
    fields={"semantic": ["description"], "filter": ["description"]},
)

transcript_index.wait_until_complete()
slides_index.wait_until_complete()
print(transcript_index.status, slides_index.status)

Step 4: Search pipeline implementation

The heart of this approach combines an exact transcript query with the timed slide artifact.

The pipeline does the following:

1. Queries the sentence-level transcript index for the remembered phrase
2. Extracts time ranges from the search results
3. Filters the slide artifact by overlap with those ranges
4. Returns the descriptions of these scenes (our slide content) and their time ranges

scene_index = [
    {
        "start": float(scene["start"]),
        "end": float(scene["end"]),
        "description": scene["data"]["description"],
    }
    for scene in slides_output["scenes"]
]
scene_index.sort(key=lambda scene: (scene["start"], scene["end"]))


def filter_overlapping_scenes(time_ranges, scenes):
    def overlaps(scene, range_start, range_end):
        return scene["start"] < range_end and range_start < scene["end"]

    filtered_scenes = []
    for start, end in time_ranges:
        filtered_scenes.extend(scene for scene in scenes if overlaps(scene, start, end))

    # Remove duplicates while preserving order
    seen = set()
    return [
        scene
        for scene in filtered_scenes
        if not (
            (scene["start"], scene["end"]) in seen
            or seen.add((scene["start"], scene["end"]))
        )
    ]

def search_pipeline(query, video):
    transcript_result = video.query(
        index_id=transcript_index.index_id,
        filter=[{"field": "text", "op": "contains", "value": query}],
        limit=100,
        sort=[("start", "asc")],
    )
    time_ranges = [
        (shot.start, shot.end) for shot in transcript_result.get_shots()
    ]

    final_result = filter_overlapping_scenes(time_ranges, scene_index)

    result_text = "\n\n".join(
        result_entry["description"]
        for result_entry in final_result
        if result_entry.get("description", "").lower().strip() != "none"
    )
    result_timeline = [
        (result_entry.get("start"), result_entry.get("end"))
        for result_entry in final_result
    ]

    return result_text, result_timeline

Step 5: Viewing the search results

Finally, let’s use our search pipeline:

from videodb import play_stream

query = "hard and fast rule"

result_text, result_timeline = search_pipeline(query, video)

stream_link = video.generate_stream(result_timeline)
play_stream(stream_link)

print(result_text)

It returns scenes where the spoken words match your query, along with the content of any slides visible in those scenes.

Here’s the result for the search query “hard and fast rule”.

The content written on the slide is

IT'S ALL IN THE DETAILS

- Prefer American English for naming
- Avoid payment-industry jargon
- Timestamp fields should use <verbed_at>
- Amount properties should also provide a currency
- API resources with IDs are top-level
- New API resources should be retrieved and listed one way
- API resource mutations should be reflected in API responses
- Use nested structures for future extensibility
- Prefer enums to booleans for new properties
- Use a type field for polymorphic objects
- Use verbs for properties with side effects
- Use top-level namespaces for product APIs
- Evaluate new features in the Dashboard before building an API
- Use simple, unambiguous language
- Always paginate unbounded lists
- Iterate on designs with beta users with the feature behind a gate

The compiled result stream for “hard and fast rule” — only the matching moments, back to back.

Here are some other query outputs using the same search pipeline.

Search for “stripe api review”

The content written on the slide is an API review checklist:

API Review: [Insert Title Here]
Ziec: Gavel jar, link for API review join creation

Gavel block
To ping PM when Q/A and other stakeholders have

Summary
(Please include a short description of the change you would like to make...)

Result stream for “stripe api review”.

Search for “Friction Log”

The content written on the slide is a set of internal Terminal dogfooding instructions:

https://go/terminal-dogfooding-instructions

Stripe! Thanks a ton for your help in dogfooding ahead of our Terminal GA launch!

We're very close to launching Terminal in public beta and then GA.

And we could use your help! There are a ton of different use cases of these
integrations to test and polish, and we want to stress test what we've made and
ship these paths and accompanying docs and dashboard flows in a developer-friendly
as possible in the time we have before the Terminal GA launch.

If you're arriving at this doc after having signed up to dogfood, continue to Steps below.
If you haven't signed up yet, please signup here, and we'll get back to you when you
have a slot your test.

Steps
1. If you're dogfooding remotely and haven't received instructions on ordering
   hardware or attending demos, please email terminal-dogfooding@stripe.com or
   ping in #terminal-dogfooding

2. If you're dogfooding the iOS or Android SDK, you'll need to set up your
   environment as described in the links...

Result stream for “Friction Log”.

Conclusion

This tutorial outlined an approach to multimodal video search that combines a queryable transcript with timed slide-understanding artifacts. By leveraging VideoDB’s Understand → Index → Retrieve pipeline, we built a workflow that finds specific visual content, in this case slide information, from spoken queries.

The technique has applications well beyond searching for slides in talks. It adapts to any case where visual information needs to be retrieved based on audio content:

• Finding product demonstrations in long-form video content
• Identifying key moments in educational videos
• Searching for specific visual elements in recorded meetings or presentations

As video content continues to grow in importance and volume, tools and techniques like these become increasingly valuable for efficient information retrieval and analysis.

FAQ

How do you search for what was on a slide, not just what was said?

Understand and index both channels. A spoken-word analyzer creates the transcript, while a shot-segmented VLM analyzer creates the timed slide artifact. At query time, use query() to locate the remembered phrase, then pull the slide records that overlap those ranges. The transcript locates the moment; the slide artifact supplies what was on screen.

Why keyword search on the transcript instead of semantic search?

Because the transcript is only being used to locate the moment, not to answer the question. Speakers say the phrase you half-remember, so exact keyword matching gives tight, high-precision time ranges. The slide content then comes from the visual index.

What scene extraction threshold works for conference talks?

Lower than the default. Slide decks change gradually, and the default shot-detection threshold skips transitions between similar slides. We used shot-based extraction with a threshold of 10 to capture every slide change. Run the extraction and view the frames before committing to a full index.

How do I keep indexing costs down while tuning the prompt?

Use a short representative test video, inspect slides_analyzer.get_output(), and only run the finalized analyzer configuration over the complete talk. Fewer sampled frames and a lower test-video duration reduce cost while preserving the exact production API shape.

Can I get a playable clip of the results, not just text?

Yes. The pipeline returns both the slide text and the matching time ranges. Pass those ranges to generate_stream() and you get a single stream containing only the matching moments, which is what the players above are showing.

Does this work on live talks and streams?

Yes, with streaming ingestion. The same two-channel pattern runs continuously over a live source, so the index is queryable while the talk is still happening. See the RTSP + AI walkthrough for the live loop.

Further resources

To go deeper, read Understanding Artifacts for analyzer configuration and output shapes, Create an Index for artifact-to-index options, and Search and Retrieval for query and semantic-search patterns.

For the architecture behind this pattern, read Video RAG: the definitive guide. If your source is YouTube, making a YouTube video agent-readable covers the shortest path, and giving your agents eyes is the wider builder’s guide.