Your AI agent can’t watch YouTube. Here’s the fix.

Give an agent a paper, a blog post, or an entire website and it’s brilliant. Give it the URL of a 40-minute conference talk and it shrugs. The fix: an AI agent “watches” a YouTube video by ingesting the URL into video infrastructure that analyzes both what’s said and what’s shown. That infrastructure indexes the results with timestamps and exposes semantic search. The agent gets back not just answers, but the exact moments that prove them, playable as a clip.

The stack everyone builds first (and regrets)

You can wire this yourself: yt-dlp to fetch, Whisper to transcribe, ffmpeg to sample frames, a VLM to caption them, a vector DB to store both, plus glue code to keep timestamps aligned across channels. That is five dependencies, two of which break when YouTube changes something.

And the output is still just text about the video. You can’t play back the moment that answered the question, and every new video re-runs the whole pipeline by hand.

The failure isn’t any single tool. It’s that you’re rebuilding video infrastructure to answer one question.

The 20-line version

pip install videodb
export VIDEO_DB_API_KEY="your-api-key"   # free key at console.videodb.io

import videodb

conn = videodb.connect()
coll = conn.get_collection()

# 1. Ingest straight from the URL
video = coll.upload(url="https://www.youtube.com/watch?v=LPZh9BOjkQs")

# 2. Understand both channels: speech AND visuals
understanding = video.understand(analyzers=[
    {"type": "spoken_words", "name": "transcript"},
    {"type": "vlm", "name": "scene",
     "config": {"prompt": "Describe the visual content, including any text, charts, or product shots on screen."}}
])
understanding.wait_until_complete()

# 3. Index
t_index = video.index(name="transcript", source=understanding.get_analyzer("transcript"))
s_index = video.index(name="scene", source=understanding.get_analyzer("scene"))
t_index.wait_until_complete()
s_index.wait_until_complete()

# 4. Ask
results = video.semantic_search(
    query="pricing or cost claims",
    index_ids=[t_index.index_id, s_index.index_id],
    top_k=5
)
for shot in results.get_shots():
    print(f"{shot.start:.1f}s-{shot.end:.1f}s")

clip_url = results.compile()   # playable reel of just the matching moments

Three details worth noticing. The VLM prompt is yours, so it works as a schema for what to extract (“note every error message”, “flag every brand logo”, “extract every number on screen”). Search runs across speech and visuals together, so “the part where they show the dashboard” works even if nobody said the word “dashboard.”

And compile() turns search results into a watchable clip. Your agent cites video the way a good analyst cites sources.

Turning it into an agent tool

Wrap the flow and hand it to your agent as a tool:

def watch(url: str, question: str) -> dict:
    """Ingest a video URL and return timestamped moments answering the question."""
    video = coll.upload(url=url)
    u = video.understand(analyzers=[
        {"type": "spoken_words", "name": "t"},
        {"type": "vlm", "name": "s", "config": {"prompt": "Describe the visual content."}}
    ])
    u.wait_until_complete()
    ti = video.index(name="t", source=u.get_analyzer("t")); ti.wait_until_complete()
    si = video.index(name="s", source=u.get_analyzer("s")); si.wait_until_complete()
    r = video.semantic_search(query=question,
                              index_ids=[ti.index_id, si.index_id], top_k=5)
    return {
        "moments": [{"start": sh.start, "end": sh.end} for sh in r.get_shots()],
        "evidence_clip": r.compile()
    }

No-code and agent-skill routes exist too. Running npx skills add video-db/skills gives Claude this capability as tools, and the same primitives are available as n8n and Zapier nodes. Indexing is one-time per video, so every later question against the same video is just a search call.

Batches work the same way. To watch 30 launch videos and extract every pricing claim, loop the uploads into one collection, search each video, and aggregate the results. That’s a research agent that does in minutes what an analyst does in a week, with receipts.

What builders are doing with it

Competitive intel from launch videos and demo days. Research agents that turn conference-talk backlogs into cited briefs. Compliance teams scanning sponsored content for claims. Content teams finding every moment a product appears across an archive. The pattern is identical: ingest → understand → index once → interrogate forever.

FAQ

Can an AI agent summarize a YouTube video through an API?

Yes. Ingest the URL, run speech + visual analyzers, index, then either search for specific moments or ask for summaries grounded in the indexed content. Results carry timestamps, and matching moments can be compiled into a playable clip.

Does this only work for YouTube?

No. The same upload(url=...) path takes direct file URLs and local files, and the same pipeline runs on live RTSP streams and screen recordings. YouTube is just the most common source. (Process content you have the rights to work with.)

What does it cost to index a video?

Indexing is one-time per video, and later searches are cheap API calls. There’s a free tier at console.videodb.io. Current rates are on the pricing page.

Is this just transcript search?

No, and that’s the key difference. The visual channel is analyzed and indexed too, with your prompt controlling what gets extracted. “The part where the chart goes up” is findable even if it was never said aloud. For transcript-only budget workloads, video.index_spoken_words() + video.search() is a leaner path.