# Real-time visual perception for agents > How AI agents get vision: ingest YouTube videos, live RTSP cameras, and screens; index continuously; search semantically; act on plain-language events. - Category: Agents - Published: 2026-08-07 - Authors: Ashutosh Trivedi - Canonical: https://videodb.io/blog/give-your-ai-agents-eyes - HTML: https://videodb.io/blog/give-your-ai-agents-eyes · Markdown: https://videodb.io/blog/give-your-ai-agents-eyes.md - Tags: product --- Your agent can write code, book flights, search the entire web, and operate a browser like a caffeinated intern. Point it at a security camera, a YouTube video, or the screen it’s supposedly automating, and it’s helpless. It can act on the world, but it cannot watch the world. > **The short version** > > AI agents get vision by plugging into video infrastructure. That layer ingests any video source: files, YouTube URLs, live RTSP cameras, screen capture. It converts them continuously into indexed, searchable context, and exposes search, events, and alerts through an API. That’s VideoDB, and it gives your agent eyes plus the memory to act on what it sees, in a few lines of Python or TypeScript. The rest of this post is the long version: why agents are blind, why the obvious hack doesn’t scale, and three working builds. The three builds are an agent that watches YouTube, an agent that watches a live camera feed, and an agent that remembers what happened on a screen. ## Agents got hands before they got eyes Look at what the agent-infrastructure wave has actually shipped. Exa and Parallel gave agents web search and research. Ask a question, get grounded answers from the live internet. Firecrawl gave them clean page context at scale. Browserbase and browser-use gave them hands: click, type, navigate, transact. TinyFish runs fleets of web agents against enterprise workflows. Every one of those unlocked a product category. And every one of them operates on the same substrate: text and DOM. HTML in, tokens out. Meanwhile, most of what actually happens in the world never touches a DOM. Meetings happen on camera. Work happens on screens. Operations happen in front of cameras: stores, hospitals, factories, streets, drones. The largest knowledge source on the internet is a video platform, and your agent can’t watch it. AI is moving out of the chatbox. The moment it does, it hits the visual world and goes blind. > That gap is not a model problem. GPT-5, Claude, and Gemini can all describe a frame beautifully. It’s an infrastructure problem. ## Why screenshots are not enough Every builder tries the same first hack: grab a frame, base64 it, stuff it into the prompt. It demos great. Then it collapses, for five predictable reasons: 1. **Moments aren’t events.** A frame tells you a person is in the room. It can’t tell you they’ve been pacing near the server rack for ten minutes. Meaning in video lives in time, and single frames throw time away. 2. **Nothing persists.** Ask your screenshot-agent what happened an hour ago and it has no idea. Perception without memory is a party trick. 3. **Tokens explode.** Continuous understanding via frames-in-prompt means paying VLM prices on every tick of every source, forever, with no reuse. You re-buy the same understanding every time you ask a new question. 4. **Polling misses things.** Real events don’t wait for your loop interval. Real-time perception has to be push, not pull. The stream tells *you* when something happened. 5. **You can’t search what you never indexed.** “When did the delivery arrive?” “Show me every moment the user opened the settings page.” Those are retrieval queries over time. No index, no answer. Sight, for an agent, is a pipeline: **ingest → understand → remember → retrieve → act.** That’s precisely the layer VideoDB provides, and here’s what it feels like to use. ## Build 1: an agent that watches a YouTube video The task: give your agent a YouTube URL and let it answer with *evidence*. Not a transcript dump, but the exact moments that support the answer, playable as a clip. ```bash pip install videodb export VIDEO_DB_API_KEY="your-api-key" ``` ```python import videodb conn = videodb.connect() coll = conn.get_collection() # Ingest: any file or URL, including YouTube video = coll.upload(url="https://www.youtube.com/watch?v=LPZh9BOjkQs") # Understand: run analyzers over speech AND visuals understanding = video.understand(analyzers=[ {"type": "spoken_words", "name": "transcript"}, {"type": "vlm", "name": "scene", "config": {"prompt": "Describe the visual content."}} ]) understanding.wait_until_complete() # Remember: index what the analyzers produced transcript_index = video.index(name="transcript", source=understanding.get_analyzer("transcript")) scene_index = video.index(name="scene", source=understanding.get_analyzer("scene")) transcript_index.wait_until_complete() scene_index.wait_until_complete() # Retrieve: semantic search across speech + visuals together results = video.semantic_search( query="an explanation supported by diagrams and on-screen text", index_ids=[transcript_index.index_id, scene_index.index_id], top_k=5 ) for shot in results.get_shots(): print(f"{shot.start:.1f}s-{shot.end:.1f}s") # Act: compile the matching moments into a single playable evidence reel evidence_url = results.compile() ``` Notice what your agent just got that a transcript scraper can’t give it. The *visual* channel is indexed too, and that VLM analyzer prompt is yours to shape: “extract every chart”, “note every product shown”, “flag every error message”. Search runs across both modalities, every hit is timestamped, and `compile()` returns a watchable clip of just the relevant moments. Your agent doesn’t summarize a video. It cites it. This is the pattern people are hacking together with yt-dlp + Whisper + ffmpeg + a vector DB + prayer. It’s one dependency here. Wire it behind a tool call and “go watch these 40 videos and tell me every pricing claim, with receipts” becomes a routine agent task, the same way Exa made “go read the web” routine. ## Build 2: an agent with a live camera feed Recorded video is where perception starts. The interesting part is live. RTSP is the lingua franca of cameras: security cams, drones, warehouse feeds, broadcast. This is where “give your agent eyes” stops being a metaphor. ```python import asyncio import videodb RTSP_URL = "rtsp://samples.rts.videodb.io:8554/intruder" # public sample stream async def main(): conn = videodb.connect() coll = conn.get_collection() # Ingest: connect the live stream rtstream = coll.connect_rtstream(url=RTSP_URL, name="Lobby Cam", store=True) # Understand: continuous analysis in rolling windows understanding = rtstream.understand( segmentation={"type": "time", "window": "5s"}, analyzers=[{ "type": "vlm", "name": "scene", "sampling": {"frame_count": 2}, "config": {"prompt": "Describe the people, movement, and activity."} }], store=True ) # Remember: a live index that's queryable while the stream runs index = rtstream.index(name="lobby-scenes", source=understanding.outputs["scene"], use_for=["semantic"]) # Act: define an event in plain language, get pushed alerts websocket = conn.connect_websocket(coll.id) await websocket.connect() event_id = conn.create_event( event_prompt="Detect when a person appears in the monitored area.", label="person_appears" ) index.create_alert(event_id=event_id, callback_url="https://your-agent.example.com/hooks/vision", ws_connection_id=websocket.connection_id) async for message in websocket.receive(): if message.get("channel") == "alert": alert = message["data"] print(alert["label"], alert["confidence"]) print(alert["explanation"]) # why the event fired print(alert["stream_url"]) # playable clip of the moment break asyncio.run(main()) ``` Read that event definition again: it’s a sentence. Not a trained classifier, not a labeled dataset, not a CV pipeline. Just a prompt. “Detect when a person appears.” “Detect water pooling on the floor.” “Detect when the forklift enters the loading zone.” When it fires, your agent receives the label, a confidence, an explanation, and a `stream_url` of the exact moment. That’s evidence again, not vibes. The feed is continuously understood in rolling windows, whether or not anyone is asking questions. That is the difference between an agent that *can look* and an agent that *is watching*. This same cycle is the backbone for the unglamorous-but-valuable stuff: intrusion detection, ICU monitoring, flood detection, quality control, drone patrol. There are runnable versions of several of these in the [cookbook](https://github.com/video-db/videodb-cookbook). ## Build 3: agents that remember what happened on screen Computer-use and browser-use agents have a peculiar disability: they act on screens they’ll never remember. Each screenshot is consumed and gone. Ask “what did the deploy dashboard look like before the incident?” and there is no answer, because there is no past. VideoDB’s capture SDK (`pip install "videodb[capture]"`) turns a desktop or browser session into the same kind of stream as Build 2: captured, continuously understood, indexed, searchable. ```python matches = screen.search(query="source code open in an editor", index_id=screen_index.id) for shot in matches.get_shots(): print(shot.text) print(shot.generate_stream()) # replayable clip of that moment ``` That’s an agent asking questions about its own visual history. Call it [episodic memory](/blog/episodic-memory-for-agents), in the concrete sense. Your pair-programmer agent can rewind to the moment the test went red. Your QA agent can produce a replay of the exact click that broke checkout. Your meeting copilot can pull up the slide, not the paraphrase. Full walkthrough in the [quickstart](https://docs.videodb.io/pages/getting-started/quickstart). ## One layer for every video source The point isn’t three demos. It’s that all three run the *same five verbs* (ingest, understand, remember, retrieve, act) over different sources: | Your agent | Its eyes | What it can now do | | --- | --- | --- | | Research / web agent | YouTube + any video URL | Watch, extract, cite with timestamped clips | | Computer-use agent | Screen capture | Recall and replay anything it ever saw | | Browser agent | Session recordings, web video | Verify visually, debug from replays | | Ops / monitoring agent | RTSP cameras, drones | Watch continuously, act on plain-language events | | Meeting copilot | Calls and meetings | Search what was shown, not just said | | Media agent | Archives and libraries | Find any moment, compile new cuts programmatically | ## Why not send frames to GPT-5 or Gemini directly? For a single image, absolutely. You don’t need infrastructure to caption a photo. The honest boundary: raw VLM calls are the right tool for *one-shot* perception of *individual* moments. You outgrow them the day your agent needs any of these: continuous sources (a stream doesn’t fit in a context window, because it never ends), search over hours of footage, events that push instead of polls that pull, memory that outlives the session, or costs that don’t scale linearly with watching. VideoDB isn’t a competitor to the models. It’s the layer that feeds them. Analyzers are model-agnostic. The VLM does the perceiving, and VideoDB does everything that makes perceiving useful: ingestion from any source, temporal segmentation, indexing, retrieval, eventing, storage, and playback. The [playback stack](/blog/playback-vs-perception) (storage, encoding, delivery) was built so *people* could watch video. This is the equivalent stack so *software* can. ## Plug it into the agent you already have You likely don’t even need to write the integration: - **Agent Skills:** `npx skills add video-db/skills` gives Claude, Cursor, and other agents video perception as tools, plus server-side video workflows they can invoke ([skills repo](https://github.com/video-db/skills)) - **Frameworks:** [LlamaIndex retriever](/blog/llama-index), LangChain, and REST for everything else - **No-code:** n8n and Zapier nodes for the same primitives ## FAQ ### How do AI agents see video? Through a perception layer. Video infrastructure ingests sources (files, URLs, live streams, screens), runs speech and vision analyzers over them continuously, indexes the results, and exposes semantic search plus real-time events via API. The agent calls tools. The infrastructure watches. ### Can my agent watch a YouTube video and answer questions about it? Yes. `coll.upload(url=...)` accepts YouTube URLs directly. After indexing, the agent can semantically search speech and visuals, get timestamped matches, and compile evidence clips. See Build 1 above. It’s about 20 lines. ### Can agents process live streams like RTSP cameras in real time? Yes. RTSP Connect ingests live feeds, understanding runs in rolling windows (e.g. every 5 seconds), and live indexes are queryable while streaming. Plain-language events (“detect when a person appears”) push alerts over WebSocket or webhooks, with an explanation and a clip of the moment. ### What’s the difference between VideoDB and a vision model API? A vision model perceives a frame or short clip you hand it. VideoDB is the infrastructure around the model: ingestion, temporal segmentation, continuous analysis, indexing, memory, retrieval, events, and playback. It’s model-agnostic. The VLM sees, and VideoDB makes seeing continuous, searchable, and actionable. ### Does this work with Claude, Cursor, LangChain, or my agent framework? Yes. Agent Skills give Claude, Cursor, and other agents video perception directly. There is also a LlamaIndex retriever, Python/Node SDKs, and n8n/Zapier for no-code automations. ### What about privacy and deployment? VideoDB supports Zero Data Retention configurations, SOC 2 Type II, SSO/SAML, and deployment as managed cloud or in your own cloud (AWS/GCP/Azure), with VPC and edge options for sensitive camera workloads. > **To see is to know** > > The next wave of AI will understand the visual world. Give your agents eyes. Start with the [quickstart](https://docs.videodb.io/pages/getting-started/quickstart) (free tier, about 5 minutes to first search), or come argue with us in [Discord](https://discord.com/invite/py9P639jGz). ## FAQ **How do AI agents see video?** Through a perception layer: video infrastructure that ingests sources (files, URLs, live streams, screens), runs speech and vision analyzers over them continuously, indexes the results, and exposes semantic search plus real-time events via API. The agent calls tools; the infrastructure watches. **Can my agent watch a YouTube video and answer questions about it?** Yes. VideoDB ingests YouTube URLs directly; after indexing, an agent can semantically search speech and visuals, get timestamped matches, and compile evidence clips, in about 20 lines of Python. **Can agents process live streams like RTSP cameras in real time?** Yes. RTSP Connect ingests live feeds, analysis runs in rolling windows, live indexes are queryable while streaming, and plain-language events push alerts with explanations and clips of the moment. **Does VideoDB work with Claude, Cursor, LangChain, or other agent frameworks?** Yes, via the VideoDB MCP server, Agent Skills, a LlamaIndex retriever, Python and Node.js SDKs, and n8n/Zapier integrations. **How is video data handled for privacy and enterprise deployment?** VideoDB supports Zero Data Retention configurations, SOC 2 Type II, SSO, and managed-cloud or bring-your-own-cloud deployment on AWS, GCP, or Azure, with VPC and edge options.