# NFL Game Analysis: Cutting VLM Hallucinations by 80%

> Three approaches to event-dense sports footage, measured on the same game. Play-by-play segmentation cut hallucinations from 68.1% to 11.4% and cost up to 70% less than 1 fps into Gemini.

Category: Engineering
Published: 2026-08-25

---

Vision language models shine in controlled benchmarks, then stumble on real-world, event-dense footage such as an NFL game. VideoDB bridges that gap by letting developers slice video at the right semantic boundaries, combine external stats, and run multi-tier visual/LLM pipelines that cut hallucinations by >80% while costing up to 70% less than a naive "1 fps into Gemini" workflow.

Source footage: https://www.youtube.com/watch?v=pA_xAsb5hbA

Four evaluation axes:

| Evaluation metric | What it measures |
| --- | --- |
| Hallucination | Frequency of incorrect or irrelevant information produced by the VLM. |
| Temporal Context | How accurately the VLM maintains correct chronological relationships within the video. |
| Performance on Granular Queries | The VLM's effectiveness in accurately responding to detailed and specific queries. |
| VideoDB Involvement | The extent to which VideoDB's capabilities were leveraged to enhance VLM performance. |

## 1. The naive Gemini approach

Complete NFL game footage sent directly to Gemini.

| Evaluation metric | Observation | Notes |
| --- | --- | --- |
| Hallucination | 68.1% | Frequent irrelevant predictions. |
| Temporal Context | Bloated | Model often lost critical event continuity. |
| Performance on Granular Queries | Moderate | Struggled significantly. |
| VideoDB Involvement | Low | |

Known VLM limitations:

- **Finite context windows.** Even a 1M-token window can't hold one NFL quarter at 30 fps.
- **Image-tile token explosion.** Every 1080p frame splits into ~4-9 tiles (~1-4k tokens) before the model sees it.
- **Weak event reasoning.** VLMs reason per-frame, not per-play, missing temporal causality ("Was the QB still behind the line when he released?").
- **Cost scales linearly** with frames.

## 2. Uniform-length chunks (possible with VideoDB)

Fixed 2s / 5s / 10s clips via VideoDB's scene index API.

```python
import videodb

conn = videodb.connect(api_key="YOUR_API_KEY")
collection = conn.get_collection()

video = collection.upload(url="https://www.youtube.com/watch?v=pA_xAsb5hbA")

# Analyze fixed five-second windows with eight representative frames per window
uniform_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[
        {
            "type": "vlm",
            "name": "uniform_play_analysis",
            "sampling": {"strategy": "uniform", "frame_count": 8},
            "config": {
                "prompt": "Summarize the football action in this five-second segment.",
                "schema": {"summary": "string"},
            },
        }
    ],
)
uniform_understanding.wait_until_complete()

uniform_analyzer = uniform_understanding.get_analyzer("uniform_play_analysis")
uniform_output = uniform_analyzer.get_output()

# Make the summaries searchable
uniform_index = video.index(
    name="uniform_play_analysis",
    source=uniform_analyzer,
    use_for=["semantic", "query"],
    fields={"semantic": ["summary"]},
)
uniform_index.wait_until_complete()

print(f"Understanding ID: {uniform_understanding.id}")
print(f"Index ID: {uniform_index.index_id} ({uniform_index.status})")
print(uniform_output["scenes"][0])
```

| Evaluation metric | Observation |
| --- | --- |
| Hallucination | 74.2% |
| Temporal Context | Insufficient |
| Performance on Granular Queries | Moderate |
| VideoDB Involvement | Moderate |

Uniform chunking scored *worse* than sending the whole game. Arbitrary boundaries split plays (a QB throw lands across two clips), so the model can't see the full action and invents the missing half. Too long means overload, too short means no context — neither extreme works.

## 3. Play-by-play segmentation (advanced pipeline with VideoDB)

Detailed statistical reports for major sports games are public, and include exact start/end timestamps for each play. The problem: those timestamps are on the official *game clock*, not video runtime.

### Aligning game-time with video-time

The on-screen scoreboard is the bridge. It displays scores, quarter, down and yardage, and the game clock.

- **OCR-based timestamp extraction** from the scoreboard throughout the video.
- **Frame sampling optimization**: 1 fps for OCR, cutting compute without losing timestamp accuracy.
- **Timestamp mapping**: OCR results correlate official game time to video runtime, enabling per-play segmentation.

```python
# Analyze one representative frame for every second of video
scoreboard_understanding = video.understand(
    segmentation={"type": "time", "seconds": 1},
    analyzers=[
        {
            "type": "vlm",
            "name": "scoreboard",
            "sampling": {"strategy": "uniform", "frame_count": 1},
            "config": {
                "prompt": (
                    "Read the scorebar at the bottom of the frame. Extract both "
                    "team names and scores, the quarter number, and the game clock."
                ),
                "schema": {
                    "team_1_name": "string",
                    "team_1_score": "integer",
                    "team_2_name": "string",
                    "team_2_score": "integer",
                    "quarter_number": "integer",
                    "game_clock": "string",
                },
            },
        }
    ],
)
scoreboard_understanding.wait_until_complete()

scoreboard_output = scoreboard_understanding.get_analyzer("scoreboard").get_output()

# Map each video-runtime second to its structured scoreboard reading
scene_ocr_results = {
    float(scene["start"]): scene["data"]
    for scene in scoreboard_output["scenes"]
}

for video_time, scoreboard in list(scene_ocr_results.items())[:5]:
    print(video_time, scoreboard)
```

### Integrating play-by-play segmentation with VideoDB

```python
# Step 1: Use the stats PDF to filter all play timestamps (game clock) where a
# catch occurred into `catch_play_scenes` as a list of (start, end) for plays with catches

# Step 2: Map game clocks to video timestamps using OCR outputs

# Analyze short windows once; they will be joined to official play ranges below
catch_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[
        {
            "type": "vlm",
            "name": "catch_analysis",
            "sampling": {"strategy": "uniform", "frame_count": 8},
            "config": {
                "prompt": (
                    "This segment is part of a play containing a catch. Extract the "
                    "catch type, player position, and whether it is an interception."
                ),
                "schema": {
                    "catch_type": "string",
                    "player_position": "string",
                    "interception": "boolean",
                },
            },
        }
    ],
)
catch_understanding.wait_until_complete()
catch_output = catch_understanding.get_analyzer("catch_analysis").get_output()

def overlaps(scene, start, end):
    return float(scene["start"]) < end and start < float(scene["end"])


catch_details = []
for start_time, end_time in catch_play_scenes:
    evidence = [
        scene["data"]
        for scene in catch_output["scenes"]
        if overlaps(scene, start_time, end_time)
    ]
    if not evidence:
        continue

    catch_details.append(
        {
            "start": start_time,
            "end": end_time,
            "catch_type": ", ".join(
                dict.fromkeys(item["catch_type"] for item in evidence if item["catch_type"])
            ) or "none",
            "player_position": ", ".join(
                dict.fromkeys(
                    item["player_position"]
                    for item in evidence
                    if item["player_position"]
                )
            ) or "none",
            "interception": any(item["interception"] for item in evidence),
        }
    )

catch_index = video.index(
    name="catch_plays",
    source=catch_details,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["catch_type", "player_position"],
        "filter": ["interception"],
    },
)
catch_index.wait_until_complete()
```

| Evaluation metric | Observation |
| --- | --- |
| Hallucination | 11.4% |
| Temporal Context | Perfect |
| Performance on Granular Queries | High |
| VideoDB Involvement | High |

## Approach comparison

| Evaluation metric | Naive whole-video | Uniform chunks | Play-by-play |
| --- | --- | --- | --- |
| Hallucination | 68.1% | 74.2% | **11.4%** |
| Temporal Context | Poor | Insufficient | **Perfect** |
| Granular Queries | Moderate | Moderate | **High** |
| VideoDB Use | Low | Moderate | **High** |

## Key takeaways

1. **Define key sports concepts** — catch (yes/no), running play (yes/no), scoring event (yes/no).
2. **Check availability of statistical data.** Available: use it to isolate plays. Not available: use the VLM directly for visual extraction.
3. **Extract relevant plays using statistical data** via the VideoDB timeline; record timestamps and metadata.
4. **Run visual analysis with VideoDB indexing** — pass extracted scenes to the VLM for detail (catch type "overhead", position "near sidelines").
5. **Structure the output data clearly:**

```json
[
  {
    "play_start_time": 12,
    "play_end_time": 52,
    "details": {
      "catch": true,
      "type": "overhead",
      "position": "near sidelines",
      "interception": false,
      "running_play": true
    }
  }
]
```

6. **Add a query and reasoning engine (small LLM).** Feed structured data plus the user query into the VideoDB search interface for accurate play-by-play results.

## Pricing: VideoDB vs. Gemini at 1 fps

| 60-min NFL game | Frames analysed | VideoDB (Balanced tier) | Gemini 1.5 Pro* |
| --- | --- | --- | --- |
| 1 fps, 1080p | 3,600 | **$2.00** index + ~$0.35 tokens | $1.1 - $7.4 |
| 5 fps | 18,000 | **$10.00** index | $5.6 - $37.0 |
| 30 fps | 108,000 | **$12.00** index | $33 - $220 |

*Prices use Google's published rate card: $0.10/M input tokens, $0.40/M output; HD frames tokenize into 1,024-4,128 tokens each.*

As frame rate or resolution rises, VideoDB's flat visual-index pricing stays predictable while pure-Gemini costs explode.

## Why choose VideoDB

- **Event-aligned indexing** — cut by play, scene, or any custom timeline, not crude 1s slices.
- **Hybrid reasoning pipelines** — blend stats, embeddings, and VLMs to slice hallucinations to ~11%.
- **Serverless scale** — petabytes or a single clip, zero idle cost.
- **Developer-first API** — Python, JS, REST.
- **Transparent pricing** — pay once for storage + index; pick Entry / Advanced / SOTA LLM pricing per query.

## FAQ

**Why do VLMs hallucinate on sports footage?** Context windows can't hold a game at broadcast frame rates, every 1080p frame explodes into thousands of tokens, and VLMs reason per-frame rather than per-play. Naive whole-video hallucinated on 68.1% of events.

**Do smaller video chunks reduce hallucinations?** No — uniform 2s/5s/10s chunks scored worse (74.2%). Fixed-length cuts split plays across boundaries. The problem is where you cut, not how small.

**What is play-by-play segmentation?** Segmenting at real semantic boundaries instead of on a timer: official play start/end timestamps from the public game summary, mapped onto video runtime by OCR-ing the on-screen game clock at 1 fps. Hallucinations dropped to 11.4%.

**How do you align the official game clock with video timestamps?** Sample 1 fps, run OCR with a structured prompt returning scores, quarter, and game clock as JSON, and build a lookup table from game clock to video runtime.

**Is VideoDB cheaper than sending frames straight to Gemini?** Comparable at 1 fps. The gap opens as frame rate rises: at 30 fps for a 60-minute game, $12.00 of indexing against $33-$220 of Gemini tokens.

**Does this only work for American football?** No. It generalizes to any domain with an authoritative event log and an on-screen clock: cricket, basketball, soccer, esports, broadcast production.

---

Docs: https://docs.videodb.io/pages/understand/indexing-pipelines/create-an-index · Retrieval architecture: https://videodb.io/blogs/video-rag · Model selection: https://videodb.io/blogs/how-to-evaluate-multimodal-vlms-for-your-video-use-case · Questions: engg@videodb.io