Case Study: NFL Game Analysis

​

1. The Naive Gemini Approach

• Finite context windows – even a 1 M-token window can’t hold one NFL quarter at 30 fps.
• Image-tile token explosion – every 1080p frame is split into ~4–9 tiles (≈ 1–4 k tokens) before the model “sees” it.
• Weak event reasoning – current VLMs reason per-frame, not per-play; they miss temporal causality (e.g., “Was the QB still behind the line when he released?”).
• Cost scales linearly with frames, so 30 fps steals wallets fast.

​

2. Uniform-Length Chunks (Possible with VideoDB)

from videodb import SceneExtractionType

conn = videodb.connect(api_key="YOUR_API_KEY")
collection = conn.get_collection()

video = collection.upload(url="https://www.youtube.com/...")

# Extract uniform scenes every n seconds
scene_collection = video.extract_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 5, "num_frames": 150},  # Example: 5 seconds per chunk
)

print(f"Scene Collection ID: {scene_collection.id}")

• Important actions (e.g., QB throw) get split across two clips, so the model can’t see the full play and mis-judges legality or outcome.
• Same problem pops up for catches, interceptions, and other decisive moments.

• Model starts “imagining” passes and catches that never happened, simply because it lacks enough temporal evidence in a single clip.
• Resulting event timeline is noisy and bloated with false positives.

• Too long → information overload and confusion.
• Too short → not enough context.
• Neither extreme works; we need a balanced segmentation window.

​

3. Play-by-Play Segmentation (Advanced Pipeline with VideoDB)

​

3.1 Aligning Game-Time with Video-Time

​

How We Achieved This

• OCR-based Timestamp Extraction: We processed the video using an Optical Character Recognition (OCR) model to detect and extract visible game times from the scoreboard throughout the video.
• Frame Sampling Optimization: To optimize efficiency, we sampled just one frame per second (1 fps) for OCR processing. This significantly reduced computational load without compromising the accuracy of the extracted timestamps.
• Timestamp Mapping Creation: The OCR results provided an exact correlation between the official game timestamps and the actual runtime of the video. Using this mapping, we segmented the video accurately into individual play-by-play events.

from videodb import SceneExtractionType

# 1 frame per second
scene_collection = video.extract_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 1, "select_frames": ["first"]} 
)

print(f"Scene Collection ID: {scene_collection.id}")

# Perform OCR with structured prompt
scene_ocr_results = {}
for scene in scene_collection.scenes:
    for frame in scene.frames:
        structured_description = frame.describe(
            """Perform OCR to extract the information from the scorebar present.
              on the bottom of the image. Output in a structured JSON format: 
              <team1 name>: <team1 score>, <team2 name>: <team2 score>, 
              quarter_number: <quarter number>, game_clock: <time left>."""                                                                                    )

# Save the OCR result with scene start time as the key
scene_ocr_results[scene.start] = structured_description

print(scene_ocr_results)

# Example Output:
{
    1.0: {"Browns": 0, "Raiders": 0, "quarter_number": 1, "game_clock": "15:00"}
    2.0: {"Browns": 0, "Raiders": 0, "quarter_number": 1, "game_clock": "14:59"}
    ...
}

​

Integrating Play-by-Play Segmentation with VideoDB

# code snippet for detecting catches from the game footage

# Step 1: Use the stats PDF to filter all play timestamps (game clock) where a catch occurred into `catch_play_scenes` as a list of (start, end) for plays with catches

# Step 2: Map game clocks to video timestamps using OCR outputs
catch_details = []

#iterate over each play
for play_scene in catch_play_scenes:
    start_time, end_time = play_scene
    
    #create a new scene object of the play
    scene = Scene(
        video_id=video.id,
        start=start_time,
        end=end_time
    )

    # Call describe with structured prompt to get catch_details
    result = scene.describe(
        """"This clip contains a catch. 
              Extract: type of catch, 
              position of the player who caught the ball, 
              and whether it was an interception. 
              Return a structured JSON with keys
            {catch_type,player_position,interception}."""
    )
    catch_details.append({
        "scene_start": start_time,
        "scene_end": end_time,
        "details": result
    })


print(catch_details)

# Example Output:
[
    {
        "scene_start": 120.0,
        "scene_end": 165.0,
        "details": {
            "catch_type": "over the shoulder",
            "player_position": "centre of the field",
            "interception": False
        }
    },
 ...
]

# Do the same for other concepts ( other than catches ) 

​

Approach Comparison

​

Key Takeaways

1. Define Key Sports Concepts: Clearly outline and specify each concept required for analysis. For example:
   • Catch (Yes/No)
   • Running Play (Yes/No)
   • Scoring Event (Yes/No)
2. Check Availability of Statistical Data: Determine if these concepts can be reliably extracted from existing statistical data:
   • If statistical data is available: use it to isolate specific plays
   • If statistical data is not available: directly use the Vision Language Model (VLM) for visual extraction
3. Extract Relevant Plays Using Statistical Data: Use accurate statistical information to isolate relevant video scenes using VideoDB Timeline. Record timestamps and relevant metadata for these Scenes.
4. Visual Analysis with VideoDB indexing: Pass the extracted scenes into the VLM to gather detailed visual insights (e.g., identifying catch types like “overhead” and positions like “near sidelines”).
5. Clearly Structure the Output Data: Organize the extracted visual information into structured data for clarity and ease of querying. For instance:

[
{
  "play_start_time" : 12,
  "play_end_time" : 52,
  "details": {
    "catch": true,
    "type": "overhead",
    "position": "near sidelines",
    "interception" : false,
    "running_play": true
  }
},
...
]

6. Query and Reasoning Engine (Small LLM): Upon receiving a user query, feed the structured data and query into VideoDB search interface. The engine processes these inputs and returns relevant, accurate play-by-play results.

​

Pricing: VideoDB vs. Gemini @ 1 fps

​

Why Choose VideoDB

• Event-aligned indexing – cut by play, scene, or any custom timeline—not crude 1s slices
• Hybrid reasoning pipelines – blend stats, embeddings, and VLMs to slice hallucinations to ~11%
• Serverless scale – ingest petabytes or a single clip; zero idle cost
• Developer-first API – Python, JS, REST; one call for streams, one for scene queries
• Transparent pricing – pay once for storage + index; pick Entry / Advanced / SOTA LLM pricing per query

​

Explore the Docs