Create an Index

Indexes turn raw video into structured, searchable data. Create a spoken word index for dialogue and narration, or a scene index for visual content.

Quick Example

import videodb

conn = videodb.connect()
coll = conn.get_collection()
video = coll.get_video("m-xxx")

# Index spoken content (dialogue, narration)
video.index_spoken_words()

# Index visual content (scenes, objects, actions)
scene_index_id = video.index_scenes(
    prompt="Describe what's happening in the scene"
)

# Search both
results = video.search("car chase through the city")
results.play()

import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();
const video = await coll.getVideo("m-xxx");

// Index spoken content (dialogue, narration)
await video.indexSpokenWords();

// Index visual content (scenes, objects, actions)
const sceneIndexId = await video.indexScenes({
    prompt: "Describe what's happening in the scene"
});

// Search both
const results = await video.search("car chase through the city");
await results.play();

Spoken Word Index

Transcribes audio into timestamped text using automatic speech recognition (ASR).

video.index_spoken_words()

await video.indexSpokenWords();

What it captures:

Dialogue and conversations
Narration and voiceovers
Lectures and presentations
Interviews and podcasts

Language Support

Major languages are auto-detected. For others, pass the language code:

# Auto-detect (English, Spanish, French, German, Italian, Portuguese, Dutch)
video.index_spoken_words()

# Explicit language code
video.index_spoken_words(language_code="hi")  # Hindi
video.index_spoken_words(language_code="ja")  # Japanese
video.index_spoken_words(language_code="zh")  # Chinese

// Auto-detect (English, Spanish, French, German, Italian, Portuguese, Dutch)
await video.indexSpokenWords();

// Explicit language code
await video.indexSpokenWords({ languageCode: "hi" });  // Hindi
await video.indexSpokenWords({ languageCode: "ja" });  // Japanese
await video.indexSpokenWords({ languageCode: "zh" });  // Chinese

Language	Code
English (Global)	`en`
English (US/UK/AU)	`en_us`, `en_uk`, `en_au`
Spanish	`es`
French	`fr`
German	`de`
Hindi	`hi`
Japanese	`ja`
Chinese	`zh`
Korean	`ko`
Russian	`ru`

Scene Index

Analyzes video frames using vision models to describe visual content.

scene_index_id = video.index_scenes(
    prompt="Describe the scene in detail"
)

const sceneIndexId = await video.indexScenes({
    prompt: "Describe the scene in detail"
});

What it captures:

Objects and people
Actions and activities
Environments and settings
Visual transitions

Open-weight models with Sandbox Compute

To use an open-weight VLM for scene indexing, start a compatible sandbox, wait until it is active, then pass both model_name and sandbox_id to index_scenes().

from videodb import SandboxTier, SceneExtractionType

sandbox = conn.create_sandbox(
    tier=SandboxTier.medium,
    models=["google/gemma-4-31B-it-FP8"],
)
sandbox.wait_for_ready()

scene_index_id = video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 10, "frame_count": 1},
    model_name="google/gemma-4-31B-it-FP8",
    prompt="Describe the scene in a clear, concise way.",
    sandbox_id=sandbox.id,
)

VideoDB validates that the sandbox is active, belongs to your workspace/account, supports the requested model tier, and was created with a matching models or model_categories value when model selection was provided. See Sandbox Compute and Sandbox Models for setup and supported model names.

Prompt Shapes the Index

The prompt you provide determines what gets indexed:

# Focus on people
video.index_scenes(prompt="Describe the people and their actions")

# Focus on environment
video.index_scenes(prompt="Describe the location and setting")

# Focus on specific objects
video.index_scenes(prompt="Identify all vehicles and their colors")

// Focus on people
await video.indexScenes({ prompt: "Describe the people and their actions" });

// Focus on environment
await video.indexScenes({ prompt: "Describe the location and setting" });

// Focus on specific objects
await video.indexScenes({ prompt: "Identify all vehicles and their colors" });

Extraction Configuration

Control how frames are sampled - choose between frame segmentation (regular intervals) and scene segmentation (automatic transitions):

Comparison of frame segmentation and scene segmentation extraction types

from videodb import SceneExtractionType

# Time-based: every N seconds
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 10, "frame_count": 2},
    prompt="Describe the scene"
)

# Shot-based: detect visual transitions
video.index_scenes(
    extraction_type=SceneExtractionType.shot_based,
    extraction_config={"threshold": 20, "frame_count": 1},
    prompt="Describe the scene"
)

// Time-based: every N seconds
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 10, frame_count: 2 },
    prompt: "Describe the scene"
});

// Shot-based: detect visual transitions
await video.indexScenes({
    extractionType: 'shot',
    extractionConfig: { threshold: 20, frame_count: 1 },
    prompt: "Describe the scene"
});

Time-based extraction example showing consistent frame sampling at regular intervals

Method	Best For
Time-based	Consistent sampling, dynamic content
Shot-based	Edited videos with clear scene changes

Managing Indexes

List All Scene Indexes

indexes = video.list_scene_index()
for idx in indexes:
    print(f"{idx.id}: {idx.name} - {idx.status}")

const indexes = await video.listSceneIndex();
for (const idx of indexes) {
    console.log(`${idx.id}: ${idx.name} - ${idx.status}`);
}

List of scene indexes showing id, name, and status

Get Index Details

scene_index = video.get_scene_index(scene_index_id)
for scene in scene_index:
    print(f"{scene.start}-{scene.end}: {scene.description}")

const sceneIndex = await video.getSceneIndex(sceneIndexId);
for (const scene of sceneIndex) {
    console.log(`${scene.start}-${scene.end}: ${scene.description}`);
}

Delete an Index

video.delete_scene_index(scene_index_id)

await video.deleteSceneIndex(sceneIndexId);

Async Processing with Callbacks

For long videos, use callbacks to get notified when indexing completes:

scene_index_id = video.index_scenes(
    prompt="Describe the scene",
    callback_url="https://your-backend.com/webhooks/index-complete"
)

const sceneIndexId = await video.indexScenes({
    prompt: "Describe the scene",
    callbackUrl: "https://your-backend.com/webhooks/index-complete"
});

What You Can Build

Keyword Search Compilation

Index spoken words, then search to create highlight reels

Multimodal Search

Combine spoken word and scene indexes for powerful queries

Baby Crib Monitoring

Scene indexing enables real-time infant monitoring

Intrusion Detection

Index camera feeds to detect unauthorized access

Next Steps

Understanding Artifacts

Extraction strategies for video + audio

Multiple Indexes

Layer different perspectives on the same media

Start Here

Core Concepts

Ingest

Understand

Act

Automate

Open Source Frameworks

Quick Example

Spoken Word Index

Language Support

Scene Index

Open-weight models with Sandbox Compute

Prompt Shapes the Index

Extraction Configuration

Managing Indexes

List All Scene Indexes

Get Index Details

Delete an Index

Async Processing with Callbacks

What You Can Build

Keyword Search Compilation

Multimodal Search

Baby Crib Monitoring

Intrusion Detection

Next Steps

Understanding Artifacts

Multiple Indexes

​Quick Example

​Spoken Word Index

​Language Support

​Scene Index

​Open-weight models with Sandbox Compute

​Prompt Shapes the Index

​Extraction Configuration

​Managing Indexes

​List All Scene Indexes

​Get Index Details

​Delete an Index

​Async Processing with Callbacks

​What You Can Build

Keyword Search Compilation

Multimodal Search

Baby Crib Monitoring

Intrusion Detection

​Next Steps

Understanding Artifacts

Multiple Indexes

Quick Example

Spoken Word Index

Language Support

Scene Index

Open-weight models with Sandbox Compute

Prompt Shapes the Index

Extraction Configuration

Managing Indexes

List All Scene Indexes

Get Index Details

Delete an Index

Async Processing with Callbacks

What You Can Build

Next Steps