Indexes & Search

VideoDB separates video retrieval into three steps:

Understand → Index → Retrieve

The model

Step	What it does	SDK surface
Understand	Runs analyzers over video and creates reusable artifacts such as transcripts, objects, OCR, brand signals, activity, location, or VLM scene outputs.	`video.understand(...)`
Index	Chooses which artifact fields should become searchable, filterable, aggregatable, or sortable.	`video.index(...)`
Retrieve	Searches, filters, aggregates, and returns timestamped moments as playable evidence with selected fields from indexed artifacts.	`search()`, `semantic_search()`, `query()`, `aggregate()`

Understanding finds what is in the video. Indexing organizes those findings for retrieval. Search, semantic search, query, and aggregation use those indexes to answer user questions.

Sandbox Compute is an independent runtime option for supported models. Use it during the Understand step when you want open-weight or specialized models; the Index step still uses the same video.index(...) interface.

RTStreams follow the same model with continuous jobs: rtstream.understand(...) keeps producing VLM artifacts from live windows, and rtstream.index(...) keeps materializing them for search and alerts. Initial RTStream support is VLM-only with time-based segmentation. See Understand & Index RTStreams.

Quick example

import videodb

conn = videodb.connect()
video = conn.get_collection().get_video("m-xxx")

# 1. Understand the video with defaults
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words"},
        {"type": "vlm"},
    ],
)

understanding.wait_until_complete()

transcript = understanding.get_analyzer("transcript")
scene = understanding.get_analyzer("scene")

# 2. Index selected artifacts
video.index(name="transcript", source=transcript)
video.index(name="scene", source=scene)

# 3. Retrieve playable evidence
results = video.search(query="person talking about a product while holding a phone")

for result in results:
    print(result.start, result.end, result.clip.stream_url)

What is an artifact?

An artifact is the output of one analyzer. Examples:

Analyzer	Artifact	Example contents
`spoken_words`	`transcript`	timed spoken text
`object_detection`	`objects`	labels, counts, boxes
`vlm`	`scene`	scene descriptions and structured visual reasoning
`ocr`	`ocr`	visible text
`brand_detection`	`brands`	detected brands/logos/signals

Artifacts are scene-aware: each output is tied to timestamps such as start and end.

Retrieval modes

API	Use when you want to…	Example
`search()`	use high-level natural-language search	“find scenes where someone talks about Nike while holding a phone”
`semantic_search()`	directly search semantic/vector fields	“happy reunion outside a shop” against the `scene` index
`query()`	apply explicit structured filters	`object_labels contains "phone"`
`aggregate()`	count, group, or facet indexed fields	count scenes by brand

Moment-returning retrieval APIs include playable evidence by default. A retrieval request can also combine multiple indexes. For example:

Find scenes where a person talks about Nike.

VideoDB can combine:

a scene index for visual/person context
a transcript index for “talks about Nike”
a brand index for Nike
playable evidence, transcript text, frames, and timestamps requested with return_fields

Next steps

Understanding Artifacts

Create reusable video artifacts with analyzers.

Create an Index

Turn artifacts into searchable, queryable indexes.

Sandbox Compute

Run supported open-weight and specialized models.

Start Here

Core Concepts

Ingest

Understand

Act

Automate

Open Source Frameworks

The model

Quick example

What is an artifact?

Retrieval modes

Next steps

Understanding Artifacts

Create an Index

Sandbox Compute

​The model

​Quick example

​What is an artifact?

​Retrieval modes

​Next steps

Understanding Artifacts

Create an Index

Sandbox Compute

The model

Quick example

What is an artifact?

Retrieval modes

Next steps