Create an Index

After you create an understanding, you can turn its artifacts into indexes. An index stores structured information extracted from the video, including transcript text, scene descriptions, object labels, brands, timestamps, and custom fields, in a retrieval-ready form. Your app or agent can then search, filter, query, and aggregate over that information instead of re-analyzing the video.

Understanding → Artifact → Index → search/query/aggregate

Understanding and indexing are separate. Understanding extracts reusable facts from the video. Indexing decides which of those facts should become searchable, filterable, aggregatable, or available as result context. You can create multiple indexes from the same artifact for different retrieval needs.

Try the indexing notebooks

Explore runnable examples for turning understanding artifacts into indexes.

Quick example

# Create artifacts
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words"},
        {"type": "vlm"},
    ],
)

understanding.wait_until_complete()

transcript = understanding.get_analyzer("transcript")
scene = understanding.get_analyzer("scene")

# Index artifacts with defaults
video.index(source=transcript)
video.index(source=scene)

VideoDB derives sensible defaults from your data. If name is omitted, VideoDB uses the artifact name.

Index sources

source accepts three forms:

# 1. An analyzer object is the lightest path. Only identifiers travel to the server;
#    the server fetches the analyzer's output from its own store.
analyzer = understanding.get_analyzer("scene")
video.index(name="scene", source=analyzer)

# 2. A reference dict works the same way when you only have the ids.
video.index(
    name="scene",
    source={"understanding_id": "und_123", "analyzer_id": "an_456"},
)

# 3. Your own temporal records.
video.index(name="chapters", source=custom_scenes)

Pass analyzer outputs using forms 1 and 2 so the scene payload never round-trips through your client. For custom records, see the user-provided temporal record format.

Choose an index name

location_analyzer = understanding.get_analyzer("location")

index = video.index(
    name="location",
    source=location_analyzer,
)

Use the same index name to group the same kind of data across videos in a collection. For example, name the location index on every video "location"; you can then target that logical group with collection.semantic_search(index_names=["location"]), collection.query(index_name="location"), or collection.aggregate(index_name="location").Indexes that share a name must also share the same field structure. This keeps collection-level retrieval consistent because "location" always represents the same fields. If you create another "location" index with a different structure, VideoDB returns a 400 showing the existing and incoming structures. See Collection Search for index-specific retrieval across videos.

Index all artifacts

for analyzer in understanding.list_analyzers():
    if analyzer.status == "done":
        video.index(name=analyzer.name, source=analyzer)

Declare index capabilities with `use_for`

Use use_for to say which retrieval capabilities the index should support.

video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query", "aggregate"],
)

Value	Meaning	Example
`semantic`	direct semantic/vector search	”happy reunion outside a shop”
`query`	exact structured filtering	`object_labels contains "phone"`
`aggregate`	counts, groups, facets, analytics	count scenes by brand

semantic describes an index capability. Use semantic_search() for direct vector similarity over semantic fields. Use search() for high-level natural-language search that may combine semantic indexes with filters and other signals.

If use_for is omitted, the index defaults to ["semantic", "query", "aggregate"] — semantic search works out of the box. The semantic fields come from the derived defaults below; if nothing qualifies as a default, all string fields are embedded. Pass use_for without semantic (e.g. ["query"]) to opt out of embeddings. query and aggregate are interchangeable capabilities. The use_for that get_index() reports back is the index’s effective capability, not a verbatim echo of the request: aggregate is present only when a field is actually groupable, semantic only when a field was embedded, and an index with no records reports []. This keeps list_indexes() from advertising a capability a call would then reject.

Default fields derived from your data

fields is optional, per group: any group you declare is used verbatim, and any group you omit is filled from your actual scene data. Well-known field names get product defaults:

Field / dotted path	Default groups
`text`, `scene_description`	`semantic`
`language`, `brand_names`	`filter`, `aggregate`
`activity`, `location`	`semantic`, `filter`, `aggregate`
`frames.detections.label`	`filter`, `aggregate`
`frames.detections.score`	`filter`, `sort`

Every other field is classified by shape:

Value shape	Default groups
prose strings (descriptions, summaries, transcripts)	`semantic`, plus `filter`/`aggregate` so everything stays queryable
short strings, booleans, lists of strings/numbers	`filter`, `aggregate`
numbers	`filter`, `aggregate`, `sort`
nested dicts, arrays of objects (frames, boxes, raw responses)	none; stored with the record and available via `return_fields`

Two rules to remember:

semantic defaults apply when use_for includes semantic or is omitted (semantic is on by default); if nothing qualifies, all string fields are embedded.
An explicitly empty group opts out and is not refilled: fields={"filter": []} means “no filter fields”.

# No fields config — the server derives from your data. A scene/VLM analyzer emits a
# nested `outputs` object, so each string leaf becomes its own dotted semantic field
# (nested objects are not auto-filtered/aggregated — declare `fields` for that):
index = video.index(source=analyzer, use_for=["semantic", "query"])
print(index.fields)
# {"semantic": ["outputs.scene_description", "outputs.caption", "outputs.action", ...]}

Configure fields

Use fields when you want explicit control over which artifact fields become semantic text, text fields, filters, aggregates, or sort keys. Field names may be dotted paths into nested data. For example, frames.detections.label reaches inside every frame’s detection list. See Index Fields for the full pattern. Fields must exist on the artifact you are indexing (dotted paths are resolved against the scene data; a name that resolves nowhere fails the create with a 400 listing the available fields). Scene descriptions, object detections, and brand detections usually come from separate artifacts, so index them separately:

# Scene/VLM artifact: natural-language scene fields
video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query", "aggregate"],
    fields={
        "semantic": ["scene_description", "activity", "setting"],
        "filter": ["activity", "setting"],
        "aggregate": ["activity", "setting"],
    },
)

# Object detection artifact: dotted paths reach the nested detections
video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["frames.detections.label", "frames.detections.score"],
        "aggregate": ["frames.detections.label"],
        "sort": ["frames.detections.score"],
    },
)

# Brand detection artifact, if generated: brand names and logo signals
brands = understanding.get_analyzer("brands")

video.index(
    name="brands",
    source=brands,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["brand_names"],
        "aggregate": ["brand_names"],
    },
)

Use the objects and brands artifacts as the source of truth for object labels and brand names. Keep the scene index focused on the fields produced by your VLM prompt/schema.

Field group	Used for
`semantic`	direct vector/semantic search with `semantic_search()`; also usable by high-level `search()`. Answers “find talk about X” (meaning).
`filter`	structured `query()` filters (exact match, `contains` substring, ranges)
`aggregate`	`aggregate()` group-by/count/facet operations
`sort`	result ordering

VideoDB still stores additional artifact fields with indexed records. Use return_fields during search to choose which stored fields, such as frames or raw response references, should come back in results. See Index Fields for details.

Index user-provided temporal records

You can also index your own records if they include timestamps. Pass one record dictionary or a list of record dictionaries. VideoDB moves every field other than the reserved fields below into the record’s indexable data.

Field	Type	Required	Description
`start`	number	yes	Start time in seconds.
`end`	number	yes	End time in seconds; it must be later than `start`.
`scene_id`	string	no	Stable record identifier. VideoDB generates one when omitted.
`metadata`	dict	no	Additional scene metadata.
any other field	supported field type	no	Custom data you can configure for semantic search, filtering, aggregation, sorting, or retrieval.

Index field paths refer to custom fields directly. For example, use summary, not data.summary.

custom_scenes = [
    {
        "start": 0.0,
        "end": 12.4,
        "summary": "Opening credits with city skyline.",
        "scene_type": "intro",
    },
    {
        "start": 12.4,
        "end": 45.0,
        "summary": "CEO discusses Q4 results.",
        "scene_type": "presentation",
    },
]

video.index(
    name="chapters",
    source=custom_scenes,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["summary"],
        "filter": ["scene_type"],
    },
)

Index artifacts from sandbox models

Sandbox Compute affects where the analyzer runs. The indexing interface stays the same. If an artifact was produced by a sandbox-backed analyzer, pass that artifact to video.index(...) exactly like a hosted-model artifact:

video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["scene_description", "activity", "setting"],
        "filter": ["activity", "setting"],
    },
)

Use the sandbox docs for sandbox creation, model selection, and sandbox_id validation. Use this page for the artifact-to-index step.

Sandbox Compute

Create a sandbox and route supported model jobs to it.

Sandbox Models

See supported models, tiers, and where to pass model names.

`video.index()` options

Option	Type	Required	Default	Description
`source`	analyzer, dict, list	yes	—	Analyzer object, `{understanding_id, analyzer_id}` reference, artifact output dict, or user-provided temporal records.
`name`	string	no	artifact/source name	User-facing index name (a per-collection schema contract — see above).
`use_for`	list	no	`["semantic", "query", "aggregate"]`	Any of `semantic` (`search` is accepted as an alias), `query`, `aggregate`. The value `get_index()` returns is the index’s effective capability — see the note above.
`fields`	dict	no	derived per group from your data	Field-level indexing configuration; dotted paths supported.
`callback_url`	string	no	none	URL called when indexing completes.

Index status and inspection

Everything is validated synchronously — bad fields, missing sources, and name-contract conflicts fail the create call immediately with a descriptive 400. What runs asynchronously is the embedding work: every semantic index returns as building and flips to ready when its vectors are written — typically within seconds for small indexes. Query-only indexes (no semantic embedding) return ready immediately. Indexing is rows-first: the structured records are stored before the create call returns, so query() and aggregate() work on a building index right away. semantic_search() serves an index only once it is ready. Because the build is asynchronous, the Index object has lifecycle helpers so you do not have to poll get_index() by hand:

Method / property	Purpose
`index.wait_until_complete(timeout=1800, poll_interval=10)`	Block until the build reaches a terminal status (`ready` or `failed`); raises `TimeoutError` if it does not finish in `timeout` seconds.
`index.refresh()`	Re-fetch the manifest and status once, without waiting.
`index.is_complete`	`True` when the status is terminal (`ready` or `failed`).
`index.is_successful`	`True` only when the status is `ready`.

index = video.index(name="scene", source=analyzer, use_for=["semantic", "query"])

# Wait for embeddings to finish (query-only indexes are already ready and return at once):
index.wait_until_complete(timeout=1800, poll_interval=10)

if index.is_successful:
    print(index.status, index.use_for, index.fields)
else:
    print("indexing failed:", index.error)

Common statuses:

Status	Meaning
`building`	Rows are stored and queryable; embeddings are still being written.
`ready`	Fully ready, including semantic search.
`failed`	Indexing failed; the manifest’s `error` carries the reason.

Use refresh() to fetch the latest manifest and update the existing index object:

index.refresh()
print(index.status)

Use wait_until_complete() when your next operation requires a fully built index, such as semantic search. It polls until the index reaches a terminal status: ready or failed.

index.wait_until_complete(timeout=3600, poll_interval=15)

if index.is_successful:
    print("Index is ready for semantic search")
else:
    print(index.error)

The default timeout is 1,800 seconds and the default polling interval is 10 seconds. If the timeout expires first, wait_until_complete() raises TimeoutError. A failed build is a completed build, so the method returns the index with status == "failed"; inspect index.error for the reason. Use video.list_indexes() and video.get_index(...) to reopen or inspect index manifests, field schemas, and record previews. See View Indexes.

Start Here

Core Concepts

Ingest

Understand

Act

Automate

Open Source Frameworks

Try the indexing notebooks

Quick example

Index sources

Choose an index name

Index all artifacts

Declare index capabilities with `use_for`

Default fields derived from your data

Configure fields

Index user-provided temporal records

Index artifacts from sandbox models

Sandbox Compute

Sandbox Models

`video.index()` options

Index status and inspection

Next steps

Index Fields

View Indexes

Try the indexing notebooks

​Quick example

​Index sources

​Choose an index name

​Index all artifacts

​Declare index capabilities with use_for

​Default fields derived from your data

​Configure fields

​Index user-provided temporal records

​Index artifacts from sandbox models

Sandbox Compute

Sandbox Models

​video.index() options

​Index status and inspection

​Next steps

Index Fields

View Indexes

Quick example

Index sources

Choose an index name

Index all artifacts

Declare index capabilities with `use_for`

Default fields derived from your data

Configure fields

Index user-provided temporal records

Index artifacts from sandbox models

`video.index()` options

Index status and inspection

Next steps