> ## Documentation Index
> Fetch the complete documentation index at: https://docs.videodb.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Create an Index
> Turn understanding artifacts into semantic, queryable, and aggregatable indexes.
After you create an understanding, you can turn its artifacts into indexes.
An index stores structured information extracted from the video, including transcript text, scene descriptions, object labels, brands, timestamps, and custom fields, in a retrieval-ready form. Your app or agent can then search, filter, query, and aggregate over that information instead of re-analyzing the video.
```text theme={null}
Understanding → Artifact → Index → search/query/aggregate
```
Understanding and indexing are separate. Understanding extracts reusable facts from the video. Indexing decides which of those facts should become searchable, filterable, aggregatable, or available as result context. You can create multiple indexes from the same artifact for different retrieval needs.
Explore runnable examples for turning understanding artifacts into indexes.
***
## Quick example
```python theme={null}
# Create artifacts
understanding = video.understand(
analyzers=[
{"type": "spoken_words"},
{"type": "vlm"},
],
)
understanding.wait_until_complete()
transcript = understanding.get_analyzer("transcript")
scene = understanding.get_analyzer("scene")
# Index artifacts with defaults
video.index(source=transcript)
video.index(source=scene)
```
VideoDB derives sensible defaults from your data. If `name` is omitted, VideoDB uses the artifact name.
***
## Index sources
`source` accepts three forms:
```python theme={null}
# 1. An analyzer object is the lightest path. Only identifiers travel to the server;
# the server fetches the analyzer's output from its own store.
analyzer = understanding.get_analyzer("scene")
video.index(name="scene", source=analyzer)
# 2. A reference dict works the same way when you only have the ids.
video.index(
name="scene",
source={"understanding_id": "und_123", "analyzer_id": "an_456"},
)
# 3. Your own temporal records.
video.index(name="chapters", source=custom_scenes)
```
Pass analyzer outputs using forms 1 and 2 so the scene payload never round-trips through your client. For custom records, see the [user-provided temporal record format](#index-user-provided-temporal-records).
## Choose an index name
```python theme={null}
location_analyzer = understanding.get_analyzer("location")
index = video.index(
name="location",
source=location_analyzer,
)
```
Use the same index name to group the same kind of data across videos in a collection. For example, name the location index on every video `"location"`; you can then target that logical group with `collection.semantic_search(index_names=["location"])`, `collection.query(index_name="location")`, or `collection.aggregate(index_name="location")`.
Indexes that share a name must also share the same field structure. This keeps collection-level retrieval consistent because `"location"` always represents the same fields. If you create another `"location"` index with a different structure, VideoDB returns a `400` showing the existing and incoming structures. See [Collection Search](/pages/understand/search-and-retrieval/collection-search) for index-specific retrieval across videos.
## Index all artifacts
```python theme={null}
for analyzer in understanding.list_analyzers():
if analyzer.status == "done":
video.index(name=analyzer.name, source=analyzer)
```
***
## Declare index capabilities with `use_for`
Use `use_for` to say which retrieval capabilities the index should support.
```python theme={null}
video.index(
name="scene",
source=scene,
use_for=["semantic", "query", "aggregate"],
)
```
| Value | Meaning | Example |
| - | - | - |
| `semantic` | direct semantic/vector search | "happy reunion outside a shop" |
| `query` | exact structured filtering | `object_labels contains "phone"` |
| `aggregate` | counts, groups, facets, analytics | count scenes by brand |
`semantic` describes an index capability. Use `semantic_search()` for direct vector similarity over semantic fields. Use `search()` for high-level natural-language search that may combine semantic indexes with filters and other signals.
If `use_for` is omitted, the index defaults to `["semantic", "query", "aggregate"]` — semantic search works out of the box. The semantic fields come from the derived defaults below; if nothing qualifies as a default, **all string fields are embedded**. Pass `use_for` without `semantic` (e.g. `["query"]`) to opt out of embeddings. `query` and `aggregate` are interchangeable capabilities.
The `use_for` that `get_index()` reports back is the index's **effective** capability, not a verbatim echo of the request: `aggregate` is present only when a field is actually groupable, `semantic` only when a field was embedded, and an index with no records reports `[]`. This keeps `list_indexes()` from advertising a capability a call would then reject.
***
## Default fields derived from your data
`fields` is optional, **per group**: any group you declare is used verbatim, and any group you omit is filled from your actual scene data.
Well-known field names get product defaults:
| Field / dotted path | Default groups |
| - | - |
| `text`, `scene_description` | `semantic` |
| `language`, `brand_names` | `filter`, `aggregate` |
| `activity`, `location` | `semantic`, `filter`, `aggregate` |
| `frames.detections.label` | `filter`, `aggregate` |
| `frames.detections.score` | `filter`, `sort` |
Every other field is classified by shape:
| Value shape | Default groups |
| - | - |
| prose strings (descriptions, summaries, transcripts) | `semantic`, plus `filter`/`aggregate` so everything stays queryable |
| short strings, booleans, lists of strings/numbers | `filter`, `aggregate` |
| numbers | `filter`, `aggregate`, `sort` |
| nested dicts, arrays of objects (frames, boxes, raw responses) | none; stored with the record and available via `return_fields` |
Two rules to remember:
* `semantic` defaults apply when `use_for` includes `semantic` or is omitted (semantic is on by default); if nothing qualifies, all string fields are embedded.
* An explicitly empty group opts out and is **not** refilled: `fields={"filter": []}` means "no filter fields".
```python theme={null}
# No fields config — the server derives from your data. A scene/VLM analyzer emits a
# nested `outputs` object, so each string leaf becomes its own dotted semantic field
# (nested objects are not auto-filtered/aggregated — declare `fields` for that):
index = video.index(source=analyzer, use_for=["semantic", "query"])
print(index.fields)
# {"semantic": ["outputs.scene_description", "outputs.caption", "outputs.action", ...]}
```
***
## Configure fields
Use `fields` when you want explicit control over which artifact fields become semantic text, text fields, filters, aggregates, or sort keys.
Field names may be **dotted paths into nested data**. For example, `frames.detections.label` reaches inside every frame's detection list. See [Index Fields](/pages/understand/indexing-pipelines/multiple-indexes) for the full pattern.
Fields must exist on the artifact you are indexing (dotted paths are resolved against the scene data; a name that resolves nowhere fails the create with a `400` listing the available fields). Scene descriptions, object detections, and brand detections usually come from separate artifacts, so index them separately:
```python theme={null}
# Scene/VLM artifact: natural-language scene fields
video.index(
name="scene",
source=scene,
use_for=["semantic", "query", "aggregate"],
fields={
"semantic": ["scene_description", "activity", "setting"],
"filter": ["activity", "setting"],
"aggregate": ["activity", "setting"],
},
)
# Object detection artifact: dotted paths reach the nested detections
video.index(
name="objects",
source=objects,
use_for=["query", "aggregate"],
fields={
"filter": ["frames.detections.label", "frames.detections.score"],
"aggregate": ["frames.detections.label"],
"sort": ["frames.detections.score"],
},
)
# Brand detection artifact, if generated: brand names and logo signals
brands = understanding.get_analyzer("brands")
video.index(
name="brands",
source=brands,
use_for=["query", "aggregate"],
fields={
"filter": ["brand_names"],
"aggregate": ["brand_names"],
},
)
```
Use the `objects` and `brands` artifacts as the source of truth for object labels and brand names. Keep the `scene` index focused on the fields produced by your VLM prompt/schema.
| Field group | Used for |
| - | - |
| `semantic` | direct vector/semantic search with `semantic_search()`; also usable by high-level `search()`. Answers "find talk **about** X" (meaning). |
| `filter` | structured `query()` filters (exact match, `contains` substring, ranges) |
| `aggregate` | `aggregate()` group-by/count/facet operations |
| `sort` | result ordering |
VideoDB still stores additional artifact fields with indexed records. Use `return_fields` during search to choose which stored fields, such as frames or raw response references, should come back in results.
See [Index Fields](/pages/understand/indexing-pipelines/multiple-indexes) for details.
***
## Index user-provided temporal records
You can also index your own records if they include timestamps. Pass one record dictionary or a list of record dictionaries. VideoDB moves every field other than the reserved fields below into the record's indexable `data`.
| Field | Type | Required | Description |
| - | - | -: | - |
| `start` | number | yes | Start time in seconds. |
| `end` | number | yes | End time in seconds; it must be later than `start`. |
| `scene_id` | string | no | Stable record identifier. VideoDB generates one when omitted. |
| `metadata` | dict | no | Additional scene metadata. |
| any other field | [supported field type](/pages/understand/indexing-pipelines/multiple-indexes#supported-field-value-types) | no | Custom data you can configure for semantic search, filtering, aggregation, sorting, or retrieval. |
Index field paths refer to custom fields directly. For example, use `summary`, not `data.summary`.
```python theme={null}
custom_scenes = [
{
"start": 0.0,
"end": 12.4,
"summary": "Opening credits with city skyline.",
"scene_type": "intro",
},
{
"start": 12.4,
"end": 45.0,
"summary": "CEO discusses Q4 results.",
"scene_type": "presentation",
},
]
video.index(
name="chapters",
source=custom_scenes,
use_for=["semantic", "query"],
fields={
"semantic": ["summary"],
"filter": ["scene_type"],
},
)
```
***
## Index artifacts from sandbox models
Sandbox Compute affects where the analyzer runs. The indexing interface stays the same.
If an artifact was produced by a sandbox-backed analyzer, pass that artifact to `video.index(...)` exactly like a hosted-model artifact:
```python theme={null}
video.index(
name="scene",
source=scene,
use_for=["semantic", "query"],
fields={
"semantic": ["scene_description", "activity", "setting"],
"filter": ["activity", "setting"],
},
)
```
Use the sandbox docs for sandbox creation, model selection, and `sandbox_id` validation. Use this page for the artifact-to-index step.
Create a sandbox and route supported model jobs to it.
See supported models, tiers, and where to pass model names.
***
## `video.index()` options
| Option | Type | Required | Default | Description |
| - | -: | -: | - | - |
| `source` | analyzer, dict, list | yes | — | Analyzer object, `{understanding_id, analyzer_id}` reference, artifact output dict, or user-provided temporal records. |
| `name` | string | no | artifact/source name | User-facing index name (a per-collection schema contract — see above). |
| `use_for` | list | no | `["semantic", "query", "aggregate"]` | Any of `semantic` (`search` is accepted as an alias), `query`, `aggregate`. The value `get_index()` returns is the index's *effective* capability — see the note above. |
| `fields` | dict | no | derived per group from your data | Field-level indexing configuration; dotted paths supported. |
| `callback_url` | string | no | none | URL called when indexing completes. |
***
## Index status and inspection
Everything is validated synchronously — bad fields, missing sources, and name-contract conflicts fail the create call immediately with a descriptive `400`. What runs asynchronously is the embedding work: every semantic index returns as `building` and flips to `ready` when its vectors are written — typically within seconds for small indexes. Query-only indexes (no semantic embedding) return `ready` immediately.
Indexing is **rows-first**: the structured records are stored before the create call returns, so `query()` and `aggregate()` work on a `building` index right away. `semantic_search()` serves an index only once it is `ready`.
Because the build is asynchronous, the `Index` object has lifecycle helpers so you do not have to poll `get_index()` by hand:
| Method / property | Purpose |
| - | - |
| `index.wait_until_complete(timeout=1800, poll_interval=10)` | Block until the build reaches a terminal status (`ready` or `failed`); raises `TimeoutError` if it does not finish in `timeout` seconds. |
| `index.refresh()` | Re-fetch the manifest and status once, without waiting. |
| `index.is_complete` | `True` when the status is terminal (`ready` or `failed`). |
| `index.is_successful` | `True` only when the status is `ready`. |
```python theme={null}
index = video.index(name="scene", source=analyzer, use_for=["semantic", "query"])
# Wait for embeddings to finish (query-only indexes are already ready and return at once):
index.wait_until_complete(timeout=1800, poll_interval=10)
if index.is_successful:
print(index.status, index.use_for, index.fields)
else:
print("indexing failed:", index.error)
```
Common statuses:
| Status | Meaning |
| - | - |
| `building` | Rows are stored and queryable; embeddings are still being written. |
| `ready` | Fully ready, including semantic search. |
| `failed` | Indexing failed; the manifest's `error` carries the reason. |
Use `refresh()` to fetch the latest manifest and update the existing index object:
```python theme={null}
index.refresh()
print(index.status)
```
Use `wait_until_complete()` when your next operation requires a fully built index, such as semantic search. It polls until the index reaches a terminal status: `ready` or `failed`.
```python theme={null}
index.wait_until_complete(timeout=3600, poll_interval=15)
if index.is_successful:
print("Index is ready for semantic search")
else:
print(index.error)
```
The default timeout is 1,800 seconds and the default polling interval is 10 seconds. If the timeout expires first, `wait_until_complete()` raises `TimeoutError`. A failed build is a completed build, so the method returns the index with `status == "failed"`; inspect `index.error` for the reason.
Use `video.list_indexes()` and `video.get_index(...)` to reopen or inspect index manifests, field schemas, and record previews. See [View Indexes](/pages/understand/indexing-pipelines/view-indexes).
***
## Next steps
Learn how fields become semantic text, filters, facets, and sort keys.
Inspect index status, fields, schema, and records.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.