Index Fields

fields controls how an artifact becomes an index. An artifact may contain many fields. VideoDB stores the additional artifact data with the indexed records, but only selected fields are optimized for retrieval capabilities:

direct semantic search
exact/text matching
structured filters
aggregation
sorting

Use return_fields at search time to choose which stored fields should come back in results.

Example artifact

A VLM scene artifact may contain scene records like this:

{
  "scene_id": "scene-000001",
  "start": 12.4,
  "end": 18.9,
  "data": {
    "scene_description": "A person walks through a retail aisle while holding a phone.",
    "activity": "walking through store",
    "setting": "retail aisle",
    "frames": [{"timestamp": 14.2, "asset": {"type": "s3_object", "key": "..."}}]
  }
}

A fields config tells VideoDB how to index values that exist on that artifact:

fields={
    "semantic": ["scene_description", "activity", "setting"],
    "filter": ["activity", "setting"],
    "aggregate": ["activity", "setting"],
}

The frames field is still stored with the record. Request it later with return_fields when you need it in search results. Object labels and brand names usually live on separate objects and brands artifacts.

Dotted paths for nested data

Field names may be dotted paths that reach inside nested objects, so you do not need to flatten your data. When a path crosses a list, values are collected from every element. An object-detection scene looks like this:

{
  "scene_id": "scene-0.000-8.634",
  "start": 0.0,
  "end": 8.634,
  "data": {
    "frames": [
      {
        "frame_id": "frame-scene-0.000-8.634-7.000",
        "timestamp": 7.0,
        "detections": [
          {
            "label": "person",
            "score": 0.5856,
            "box": {
              "box": [0.0002, 0.2332, 0.6967, 0.9947],
              "unit": "normalized"
            }
          },
          {
            "label": "dog",
            "score": 0.5041,
            "box": {
              "box": [0.0018, 0.2346, 0.6908, 0.7378],
              "unit": "normalized"
            }
          }
        ]
      }
    ]
  },
  "metadata": {}
}

Index the nested values directly:

video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["frames.detections.label", "frames.detections.score"],
        "aggregate": ["frames.detections.label"],
        "sort": ["frames.detections.score"],
    },
)

frames.detections.label becomes the list of every detection label in the scene (["person", "dog"]); frames.detections.score becomes the list of every score. Bounding boxes remain nested under frames[].detections[].box, and the raw frames structure stays stored-only; request the objects index data with return_fields when you need timestamps or boxes. Dotted paths work everywhere a field name does: fields groups, query() filters, aggregate(group_by=...), and targeted semantic refs (index_names=["scene.outputs.location"]).

Field groups

Group	Values	Used for
`semantic`	list of text-like field names	direct semantic/vector search with `semantic_search()`; also usable by high-level `search()`
`filter`	list of field names	structured `query()` filters
`aggregate`	list of field names	`aggregate()` count/group/facet operations
`sort`	list of field names	result ordering

Do not add clip to fields. Clips are generated automatically from video_id, start, and end for moment results. Use include_clip=False at retrieval time if you do not need playable clips.

Supported field value types

Type	Example	Common use
`string`	`"Nike"`	filter, aggregate
`text`	`"A person walks through a store."`	semantic/text retrieval
`number`	`0.94`	filter, sort, aggregate
`boolean`	`true`	filter, aggregate
`string_array`	`["person", "phone"]`	contains filters, facets (an array of objects is reported as `string_array`)
`number_array`	`[0.91, 0.88]`	advanced filters, returned fields
`object`	`{...}`	stored record detail, returned fields; index inner values with dotted paths

Nested values you filter or aggregate on should be declared with their dotted path. Declaring them projects the values into the queryable store at ingest time, so retrieval stays fast.

Default fields

fields is optional per group. Any group you omit is derived from your data: well-known names get product defaults (text/scene_description → semantic, language/brand_names → filter + aggregate, frames.detections.label → filter + aggregate, frames.detections.score → filter + sort), and everything else is classified by shape — prose becomes semantic, scalars and scalar lists become filter + aggregate (numbers also sort), and nested structures stay stored-only. Groups you declare are used verbatim, and an explicit empty group ("filter": []) opts out. See Create an Index for the full derivation tables.

# Declare only what you want to control; the rest is derived:
video.index(
    name="scene",
    source=analyzer,
    use_for=["semantic", "query"],
    fields={"semantic": ["outputs.description"]},  # filter/aggregate still derived
)

User-provided temporal records

If you index your own timestamped records, choose fields the same way:

fields={
    "semantic": ["summary"],
    "filter": ["scene_type"],
    "aggregate": ["scene_type"],
}

Return fields at retrieval time

Indexing does not decide the result payload. Search does. Use return_fields to choose which stored fields to include in each result:

results = video.semantic_search(
    query="happy reunion outside a shop",
    index_names=["scene"],
    return_fields=["scene_description", "activity", "frames"],
)

For multi-index search, namespace fields by index name:

results = collection.search(
    query="person talking about Nike while holding a phone",
    return_fields={
        "scene": ["scene_description", "frames"],
        "transcript": ["text"],
        "objects": ["object_labels", "frames"],
        "brands": ["brand_names"],
    },
)

For debugging or inspection, request all stored fields:

results = video.search(
    query="product demo moments",
    return_fields="all",
)

Returned fields can make responses larger. Keep them narrow for broad searches and request detailed fields only when you need to display or inspect them.

Filter syntax

Query filters use fields that were indexed in fields["filter"], including dotted paths.

Filter type	Example
equality	`{"activity": "walking through store"}`
contains	`{"frames.detections.label": {"contains": "phone"}}`
numeric range	`{"frames.detections.score": {">=": 0.9}}`
boolean	`{"is_song": True}`
multiple conditions	`{"activity": "walking", "setting": "retail aisle"}`

How filters behave on list-valued fields (anything a dotted path collected across a list, or a plain array field):

equality means element membership: {"frames.detections.label": "phone"} matches a scene where any detection is a phone
numeric comparators match any element: {"frames.detections.score": {">=": 0.9}} matches if any detection scores ≥ 0.9
contains is a per-element substring match: "car" matches ["car", "truck"] but not ["cartoon"]

Aggregation syntax

Aggregation uses fields that were indexed in fields["aggregate"].

collection.aggregate(
    index_name="brands",
    group_by="brand_names",
)

# Dotted paths group by nested values, with one group per detection label
collection.aggregate(
    index_name="objects",
    group_by="frames.detections.label",
    metric="count",
)

List-valued fields explode to one group per element (each detection label counts separately). Dict-valued fields ({"person": 3, "phone": 1}) explode to one group per key, and numeric metrics (sum/avg/min/max) apply to the dict’s values. Common metrics:

Metric	Meaning
`count`	count matching scenes/records
`sum(field)`	sum numeric field values
`avg(field)`	average numeric field values
`min(field)`	minimum value
`max(field)`	maximum value

Best practices

Put natural-language descriptions in fields["semantic"].
Put exact labels, enums, numbers, and booleans in fields["filter"].
Put fields you want to count or facet in fields["aggregate"].
Put fields you want to order by in fields["sort"].
Keep indexing focused on retrieval capabilities; request display/debug details with return_fields during search.
Reach into nested data with dotted paths (frames.detections.label) instead of reshaping your artifacts. Declare the paths in fields so the values are projected for fast retrieval.

Start Here

Core Concepts

Ingest

Understand

Act

Automate

Open Source Frameworks

Example artifact

Dotted paths for nested data

Field groups

Supported field value types

Default fields

User-provided temporal records

Return fields at retrieval time

Filter syntax

Aggregation syntax

Best practices

Next steps

Create an Index

View Indexes

​Example artifact

​Dotted paths for nested data

​Field groups

​Supported field value types

​Default fields

​User-provided temporal records

​Return fields at retrieval time

​Filter syntax

​Aggregation syntax

​Best practices

​Next steps

Create an Index

View Indexes

Example artifact

Dotted paths for nested data

Field groups

Supported field value types

Default fields

User-provided temporal records

Return fields at retrieval time

Filter syntax

Aggregation syntax

Best practices

Next steps