> ## Documentation Index
> Fetch the complete documentation index at: https://docs.videodb.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Understand & Index RTStreams

> Use the new Understand → Index interface with RTStreams for continuous VLM analysis and search.

RTStreams use the same retrieval model as videos:

```text theme={null}
Understand → Index → Retrieve
```

The difference is lifecycle. A video understanding job finishes when the file is processed. An RTStream understanding job is continuous: it keeps processing new windows while the stream and job are running.

<Note>
  Initial RTStream support for the new interface is limited to **VLM visual understanding** with **time-based segmentation**. Other analyzers and segmentation modes will be added later.
</Note>

***

## What is supported

| Capability   | Initial RTStream support             |
| ------------ | ------------------------------------ |
| Analyzer     | `vlm` only                           |
| Segmentation | `{"type": "time"}` only              |
| Sampling     | Frames per time window               |
| Index source | VLM understanding output             |
| Job mode     | Continuous                           |
| Retrieval    | RTStream search over indexed windows |
| Alerts       | Alerts attached to RTStream indexes  |

Unsupported analyzer types such as `spoken_words`, `object_detection`, `ocr`, `faces`, `brands`, `activity_recognition`, and `location_detection` should return a validation error until support is rolled out.

***

## Quick example

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
rtstream = coll.get_rtstream("rts-xxx")

# 1. Continuously understand the live stream with a VLM
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "10s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 5},
            "config": {
                "prompt": "Describe the scene, people, activity, and unusual events.",
                "model": "base",
            },
        }
    ],
    store=True,
)

# 2. Continuously index the VLM output for retrieval
index = rtstream.index(
    name="scene_search",
    source=understanding.outputs["scene"],
    use_for=["semantic"],
)

# 3. Search indexed stream windows
results = rtstream.search(
    query="person entering with a package",
    index_id=index.id,
)

for shot in results.shots:
    print(shot.start, shot.end, shot.text)
    shot.play()
```

***

## 1. Understand the stream

Use `rtstream.understand(...)` to start a continuous understanding job. It returns an `RTStreamUnderstanding` object containing the job ID, status, outputs, and lifecycle methods.

```python theme={null}
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {
                "prompt": "Describe what is happening in this time window."
            },
        }
    ],
    store=True,
    ws_connection_id=ws.connection_id,  # optional
)
```

### Parameters

| Field                              | Description                                                    |
| ---------------------------------- | -------------------------------------------------------------- |
| `segmentation.type`                | Must be `"time"` in the initial rollout.                       |
| `segmentation.window`              | Duration of each stream window, for example `"5s"` or `"10s"`. |
| `analyzers`                        | Must contain one `vlm` analyzer in the initial rollout.        |
| `analyzers[].name`                 | Output name. Use `"scene"` unless you need a custom key.       |
| `analyzers[].sampling.frame_count` | Number of frames sampled per time window.                      |
| `analyzers[].config.prompt`        | Prompt used to describe or structure each window.              |
| `store`                            | Use `True` when you plan to index the output.                  |
| `ws_connection_id`                 | Optional WebSocket connection for real-time updates.           |

***

## 2. Get the continuous output

The object returned by `rtstream.understand(...)` already contains the output descriptors:

```python theme={null}
scene_output = understanding.outputs["scene"]
```

To reopen the job later, fetch it by its string ID:

```python theme={null}
understanding = rtstream.get_understanding("und-xxx")
```

For videos, an understanding output can be a finite artifact. For RTStreams, the output is a continuous source descriptor that points to records produced over time.

```json theme={null}
{
  "type": "understanding",
  "asset_type": "rtstream",
  "rtstream_id": "rts-xxx",
  "understanding_id": "und-xxx",
  "output": "scene",
  "extract_type": "vlm",
  "mode": "continuous"
}
```

You can inspect produced records by time range:

```python theme={null}
records = understanding.get_records(
    start=1711000000,
    end=1711000600,
    page=1,
    page_size=100,
)
```

***

## 3. Index the output

Use `rtstream.index(...)` to materialize the continuous VLM output for search. It returns an `RTStreamIndex` object with its ID and lifecycle methods.

```python theme={null}
index = rtstream.index(
    name="front_door_activity",
    source=understanding.outputs["scene"],
    use_for=["semantic"],
)
```

If `use_for` is omitted, VideoDB chooses the default capabilities for the VLM scene output. In the initial RTStream rollout, the primary capability is semantic search over scene descriptions.

***

## 4. Manage the continuous jobs

RTStream understanding and index jobs have continuous lifecycle controls.

```python theme={null}
# Reopen jobs later when you only have their string IDs
understanding = rtstream.get_understanding("und-xxx")
index = rtstream.get_index("idx-xxx")

# Pause or resume understanding
understanding.stop()
understanding.start()

# Pause or resume indexing
index.stop()
index.start()
```

| State     | Meaning                                                                  |
| --------- | ------------------------------------------------------------------------ |
| `running` | The job processes new stream windows.                                    |
| `stopped` | The job stops processing new windows. Existing records remain available. |
| `failed`  | The job stopped because of an error.                                     |

Stopping the RTStream itself disconnects the media source. Stopping understanding or indexing only pauses that processing layer.

***

## 5. Retrieve and play results

Search works like existing RTStream search: results are timestamped stream shots.

```python theme={null}
results = rtstream.search(
    query="delivery person at the entrance",
    index_id=index.id,
    score_threshold=0.5,
)

for shot in results.shots:
    print(shot.start, shot.end, shot.text)
    stream_url = shot.generate_stream()
```

Each shot includes Unix timestamps. Use `shot.generate_stream()` or `shot.play()` to create a playable stream for that time range.

***

## 6. Add alerts

Alerts remain attached to RTStream indexes.

```python theme={null}
event_id = conn.create_event(
    event_prompt="Detect when someone enters the restricted area",
    label="restricted_area_entry",
)

alert_id = index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/webhooks/alerts",
    ws_connection_id=ws.connection_id,  # optional
)
```

Manage alerts from the index:

```python theme={null}
alerts = index.list_alerts()
index.disable_alert(alert_id)
index.enable_alert(alert_id)
```

***

## How this differs from videos

| Video                                                     | RTStream                                               |
| --------------------------------------------------------- | ------------------------------------------------------ |
| Finite media asset                                        | Continuous live source                                 |
| `video.understand(...)` eventually completes              | `rtstream.understand(...)` keeps running               |
| `video.index(...)` indexes a completed or stored artifact | `rtstream.index(...)` continuously indexes new records |
| Timestamps are relative seconds in the video              | Timestamps are Unix timestamps                         |
| Alerts are not part of the initial video index lifecycle  | Alerts are supported on RTStream indexes               |

***

## Compatibility with existing helpers

Existing RTStream helpers remain supported during migration:

```python theme={null}
scene_index = rtstream.index_visuals(
    prompt="Describe activity and detect unusual behavior",
    batch_config={"type": "time", "value": 5, "frame_count": 2},
)
```

The new interface separates that flow into two explicit steps:

```python theme={null}
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe activity and detect unusual behavior"},
        }
    ],
    store=True,
)

index = rtstream.index(source=understanding.outputs["scene"])
```

Use the new interface for new VLM-based RTStream workloads. Use the existing helpers while migrating older applications.

***

## Target API routes

The SDK methods map to these RTStream routes:

| SDK method                                           | Route                                                                    |
| ---------------------------------------------------- | ------------------------------------------------------------------------ |
| `rtstream.understand(...)`                           | `POST /rtstream/{stream_id}/understand`                                  |
| `rtstream.list_understanding()`                      | `GET /rtstream/{stream_id}/understand`                                   |
| `rtstream.get_understanding(id)`                     | `GET /rtstream/{stream_id}/understand/{understanding_id}`                |
| `understanding.start()` / `understanding.stop()`     | `PATCH /rtstream/{stream_id}/understand/{understanding_id}/status`       |
| `understanding.get_records(...)`                     | `GET /rtstream/{stream_id}/understand/{understanding_id}/records`        |
| `rtstream.index(...)`                                | `POST /rtstream/{stream_id}/indexes`                                     |
| `rtstream.list_indexes()`                            | `GET /rtstream/{stream_id}/indexes`                                      |
| `rtstream.get_index(id)`                             | `GET /rtstream/{stream_id}/indexes/{index_id}`                           |
| `index.start()` / `index.stop()`                     | `PATCH /rtstream/{stream_id}/indexes/{index_id}/status`                  |
| `index.get_records(...)`                             | `GET /rtstream/{stream_id}/indexes/{index_id}/records`                   |
| `index.create_alert(...)`                            | `POST /rtstream/{stream_id}/indexes/{index_id}/alert`                    |
| `index.list_alerts()`                                | `GET /rtstream/{stream_id}/indexes/{index_id}/alert`                     |
| `index.enable_alert(id)` / `index.disable_alert(id)` | `PATCH /rtstream/{stream_id}/indexes/{index_id}/alert/{alert_id}/status` |

***

## Next steps

<CardGroup cols={2}>
  <Card icon="radio" title="Real-time APIs" href="/pages/ingest/live-streams/realtime-apis">
    Existing RTStream transcription, search, alerts, and stream playback APIs.
  </Card>

  <Card icon="zap" title="Events & Real-time" href="/pages/core-concepts/events-and-realtime">
    Build event rules and deliver alerts over webhooks or WebSocket.
  </Card>
</CardGroup>