Core Concepts Overview

The Platform Loop

See (Ingest) → Process → Understand (Indexes) → Remember → Retrieve (Search) → Act

import videodb

conn = videodb.connect()
coll = conn.get_collection()

# SEE: Ingest from any source
video = coll.upload(url="https://example.com/video.mp4")

# UNDERSTAND: Create a reusable scene artifact and index it
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {"prompt": "Extract key moments"},
        }
    ]
)
understanding.wait_until_complete()

scene = understanding.get_analyzer("scene")
scene_index = video.index(name="key_moments", source=scene)
scene_index.wait_until_complete()

# RETRIEVE: Search with natural language
results = video.semantic_search(
    query="important announcement",
    index_ids=[scene_index.index_id],
    top_k=5,
)

# ACT: Generate outputs, trigger actions
for shot in results.shots:
    print(f"{shot.start}s - {shot.end}s")
    shot.play()  # Playable evidence

See (Ingest)

Get video and audio from anywhere into VideoDB.

Source	Method
File URL	`coll.upload(url="https://...")`
Local file	`coll.upload(file_path="./video.mp4")`
RTSP stream	`coll.connect_rtstream(url="rtsp://...", name="Camera")`
Desktop capture	Capture SDK (screen, mic, camera)

# File-based
video = coll.upload(url="https://example.com/meeting.mp4")

# Live stream
rtstream = coll.connect_rtstream(
    name="Security Camera",
    url="rtsp://user:pass@host:554/stream"
)

Process

Understanding configuration converts raw media into reusable, processable artifacts before indexing.

Scene segmentation - Time-based or shot-based
Frame sampling - Control which frames to analyze
Audio chunking - Word, sentence, or time-based segments

# Time-based finite-video analysis: one frame every 10 seconds
understanding = video.understand(
    segmentation={"type": "time", "seconds": 10},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe the scene."},
        }
    ],
)
understanding.wait_until_complete()

# RTStreams continuously produce VLM output descriptors.
live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe this stream window."},
        }
    ],
    store=True,
)
live_scene_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)

This is where cost control happens - sampling policies trade compute for recall.

Understand (Indexes)

Indexes are programmable interpretation layers. You define what to extract with prompts.

Prompt-driven - Natural language instructions
Model-orchestrated - LLMs and VLMs do the work
Additive - Multiple indexes on same media
Multimodal - Visual and spoken

# Create visual and spoken artifacts in one finite understanding run.
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "visual",
            "config": {"prompt": "Identify key moments and describe activities"},
        },
        {"type": "spoken_words", "name": "transcript"},
    ]
)
understanding.wait_until_complete()

visual = understanding.get_analyzer("visual")
transcript = understanding.get_analyzer("transcript")
visual_index = video.index(name="visual", source=visual)
transcript_index = video.index(name="transcript", source=transcript)
visual_index.wait_until_complete()
transcript_index.wait_until_complete()

Remember

Indexes are stored as episodic memory. This is automatic by default. What gets stored:

Transcripts and embeddings
Scene descriptions and tags
Structured metadata
Retrieval structures

For a live stream, set store=True when you plan to index its continuous VLM output:

live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe this stream window."},
        }
    ],
    store=True,
)
live_scene_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)

Retrieve (Search)

Search across indexed content with natural language. Results include playable evidence.

# Single video
results = video.semantic_search(
    query="product demo",
    index_ids=[visual_index.index_id],
    top_k=10,
)

# Single stream
results = rtstream.search("intrusion", index_id=live_scene_index.id)

# Collection-wide
results = coll.search("quarterly results", top_k=10)

Results include:

Timestamps - Exact start/end times
Text - What was detected
Score - Relevance ranking
Stream URL - Playable link

for shot in results.shots:
    print(f"{shot.start}s - {shot.end}s")
    shot.play()  # Verify the result

Act

Go from understanding to automation and outputs.

Event Detection

Continuously understand and index the live VLM output before adding an alert:

live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe activity in this stream window."},
        }
    ],
    store=True,
)
live_scene_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)

event_id = conn.create_event(
    event_prompt="Detect intruder",
    label="security_alert"
)

live_scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts"
)

Programmable Editing

Compose outputs using the 4-layer editor architecture:

from videodb.editor import Timeline, Track, Clip, VideoAsset

video_asset = VideoAsset(id=video.id, start=10)
clip = Clip(asset=video_asset, duration=20)

track = Track()
track.add_clip(0, clip)

timeline = Timeline(conn)
timeline.add_track(track)
output = timeline.generate_stream()

Architecture Patterns

The loop applies to different use cases:

Use Case	See	Understand	Act
Video Search	Upload files	Index with domain prompts	Search + retrieve
Monitoring	Connect RTSP	Real-time indexing	Alerts + webhooks
Desktop Agent	Capture SDK	Index screen/mic	Context for LLM
Media Automation	Upload + transcode	Index for editing	Timeline + export

Next Steps

Data Model

Collections, Videos, RTStreams, and other core objects

Indexes

Turn media into searchable knowledge

Search & Retrieval

How search returns playable evidence

Events & Alerts

Real-time detection and automation

Start Here

Core Concepts

Ingest

Understand

Act

Automate

MCP Server

Open Source Frameworks

The Platform Loop

See (Ingest)

Process

Understand (Indexes)

Remember

Retrieve (Search)

Act

Event Detection

Programmable Editing

Architecture Patterns

Next Steps

Data Model

Indexes

Search & Retrieval

Events & Alerts

​The Platform Loop

​See (Ingest)

​Process

​Understand (Indexes)

​Remember

​Retrieve (Search)

​Act

​Event Detection

​Programmable Editing

​Architecture Patterns

​Next Steps

Data Model

Indexes

Search & Retrieval

Events & Alerts

The Platform Loop

See (Ingest)

Process

Understand (Indexes)

Remember

Retrieve (Search)

Act

Event Detection

Programmable Editing

Architecture Patterns

Next Steps