# Dub video with AI-generated audio
Source: https://docs.videodb.io/api-reference/ai-generation/dub-video-with-ai-generated-audio

/openapi.yaml post /collection/{collection_id}/generate/video/dub



# Translate video content
Source: https://docs.videodb.io/api-reference/ai-generation/translate-video-content

/openapi.yaml post /collection/{collection_id}/video/{video_id}/translate



# Assets
Source: https://docs.videodb.io/api-reference/assets/index

List and filter assets across collections

Browse and filter all media assets (videos, audio, images) across your collections.

## Endpoints

> **[List Assets](/api-reference/assets/list_assets)**
>
> List and filter all assets


# List all assets across collections
Source: https://docs.videodb.io/api-reference/assets/list-all-assets-across-collections

/openapi.yaml get /assets



# List Assets
Source: https://docs.videodb.io/api-reference/assets/list_assets

GET /assets
List all assets across collections with filtering and pagination

List all media assets across your collections with powerful filtering options. Filter by asset type, duration, size, name pattern, and more.

> [!NOTE]
>
> * Filter by `asset_type`: comma-separated values like `video,audio,image`
> * Sort by `name`, `duration`, `size`, or `created_at`
> * Use `min_duration`/`max_duration` and `min_size`/`max_size` for range filters
> * Use `name_pattern` for regex-based name filtering
> * Filter by `collection_id` to scope to a specific collection

> **[List Videos](/api-reference/videos/list_videos)**
>
> List videos in a collection

> **[List Audio](/api-reference/audio/list_audio)**
>
> List audio in a collection


# Create Audio Transcription
Source: https://docs.videodb.io/api-reference/audio/create_audio_transcription

POST /audio/{audio_id}/transcription/
Generate a transcription for an audio file

Generate a speech-to-text transcription for a specific audio file.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
audio = coll.get_audio("audio_id")

# Generate transcript for audio
audio.generate_transcript()

# Generate with language hint
audio.generate_transcript(language_code="es")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const audio = await coll.getAudio('audio_id');

// Generate transcript for audio
await audio.generateTranscript();

// Generate with language hint
await audio.generateTranscript(false, 'es');
```

> [!NOTE]
>
> * Transcription is asynchronous; use the GET endpoint to retrieve results
> * Supports different transcription engines via the `engine` parameter
> * Use `callback_url` for webhook notification when transcription completes

> **[Get Transcription](/api-reference/audio/get_audio_transcription)**
>
> Retrieve completed transcription

> **[Video Transcription](/api-reference/videos/transcription/create_video_transcription)**
>
> Transcribe video audio tracks


# Delete Audio File
Source: https://docs.videodb.io/api-reference/audio/delete_audio

DELETE /audio/{audio_id}
Permanently delete an audio file from your collection

Permanently remove an audio file and its metadata from your collection.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
audio = coll.get_audio("a-abc123def")

# Delete audio file
audio.delete()
print("Audio file deleted")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const audio = await coll.getAudio("a-abc123def");

// Delete audio file
await audio.delete();
console.log("Audio file deleted");
```

> [!WARNING]
>
> This action is irreversible. The audio file and all associated data (transcripts, indexes) will be permanently deleted from your collection.


# Generate audio transcription
Source: https://docs.videodb.io/api-reference/audio/generate-audio-transcription

/openapi.yaml post /audio/{audio_id}/transcription/



# Generate Audio URL
Source: https://docs.videodb.io/api-reference/audio/generate_audio_url

POST /audio/{audio_id}/generate_url
Generate a shareable streaming URL for an audio file

Generate shareable URLs for streaming or downloading audio files.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
audio = coll.get_audio("a-abc123def")

# Generate download/streaming URL
url = audio.generate_url()
print(f"Audio URL: {url}")
# https://cdn.videodb.io/v3/{collection}/{audio}.mp3
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const audio = await coll.getAudio("a-abc123def");

// Generate download/streaming URL
const url = await audio.generateUrl();
console.log(`Audio URL: ${url}`);
// https://cdn.videodb.io/v3/{collection}/{audio}.mp3
```

> [!NOTE]
>
> * URLs are CDN-hosted and can be shared directly
> * Format is MP3 or original audio format
> * URLs remain valid for extended periods
> * Useful for embedding in web players or sending via email

> **[Streams and Exports](/pages/act/output-and-delivery/streams-and-exports)**
>
> Learn about generating URLs and streaming options

> **[Get Audio Details](/api-reference/audio/get_audio)**
>
> Retrieve audio metadata before generating URLs


# Get audio transcription
Source: https://docs.videodb.io/api-reference/audio/get-audio-transcription

/openapi.yaml get /audio/{audio_id}/transcription/



# Get Audio Details
Source: https://docs.videodb.io/api-reference/audio/get_audio

GET /audio/{audio_id}
Retrieve detailed information about a specific audio file

Retrieve metadata for a specific audio file by its ID.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Get audio details
audio = coll.get_audio("a-abc123def")
print(f"ID: {audio.id}")
print(f"Name: {audio.name}")
print(f"Duration: {audio.length}s")
print(f"Collection: {audio.collection_id}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Get audio details
const audio = await coll.getAudio("a-abc123def");
console.log(`ID: ${audio.id}`);
console.log(`Name: ${audio.name}`);
console.log(`Duration: ${audio.length}s`);
console.log(`Collection: ${audio.collectionId}`);
```

> [!NOTE]
>
> * Audio IDs always start with `a-` prefix
> * The `length` property contains duration in seconds
> * Use this to retrieve metadata before generating streams or download URLs

> **[List All Audio](/api-reference/audio/list_audio)**
>
> View all audio files in collection

> **[Generate URL](/api-reference/audio/generate_audio_url)**
>
> Create streaming or download URLs


# Get Audio Transcription
Source: https://docs.videodb.io/api-reference/audio/get_audio_transcription

GET /audio/{audio_id}/transcription/
Retrieve the transcription for an audio file

Retrieve the transcription data for a specific audio file, including word-level timestamps.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
audio = coll.get_audio("audio_id")

# Get full transcript with timestamps
transcript = audio.get_transcript()

# Get transcript for a time range
transcript = audio.get_transcript(start=10, end=60)

# Segment by sentence
transcript = audio.get_transcript(segmenter="sentence", force=True)

# Group every 10 words
transcript = audio.get_transcript(segmenter="word", length=10, force=True)

# Fixed-duration audio segments (e.g., 5-second chunks)
transcript = audio.get_transcript(segmenter="time", length=5, force=True)

# Get plain text transcript
text = audio.get_transcript_text()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const audio = await coll.getAudio('audio_id');

// Get full transcript with timestamps
const transcript = await audio.getTranscript();

// Get transcript for a time range
const rangeTranscript = await audio.getTranscript(10, 60);

// Segment by sentence
const sentenceTranscript = await audio.getTranscript(0, audio.length, 'sentence', 1, true);

// Group every 10 words
const wordGroups = await audio.getTranscript(0, audio.length, 'word', 10, true);

// Fixed-duration audio segments
const timeTranscript = await audio.getTranscript(0, audio.length, 'time', 5, true);

// Get plain text transcript
const text = await audio.getTranscriptText();
```

### Audio segmentation

Audio transcription can be segmented independently from any video frame or scene segmentation.

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `segmenter` | str | `"word"` | How to split the transcript: `"word"`, `"sentence"`, or `"time"` |
| `length` | int | `1` | Group size: words for `word`, sentences for `sentence`, or seconds for `time` |
| `start` | float | — | Start time in seconds |
| `end` | float | — | End time in seconds |

> [!NOTE]
>
> * Use `start` and `end` parameters to retrieve a specific time range
> * Use `segmenter="word"` with `length=1` for word-level timestamps
> * SDK helpers return timestamp entries with `start`, `end`, and `text`
> * The raw API response wraps those entries as `data.word_timestamps` plus `data.text`
> * The `engine` parameter selects which transcription engine's output to retrieve

### Raw response shape

```json theme={null}
{
  "success": true,
  "status": "completed",
  "data": {
    "text": "Welcome to VideoDB.",
    "word_timestamps": [
      {"start": 0.2, "end": 0.7, "text": "Welcome"},
      {"start": 0.7, "end": 0.9, "text": "to"},
      {"start": 0.9, "end": 1.4, "text": "VideoDB."}
    ]
  }
}
```

Silence ranges may be included with `text: "-"`.

> **[Create Transcription](/api-reference/audio/create_audio_transcription)**
>
> Generate a new transcription

> **[Get Audio](/api-reference/audio/get_audio)**
>
> Retrieve audio file details


# Audio
Source: https://docs.videodb.io/api-reference/audio/index

Manage audio files and streams

Manage audio files and generate streaming URLs.

## Endpoints

> **[List Audio](/api-reference/audio/list_audio)**
>
> Get all audio files

> **[Get Audio](/api-reference/audio/get_audio)**
>
> Retrieve audio details

> **[Delete Audio](/api-reference/audio/delete_audio)**
>
> Remove audio file

> **[Generate URL](/api-reference/audio/generate_audio_url)**
>
> Create streaming URL

> **[Update Audio](/api-reference/audio/update_audio)**
>
> Update audio metadata

> **[Create Transcription](/api-reference/audio/create_audio_transcription)**
>
> Generate audio transcription

> **[Get Transcription](/api-reference/audio/get_audio_transcription)**
>
> Retrieve audio transcription


# List All Audio Files
Source: https://docs.videodb.io/api-reference/audio/list_audio

GET /audio/
Retrieve a paginated list of all audio files in your collection

Retrieve all audio files in a collection for browsing and management.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# List all audio files
audios = coll.get_audios()
for audio in audios:
    print(f"{audio.id}: {audio.name} ({audio.length}s)")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// List all audio files
const audios = await coll.getAudios();
for (const audio of audios) {
    console.log(`${audio.id}: ${audio.name} (${audio.length}s)`);
}
```

> [!NOTE]
>
> * Audio IDs have an `a-` prefix (e.g., `a-abc123def`)
> * Each audio file includes duration in seconds via the `length` property
> * Use this to build audio galleries or management interfaces

> **[Collection Management](/pages/ingest/files-and-collections/create-collection)**
>
> Learn about organizing media in collections

> **[Get Audio Details](/api-reference/audio/get_audio)**
>
> Retrieve details for a specific audio file


# Update audio
Source: https://docs.videodb.io/api-reference/audio/update-audio

/openapi.yaml patch /audio/{audio_id}



# Update Audio
Source: https://docs.videodb.io/api-reference/audio/update_audio

PATCH /audio/{audio_id}
Update audio file metadata

Update metadata for a specific audio file, including its name and description.

> [!NOTE]
>
> * Audio IDs start with the prefix `a-`
> * Only provided fields are updated; omitted fields remain unchanged


# Create API Key
Source: https://docs.videodb.io/api-reference/authentication/create_api_key

POST /user/api_key
Generate a new API key for your account

Generate a new API key for authenticating API requests. Each key provides full access to your account resources.

> [!NOTE]
>
> * API keys provide full access to your account; store securely
> * Each new key can be used immediately for authentication
> * You can generate multiple keys for different applications or environments
> * Keep track of keys used to rotate or revoke them when needed

> **[List API Keys](/api-reference/authentication/list_api_keys)**
>
> View all your generated API keys

> **[Delete API Key](/api-reference/authentication/delete_api_key)**
>
> Revoke an API key


# Delete API Key
Source: https://docs.videodb.io/api-reference/authentication/delete_api_key

DELETE /user/api_key/{api_key}
Revoke an API key permanently

Revoke an API key to prevent it from being used for future authentication requests. This action is permanent.

> [!WARNING]
>
> * Deleting an API key is permanent and irreversible
> * Any applications using this key will immediately lose access
> * Ensure no active integrations depend on the key before deleting
> * Any ongoing API calls using this key will fail after deletion

> **[List API Keys](/api-reference/authentication/list_api_keys)**
>
> View all your API keys

> **[Create API Key](/api-reference/authentication/create_api_key)**
>
> Generate a replacement API key


# Get User Info
Source: https://docs.videodb.io/api-reference/authentication/get_user

GET /user
Retrieve current user account information and credits

Retrieve the current user's account details including email and available credits.

> [!NOTE]
>
> * Requires valid API key for authentication via `x-access-token` header
> * Returns user email and current credit balance
> * Use this to check account status before running operations

> **[Create API Key](/api-reference/authentication/create_api_key)**
>
> Generate a new API key for access

> **[List API Keys](/api-reference/authentication/list_api_keys)**
>
> View all your API keys


# Authentication
Source: https://docs.videodb.io/api-reference/authentication/index

Manage API keys and user account information

Manage API keys and retrieve user account details.

## Endpoints

> **[Create API Key](/api-reference/authentication/create_api_key)**
>
> Generate a new API key

> **[List API Keys](/api-reference/authentication/list_api_keys)**
>
> Get all API keys

> **[Get User Info](/api-reference/authentication/get_user)**
>
> Retrieve account details

> **[Delete API Key](/api-reference/authentication/delete_api_key)**
>
> Remove an API key

> **[Set API Key Collections](/api-reference/authentication/set_api_key_collections)**
>
> Scope API key to collections


# List API Keys
Source: https://docs.videodb.io/api-reference/authentication/list_api_keys

GET /user/api_key
Retrieve all API keys for the authenticated user

Retrieve a list of all API keys associated with your account. Useful for auditing which keys are in use.

> [!NOTE]
>
> * Returns all API keys created under your account
> * Includes creation date and last usage timestamp for each key
> * Use `last_used_at` to identify stale or unused keys for cleanup
> * Each key provides full account access; review regularly for security

> **[Create API Key](/api-reference/authentication/create_api_key)**
>
> Generate a new API key

> **[Delete API Key](/api-reference/authentication/delete_api_key)**
>
> Remove a specific API key


# Set API key collection scoping
Source: https://docs.videodb.io/api-reference/authentication/set-api-key-collection-scoping

/openapi.yaml put /user/api-key-collections



# Set API Key Collections
Source: https://docs.videodb.io/api-reference/authentication/set_api_key_collections

PUT /user/api-key-collections
Scope an API key to specific collections

Restrict an API key to only access specific collections. This is useful for creating scoped keys for third-party integrations or limiting access in multi-tenant setups.

> [!NOTE]
>
> * This is a Pro-only feature
> * Pass an empty `collection_ids` array to remove all collection restrictions
> * The scoped key will only be able to access the specified collections

> **[Create API Key](/api-reference/authentication/create_api_key)**
>
> Generate a new API key

> **[List API Keys](/api-reference/authentication/list_api_keys)**
>
> View all API keys


# Create Checkout Session
Source: https://docs.videodb.io/api-reference/billing/create_checkout

POST /billing/checkout
Create a payment checkout session to purchase credits

Initiate a checkout session to purchase credits using your preferred payment method. Returns a checkout URL for the payment flow.

```python Python theme={null}
import requests

payload = {
    "amount": 50,
    "success_url": "https://yourapp.com/billing/success",
    "cancel_url": "https://yourapp.com/billing/cancelled"
}

response = requests.post(
    "https://api.videodb.io/billing/checkout",
    json=payload,
    headers={"x-access-token": "your_api_key"}
)

checkout_data = response.json()
print(f"Checkout URL: {checkout_data.get('checkout_url')}")
print(f"Session ID: {checkout_data.get('id')}")
```

```javascript Node.js theme={null}
const payload = {
    amount: 50,
    successUrl: "https://yourapp.com/billing/success",
    cancelUrl: "https://yourapp.com/billing/cancelled"
};

const response = await fetch("https://api.videodb.io/billing/checkout", {
    method: "POST",
    headers: {
        "x-access-token": "your_api_key",
        "Content-Type": "application/json"
    },
    body: JSON.stringify(payload)
});

const checkoutData = await response.json();
console.log(`Checkout URL: ${checkoutData.checkoutUrl}`);
console.log(`Session ID: ${checkoutData.id}`);
```

> [!NOTE]
>
> * Amount is specified in USD (whole dollars)
> * Success and cancel URLs redirect users after payment completion or cancellation
> * Checkout sessions expire after 24 hours if not completed
> * Credits are added to your account immediately upon successful payment

> **[List Checkouts](/api-reference/billing/list_checkouts)**
>
> View all checkout sessions and their status

> **[Get Usage Statistics](/api-reference/billing/get_usage)**
>
> Monitor credit usage and balance


# Manual Credit Top-up
Source: https://docs.videodb.io/api-reference/billing/create_topup

POST /billing/topup
Manually add credits to your account balance

Manually purchase and add credits to your account balance using your preferred payment method.

```python Python theme={null}
import requests

payload = {
    "amount": 100,
    "redirect_url": "https://yourapp.com/dashboard"
}

response = requests.post(
    "https://api.videodb.io/billing/topup",
    json=payload,
    headers={"x-access-token": "your_api_key"}
)

result = response.json()
print(f"Payment URL: {result.get('payment_url')}")
print(f"Session ID: {result.get('session_id')}")
```

```javascript Node.js theme={null}
const payload = {
    amount: 100,
    redirectUrl: "https://yourapp.com/dashboard"
};

const response = await fetch("https://api.videodb.io/billing/topup", {
    method: "POST",
    headers: {
        "x-access-token": "your_api_key",
        "Content-Type": "application/json"
    },
    body: JSON.stringify(payload)
});

const result = await response.json();
console.log(`Payment URL: ${result.paymentUrl}`);
console.log(`Session ID: ${result.sessionId}`);
```

> [!NOTE]
>
> * Amount is specified in USD (whole dollars)
> * Redirect URL is optional and directs users after payment completes
> * Topup credits are added immediately upon successful payment
> * Your payment method must be configured in your account settings
> * Topup is independent of auto-recharge and can be used alongside it

> **[List Checkouts](/api-reference/billing/list_checkouts)**
>
> View all payment checkout sessions

> **[Get Usage Statistics](/api-reference/billing/get_usage)**
>
> Monitor current credit balance and consumption


# Get Auto-Recharge Settings
Source: https://docs.videodb.io/api-reference/billing/get_auto_recharge

GET /billing/auto_recharge
Retrieve automatic credit recharge configuration

Retrieve your current auto-recharge settings, including whether auto-recharge is enabled and the threshold amount.

```python Python theme={null}
import requests

response = requests.get(
    "https://api.videodb.io/billing/auto_recharge",
    headers={"x-access-token": "your_api_key"}
)

auto_recharge = response.json()

print(f"Enabled: {auto_recharge.get('enabled')}")
print(f"Threshold: {auto_recharge.get('threshold_credits')}")
print(f"Recharge Amount: {auto_recharge.get('recharge_amount')}")
```

```javascript Node.js theme={null}
const response = await fetch("https://api.videodb.io/billing/auto_recharge", {
    method: "GET",
    headers: {
        "x-access-token": "your_api_key"
    }
});

const autoRecharge = await response.json();

console.log(`Enabled: ${autoRecharge.enabled}`);
console.log(`Threshold: ${autoRecharge.thresholdCredits}`);
console.log(`Recharge Amount: ${autoRecharge.rechargeAmount}`);
```

> [!NOTE]
>
> * Auto-recharge triggers when your balance falls below the threshold amount
> * Recharge amount is automatically charged to your payment method on file
> * Feature is optional and disabled by default
> * You can enable, disable, or modify settings at any time

> **[Configure Auto-Recharge](/api-reference/billing/set_auto_recharge)**
>
> Enable and modify auto-recharge settings

> **[Get Usage Statistics](/api-reference/billing/get_usage)**
>
> Monitor your current credit balance


# Get Usage Statistics
Source: https://docs.videodb.io/api-reference/billing/get_usage

GET /billing/usage
Retrieve detailed usage statistics and credit consumption

Retrieve detailed usage statistics showing credit consumption across different operations in your account.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")

# Get usage statistics
usage_data = conn.check_usage()

print(f"Total Credits Used: {usage_data.get('total_credits_used')}")
print(f"Monthly Credit Limit: {usage_data.get('monthly_limit')}")
print(f"Credits Remaining: {usage_data.get('credits_remaining')}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });

// Get usage statistics
const usageData = await conn.checkUsage();

console.log(`Total Credits Used: ${usageData.totalCreditsUsed}`);
console.log(`Monthly Credit Limit: ${usageData.monthlyLimit}`);
console.log(`Credits Remaining: ${usageData.creditsRemaining}`);
```

> [!NOTE]
>
> * Usage data includes cumulative credits consumed across all operations (video processing, AI generation, storage)
> * Credits reset monthly based on your billing plan
> * Track usage regularly to avoid unexpected bill increases
> * Real-time updates may have slight delays (typically 5-15 minutes)

> **[Billing Overview](/api-reference/billing/index)**
>
> Manage billing and usage information

> **[Get Invoices](/api-reference/billing/list_invoices)**
>
> Retrieve billing invoices and payment records


# Billing
Source: https://docs.videodb.io/api-reference/billing/index

Manage billing and usage information

Monitor usage, manage billing, and configure auto-recharge.

## Endpoints

> **[List Checkouts](/api-reference/billing/list_checkouts)**
>
> Get all checkouts

> **[Get Usage](/api-reference/billing/get_usage)**
>
> Retrieve usage stats

> **[Get Invoices](/api-reference/billing/list_invoices)**
>
> Retrieve invoices

> **[Get Auto-Recharge](/api-reference/billing/get_auto_recharge)**
>
> Check auto-recharge settings

> **[Set Auto-Recharge](/api-reference/billing/set_auto_recharge)**
>
> Configure auto-recharge

> **[Create Checkout](/api-reference/billing/create_checkout)**
>
> Initiate checkout

> **[Top Up Credits](/api-reference/billing/create_topup)**
>
> Add credits to account


# List Checkout Sessions
Source: https://docs.videodb.io/api-reference/billing/list_checkouts

GET /billing/checkouts
Retrieve a list of all checkout sessions and their status

Retrieve all checkout sessions initiated through your account, including their status and payment information.

```python Python theme={null}
import requests

response = requests.get(
    "https://api.videodb.io/billing/checkouts",
    headers={"x-access-token": "your_api_key"}
)

checkouts = response.json()

for checkout in checkouts.get('checkouts', []):
    print(f"Checkout ID: {checkout.get('id')}")
    print(f"Amount: ${checkout.get('amount')}")
    print(f"Status: {checkout.get('status')}")
    print(f"Created: {checkout.get('created_at')}")
    print("---")
```

```javascript Node.js theme={null}
const response = await fetch("https://api.videodb.io/billing/checkouts", {
  method: "GET",
  headers: {
    "x-access-token": "your_api_key"
  }
});

const data = await response.json();

for (const checkout of data.checkouts || []) {
    console.log(`Checkout ID: ${checkout.id}`);
    console.log(`Amount: $${checkout.amount}`);
    console.log(`Status: ${checkout.status}`);
    console.log(`Created: ${checkout.createdAt}`);
    console.log("---");
}
```

> [!NOTE]
>
> * Checkout sessions include both completed and abandoned sessions
> * Status values include "completed", "pending", and "expired"
> * Completed checkouts result in credit additions to your account
> * Each checkout session is associated with a specific purchase amount

> **[Create Checkout](/api-reference/billing/create_checkout)**
>
> Initiate a new checkout session to purchase credits

> **[Top Up Credits](/api-reference/billing/create_topup)**
>
> Manually add credits to your account


# List Invoices
Source: https://docs.videodb.io/api-reference/billing/list_invoices

GET /billing/invoices
Retrieve billing invoices and payment records

Retrieve a list of all invoices and payment records for your account, including dates, amounts, and payment status.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")

# Get all invoices
invoices = conn.get_invoices()

for invoice in invoices:
    print(f"Invoice ID: {invoice.get('id')}")
    print(f"Amount: ${invoice.get('amount')}")
    print(f"Date: {invoice.get('date')}")
    print(f"Status: {invoice.get('status')}")
    print("---")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });

// Get all invoices
const invoices = await conn.getInvoices();

for (const invoice of invoices) {
    console.log(`Invoice ID: ${invoice.id}`);
    console.log(`Amount: $${invoice.amount}`);
    console.log(`Date: ${invoice.date}`);
    console.log(`Status: ${invoice.status}`);
    console.log("---");
}
```

> [!NOTE]
>
> * Invoices are generated monthly based on your billing cycle
> * Each invoice includes itemized charges for different operations and features
> * Payment status may be "paid", "pending", or "failed"
> * Invoices are available for the current billing period and all previous periods

> **[Billing Overview](/api-reference/billing/index)**
>
> Manage billing and usage information

> **[Get Usage Statistics](/api-reference/billing/get_usage)**
>
> Check current usage and credit consumption


# Configure Auto-Recharge
Source: https://docs.videodb.io/api-reference/billing/set_auto_recharge

POST /billing/auto_recharge
Enable and configure automatic credit recharge settings

Enable or disable automatic credit recharge and configure the threshold and recharge amounts for your account.

```python Python theme={null}
import requests

payload = {
    "enabled": True,
    "threshold_credits": 100,
    "recharge_amount": 500
}

response = requests.post(
    "https://api.videodb.io/billing/auto_recharge",
    json=payload,
    headers={"x-access-token": "your_api_key"}
)

result = response.json()
print(f"Auto-Recharge Updated: {result.get('success')}")
print(f"New Settings: {result.get('settings')}")
```

```javascript Node.js theme={null}
const payload = {
    enabled: true,
    thresholdCredits: 100,
    rechargeAmount: 500
};

const response = await fetch("https://api.videodb.io/billing/auto_recharge", {
    method: "POST",
    headers: {
        "x-access-token": "your_api_key",
        "Content-Type": "application/json"
    },
    body: JSON.stringify(payload)
});

const result = await response.json();
console.log(`Auto-Recharge Updated: ${result.success}`);
console.log(`New Settings: ${JSON.stringify(result.settings)}`);
```

> [!NOTE]
>
> * Threshold amount determines when auto-recharge triggers
> * Recharge amount is the credit quantity added when threshold is met
> * Requires a valid payment method on file
> * Set `enabled` to `false` to disable auto-recharge without losing your configuration
> * Changes take effect immediately

> **[Get Auto-Recharge Settings](/api-reference/billing/get_auto_recharge)**
>
> Check current auto-recharge configuration

> **[Get Usage Statistics](/api-reference/billing/get_usage)**
>
> Monitor credit consumption and balance


# Create capture session
Source: https://docs.videodb.io/api-reference/capture/create-capture-session

/openapi.yaml post /collection/{collection_id}/capture/session



# Create capture session token
Source: https://docs.videodb.io/api-reference/capture/create-capture-session-token

/openapi.yaml post /capture/session/token



# Create Capture Session
Source: https://docs.videodb.io/api-reference/capture/create_session

POST /collection/{collection_id}/capture/session
Create a new capture session for real-time media ingestion

Create a new capture session within a collection. Capture sessions enable real-time media ingestion from client applications.

> [!NOTE]
>
> * `end_user_id` is required to identify the end user
> * Use `ws_connection_id` to link to an existing WebSocket connection for real-time updates
> * Use `metadata` to attach custom key-value data to the session

> **[Start Session](/api-reference/capture/start_session)**
>
> Start the capture session

> **[Capture SDK Guide](/pages/ingest/capture-sdks/overview)**
>
> Learn about capture SDKs


# Create Session Token
Source: https://docs.videodb.io/api-reference/capture/create_session_token

POST /capture/session/token
Create a temporary token for capture session authentication

Create a temporary authentication token for capture sessions. Use this to generate short-lived tokens for client-side SDK authentication without exposing your API key.

> [!NOTE]
>
> * Tokens are short-lived and expire after the specified duration (default: 24 hours)
> * `user_id` identifies the end user for partner tracking
> * Use these tokens in client-side applications instead of your main API key

> **[Create Session](/api-reference/capture/create_session)**
>
> Create a capture session

> **[Privacy Controls](/pages/ingest/capture-sdks/privacy-controls)**
>
> Capture privacy controls


# Get capture session details
Source: https://docs.videodb.io/api-reference/capture/get-capture-session-details

/openapi.yaml get /collection/{collection_id}/capture/session/{session_id}



# Get capture session with RTSP URLs
Source: https://docs.videodb.io/api-reference/capture/get-capture-session-with-rtsp-urls

/openapi.yaml get /capture/session/{streaming_session_id}



# Get Capture Session
Source: https://docs.videodb.io/api-reference/capture/get_session

GET /collection/{collection_id}/capture/session/{session_id}
Get details of a specific capture session

Retrieve detailed information about a specific capture session.

> [!NOTE]
>
> * Returns full session details including status, channels, and metadata


# Get Streaming Session
Source: https://docs.videodb.io/api-reference/capture/get_streaming_session

GET /capture/session/{streaming_session_id}
Get capture session details with RTSP URLs

Retrieve details of an active streaming session, including RTSP URLs for each channel.

> [!NOTE]
>
> * RTSP URLs are available once the session is in `active` status
> * Each channel has its own RTSP URL for independent streaming

> **[Start Session](/api-reference/capture/start_session)**
>
> Start a capture session

> **[Stop Session](/api-reference/capture/stop_session)**
>
> Stop the capture session


# Capture
Source: https://docs.videodb.io/api-reference/capture/index

Capture SDK session management for real-time media ingestion

Manage capture sessions for real-time media ingestion from client applications using the Capture SDK.

## Endpoints

> **[Create Session](/api-reference/capture/create_session)**
>
> Create a capture session

> **[List Sessions](/api-reference/capture/list_sessions)**
>
> List capture sessions

> **[Get Session](/api-reference/capture/get_session)**
>
> Get session details

> **[Create Token](/api-reference/capture/create_session_token)**
>
> Generate session token

> **[Start Session](/api-reference/capture/start_session)**
>
> Start a capture session

> **[Get Streaming Session](/api-reference/capture/get_streaming_session)**
>
> Get streaming session with RTSP URLs

> **[Stop Session](/api-reference/capture/stop_session)**
>
> Stop a capture session


# List capture sessions
Source: https://docs.videodb.io/api-reference/capture/list-capture-sessions

/openapi.yaml get /collection/{collection_id}/capture/session



# List Capture Sessions
Source: https://docs.videodb.io/api-reference/capture/list_sessions

GET /collection/{collection_id}/capture/session
List capture sessions in a collection

List all capture sessions in a collection, with optional filtering by status.

> [!NOTE]
>
> * Filter by `status`: `created`, `starting`, `active`, `stopped`, `failed`
> * Results are paginated; use `next_page` for additional results


# Start capture session
Source: https://docs.videodb.io/api-reference/capture/start-capture-session

/openapi.yaml post /capture/session/start



# Start Capture Session
Source: https://docs.videodb.io/api-reference/capture/start_session

POST /capture/session/start
Start an existing capture session with channel configuration

Start a previously created capture session with specific channel configurations. Define the audio and video channels to capture.

> [!NOTE]
>
> * `session_id` and `channels` are required
> * Each channel needs a `channel_id` and can be of type `audio` or `video`
> * Set `store` to `true` to persist captured media in VideoDB
> * Use `callback_url` for webhook notifications on session events

> **[Get Streaming Session](/api-reference/capture/get_streaming_session)**
>
> Get RTSP URLs for the session

> **[Stop Session](/api-reference/capture/stop_session)**
>
> Stop the capture session


# Stop capture session
Source: https://docs.videodb.io/api-reference/capture/stop-capture-session

/openapi.yaml post /capture/session/{streaming_session_id}/stop



# Stop Capture Session
Source: https://docs.videodb.io/api-reference/capture/stop_session

POST /capture/session/{streaming_session_id}/stop
Stop an active capture session

Stop an active capture session. This ends the media ingestion and finalizes any stored recordings.

> [!NOTE]
>
> * Stopping a session finalizes all channel recordings
> * Stored media becomes available for indexing and search after processing

> **[Create Session](/api-reference/capture/create_session)**
>
> Create a new capture session

> **[List Sessions](/api-reference/capture/list_sessions)**
>
> View all capture sessions


# Create Collection
Source: https://docs.videodb.io/api-reference/collections/create_collection

POST /collection
Create a new collection to organize videos

Collections organize your media files and provide a scope for search and other operations.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.create_collection(
    name="Product Demos",
    description="Demo videos for Q1 2024"
)
print(f"Created collection: {coll.id}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.createCollection(
    'Product Demos',
    'Demo videos for Q1 2024'
);
console.log(`Created collection: ${coll.id}`);
```

> [!NOTE]
>
> * Collection IDs have a `c-` prefix (e.g., `c-a1b2c3d4`)
> * Use `is_public=True` to make collections accessible via ID without authentication
> * Each account has a "default" collection created automatically

> **[Collection Management Guide](/pages/ingest/files-and-collections/create-collection)**
>
> Complete guide to creating and managing collections

> **[Upload Media](/api-reference/collections/upload_to_collection)**
>
> Add videos, audio, or images to your collection


# Delete Collection
Source: https://docs.videodb.io/api-reference/collections/delete_collection

DELETE /collection/{collection_id}
Permanently remove a collection and all its videos

Permanently delete a collection and all its media files.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection("c-a1b2c3d4")
coll.delete()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection('c-a1b2c3d4');
await coll.delete();
```

> [!WARNING]
>
> This action is irreversible. All videos, audio files, images, and associated data (transcripts, indexes, thumbnails) in the collection will be permanently deleted.


# Get presigned upload URL
Source: https://docs.videodb.io/api-reference/collections/get-presigned-upload-url

/openapi.yaml get /collection/{collection_id}/upload_url



# Get WebSocket connection URL
Source: https://docs.videodb.io/api-reference/collections/get-websocket-connection-url

/openapi.yaml get /collection/{collection_id}/websocket



# Get Collection
Source: https://docs.videodb.io/api-reference/collections/get_collection

GET /collection/{collection_id}
Retrieve details of a specific collection

Retrieve details for a specific collection by ID. Use `"default"` to get your default collection.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")

# Get default collection
coll = conn.get_collection()

# Get specific collection
coll = conn.get_collection("c-a1b2c3d4")

print(f"Collection: {coll.name}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });

// Get default collection
const defaultCollection = await conn.getCollection();

// Get specific collection
const coll = await conn.getCollection('c-a1b2c3d4');

console.log(`Collection: ${coll.name}`);
```

> [!NOTE]
>
> * Pass no arguments or `"default"` to access your default collection
> * Public collections can be accessed by anyone with the collection ID


# Get Upload URL
Source: https://docs.videodb.io/api-reference/collections/get_upload_url

GET /collection/{collection_id}/upload_url
Get a presigned URL for direct file upload

Get a presigned upload URL for directly uploading files to a collection. This is useful for client-side uploads where you want to avoid routing file data through your server.

> [!NOTE]
>
> * The presigned URL expires after a limited time
> * Upload the file directly to the returned URL using a PUT request
> * A video ID is pre-assigned and returned immediately

> **[Upload to Collection](/api-reference/collections/upload_to_collection)**
>
> Standard server-side upload


# Get WebSocket URL
Source: https://docs.videodb.io/api-reference/collections/get_websocket

GET /collection/{collection_id}/websocket
Get a WebSocket connection URL for real-time updates

Get a WebSocket connection URL for receiving real-time updates from a collection. Use this for live notifications about upload progress, transcription completion, and other async events.

> [!NOTE]
>
> * WebSocket URLs are temporary and should be reconnected if the connection drops
> * Use the `ws_connection_id` from the WebSocket handshake in other API calls to receive targeted updates


# Collections
Source: https://docs.videodb.io/api-reference/collections/index

Organize and manage video collections

Collections group videos for organization and scoped operations.

## Endpoints

> **[Create Collection](/api-reference/collections/create_collection)**
>
> Create a new collection

> **[List Collections](/api-reference/collections/list_collections)**
>
> Get all collections

> **[Get Collection](/api-reference/collections/get_collection)**
>
> Retrieve collection details

> **[Update Collection](/api-reference/collections/update_collection)**
>
> Modify collection metadata

> **[Delete Collection](/api-reference/collections/delete_collection)**
>
> Remove a collection

> **[Upload to Collection](/api-reference/collections/upload_to_collection)**
>
> Add media to collection

> **[Search Collection](/api-reference/search-v2/search)**
>
> Search across collection

> **[Get Upload URL](/api-reference/collections/get_upload_url)**
>
> Get presigned upload URL

> **[Get WebSocket](/api-reference/collections/get_websocket)**
>
> Get real-time WebSocket URL

> **[Search by Title](/api-reference/collections/search_title)**
>
> Search media by title

> **[Web Search](/api-reference/collections/search_web)**
>
> Web search in collection


# List Collections
Source: https://docs.videodb.io/api-reference/collections/list_collections

GET /collection
Retrieve all collections in your account

Retrieve all collections in your account, including the default collection.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
collections = conn.get_collections()

for coll in collections:
    print(f"{coll.id}: {coll.name}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const collections = await conn.getCollections();

for (const coll of collections) {
    console.log(`${coll.id}: ${coll.name}`);
}
```

> [!NOTE]
>
> Returns all collections you have access to, including public collections you've accessed.


# Legacy Collection Search
Source: https://docs.videodb.io/api-reference/collections/search_collections

POST /collection/{collection_id}/search/
Search across all videos in a collection

Search across all media files in a collection using semantic or keyword search.

> [!NOTE]
>
> ### Legacy Compatibility
>
> This endpoint documents the legacy `coll.search(..., SearchType/IndexType)` API below. For current retrieval, run Understand, index the resulting analyzer artifact, then use semantic search or a structured query. See [Create an Index](/pages/understand/indexing-pipelines/create-an-index) and [Collection Search](/pages/understand/search-and-retrieval/collection-search).

```python Python theme={null}
import videodb
from videodb import SearchType, IndexType

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Search spoken content
results = coll.search(
    query="product launch",
    search_type=SearchType.semantic,
    index_type=IndexType.spoken_word
)

for result in results.shots:
    print(f"Video: {result.video_id}")
    print(f"Match: {result.text}")
    print(f"Time: {result.start} - {result.end}")
```

```javascript Node.js theme={null}
import { connect, SearchTypeValues, IndexTypeValues } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Search spoken content
const results = await coll.search(
    'product launch',
    SearchTypeValues.semantic,
    IndexTypeValues.spoken
);

for (const result of results.shots) {
    console.log(`Video: ${result.videoId}`);
    console.log(`Match: ${result.text}`);
    console.log(`Time: ${result.start} - ${result.end}`);
}
```

> [!NOTE]
>
> * Videos must be indexed before search works (use `video.index_spoken_words()` or `video.index_scenes()`)
> * Use `IndexType.scene` for visual search across scenes
> * Add metadata during indexing for filtered searches

> **[Collection Search Guide](/pages/understand/search-and-retrieval/collection-search)**
>
> Advanced search with metadata filtering

> **[Video Indexing](/api-reference/videos/indexing/create_video_index)**
>
> Create indexes for searchable content


# Search by Title
Source: https://docs.videodb.io/api-reference/collections/search_title

POST /collection/{collection_id}/search/title/
Search for media by title within a collection

Search for media files by title within a specific collection.

> [!NOTE]
>
> * Searches across video, audio, and image titles in the collection
> * Use this for quick metadata-based lookups without requiring an index

> **[Search Collection](/api-reference/search-v2/semantic-search)**
>
> Semantic search across collection content

> **[Web Search](/api-reference/collections/search_web)**
>
> Web search within collection


# Web Search
Source: https://docs.videodb.io/api-reference/collections/search_web

POST /collection/{collection_id}/search/web/
Perform web search within a collection context

Perform a web-style search within a specific collection.

> [!NOTE]
>
> * Performs web-style search across collection media
> * Returns relevant results based on the search query

> **[Search Collection](/api-reference/search-v2/semantic-search)**
>
> Semantic search across collection content

> **[Title Search](/api-reference/collections/search_title)**
>
> Search by media title


# Update Collection
Source: https://docs.videodb.io/api-reference/collections/update_collection

PATCH /collection/{collection_id}
Modify collection metadata and settings

Update a collection's name, description, or visibility.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.update_collection(
    id="c-a1b2c3d4",
    name="Q1 2024 Demos",
    description="Updated product demos"
)

# Or update via collection object
coll = conn.get_collection("c-a1b2c3d4")
coll.make_public()  # Make collection publicly accessible
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const updatedCollection = await conn.updateCollection(
    'c-a1b2c3d4',
    'Q1 2024 Demos',
    'Updated product demos'
);

// Or update via collection object
const coll = await conn.getCollection('c-a1b2c3d4');
await coll.makePublic();  // Make collection publicly accessible
```

> [!NOTE]
>
> Use `make_public()` / `makePublic()` or `make_private()` / `makePrivate()` methods for visibility control.

> **[Public Collections](/pages/ingest/files-and-collections/create-collection#public-collections)**
>
> Learn about sharing collections publicly


# Upload to Collection
Source: https://docs.videodb.io/api-reference/collections/upload_to_collection

POST /collection/{collection_id}/upload
Upload videos, audio, or images to a collection

Upload videos, audio files, or images to a collection from URLs or local files.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Upload from URL
url_video = coll.upload(url="https://example.com/video.mp4")

# Upload from local file
file_video = coll.upload(file_path="/path/to/video.mp4")

# Upload with callback for async processing
coll.upload(
    url="https://example.com/video.mp4",
    callback_url="https://your-server.com/webhook"
)

print(f"Video ID: {url_video.id}")
print(f"Stream URL: {url_video.stream_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Upload from URL
const urlVideo = await coll.uploadURL({
    url: 'https://example.com/video.mp4'
});

// Upload from local file
const fileVideo = await coll.uploadFile({
    filePath: '/path/to/video.mp4'
});

// Upload with callback for async processing
await coll.uploadURL({
    url: 'https://example.com/video.mp4',
    callbackUrl: 'https://your-server.com/webhook'
});

console.log(`Video ID: ${urlVideo.id}`);
console.log(`Stream URL: ${urlVideo.streamUrl}`);
```

> [!NOTE]
>
> * Supports video, audio, and image files
> * Use `media_type` parameter to specify type explicitly
> * Large files should use `callback_url` for async processing

> **[Upload Guide](/pages/ingest/files-and-collections/upload-video)**
>
> Supported formats and upload methods

> **[Production Patterns](/pages/ingest/files-and-collections/collection-patterns)**
>
> Batch uploads, webhooks, and retry logic


# Compile media content
Source: https://docs.videodb.io/api-reference/compile-media-content

/openapi.yaml post /compile/



# Compile timeline
Source: https://docs.videodb.io/api-reference/compile-timeline

/openapi.yaml post /timeline



# Compile timeline (v2)
Source: https://docs.videodb.io/api-reference/compile-timeline-v2

/openapi.yaml post /timeline_v2



# Create billing checkout session
Source: https://docs.videodb.io/api-reference/create-billing-checkout-session

/openapi.yaml post /billing/checkout



# Create download request
Source: https://docs.videodb.io/api-reference/create-download-request

/openapi.yaml post /download



# Create new API key
Source: https://docs.videodb.io/api-reference/create-new-api-key

/openapi.yaml post /user/api_key



# Create new collection
Source: https://docs.videodb.io/api-reference/create-new-collection

/openapi.yaml post /collection



# Create topup payment
Source: https://docs.videodb.io/api-reference/create-topup-payment

/openapi.yaml post /billing/topup



# Create video index
Source: https://docs.videodb.io/api-reference/create-video-index

/openapi.yaml post /video/{video_id}/index/



# Create video scene index
Source: https://docs.videodb.io/api-reference/create-video-scene-index

/openapi.yaml post /video/{video_id}/index/scene/



# Create video scenes
Source: https://docs.videodb.io/api-reference/create-video-scenes

/openapi.yaml post /video/{video_id}/scenes/



# Create video stream
Source: https://docs.videodb.io/api-reference/create-video-stream

/openapi.yaml post /video/{video_id}/stream/



# Delete API key
Source: https://docs.videodb.io/api-reference/delete-api-key

/openapi.yaml delete /user/api_key/{api_key}



# Delete audio
Source: https://docs.videodb.io/api-reference/delete-audio

/openapi.yaml delete /audio/{audio_id}



# Delete collection
Source: https://docs.videodb.io/api-reference/delete-collection

/openapi.yaml delete /collection/{collection_id}



# Delete download entry
Source: https://docs.videodb.io/api-reference/delete-download-entry

/openapi.yaml delete /download



# Delete image
Source: https://docs.videodb.io/api-reference/delete-image

/openapi.yaml delete /image/{image_id}



# Delete scene index
Source: https://docs.videodb.io/api-reference/delete-scene-index

/openapi.yaml delete /video/{video_id}/index/scene/{scene_index_id}



# Delete video
Source: https://docs.videodb.io/api-reference/delete-video

/openapi.yaml delete /video/{video_id}



# Delete video storage
Source: https://docs.videodb.io/api-reference/delete-video-storage

/openapi.yaml delete /video/{video_id}/storage/



# Describe video frame
Source: https://docs.videodb.io/api-reference/describe-video-frame

/openapi.yaml post /video/{video_id}/frame/{frame_id}/describe/



# Describe video scene
Source: https://docs.videodb.io/api-reference/describe-video-scene

/openapi.yaml post /video/{video_id}/scene/{scene_id}/describe/



# Download compiled timeline
Source: https://docs.videodb.io/api-reference/download-compiled-timeline

/openapi.yaml post /timeline_v2/download



# Create Download Job
Source: https://docs.videodb.io/api-reference/downloads/create_download

POST /download
Create a new video download job to generate a downloadable file

Initiate a download job to convert a stream URL into a downloadable file. This is an asynchronous operation that returns a download ID for tracking progress.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")

# Create a download job
result = conn.download(
    stream_link="https://stream.videodb.io/v/abc123def",
    name="my_export.mp4"
)

print(f"Download ID: {result.get('download_id')}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });

// Create a download job
const result = await conn.download(
    'https://stream.videodb.io/v/abc123def',
    'my_export.mp4'
);

console.log(`Download ID: ${result.download_id}`);
```

> [!NOTE]
>
> * Download jobs are asynchronous—use the returned `download_id` to check status
> * Provide a stream URL from `generateStream()` or a timeline stream URL
> * The `name` parameter sets the output filename (include file extension)
> * Download URLs expire after 24 hours from completion

> **[Streams and Exports Guide](/pages/act/output-and-delivery/streams-and-exports)**
>
> Learn how to generate streams before downloading

> **[Get Download Status](/api-reference/downloads/get_download)**
>
> Check download progress and retrieve the final URL


# Delete Download Job
Source: https://docs.videodb.io/api-reference/downloads/delete_download

DELETE /download
Cancel or delete a download job

Cancel or delete a download job. Once deleted, the download job record is removed and cannot be recovered.

```python Python theme={null}
import requests

response = requests.delete(
    "https://api.videodb.io/download",
    headers={"x-access-token": "your_api_key"},
    json={"download_id": "download-abc123"}
)

result = response.json()
if result.get('success'):
    print("Download job deleted successfully")
else:
    print("Failed to delete download job")
```

```javascript Node.js theme={null}
const response = await fetch(
    "https://api.videodb.io/download",
    {
        method: "DELETE",
        headers: {
            "x-access-token": "your_api_key"
        },
        body: JSON.stringify({
            download_id: "download-abc123"
        })
    }
);

const result = await response.json();
if (result.success) {
    console.log('Download job deleted successfully');
} else {
    console.log('Failed to delete download job');
}
```

> [!WARNING]
>
> * Deletion removes the download job record permanently
> * Cannot recover deleted jobs or their associated data
> * If download is still processing, it will be cancelled
> * Already completed downloads will have their records removed from your download history

> **[Get Download Status](/api-reference/downloads/get_download)**
>
> Check download status before deleting

> **[List Downloads](/api-reference/downloads/list_downloads)**
>
> View all downloads before deleting


# Get Download Status
Source: https://docs.videodb.io/api-reference/downloads/get_download

GET /download/{download_id}
Get detailed status and information for a specific download job

Check the status of a specific download job and retrieve the final download URL when ready. Poll this endpoint to track progress.

```python Python theme={null}
import requests

response = requests.get(
    "https://api.videodb.io/download/download-abc123",
    headers={"x-access-token": "your_api_key"}
)

result = response.json()
data = result.get('data', {})
print(f"Status: {data.get('status')}")
print(f"Name: {data.get('name')}")

if data.get('status') == 'done':
    print(f"Download URL: {data.get('download_url')}")
elif data.get('status') == 'error':
    print(f"Error: {data.get('error')}")
```

```javascript Node.js theme={null}
const response = await fetch(
    "https://api.videodb.io/download/download-abc123",
    {
        method: "GET",
        headers: {
            "x-access-token": "your_api_key"
        }
    }
);

const result = await response.json();
const data = result.data || {};
console.log(`Status: ${data.status}`);
console.log(`Name: ${data.name}`);

if (data.status === 'done') {
    console.log(`Download URL: ${data.download_url}`);
} else if (data.status === 'error') {
    console.log(`Error: ${data.error}`);
}
```

> [!NOTE]
>
> * Status values: `processing` (in progress), `done` (ready to download), `error` (failed)
> * Download URLs are only available when status is `done`
> * URLs expire 24 hours after generation—regenerate if needed
> * Includes metadata: creation time, filename, completion status

> **[Create Download](/api-reference/downloads/create_download)**
>
> Start a new download job

> **[Update Download](/api-reference/downloads/update_download)**
>
> Retry a failed download


# Downloads
Source: https://docs.videodb.io/api-reference/downloads/index

Manage video download jobs

Create and manage video download jobs.

## Endpoints

> **[Create Download](/api-reference/downloads/create_download)**
>
> Initiate download job

> **[List Downloads](/api-reference/downloads/list_downloads)**
>
> Get all downloads

> **[Get Download](/api-reference/downloads/get_download)**
>
> Retrieve download status

> **[Update Download](/api-reference/downloads/update_download)**
>
> Modify download settings

> **[Delete Download](/api-reference/downloads/delete_download)**
>
> Cancel download job


# List Download Jobs
Source: https://docs.videodb.io/api-reference/downloads/list_downloads

GET /download
Retrieve a list of all video download jobs and their status

Retrieve paginated list of all download jobs in your account. Each entry shows download status, timestamps, and download URLs for completed jobs.

```python Python theme={null}
import requests

response = requests.get(
    "https://api.videodb.io/download",
    headers={"x-access-token": "your_api_key"},
    params={
        "page_index": 0,
        "count": 50
    }
)

result = response.json()
downloads = result.get('data', {}).get('downloads', [])
for download in downloads:
    print(f"{download['id']}: {download['name']} - {download['status']}")
```

```javascript Node.js theme={null}
const response = await fetch(
    "https://api.videodb.io/download?page_index=0&count=50",
    {
        method: "GET",
        headers: {
            "x-access-token": "your_api_key"
        }
    }
);

const result = await response.json();
const downloads = result.data?.downloads || [];
for (const download of downloads) {
    console.log(`${download.id}: ${download.name} - ${download.status}`);
}
```

> [!NOTE]
>
> * Results are paginated with `page_index` and `count` parameters
> * `count` has a maximum of 5,000 items per request
> * Status can be `processing`, `done`, or `error`
> * Completed downloads include a `download_url` valid for 24 hours

> **[Get Download Status](/api-reference/downloads/get_download)**
>
> Get detailed information for a specific download

> **[Create Download](/api-reference/downloads/create_download)**
>
> Start a new download job


# Update Download Job
Source: https://docs.videodb.io/api-reference/downloads/update_download

POST /download/{download_id}
Update or modify a download job configuration

Retry a failed download job. Use this endpoint to re-attempt a download that encountered an error during processing.

```python Python theme={null}
import requests

response = requests.post(
    "https://api.videodb.io/download/download-abc123",
    headers={"x-access-token": "your_api_key"}
)

result = response.json()
print(f"Download ID: {result.get('download_id')}")
print("Retry initiated")
```

```javascript Node.js theme={null}
const response = await fetch(
    "https://api.videodb.io/download/download-abc123",
    {
        method: "POST",
        headers: {
            "x-access-token": "your_api_key"
        }
    }
);

const result = await response.json();
console.log(`Download ID: ${result.download_id}`);
console.log('Retry initiated');
```

> [!NOTE]
>
> * Use this endpoint to retry downloads that are in `error` status
> * Creates a new processing request with the same parameters
> * Returns immediately—use Get Download Status to track progress
> * Failed downloads may have encountered temporary issues (network, timeout)

> **[Get Download Status](/api-reference/downloads/get_download)**
>
> Check the current download status

> **[Delete Download](/api-reference/downloads/delete_download)**
>
> Cancel a download job


# Compile editor timeline
Source: https://docs.videodb.io/api-reference/editor/compile-editor-timeline

/openapi.yaml post /editor



# Compile Editor Timeline
Source: https://docs.videodb.io/api-reference/editor/compile_editor_timeline

POST /editor
Compile a visual editor timeline into a playable stream

Compile a timeline created in the visual editor into a playable stream. The timeline payload includes tracks with clips and their configurations.

> [!NOTE]
>
> * Compilation is asynchronous; the response includes an async response ID
> * The timeline payload should include tracks with clip definitions
> * Use the async response endpoint to poll for completion

> **[Download Timeline](/api-reference/editor/download_editor_timeline)**
>
> Download the compiled output

> **[Create Timeline v2](/api-reference/timeline/create_timeline_v2)**
>
> Programmatic timeline creation


# Download compiled editor timeline
Source: https://docs.videodb.io/api-reference/editor/download-compiled-editor-timeline

/openapi.yaml post /editor/download



# Download Editor Timeline
Source: https://docs.videodb.io/api-reference/editor/download_editor_timeline

POST /editor/download
Download a compiled editor timeline as a file

Generate a download URL for a compiled editor timeline. Provide the stream URL from a previous compilation to get a downloadable file.

> [!NOTE]
>
> * Requires a `stream_url` from a previous compilation
> * The download URL is temporary and should be used promptly

> **[Compile Timeline](/api-reference/editor/compile_editor_timeline)**
>
> Compile an editor timeline first


# Editor
Source: https://docs.videodb.io/api-reference/editor/index

Visual editor timeline compilation and download

Compile and download timelines created in the visual editor.

## Endpoints

> **[Compile Timeline](/api-reference/editor/compile_editor_timeline)**
>
> Compile an editor timeline

> **[Download Timeline](/api-reference/editor/download_editor_timeline)**
>
> Download compiled timeline


# Execute video workflow
Source: https://docs.videodb.io/api-reference/execute-video-workflow

/openapi.yaml post /video/{video_id}/workflow/



# Generate audio stream URL
Source: https://docs.videodb.io/api-reference/generate-audio-stream-url

/openapi.yaml post /audio/{audio_id}/generate_url



# Generate audio using AI
Source: https://docs.videodb.io/api-reference/generate-audio-using-ai

/openapi.yaml post /collection/{collection_id}/generate/audio/



# Generate custom thumbnail
Source: https://docs.videodb.io/api-reference/generate-custom-thumbnail

/openapi.yaml post /video/{video_id}/thumbnail/



# Generate image URL
Source: https://docs.videodb.io/api-reference/generate-image-url

/openapi.yaml post /image/{image_id}/generate_url



# Generate image using AI
Source: https://docs.videodb.io/api-reference/generate-image-using-ai

/openapi.yaml post /collection/{collection_id}/generate/image/



# Generate text using AI
Source: https://docs.videodb.io/api-reference/generate-text-using-ai

/openapi.yaml post /collection/{collection_id}/generate/text/



# Generate video clip
Source: https://docs.videodb.io/api-reference/generate-video-clip

/openapi.yaml post /video/{video_id}/clip



# Generate video transcription
Source: https://docs.videodb.io/api-reference/generate-video-transcription

/openapi.yaml post /video/{video_id}/transcription/



# Generate video using AI
Source: https://docs.videodb.io/api-reference/generate-video-using-ai

/openapi.yaml post /collection/{collection_id}/generate/video/



# Dub Video
Source: https://docs.videodb.io/api-reference/generation/dub_video

POST /collection/{collection_id}/generate/video/dub
Dub a video with AI-generated audio in a different language

Dub a video with AI-generated audio, replacing the original soundtrack with a new language version while preserving the speaker's voice characteristics.

> [!NOTE]
>
> * Dubbing is asynchronous; use `callback_url` for completion notification
> * `language_code` uses ISO 639-1 language codes (e.g., `es`, `fr`, `de`, `hi`)
> * The original audio is replaced with the dubbed version

> **[Translate Video](/api-reference/generation/translate_video)**
>
> Translate video content

> **[Dubbing Guide](/pages/act/generative-media/translation-and-dubbing)**
>
> Learn about dubbing and translation


# Generate Audio with AI
Source: https://docs.videodb.io/api-reference/generation/generate_audio

POST /collection/{collection_id}/generate/audio/
Generate audio, music, and voice content using AI models

Generate music, sound effects, and AI voices from text. The SDK provides specialized methods for each audio type with appropriate parameters.

## Music Generation

Create background music and ambient soundtracks from text prompts.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

music = coll.generate_music(
    prompt="Upbeat electronic dance music",
    duration=10
)

if music is None:
    raise RuntimeError("Expected a generated Audio asset.")

print(music.id)
print(music.generate_url())
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const music = await coll.generateMusic(
    "Upbeat electronic dance music",
    10
);

if (!music) {
    throw new Error("Expected a generated Audio asset.");
}

console.log(music.id);
console.log(await music.generateUrl());
```

## Sound Effect Generation

Generate sound effects and environmental audio from descriptions.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

sfx = coll.generate_sound_effect(
    prompt="Heavy rain with distant thunder",
    duration=5
)

if sfx is None:
    raise RuntimeError("Expected a generated Audio asset.")

print(sfx.id)
print(sfx.generate_url())
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const sfx = await coll.generateSoundEffect(
    "Heavy rain with distant thunder",
    5
);

if (!sfx) {
    throw new Error("Expected a generated Audio asset.");
}

console.log(sfx.id);
console.log(await sfx.generateUrl());
```

## Voice Generation

Convert text to natural-sounding speech in multiple voices.

```python Python theme={null}
import videodb
from videodb import GenerationJob

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

voice = coll.generate_voice(
    text="Welcome to our video demo",
    voice_name="Default",
    wait=True,
)

if voice is None or isinstance(voice, GenerationJob):
    raise RuntimeError("Expected the Audio asset after waiting.")

print(voice.id)
print(voice.generate_url())
```

```javascript Node.js theme={null}
import { connect, Audio, GenerationJob } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const voiceCandidate = await coll.generateVoice(
    "Welcome to our video demo",
    "Default"
);
const voice = voiceCandidate instanceof GenerationJob
    ? await voiceCandidate.wait()
    : voiceCandidate;

if (!(voice instanceof Audio)) {
    throw new Error("Expected a generated Audio asset.");
}

console.log(voice.id);
console.log(await voice.generateUrl());
```

> [!NOTE]
>
> * Default duration is 5 seconds for music, 2 seconds for sound effects
> * Audio IDs have an `a-` prefix (e.g., `a-a1b2c3d4`)
> * Voice self-inference can return a job; use `wait=True` or `{ wait: true }` to wait for Audio. Poll intervals and timeouts are measured in seconds.

> **[Generative Media Guide](/pages/act/generative-media/index)**
>
> Learn about all AI generation capabilities

> **[Voiceovers Tutorial](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add narration to silent footage with AI voices


# Generate Image with AI
Source: https://docs.videodb.io/api-reference/generation/generate_image

POST /collection/{collection_id}/generate/image/
Generate images using AI models from text prompts

Create AI-generated images from text descriptions with customizable aspect ratios. For self-inference jobs, these examples wait for the Image asset.

```python Python theme={null}
import videodb
from videodb import GenerationJob

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

image = coll.generate_image(
    prompt="Serene mountain landscape with golden sunset",
    aspect_ratio="16:9",
    wait=True,
)

if image is None or isinstance(image, GenerationJob):
    raise RuntimeError("Expected the Image asset after waiting.")

print(image.id)
print(image.generate_url())
```

```javascript Node.js theme={null}
import { connect, GenerationJob, Image } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const imageCandidate = await coll.generateImage(
    "Serene mountain landscape with golden sunset",
    "16:9"
);
const image = imageCandidate instanceof GenerationJob
    ? await imageCandidate.wait()
    : imageCandidate;

if (!(image instanceof Image)) {
    throw new Error("Expected an Image asset.");
}

console.log(image.id);
console.log(await image.generateUrl());
```

> [!NOTE]
>
> * Supported aspect ratios: `1:1`, `9:16`, `16:9`, `4:3`, `3:4`
> * Default aspect ratio is `1:1` if not specified
> * Self-inference can run asynchronously; use `callback_url` or wait for completion
> * Image IDs have an `img-` prefix (e.g., `img-a1b2c3d4`)

> **[Generative Media Guide](/pages/act/generative-media/index)**
>
> Learn about all AI generation capabilities

> **[Related Endpoint](/api-reference/generation/generate_video)**
>
> Generate Video


# Generate Text with AI
Source: https://docs.videodb.io/api-reference/generation/generate_text

POST /collection/{collection_id}/generate/text/
Generate text content using AI language models for captions, scripts, and summaries

Generate text content using AI language models. Use the basic default, a mini, pro, or ultra built-in, or a self-hosted model name, and get results as plain text or JSON.

```python Python theme={null}
import json
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Generate plain text
script_response = coll.generate_text(
    prompt="Write a 30-second commercial script for a coffee brand",
    model_name="pro",
    response_type="text"
)
script = (
    script_response
    if isinstance(script_response, str)
    else script_response.get("output", script_response)
)

print(script)

# Generate JSON response
data_response = coll.generate_text(
    prompt="Extract key points from a video summary",
    model_name="pro",
    response_type="json"
)
json_output = (
    data_response
    if isinstance(data_response, str)
    else data_response.get("output", data_response)
)
data = json.loads(json_output) if isinstance(json_output, str) else json_output

print(data)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Generate plain text
const scriptResult = await coll.generateText(
    "Write a 30-second commercial script for a coffee brand",
    "pro",
    "text"
);
const script = typeof scriptResult === "string"
    ? scriptResult
    : scriptResult.output;

if (typeof script !== "string") {
    throw new Error("Expected a text response.");
}

console.log(script);

// Generate JSON response
const jsonResult = await coll.generateText(
    "Extract key points from a video summary",
    "pro",
    "json"
);

const jsonOutput = typeof jsonResult === "string"
    ? jsonResult
    : jsonResult.output;
const data = typeof jsonOutput === "string"
    ? JSON.parse(jsonOutput)
    : jsonOutput;
console.log(data);
```

> [!NOTE]
>
> * Model options: `basic` (default), `mini`, `pro`, and `ultra` built-ins, or a self-hosted model name
> * Response types: `text` for plain text or `json` for JSON-formatted output
> * `generate_text()` returns a string or object; some responses wrap the generated value in `output`. Parse JSON text when structured data is needed
> * Text generation supports `wait` and callback options
> * Useful for generating captions, scripts, summaries, and extracted data

> **[Generative Media Guide](/pages/act/generative-media/index)**
>
> Learn about all AI generation capabilities

> **[Text Prompts Tutorial](/examples-and-tutorials/content-factory/text-prompts)**
>
> Master text prompting strategies for generation


# Generate Video with AI
Source: https://docs.videodb.io/api-reference/generation/generate_video

POST /collection/{collection_id}/generate/video/
Generate videos using AI models from text prompts

Generate short AI-powered video clips from text prompts.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

video = coll.generate_video(
    prompt="A drone flying over a futuristic city",
    duration=6
)

if video is None:
    raise RuntimeError("Expected a generated Video asset.")

print(video.id)
print(video.stream_url)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const video = await coll.generateVideo(
    "A drone flying over a futuristic city",
    6
);

if (!video) {
    throw new Error("Expected a generated Video asset.");
}

console.log(video.id);
console.log(video.streamUrl);
```

> [!NOTE]
>
> * Use Python's `callback_url=` or Node's third positional `callbackUrl` for webhook notification when generation completes
> * Video IDs have an `m-` prefix (e.g., `m-a1b2c3d4`)

> **[Generative Media Guide](/pages/act/generative-media/index)**
>
> Learn about all AI generation capabilities

> **[AI Ad Films Tutorial](/examples-and-tutorials/content-factory/ai-ad-films)**
>
> Create compelling AI-generated advertisements


# AI Generation
Source: https://docs.videodb.io/api-reference/generation/index

Generate media using AI models

Generate images, videos, audio, and text using AI models.

## Endpoints

> **[Generate Image](/api-reference/generation/generate_image)**
>
> Create AI-generated images

> **[Generate Video](/api-reference/generation/generate_video)**
>
> Create AI-generated videos

> **[Generate Audio](/api-reference/generation/generate_audio)**
>
> Create AI-generated audio

> **[Generate Text](/api-reference/generation/generate_text)**
>
> Create AI-generated text

> **[Dub Video](/api-reference/generation/dub_video)**
>
> Dub video with AI audio

> **[Translate Video](/api-reference/generation/translate_video)**
>
> Translate video content


# Translate Video
Source: https://docs.videodb.io/api-reference/generation/translate_video

POST /collection/{collection_id}/video/{video_id}/translate
Translate video content to a different language

Translate a video's spoken transcript for subtitle or text workflows.

> [!NOTE]
>
> * Translation is asynchronous; use `callback_url` for completion notification
> * `target_language` uses ISO 639-1 language codes (e.g., `es`, `fr`, `de`, `hi`)
> * The SDK translation helper returns translated transcript entries and does not create or mutate a video asset

> **[Dub Video](/api-reference/generation/dub_video)**
>
> Dub video with AI audio

> **[Translation Guide](/pages/act/generative-media/translation-and-dubbing)**
>
> Learn about dubbing and translation


# Get all video thumbnails
Source: https://docs.videodb.io/api-reference/get-all-video-thumbnails

/openapi.yaml get /video/{video_id}/thumbnails/



# Get async operation result
Source: https://docs.videodb.io/api-reference/get-async-operation-result

/openapi.yaml get /async-response/{response_id}



# Get audio details
Source: https://docs.videodb.io/api-reference/get-audio-details

/openapi.yaml get /audio/{audio_id}



# Get auto recharge settings
Source: https://docs.videodb.io/api-reference/get-auto-recharge-settings

/openapi.yaml get /billing/auto_recharge



# Get billing checkout history
Source: https://docs.videodb.io/api-reference/get-billing-checkout-history

/openapi.yaml get /billing/checkouts



# Get billing invoices
Source: https://docs.videodb.io/api-reference/get-billing-invoices

/openapi.yaml get /billing/invoices



# Get billing usage information
Source: https://docs.videodb.io/api-reference/get-billing-usage-information

/openapi.yaml get /billing/usage



# Get collection details
Source: https://docs.videodb.io/api-reference/get-collection-details

/openapi.yaml get /collection/{collection_id}



# Get download status/details
Source: https://docs.videodb.io/api-reference/get-download-statusdetails

/openapi.yaml get /download/{download_id}



# Get image details
Source: https://docs.videodb.io/api-reference/get-image-details

/openapi.yaml get /image/{image_id}



# Get scene index details
Source: https://docs.videodb.io/api-reference/get-scene-index-details

/openapi.yaml get /video/{video_id}/index/scene/{scene_index_id}



# Get self-inference generation job status
Source: https://docs.videodb.io/api-reference/get-self-inference-generation-job-status

/openapi.yaml get /job/{job_id}



# Get service information
Source: https://docs.videodb.io/api-reference/get-service-information

/openapi.yaml get /
Returns basic service information



# Get user API keys
Source: https://docs.videodb.io/api-reference/get-user-api-keys

/openapi.yaml get /user/api_key



# Get user collections
Source: https://docs.videodb.io/api-reference/get-user-collections

/openapi.yaml get /collection



# Get user information
Source: https://docs.videodb.io/api-reference/get-user-information

/openapi.yaml get /user



# Get video details
Source: https://docs.videodb.io/api-reference/get-video-details

/openapi.yaml get /video/{video_id}



# Get video index status
Source: https://docs.videodb.io/api-reference/get-video-index-status

/openapi.yaml get /video/{video_id}/index/



# Get video scene index status
Source: https://docs.videodb.io/api-reference/get-video-scene-index-status

/openapi.yaml get /video/{video_id}/index/scene/



# Get video scenes
Source: https://docs.videodb.io/api-reference/get-video-scenes

/openapi.yaml get /video/{video_id}/scenes/



# Get video thumbnail
Source: https://docs.videodb.io/api-reference/get-video-thumbnail

/openapi.yaml get /video/{video_id}/thumbnail/



# Get video transcription
Source: https://docs.videodb.io/api-reference/get-video-transcription

/openapi.yaml get /video/{video_id}/transcription/



# Delete Image
Source: https://docs.videodb.io/api-reference/images/delete_image

DELETE /image/{image_id}
Permanently delete an image from your collection

Permanently deletes an image from a collection. This action is irreversible and cannot be undone.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

coll.delete_image("i-xyz789")
print("Image deleted successfully")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

await coll.deleteImage('i-xyz789');
console.log('Image deleted successfully');
```

> [!WARNING]
>
> * This permanently removes the image and cannot be recovered
> * Ensure you have the correct image ID before proceeding
> * Deletion is immediate and applies to all associated data

> **[List Images](/api-reference/images/list_images)**
>
> View all images before deletion

> **[Get Image Details](/api-reference/images/get_image)**
>
> Verify image details before deleting


# Generate Image URL
Source: https://docs.videodb.io/api-reference/images/generate_image_url

POST /image/{image_id}/generate_url
Generate a shareable URL for an image

Generates a signed, shareable URL for an image. The URL can be used to access the image without authentication.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

image = coll.get_image("i-xyz789")
signed_url = image.generate_url()
print(f"Signed URL: {signed_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const image = await coll.getImage('i-xyz789');
const signedUrl = await image.generateUrl();
console.log(`Signed URL: ${signedUrl}`);
```

> [!NOTE]
>
> * Returns a signed URL string that can be shared publicly
> * Signed URLs provide temporary access without requiring authentication
> * URLs expire after a certain period (check API documentation for expiration time)
> * Collection ID must be passed as a parameter with the request

> **[Get Image Details](/api-reference/images/get_image)**
>
> Retrieve image information before generating URL

> **[List Images](/api-reference/images/list_images)**
>
> Browse available images


# Get Image Details
Source: https://docs.videodb.io/api-reference/images/get_image

GET /image/{image_id}
Retrieve detailed information about a specific image

Retrieves detailed information about a specific image including its name, URL, and collection association.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

image = coll.get_image("i-xyz789")
print(f"Name: {image.name}")
print(f"URL: {image.url}")
print(f"Collection ID: {image.collection_id}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const image = await coll.getImage('i-xyz789');
console.log(`Name: ${image.name}`);
console.log(`URL: ${image.url}`);
console.log(`Collection ID: ${image.collectionId}`);
```

> [!NOTE]
>
> * Requires the `collection_id` parameter to be passed with the request
> * Returns an Image object with properties: id, name, url, and collection\_id
> * Image ID must be valid and belong to the specified collection

> **[List All Images](/api-reference/images/list_images)**
>
> Get all images in your collection

> **[Generate URL](/api-reference/images/generate_image_url)**
>
> Create a shareable signed URL for the image


# Images
Source: https://docs.videodb.io/api-reference/images/index

Manage images and generate URLs

Manage images and generate access URLs.

## Endpoints

> **[List Images](/api-reference/images/list_images)**
>
> Get all images

> **[Get Image](/api-reference/images/get_image)**
>
> Retrieve image details

> **[Delete Image](/api-reference/images/delete_image)**
>
> Remove image

> **[Generate URL](/api-reference/images/generate_image_url)**
>
> Create access URL

> **[Update Image](/api-reference/images/update_image)**
>
> Update image metadata


# List All Images
Source: https://docs.videodb.io/api-reference/images/list_images

GET /image/
Retrieve a paginated list of all images in your collection

Retrieves all images in a collection. Returns a list of image objects with metadata like name, URL, and ID.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

images = coll.get_images()
for image in images:
    print(f"Image: {image.name} (ID: {image.id})")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

const images = await coll.getImages();
images.forEach(img => {
  console.log(`Image: ${img.name} (ID: ${img.id})`);
});
```

> [!NOTE]
>
> * Returns all images in the collection specified by the `collection_id` parameter
> * Image IDs have an `i-` prefix (e.g., `i-a1b2c3d4`)
> * Each image has properties: id, collection\_id, name, and url

> **[Collection Guide](/pages/ingest/files-and-collections/create-collection)**
>
> Learn how to organize images in collections

> **[Get Image Details](/api-reference/images/get_image)**
>
> Retrieve a specific image


# Update image
Source: https://docs.videodb.io/api-reference/images/update-image

/openapi.yaml patch /image/{image_id}



# Update Image
Source: https://docs.videodb.io/api-reference/images/update_image

PATCH /image/{image_id}
Update image metadata

Update metadata for a specific image, including its name and description.

> [!NOTE]
>
> * Only provided fields are updated; omitted fields remain unchanged


# API Reference
Source: https://docs.videodb.io/api-reference/introduction

REST API for video processing, AI generation, and semantic search

Complete REST API for VideoDB - the serverless video database.

**Base URL:** `https://api.videodb.io`

## Authentication

All requests require an API key in the `x-access-token` header.

```bash theme={null}
curl -H "x-access-token: your_api_key" https://api.videodb.io/collection
```

## API Groups

> **[Authentication](/api-reference/authentication/index)**
>
> API keys and user account

> **[Collections](/api-reference/collections/index)**
>
> Organize media content

> **[Videos](/api-reference/videos/index)**
>
> Core video operations

> **[Streaming & Playback](/api-reference/videos/streaming/index)**
>
> Generate streams and thumbnails

> **[Transcription](/api-reference/videos/transcription/index)**
>
> Speech-to-text transcripts

> **[Scene & Frame Analysis](/api-reference/videos/scenes/index)**
>
> Computer vision analysis

> **[Advanced Operations](/api-reference/videos/advanced/index)**
>
> Video clipping and workflows

> **[Audio](/api-reference/audio/index)**
>
> Audio file management

> **[Images](/api-reference/images/index)**
>
> Image management

> **[AI Generation](/api-reference/generation/index)**
>
> Generate media with AI

> **[Sandbox](/api-reference/sandbox/create_sandbox)**
>
> Dedicated compute lifecycle

> **[Timeline & Compilation](/api-reference/timeline/index)**
>
> Programmatic video editing

> **[Billing](/api-reference/billing/index)**
>
> Usage and billing

> **[Downloads](/api-reference/downloads/index)**
>
> Export video files

> **[Utilities](/api-reference/utilities/index)**
>
> System utilities

> **[Assets](/api-reference/assets/index)**
>
> Browse all media assets

> **[Editor](/api-reference/editor/index)**
>
> Visual editor compilation

> **[Transcode](/api-reference/transcode/index)**
>
> Video format conversion

> **[Meeting](/api-reference/meeting/index)**
>
> Meeting recording

> **[Capture](/api-reference/capture/index)**
>
> Real-time media capture

> **[RTStream](/api-reference/rtstream/index)**
>
> Live stream processing

> **[RTStream Transcription](/api-reference/rtstream/transcription/index)**
>
> Real-time transcription

> **[RTStream Search](/api-reference/rtstream/scene-indexing/search_rtstream)**
>
> Search indexed stream content

> **[RTStream Events](/api-reference/rtstream/events-alerts/index)**
>
> Define event detection templates

## Response Format

All responses follow this structure:

```json theme={null}
{
  "success": true,
  "data": { }
}
```

## Error Codes

| Code | Description |
| - | - |
| `400` | Invalid request parameters |
| `401` | Invalid or missing API key |
| `404` | Resource not found |
| `429` | Rate limit exceeded |
| `500` | Internal server error |

## SDKs

> **[Python SDK](https://github.com/video-db/videodb-python)**
>
> ```bash theme={null}
> pip install videodb
> ```

> **[Node.js SDK](https://github.com/video-db/videodb-node)**
>
> ```bash theme={null}
> npm install videodb
> ```


# List audios
Source: https://docs.videodb.io/api-reference/list-audios

/openapi.yaml get /audio/



# List download entries
Source: https://docs.videodb.io/api-reference/list-download-entries

/openapi.yaml get /download



# List images
Source: https://docs.videodb.io/api-reference/list-images

/openapi.yaml get /image/



# List videos
Source: https://docs.videodb.io/api-reference/list-videos

/openapi.yaml get /video/



# Get meeting by video
Source: https://docs.videodb.io/api-reference/meeting/get-meeting-by-video

/openapi.yaml get /video/{video_id}/meeting



# Get meeting recording information
Source: https://docs.videodb.io/api-reference/meeting/get-meeting-recording-information

/openapi.yaml get /collection/{collection_id}/meeting/{bot_id}



# Get Meeting
Source: https://docs.videodb.io/api-reference/meeting/get_meeting

GET /collection/{collection_id}/meeting/{bot_id}
Get meeting recording information

Retrieve information about a meeting recording, including its status and video URL.

> [!NOTE]
>
> * Status indicates the current state of the recording (e.g., `recording`, `done`)
> * The `video_url` is available once the recording is complete
> * Use the `bot_id` returned from the record meeting endpoint

> **[Record Meeting](/api-reference/meeting/record_meeting)**
>
> Start a new recording

> **[Get Meeting by Video](/api-reference/meeting/get_meeting_by_video)**
>
> Look up meeting from video ID


# Get Meeting by Video
Source: https://docs.videodb.io/api-reference/meeting/get_meeting_by_video

GET /video/{video_id}/meeting
Get meeting information associated with a video

Retrieve meeting information for a video that was recorded from a meeting. This is useful when you have a video ID and want to look up its meeting context.

> [!NOTE]
>
> * Only returns data for videos that were recorded from meetings
> * Returns meeting metadata including bot details and recording status

> **[Record Meeting](/api-reference/meeting/record_meeting)**
>
> Start a new meeting recording

> **[Get Video](/api-reference/videos/get_video)**
>
> Get video details


# Meeting
Source: https://docs.videodb.io/api-reference/meeting/index

Record and manage meeting recordings

Record meetings from video conferencing platforms and retrieve meeting data.

## Endpoints

> **[Record Meeting](/api-reference/meeting/record_meeting)**
>
> Start recording a meeting

> **[Get Meeting](/api-reference/meeting/get_meeting)**
>
> Get meeting recording info

> **[Get Meeting by Video](/api-reference/meeting/get_meeting_by_video)**
>
> Get meeting info from video ID


# Record a meeting
Source: https://docs.videodb.io/api-reference/meeting/record-a-meeting

/openapi.yaml post /collection/{collection_id}/meeting/record



# Record Meeting
Source: https://docs.videodb.io/api-reference/meeting/record_meeting

POST /collection/{collection_id}/meeting/record
Start recording a meeting from a video conferencing platform

Start recording a meeting by sending a bot to join a video conferencing session. Supports Google Meet, Zoom, and other platforms.

> [!NOTE]
>
> * The bot joins the meeting and records audio/video
> * Set `bot_name` to customize the display name in the meeting
> * Use `bot_image_url` to set a custom avatar for the bot
> * Enable `realtime_stream` for live streaming during the meeting
> * Use `callback_url` for webhook notifications on meeting events

> **[Get Meeting](/api-reference/meeting/get_meeting)**
>
> Check recording status

> **[Meeting Intelligence](/pages/automate/n8n/n8n-workflow-meeting-intelligence)**
>
> Build meeting intelligence workflows


# OpenAI-compatible chat completions proxy
Source: https://docs.videodb.io/api-reference/openai-compatible-chat-completions-proxy

/openapi.yaml post /chat/completions
Proxy endpoint for OpenAI chat completions API with VideoDB billing



# Retry download
Source: https://docs.videodb.io/api-reference/retry-download

/openapi.yaml post /download/{download_id}



# Create collection event
Source: https://docs.videodb.io/api-reference/rtstream/create-collection-event

/openapi.yaml post /rtstream/event



# Create RTStream
Source: https://docs.videodb.io/api-reference/rtstream/create-rtstream

/openapi.yaml post /rtstream/



# Create RTStream alert
Source: https://docs.videodb.io/api-reference/rtstream/create-rtstream-alert

/openapi.yaml post /rtstream/{stream_id}/index/{scene_index_id}/alert



# Create RTStream scene index
Source: https://docs.videodb.io/api-reference/rtstream/create-rtstream-scene-index

/openapi.yaml post /rtstream/{stream_id}/index/scene



# Create RTStream
Source: https://docs.videodb.io/api-reference/rtstream/create_rtstream

POST /rtstream/
Create a new real-time stream connection

Create a new real-time stream by connecting to an RTSP or YouTube stream URL. The stream begins processing immediately.

> [!NOTE]
>
> * `url` is required and must be a valid RTSP or YouTube stream URL
> * `sample_rate` controls the frame sampling rate (default: 30)
> * `media_types` specifies which media to capture: `video`, `audio`, or both
> * Set `store` to `true` to persist stream recordings
> * Enable `enable_transcript` for automatic speech transcription

> **[RTSP Ingest Guide](/pages/ingest/live-streams/rtsp-ingest)**
>
> Learn about RTSP stream ingestion

> **[Stream Lifecycle](/pages/ingest/live-streams/stream-lifecycle)**
>
> Understand stream states


# Create Alert
Source: https://docs.videodb.io/api-reference/rtstream/events-alerts/create_alert

POST /rtstream/{stream_id}/index/{scene_index_id}/alert
Create an alert on a stream scene index

Create an alert that triggers when a detection event matches on a stream's scene index. Alerts can send webhooks or WebSocket notifications.

> [!NOTE]
>
> This singular scene-index endpoint is retained for compatibility. For current RTStream V2, attach alerts to plural `/indexes` as shown in the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index).

> [!NOTE]
>
> * `event_id` is required - must reference an existing detection event
> * Use `callback_url` for HTTP webhook notifications when the event is detected
> * Use `ws_connection_id` for real-time WebSocket notifications

> **[Create Event](/api-reference/rtstream/events-alerts/create_event)**
>
> Define a detection event first

> **[Alerts Guide](/pages/act/live-action/alerts-and-callbacks)**
>
> Learn about alerts and callbacks


# Create Event
Source: https://docs.videodb.io/api-reference/rtstream/events-alerts/create_event

POST /rtstream/event
Create a detection event for real-time stream monitoring

Create a detection event that defines what to look for in real-time streams. Events are reusable templates that can be attached to stream indexes as alerts.

> [!NOTE]
>
> * `event_prompt` describes the condition to detect in natural language
> * `label` is a short identifier for the event type
> * Events are reusable across multiple stream indexes and alerts

> **[Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index)**
>
> Set up a stream understanding and index

> **[Event Detection Guide](/pages/act/live-action/event-detection-patterns)**
>
> Learn about event detection


# RTStream Events
Source: https://docs.videodb.io/api-reference/rtstream/events-alerts/index

Define detection events for live streams

Define reusable detection-event templates for real-time stream monitoring.

## Endpoints

> **[Create Event](/api-reference/rtstream/events-alerts/create_event)**
>
> Define a detection event

> **[List Events](/api-reference/rtstream/events-alerts/list_events)**
>
> Get all events


# List Alerts
Source: https://docs.videodb.io/api-reference/rtstream/events-alerts/list_alerts

GET /rtstream/{stream_id}/index/{scene_index_id}/alert
List all alerts on a stream scene index

List all alerts configured on a specific scene index of a real-time stream.

> [!NOTE]
>
> **Legacy scene-index compatibility endpoint.** For current RTStream V2, attach alerts to plural `/indexes` as shown in the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index).

> [!NOTE]
>
> * Returns all alerts with their event details, callback URLs, and status
> * Alert status is either `enabled` or `disabled`


# List Events
Source: https://docs.videodb.io/api-reference/rtstream/events-alerts/list_events

GET /rtstream/event
List all detection events

List all detection events in your account.

> [!NOTE]
>
> * Returns all events with their IDs, prompts, and labels
> * Events can be reused across multiple alerts and stream indexes


# Update Alert Status
Source: https://docs.videodb.io/api-reference/rtstream/events-alerts/update_alert_status

PATCH /rtstream/{stream_id}/index/{scene_index_id}/alert/{alert_id}/status
Enable or disable an alert

Enable or disable an alert on a stream's scene index.

> [!NOTE]
>
> This singular scene-index endpoint is retained for compatibility. For current RTStream V2, attach alerts to plural `/indexes` as shown in the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index).

> [!NOTE]
>
> * Request body: `{"action":"enable"}` or `{"action":"disable"}`
> * Disabled alerts stop sending notifications but are not deleted
> * Re-enable with `{"action":"enable"}`


# Export RTStream recording as VideoDB asset
Source: https://docs.videodb.io/api-reference/rtstream/export-rtstream-recording-as-videodb-asset

/openapi.yaml post /rtstream/{stream_id}/export



# Export RTStream
Source: https://docs.videodb.io/api-reference/rtstream/export_rtstream

POST /rtstream/{stream_id}/export
Export an RTStream recording as a VideoDB video asset

Export a recording from a real-time stream as a VideoDB video asset. This creates a standard video that can be indexed, searched, and edited.

> [!NOTE]
>
> * The exported video gets a standard video ID (`m-` prefix)
> * Includes stream URL and player URL for immediate playback
> * The recording must exist (stream must have `store` enabled or have recorded data)

> **[Get Video](/api-reference/videos/get_video)**
>
> Access the exported video

> **[Streams & Exports Guide](/pages/act/output-and-delivery/streams-and-exports)**
>
> Learn about exports


# Get RTStream details
Source: https://docs.videodb.io/api-reference/rtstream/get-rtstream-details

/openapi.yaml get /rtstream/{stream_id}/



# Get RTStream playback URL
Source: https://docs.videodb.io/api-reference/rtstream/get-rtstream-playback-url

/openapi.yaml get /rtstream/{stream_id}/stream



# Get RTStream scene index details
Source: https://docs.videodb.io/api-reference/rtstream/get-rtstream-scene-index-details

/openapi.yaml get /rtstream/{stream_id}/index/{scene_index_id}



# Get RTStream scene records
Source: https://docs.videodb.io/api-reference/rtstream/get-rtstream-scene-records

/openapi.yaml get /rtstream/{stream_id}/index/scene/{scene_index_id}



# Get RTStream transcription data
Source: https://docs.videodb.io/api-reference/rtstream/get-rtstream-transcription-data

/openapi.yaml get /rtstream/{stream_id}/transcription/



# Get RTStream transcription status
Source: https://docs.videodb.io/api-reference/rtstream/get-rtstream-transcription-status

/openapi.yaml get /rtstream/{stream_id}/transcription/status



# Get RTStream
Source: https://docs.videodb.io/api-reference/rtstream/get_rtstream

GET /rtstream/{stream_id}/
Retrieve details of a specific RTStream

Retrieve detailed information about a specific real-time stream, including its configuration and current status.

> [!NOTE]
>
> * Stream IDs start with `rts-`
> * Returns full configuration including URL, sample rate, media types, and status


# Get RTStream Playback URL
Source: https://docs.videodb.io/api-reference/rtstream/get_rtstream_stream

GET /rtstream/{stream_id}/stream
Get a playback URL for an RTStream

Get a playback URL for viewing the real-time stream. Optionally specify a time range for historical playback.

> [!NOTE]
>
> * Omit `start` and `end` for live playback
> * Use Unix timestamps for `start` and `end` to retrieve historical segments
> * `frame_rate` controls the playback frame rate


# RTStream
Source: https://docs.videodb.io/api-reference/rtstream/index

Real-time stream processing, indexing, and search

Manage real-time streams from RTSP sources and YouTube. Connect live video feeds, index content in real-time, and search across streams.

> [!NOTE]
>
> The new RTStream Understand → Index interface is documented in [Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index). Initial support is limited to continuous VLM understanding with time-based segmentation.

## Core

> **[List RTStreams](/api-reference/rtstream/list_rtstreams)**
>
> Get all RTStreams

> **[Create RTStream](/api-reference/rtstream/create_rtstream)**
>
> Connect a new stream

> **[Get RTStream](/api-reference/rtstream/get_rtstream)**
>
> Retrieve stream details

> **[Update RTStream](/api-reference/rtstream/update_rtstream)**
>
> Modify stream settings

> **[Update Status](/api-reference/rtstream/update_rtstream_status)**
>
> Start or stop a stream

> **[Export RTStream](/api-reference/rtstream/export_rtstream)**
>
> Export recording as asset

> **[Get Stream URL](/api-reference/rtstream/get_rtstream_stream)**
>
> Get playback URL

## Transcription

> **[Get Transcription](/api-reference/rtstream/transcription/get_rtstream_transcription)**
>
> Retrieve transcription data

> **[Manage Transcription](/api-reference/rtstream/transcription/manage_rtstream_transcription)**
>
> Start or stop transcription

> **[Transcription Status](/api-reference/rtstream/transcription/get_rtstream_transcription_status)**
>
> Check transcription status

## Understand & Index

> **[Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index)**
>
> Set up continuous understanding and indexes

## Search

> **[Search RTStream](/api-reference/rtstream/scene-indexing/search_rtstream)**
>
> Search indexed content

## Events

> **[Create Event](/api-reference/rtstream/events-alerts/create_event)**
>
> Define a detection event

> **[List Events](/api-reference/rtstream/events-alerts/list_events)**
>
> Get all events


# List collection events
Source: https://docs.videodb.io/api-reference/rtstream/list-collection-events

/openapi.yaml get /rtstream/event



# List RTStream alerts
Source: https://docs.videodb.io/api-reference/rtstream/list-rtstream-alerts

/openapi.yaml get /rtstream/{stream_id}/index/{scene_index_id}/alert



# List RTStream scene indexes
Source: https://docs.videodb.io/api-reference/rtstream/list-rtstream-scene-indexes

/openapi.yaml get /rtstream/{stream_id}/index/scene



# List RTStreams
Source: https://docs.videodb.io/api-reference/rtstream/list-rtstreams

/openapi.yaml get /rtstream/



# List RTStreams
Source: https://docs.videodb.io/api-reference/rtstream/list_rtstreams

GET /rtstream/
List all RTStreams with filtering and pagination

List all real-time streams with optional filtering by status, name, and sorting.

> [!NOTE]
>
> * Filter by `status`: `connected` or `stopped`
> * Filter by `name` (substring match)
> * Use `ordering` for sorting (prefix with `-` for descending, e.g., `-created_at`)
> * Results are paginated with `limit` and `offset`


# Create RTStream Scene Index
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/create_rtstream_scene_index

POST /rtstream/{stream_id}/index/scene
Create a scene index for real-time visual analysis

Create a scene index on a real-time stream for continuous visual analysis. The index extracts and describes scenes as the stream progresses.

> [!NOTE]
>
> **Legacy compatibility endpoint.** This singular `/index/scene` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

> [!NOTE]
>
> * `extraction_type` can be `time` or `transcript`
> * For time-based extraction, set `extraction_config.time` for interval in seconds
> * `prompt` defines what the model should look for in each scene
> * Use `ws_connection_id` to receive real-time index updates via WebSocket

> **[List Scene Indexes](/api-reference/rtstream/scene-indexing/list_rtstream_scene_indexes)**
>
> View all indexes on a stream

> **[Search RTStream](/api-reference/rtstream/scene-indexing/search_rtstream)**
>
> Search indexed content


# Get RTStream Scene Index
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/get_rtstream_scene_index

GET /rtstream/{stream_id}/index/{scene_index_id}
Get details of a specific scene index

Retrieve detailed information about a specific scene index on a real-time stream.

> [!NOTE]
>
> This singular `/index/{scene_index_id}` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

> [!NOTE]
>
> * Returns full index configuration including extraction type, config, prompt, and status


# Get RTStream Scene Records
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/get_rtstream_scene_records

GET /rtstream/{stream_id}/index/scene/{scene_index_id}
Retrieve indexed scene records from a real-time stream

Retrieve the scene records (descriptions and timestamps) generated by a scene index on a real-time stream.

> [!NOTE]
>
> **Legacy compatibility endpoint.** This singular `/index/scene` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

> [!NOTE]
>
> * Use `start` and `end` (Unix timestamps) to filter by time range
> * Paginated with `page` and `page_size` (default: 100)
> * Each record includes timestamps and the AI-generated scene description


# RTStream Scene Indexing
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/index

Index and search visual content in real-time streams

Create and manage scene indexes for real-time visual analysis on live streams. Index visual content as it streams, search indexed scenes, and configure detection prompts.

> [!NOTE]
>
> **Legacy compatibility endpoint.** This singular `/index/scene` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

## Endpoints

> **[Create Scene Index](/api-reference/rtstream/scene-indexing/create_rtstream_scene_index)**
>
> Index stream scenes

> **[List Scene Indexes](/api-reference/rtstream/scene-indexing/list_rtstream_scene_indexes)**
>
> Get all scene indexes

> **[Get Scene Index](/api-reference/rtstream/scene-indexing/get_rtstream_scene_index)**
>
> Retrieve index details

> **[Get Scene Records](/api-reference/rtstream/scene-indexing/get_rtstream_scene_records)**
>
> Retrieve indexed scene data

> **[Update Scene Index](/api-reference/rtstream/scene-indexing/update_rtstream_scene_index)**
>
> Update index prompt

> **[Update Index Status](/api-reference/rtstream/scene-indexing/update_rtstream_scene_index_status)**
>
> Start or stop indexing

> **[Search RTStream](/api-reference/rtstream/scene-indexing/search_rtstream)**
>
> Search indexed content


# List RTStream Scene Indexes
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/list_rtstream_scene_indexes

GET /rtstream/{stream_id}/index/scene
List all scene indexes on a real-time stream

List all scene indexes configured on a specific real-time stream.

> [!NOTE]
>
> **Legacy compatibility endpoint.** This singular `/index/scene` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

> [!NOTE]
>
> * Returns all scene indexes with their status, prompt, and extraction type
> * A stream can have multiple scene indexes with different prompts


# Search RTStream
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/search_rtstream

POST /rtstream/{stream_id}/search
Search indexed content within a real-time stream

Search for specific content within a real-time stream's scene index using semantic search.

> [!NOTE]
>
> * `query` is required; `scene_index_id` (SDK `indexId`) is optional
> * `result_threshold`, `score_threshold`, `dynamic_score_percentage`, and `filter` are optional

> **[Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index)**
>
> Set up continuous understanding and indexes

> **[Event Detection](/pages/act/live-action/event-detection-patterns)**
>
> Build event detection patterns


# Update RTStream Scene Index
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/update_rtstream_scene_index

PATCH /rtstream/{stream_id}/index/scene/{scene_index_id}
Update the prompt of a scene index

Update the prompt used by a scene index on a real-time stream. The new prompt takes effect for subsequent scene descriptions.

> [!NOTE]
>
> **Legacy compatibility endpoint.** This singular `/index/scene` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

> [!NOTE]
>
> * Only the `prompt` field can be updated
> * Changes apply to future scene descriptions, not retroactively


# Update RTStream Scene Index Status
Source: https://docs.videodb.io/api-reference/rtstream/scene-indexing/update_rtstream_scene_index_status

PATCH /rtstream/{stream_id}/index/scene/{scene_index_id}/status
Start or stop a scene index on a real-time stream

Start or stop a scene index on a real-time stream.

> [!NOTE]
>
> **Legacy compatibility endpoint.** This singular `/index/scene` route is retained for compatibility. Use the [RTStream V2 Understand → Index guide](/pages/ingest/live-streams/rtstream-understand-index) for plural `/indexes`.

> [!NOTE]
>
> * `action` must be `start` or `stop`
> * Stopping an index pauses scene description generation but retains existing records


# Search RTStream scene index
Source: https://docs.videodb.io/api-reference/rtstream/search-rtstream-scene-index

/openapi.yaml post /rtstream/{stream_id}/search



# Start or stop RTStream
Source: https://docs.videodb.io/api-reference/rtstream/start-or-stop-rtstream

/openapi.yaml patch /rtstream/{stream_id}/status/



# Start or stop RTStream transcription
Source: https://docs.videodb.io/api-reference/rtstream/start-or-stop-rtstream-transcription

/openapi.yaml post /rtstream/{stream_id}/transcription/



# Get RTStream Transcription
Source: https://docs.videodb.io/api-reference/rtstream/transcription/get_rtstream_transcription

GET /rtstream/{stream_id}/transcription/
Retrieve transcription data from a real-time stream

Retrieve transcription records from a real-time stream, including word-level timestamps and pagination support.

> [!NOTE]
>
> * Use `start` and `end` (Unix timestamps) to filter by time range
> * Use `since` for polling - returns only entries newer than the given timestamp
> * Paginated with `page` and `page_size` (max 1000)
> * Includes word-level timestamps for each transcription record


# Get RTStream Transcription Status
Source: https://docs.videodb.io/api-reference/rtstream/transcription/get_rtstream_transcription_status

GET /rtstream/{stream_id}/transcription/status
Check the status of transcription on a real-time stream

Check whether transcription is currently active on a real-time stream.

> [!NOTE]
>
> * Status values: `running`, `stopped`, `not_configured`
> * Returns the engine, language, and timestamps for the transcription configuration


# RTStream Transcription
Source: https://docs.videodb.io/api-reference/rtstream/transcription/index

Manage real-time transcription on live streams

Control and retrieve real-time transcription from live streams. Enable speech-to-text processing on active streams and access transcribed content.

## Endpoints

> **[Get Transcription](/api-reference/rtstream/transcription/get_rtstream_transcription)**
>
> Retrieve transcription data

> **[Manage Transcription](/api-reference/rtstream/transcription/manage_rtstream_transcription)**
>
> Start or stop transcription

> **[Transcription Status](/api-reference/rtstream/transcription/get_rtstream_transcription_status)**
>
> Check transcription status


# Manage RTStream Transcription
Source: https://docs.videodb.io/api-reference/rtstream/transcription/manage_rtstream_transcription

POST /rtstream/{stream_id}/transcription/
Start or stop transcription on a real-time stream

Start or stop the transcription service on a real-time stream.

> [!NOTE]
>
> * `action` must be `start` or `stop`
> * Use `engine` to select the transcription engine (default: `default`)
> * Use `ws_connection_id` to receive real-time transcription updates via WebSocket

> **[Get Transcription](/api-reference/rtstream/transcription/get_rtstream_transcription)**
>
> Retrieve transcription data

> **[Transcription Status](/api-reference/rtstream/transcription/get_rtstream_transcription_status)**
>
> Check transcription status


# Update RTStream
Source: https://docs.videodb.io/api-reference/rtstream/update-rtstream

/openapi.yaml patch /rtstream/{stream_id}/



# Update RTStream alert status
Source: https://docs.videodb.io/api-reference/rtstream/update-rtstream-alert-status

/openapi.yaml patch /rtstream/{stream_id}/index/{scene_index_id}/alert/{alert_id}/status



# Update RTStream scene index prompt
Source: https://docs.videodb.io/api-reference/rtstream/update-rtstream-scene-index-prompt

/openapi.yaml patch /rtstream/{stream_id}/index/scene/{scene_index_id}



# Update RTStream scene index status
Source: https://docs.videodb.io/api-reference/rtstream/update-rtstream-scene-index-status

/openapi.yaml patch /rtstream/{stream_id}/index/scene/{scene_index_id}/status



# Update RTStream
Source: https://docs.videodb.io/api-reference/rtstream/update_rtstream

PATCH /rtstream/{stream_id}/
Update RTStream configuration

Update the configuration of an existing real-time stream, such as its name or sample rate.

> [!NOTE]
>
> * Only provided fields are updated; omitted fields remain unchanged
> * Changes to `sample_rate` take effect immediately on the running stream


# Update RTStream Status
Source: https://docs.videodb.io/api-reference/rtstream/update_rtstream_status

PATCH /rtstream/{stream_id}/status/
Start or stop an RTStream

Start or stop a real-time stream. Use this to control the stream lifecycle without deleting it.

> [!NOTE]
>
> * `action` must be `start` or `stop`
> * Stopping a stream pauses processing but retains its configuration
> * Restarting a stopped stream resumes from the live feed


# Create a Sandbox Compute pool
Source: https://docs.videodb.io/api-reference/sandbox/create-a-sandbox-compute-pool

/openapi.yaml post /sandbox



# Create Sandbox
Source: https://docs.videodb.io/api-reference/sandbox/create_sandbox

POST /sandbox
Create a Sandbox Compute pool



# Get a Sandbox Compute pool
Source: https://docs.videodb.io/api-reference/sandbox/get-a-sandbox-compute-pool

/openapi.yaml get /sandbox/{sandbox_id}



# Get Job
Source: https://docs.videodb.io/api-reference/sandbox/get_job

GET /job/{job_id}
Get self-inference generation job status



# Get Sandbox
Source: https://docs.videodb.io/api-reference/sandbox/get_sandbox

GET /sandbox/{sandbox_id}
Get a Sandbox Compute pool



# List Sandbox Compute pools
Source: https://docs.videodb.io/api-reference/sandbox/list-sandbox-compute-pools

/openapi.yaml get /sandbox



# List Sandboxes
Source: https://docs.videodb.io/api-reference/sandbox/list_sandboxes

GET /sandbox
List Sandbox Compute pools



# Stop a Sandbox Compute pool
Source: https://docs.videodb.io/api-reference/sandbox/stop-a-sandbox-compute-pool

/openapi.yaml post /sandbox/{sandbox_id}/stop



# Stop Sandbox
Source: https://docs.videodb.io/api-reference/sandbox/stop_sandbox

POST /sandbox/{sandbox_id}/stop
Stop a Sandbox Compute pool



# Aggregate
Source: https://docs.videodb.io/api-reference/search-v2/aggregate

Run counts, groups, facets, and metrics over one aggregatable index.

## Endpoints

```http theme={null}
POST /video/{video_id}/aggregate
POST /collection/{collection_id}/aggregate
```

Use `aggregate()` for counts, groups, facets, and metrics.

## Request body

| Field | Type | Required | Description |
| - | -: | -: | - |
| `index_name` | string | no | Optional index name to aggregate. |
| `index_id` | string | no | Optional index ID to aggregate. |
| `group_by` | string | no | Optional aggregatable field to group by. |
| `metric` | string | no | Aggregation metric. Defaults to `count`. |
| `filter` | list/object | no | Filter applied before aggregation. |
| `limit` | integer | no | Maximum returned groups. Defaults to `100`. |
| `sort` | list/string/object | no | Sort aggregate rows when supported. |

`aggregate()` does not accept `return_fields` and does not return `Shot` objects.

## Example

```python theme={null}
response = collection.aggregate(
    index_name="objects",
    group_by="label",
    metric="count",
    limit=100,
)

counts = response["results"]
```

## Example with filters

```python theme={null}
response = collection.aggregate(
    index_name="objects",
    filter=[{"field": "max_score", "op": ">=", "value": 0.8}],
    group_by="label",
    metric="count",
    limit=100,
)

counts = response["results"]
```

## Response

`aggregate()` returns an envelope containing the aggregate rows and any migration warnings:

```json theme={null}
{
  "results": [
    {"label": "person", "count": 18},
    {"label": "car", "count": 7}
  ],
  "warnings": []
}
```

Use `response["results"]` for the aggregate rows. When the server includes warnings, the SDK also emits them as Python `UserWarning` messages.


# Ask
Source: https://docs.videodb.io/api-reference/search-v2/ask

Ask questions over indexed video or collection context.

## Endpoints

```http theme={null}
POST /video/{video_id}/ask
POST /collection/{collection_id}/ask
```

Use `ask()` when you want a generated answer grounded in indexed video context.

## Request body

| Field | Type | Required | Description |
| - | -: | -: | - |
| `question` | string | yes | Question to answer from indexed video context. |
| `top_k` | integer | no | Number of retrieved candidate moments. Defaults to `15`. |
| `mode` | string | no | Search mode used for source retrieval. Defaults to `"default"`. |
| `include_sources` | boolean | no | Include supporting source shots in the response. Defaults to `false`. |

`ask()` does not accept index selectors such as `index_name`, `index_names`, `index_id`, or `index_ids`.

## Example

```python theme={null}
answer = collection.ask(
    question="Which brands are mentioned and what is said about them?",
    top_k=15,
    mode="default",
    include_sources=True,
)

print(answer.answer)

for source in answer.sources:
    print(source.video_id, source.start, source.end)
```

## Response

Returns an `AskResponse`.

| Attribute | Description |
| - | - |
| `answer` | Generated answer text. |
| `sources` | Supporting `Shot` objects when `include_sources=True`. |


# Search V2 & Ask
Source: https://docs.videodb.io/api-reference/search-v2/index

High-level search, DeepSearch, direct semantic search, structured query, aggregation, and Ask.

VideoDB Search V2 separates intelligent search from direct retrieval methods.

| Capability | Video endpoint | Collection endpoint | Python SDK |
| - | - | - | - |
| High-level search / DeepSearch | `POST /video/{video_id}/search/v2` | `POST /collection/{collection_id}/search/v2` | `video.search(...)`, `collection.search(...)` |
| Semantic search | `POST /video/{video_id}/semantic-search` | `POST /collection/{collection_id}/semantic-search` | `semantic_search(...)` |
| Structured query | `POST /video/{video_id}/query` | `POST /collection/{collection_id}/query` | `query(...)` |
| Aggregation | `POST /video/{video_id}/aggregate` | `POST /collection/{collection_id}/aggregate` | `aggregate(...)` |
| Ask | `POST /video/{video_id}/ask` | `POST /collection/{collection_id}/ask` | `ask(...)` |

> **[High-level Search & DeepSearch](/api-reference/search-v2/search)**
>
> Natural-language retrieval and DeepSearch sessions.

> **[Semantic Search](/api-reference/search-v2/semantic-search)**
>
> Direct vector search over semantic indexes.

> **[Query](/api-reference/search-v2/query)**
>
> Structured filters over one queryable index.

> **[Aggregate](/api-reference/search-v2/aggregate)**
>
> Counts, groups, facets, and metrics.

> **[Ask](/api-reference/search-v2/ask)**
>
> Answer questions using retrieved video context.


# Legacy Search
Source: https://docs.videodb.io/api-reference/search-v2/legacy-search

Use legacy spoken-word and scene-index search behavior during migration.

## Endpoints

```http theme={null}
POST /video/{video_id}/search/
POST /collection/{collection_id}/search/
```

These endpoints are the older search surface for spoken-word and legacy scene indexes.

In the Python SDK, prefer `legacy_search()` when using legacy parameters:

```python theme={null}
from videodb import SearchType, IndexType

results = video.legacy_search(
    query="product demo",
    search_type=SearchType.semantic,
    index_type=IndexType.scene,
    scene_index_id="idx_scene_123",
    score_threshold=0.3,
)
```

Legacy-shaped parameters include:

* `search_type`
* `index_type`
* `result_threshold`
* `dynamic_score_percentage`
* `scene_index_id`
* `algorithm`
* `sort_docs_on`
* `namespace`

> [!NOTE]
>
> Do not mix legacy parameters with Search V2 parameters. Use `search()` for new high-level Search V2, and `legacy_search()` for legacy spoken-word / scene-index behavior.

For new retrieval flows, use Search V2:

* `search()` / DeepSearch for high-level natural-language retrieval
* `semantic_search()` for direct semantic index search
* `query()` for structured filtering
* `aggregate()` for counts/facets
* `ask()` for generated answers


# Query
Source: https://docs.videodb.io/api-reference/search-v2/query

Run structured filters over one queryable index.

## Endpoints

```http theme={null}
POST /video/{video_id}/query
POST /collection/{collection_id}/query
```

Use `query()` when you know the exact index and filter fields.

## Request body

| Field | Type | Required | Description |
| - | -: | -: | - |
| `index_name` | string | no | Optional index name to query. |
| `index_id` | string | no | Optional index ID to query. |
| `filter` | list/object | no | Explicit filter spec. A top-level list is an implicit AND. |
| `limit` | integer | no | Maximum returned rows/moments. Defaults to `100`. |
| `sort` | list/string/object | no | Sort field and direction. |
| `return_fields` | string/list/dict | no | Select indexes to hydrate under `shot.metadata["indexes"]`. |

Do not pass `index_names` or `index_ids` to `query()`.

## Example

```python theme={null}
results = collection.query(
    index_name="objects",
    filter=[
        {"field": "label", "op": "contains", "value": "cell phone"},
    ],
    limit=100,
    sort=[("max_score", "desc")],
    return_fields=["objects"],
)
```

## Response

Returns a `SearchResult` containing `Shot` objects.


# High-level Search & DeepSearch
Source: https://docs.videodb.io/api-reference/search-v2/search

Run Search V2 natural-language retrieval on a video or collection.

## Endpoints

```http theme={null}
POST /video/{video_id}/search/v2
POST /collection/{collection_id}/search/v2
```

Use high-level `search()` when VideoDB should plan retrieval across available indexes. Use `mode="deepsearch"` for deeper multi-step retrieval with optional follow-up sessions.

## Request body

| Field | Type | Required | Description |
| - | -: | -: | - |
| `query` | string | yes | Natural-language search request. |
| `top_k` | integer | no | Maximum returned moments. |
| `mode` | string | no | `"default"` or `"deepsearch"`. |
| `filter` | list/object | no | Default-mode structured filter applied where supported by selected indexes. Not supported in DeepSearch. |
| `sort` | list/string/object | no | Default-mode sort applied where supported. Not supported in DeepSearch. |
| `score_threshold` | number | no | Default-mode semantic score threshold where applicable. Not supported in DeepSearch. |
| `return_fields` | string/list/dict | no | Select indexes to hydrate under `shot.metadata["indexes"]`. Use `"all"` to hydrate all. |
| `session_id` | string | no | DeepSearch follow-up session ID. Only valid with `mode="deepsearch"`. |
| `config.include_trace` | boolean | no | Include planner trace for default-mode search. |

> [!NOTE]
>
> High-level Search V2 does not accept `index_name`, `index_names`, `index_id`, or `index_ids`. Use semantic search, query, or aggregate for index-specific calls.

## Example: default search

```python theme={null}
response = collection.search(
    query="person talking about Nike while holding a phone",
    top_k=10,
    mode="default",
    return_fields=["scene", "transcript", "brands"],
)

for shot in response:
    print(shot.video_id, shot.start, shot.end)
```

## Example: DeepSearch

```python theme={null}
response = collection.search(
    query="find the strongest examples of customers reacting negatively to pricing",
    mode="deepsearch",
    top_k=10,
    return_fields="all",
)

followup = collection.search(
    query="only include competitor mentions",
    mode="deepsearch",
    session_id=response.session_id,
    top_k=10,
)
```

## Response

`search()` returns a `SearchResponse`.

| Attribute | Description |
| - | - |
| `response_type` | `"shots"`, `"aggregate"`, or `"deepsearch"`. |
| `results` | `SearchResult` for shot/deepsearch responses, or aggregate rows for aggregate responses. |
| `shots` | Convenience list of `Shot` objects for shot/deepsearch responses. |
| `session_id` | DeepSearch session ID, when applicable. |
| `waiting_for` | DeepSearch waiting state, when applicable. |
| `clarification` | DeepSearch clarification, when applicable. |
| `trace` | Planner trace when requested with `config={"include_trace": True}`. |


# Semantic Search
Source: https://docs.videodb.io/api-reference/search-v2/semantic-search

Run direct vector search over semantic indexes.

## Endpoints

```http theme={null}
POST /video/{video_id}/semantic-search
POST /collection/{collection_id}/semantic-search
```

Use semantic search when you know which semantic index or indexes to search directly.

## Request body

| Field | Type | Required | Description |
| - | -: | -: | - |
| `query` | string | yes | Text to search for. |
| `index_names` | string/list | no | Semantic index names to search. |
| `index_ids` | string/list | no | Semantic index IDs to search. |
| `top_k` | integer | no | Maximum returned moments. Defaults to `10`. |
| `score_threshold` | number | no | Minimum similarity score. |
| `filter` | list/object | no | Structured filter over supported fields/vector metadata. |
| `return_fields` | string/list/dict | no | Select indexes to hydrate under `shot.metadata["indexes"]`. |

If `index_names` and `index_ids` are omitted, VideoDB searches all semantic indexes in scope.

> [!NOTE]
>
> The Python SDK argument is `index_names`, not singular `index_name`. Use `index_ids` for ID selection.

## Example

```python theme={null}
results = video.semantic_search(
    query="happy reunion outside a shop",
    index_names=["scene"],
    top_k=10,
    score_threshold=0.7,
    return_fields=["scene"],
)

for shot in results:
    print(shot.start, shot.end, shot.search_score)
```

## Example with index IDs

```python theme={null}
results = collection.semantic_search(
    query="person presenting at a podium",
    index_ids=["idx_scene_123"],
    top_k=20,
)
```

## Response

Returns a `SearchResult` containing `Shot` objects. `SearchResult.compile()` can compile all returned shots into one stream.


# Search within collection
Source: https://docs.videodb.io/api-reference/search-within-collection

/openapi.yaml post /collection/{collection_id}/search/



# Search within video
Source: https://docs.videodb.io/api-reference/search-within-video

/openapi.yaml post /video/{video_id}/search/



# Search by title within collection
Source: https://docs.videodb.io/api-reference/search/search-by-title-within-collection

/openapi.yaml post /collection/{collection_id}/search/title/



# Web search within collection
Source: https://docs.videodb.io/api-reference/search/web-search-within-collection

/openapi.yaml post /collection/{collection_id}/search/web/



# Compile Search Results
Source: https://docs.videodb.io/api-reference/timeline/compile_timeline

POST /compile/
Compile video search results into a single playable stream

Compile search results or a list of videos/scenes into a single playable stream. Useful for creating compilations from search queries or combining specific video segments.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Search for videos
search_results = coll.search(query="action scenes")

# Compile search results into a stream
stream_url = search_results.compile()

print(f"Compiled stream: {stream_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Search for videos
const searchResults = await coll.search({ query: 'action scenes' });

// Compile search results into a stream
const streamUrl = await searchResults.compile();

console.log(`Compiled stream: ${streamUrl}`);
```

> [!NOTE]
>
> * Compile is useful for creating highlight reels or compilations from search results
> * Each timeline entry needs video ID and time range (start, end in seconds)
> * Returns a playable stream URL
> * Compile automatically stitches segments together in order

> **[Create Timeline v2](/api-reference/timeline/create_timeline_v2)**
>
> Advanced composition with multiple tracks

> **[Search Collections](/api-reference/search-v2/index)**
>
> Find videos to compile


# Create Timeline (v1)
Source: https://docs.videodb.io/api-reference/timeline/create_timeline

POST /timeline
Create a video composition using the legacy Timeline v1 API

Create a video timeline by adding video clips inline and overlaying audio, images, or text assets on top.

```python Python theme={null}
import videodb
from videodb.editor import Timeline, VideoAsset, AudioAsset

conn = videodb.connect(api_key="your_api_key")

# Create a timeline
timeline = Timeline(conn)

# Add videos inline (sequentially)
video_asset = VideoAsset(asset_id="m-abc123def")
timeline.add_inline(video_asset)

# Add audio overlay (starts at 5 seconds)
audio_asset = AudioAsset(asset_id="a-xyz789")
timeline.add_overlay(start=5, asset=audio_asset)

# Generate stream
stream_url = timeline.generate_stream()
print(f"Stream: {stream_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';
import { Timeline, VideoAsset, AudioAsset } from 'videodb';

const conn = await connect({ apiKey: 'your_api_key' });

// Create a timeline
const timeline = new Timeline(conn);

// Add videos inline (sequentially)
const videoAsset = new VideoAsset('m-abc123def');
timeline.addInline(videoAsset);

// Add audio overlay (starts at 5 seconds)
const audioAsset = new AudioAsset('a-xyz789');
timeline.addOverlay(5, audioAsset);

// Generate stream
const streamUrl = await timeline.generateStream();
console.log(`Stream: ${streamUrl}`);
```

> [!NOTE]
>
> * Timeline v1 uses simple inline and overlay operations
> * Video IDs start with `m-`, audio with `a-`, images with `img-`
> * For advanced editing with tracks and effects, use Timeline v2
> * `generate_stream()` returns a playable stream URL

> **[Create Timeline v2](/api-reference/timeline/create_timeline_v2)**
>
> Advanced timeline with tracks and effects

> **[Timeline Guides](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Learn timeline composition


# Create Timeline (v2)
Source: https://docs.videodb.io/api-reference/timeline/create_timeline_v2

POST /timeline_v2
Create advanced video compositions with multiple tracks, effects, and assets

Create advanced video compositions with multiple tracks, clips, transitions, and effects. Timeline v2 provides granular control over video editing including cropping, filters, and positioning.

```python Python theme={null}
import videodb
from videodb.editor import Timeline, Track, Clip, VideoAsset

conn = videodb.connect(api_key="your_api_key")

# Create timeline with custom resolution and background
timeline = Timeline(conn)
timeline.resolution = "1920x1080"
timeline.background = "#000000"

# Create a track and add video clips
track = Track()
video_asset = VideoAsset(id="m-abc123def", start=0)
clip = Clip(asset=video_asset, duration=10)
track.add_clip(0, clip)

# Add another clip at 10 seconds
video_asset2 = VideoAsset(id="m-xyz789", start=0)
clip2 = Clip(asset=video_asset2, duration=8)
track.add_clip(10, clip2)

timeline.add_track(track)

# Generate the stream
stream_url = timeline.generate_stream()
print(f"Stream: {stream_url}")
```

```javascript Node.js theme={null}
import { connect, EditorTimeline, Track, Clip, EditorVideoAsset } from 'videodb';

const conn = await connect({ apiKey: 'your_api_key' });

// Create timeline with custom resolution and background
const timeline = new EditorTimeline(conn);
timeline.resolution = '1920x1080';
timeline.background = '#000000';

// Create a track and add video clips
const track = new Track();
const videoAsset = new EditorVideoAsset({ id: 'm-abc123def', start: 0 });
const clip = new Clip({ asset: videoAsset, duration: 10 });
track.addClip(0, clip);

// Add another clip at 10 seconds
const videoAsset2 = new EditorVideoAsset({ id: 'm-xyz789', start: 0 });
const clip2 = new Clip({ asset: videoAsset2, duration: 8 });
track.addClip(10, clip2);

timeline.addTrack(track);

// Generate the stream
const streamUrl = await timeline.generateStream();
console.log(`Stream: ${streamUrl}`);
```

> [!NOTE]
>
> * Timeline v2 supports multiple tracks for layering videos, audio, images, and text
> * Clips define what asset plays and for how long (duration)
> * Default resolution is 1280x720, default background is black
> * Supports transitions, filters, cropping, and positioning on clips
> * Download the generated timeline with the Download endpoint

> **[Download Timeline](/api-reference/timeline/download_timeline)**
>
> Export timeline as video file

> **[Editor Guides](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Learn advanced video composition


# Download Timeline Video
Source: https://docs.videodb.io/api-reference/timeline/download_timeline

POST /timeline_v2/download
Download a Timeline composition as a video file

Download a generated timeline stream as a downloadable video file in MP4 format. After creating a timeline, use this endpoint to export it for sharing or offline use.

```python Python theme={null}
import videodb
from videodb.editor import Timeline, Track, Clip, VideoAsset

conn = videodb.connect(api_key="your_api_key")

# Create and generate a timeline (see Create Timeline v2)
timeline = Timeline(conn)
track = Track()
video_asset = VideoAsset(id="m-abc123def", start=0)
clip = Clip(asset=video_asset, duration=10)
track.add_clip(0, clip)
timeline.add_track(track)

# Generate the stream
stream_url = timeline.generate_stream()

# Download the timeline as a video file
download_result = timeline.download_stream(stream_url)

print(f"Download URL: {download_result.get('download_url')}")
print(f"Status: {download_result.get('status')}")
```

```javascript Node.js theme={null}
import { connect, EditorTimeline, Track, Clip, EditorVideoAsset } from 'videodb';

const conn = await connect({ apiKey: 'your_api_key' });

// Create and generate a timeline (see Create Timeline v2)
const timeline = new EditorTimeline(conn);
const track = new Track();
const videoAsset = new EditorVideoAsset({ id: 'm-abc123def', start: 0 });
const clip = new Clip({ asset: videoAsset, duration: 10 });
track.addClip(0, clip);
timeline.addTrack(track);

// Generate the stream
const streamUrl = await timeline.generateStream();

// Download the timeline as a video file
const downloadResult = await timeline.downloadStream(streamUrl);

console.log(`Download URL: ${downloadResult.downloadUrl}`);
console.log(`Status: ${downloadResult.status}`);
```

> [!NOTE]
>
> * Download requires a valid stream URL from a generated timeline
> * Downloads are processed asynchronously and return a download URL
> * Video format is MP4 with standard codec support
> * Downloaded files can be shared, archived, or used in other applications
> * Check the status field to track download progress

> **[Create Timeline v2](/api-reference/timeline/create_timeline_v2)**
>
> Generate timeline to download

> **[Connection API](/api-reference/authentication)**
>
> API authentication and connection


# Timeline & Compilation
Source: https://docs.videodb.io/api-reference/timeline/index

Create and compile video timelines

Create video timelines and compile them into streams or downloads.

## Endpoints

> **[Create Timeline](/api-reference/timeline/create_timeline)**
>
> Create timeline (v1)

> **[Create Timeline v2](/api-reference/timeline/create_timeline_v2)**
>
> Create timeline (v2)

> **[Download Timeline](/api-reference/timeline/download_timeline)**
>
> Export timeline as file

> **[Compile Timeline](/api-reference/timeline/compile_timeline)**
>
> Generate stream from timeline

> [!NOTE]
>
> See also the [Editor](/api-reference/editor/index) endpoints for visual editor timeline compilation.


# Get transcode job status
Source: https://docs.videodb.io/api-reference/transcode/get-transcode-job-status

/openapi.yaml get /transcode/{job_id}



# Get Transcode Status
Source: https://docs.videodb.io/api-reference/transcode/get_transcode_status

GET /transcode/{job_id}
Check the status of a transcode job

Check the status of an ongoing transcode job.

> [!NOTE]
>
> * Poll this endpoint to track transcode progress
> * The job ID is returned when starting a transcode job

> **[Start Transcode](/api-reference/transcode/start_transcode)**
>
> Start a new transcode job


# Transcode
Source: https://docs.videodb.io/api-reference/transcode/index

Transcode video files to different formats

Transcode video files to optimize them for different use cases and playback requirements.

## Endpoints

> **[Start Transcode](/api-reference/transcode/start_transcode)**
>
> Start a transcode job

> **[Get Status](/api-reference/transcode/get_transcode_status)**
>
> Check transcode job status


# Start transcode job
Source: https://docs.videodb.io/api-reference/transcode/start-transcode-job

/openapi.yaml post /transcode



# Start Transcode
Source: https://docs.videodb.io/api-reference/transcode/start_transcode

POST /transcode
Start a video transcode job

Start a transcode job for a video. Transcoding optimizes the video format for better compatibility and playback performance.

> [!NOTE]
>
> * The SDK takes a downloadable source URL and requires `callback_url` / `callbackUrl`; use `get_transcode_details` / `getTranscodeDetails` to track status
> * Transcoding is asynchronous; use the status endpoint to track progress
> * The transcoded video replaces the original format

> **[Get Transcode Status](/api-reference/transcode/get_transcode_status)**
>
> Check job progress

> **[Transcoding Guide](/pages/ingest/transcoding/when-to-transcode)**
>
> When and why to transcode


# Update auto recharge settings
Source: https://docs.videodb.io/api-reference/update-auto-recharge-settings

/openapi.yaml post /billing/auto_recharge



# Update collection
Source: https://docs.videodb.io/api-reference/update-collection

/openapi.yaml patch /collection/{collection_id}



# Upload media to collection
Source: https://docs.videodb.io/api-reference/upload-media-to-collection

/openapi.yaml post /collection/{collection_id}/upload



# Chat Completions
Source: https://docs.videodb.io/api-reference/utilities/create_chat

POST /chat/completions
Generate AI chat responses with video context awareness

Generate AI-powered chat responses with awareness of your video content. This endpoint enables interactive conversations about videos in your collections, combining video intelligence with conversational AI.

```python Python theme={null}
import requests
import json

payload = {
    "messages": [
        {
            "role": "user",
            "content": "Summarize the key moments in this video"
        }
    ],
    "collection_id": "default",
    "video_id": "m-xyz789"
}

response = requests.post(
    "https://api.videodb.io/chat/completions",
    json=payload,
    headers={"x-access-token": "your_api_key"}
)

chat_response = response.json()
print(f"Response: {chat_response.get('choices')[0].get('message').get('content')}")
```

```javascript Node.js theme={null}
const payload = {
  messages: [
    {
      role: "user",
      content: "Summarize the key moments in this video"
    }
  ],
  collectionId: "default",
  videoId: "m-xyz789"
};

const response = await fetch(
  "https://api.videodb.io/chat/completions",
  {
    method: "POST",
    headers: {
      "x-access-token": "your_api_key",
      "Content-Type": "application/json"
    },
    body: JSON.stringify(payload)
  }
);

const chatResponse = await response.json();
const content = chatResponse.choices[0].message.content;
console.log("Response:", content);
```

> [!NOTE]
>
> * Requires `collection_id` to specify which collection to search
> * Optional `video_id` to focus on a specific video
> * Supports multi-turn conversations by including message history in the `messages` array
> * Returns response in OpenAI-compatible format with `choices` array

> **[Health Check](/api-reference/utilities/health_check)**
>
> Verify API availability

> **[Async Response](/api-reference/utilities/get_async_response)**
>
> Poll async operation results


# Get Async Response
Source: https://docs.videodb.io/api-reference/utilities/get_async_response

GET /async-response/{response_id}
Retrieve results from asynchronous operations

Retrieves the results of long-running asynchronous operations such as transcoding, processing, or indexing. Use this endpoint to poll the status and final results of async requests that were initiated with a callback or that return a response ID.

```python Python theme={null}
import requests

response_id = "async-response-id-123"

response = requests.get(
    f"https://api.videodb.io/async-response/{response_id}",
    headers={"x-access-token": "your_api_key"}
)

result = response.json()
print(f"Status: {result.get('status')}")
print(f"Result: {result.get('result')}")
```

```javascript Node.js theme={null}
const responseId = "async-response-id-123";

const response = await fetch(
  `https://api.videodb.io/async-response/${responseId}`,
  {
    method: "GET",
    headers: {
      "x-access-token": "your_api_key"
    }
  }
);

const result = await response.json();
console.log("Status:", result.status);
console.log("Result:", result.result);
```

> [!NOTE]
>
> * `response_id` is returned by async operations and included in the URL path
> * Check the `status` field to determine if processing is complete
> * Results are available once status is "completed" or "done"
> * Typical status values: "pending", "processing", "completed", "failed"

> **[Health Check](/api-reference/utilities/health_check)**
>
> Verify API status

> **[Chat Completions](/api-reference/utilities/create_chat)**
>
> Generate AI chat responses


# API Health Check
Source: https://docs.videodb.io/api-reference/utilities/health_check

GET /
Check API status and service availability

Checks the status of the VideoDB API and verifies service availability. Returns a simple health status response to confirm the API is operational.

```python Python theme={null}
import requests

response = requests.get(
    "https://api.videodb.io/",
    headers={"x-access-token": "your_api_key"}
)

status = response.json()
print(f"API Status: {status}")
```

```javascript Node.js theme={null}
const response = await fetch("https://api.videodb.io/", {
  method: "GET",
  headers: {
    "x-access-token": "your_api_key"
  }
});

const status = await response.json();
console.log("API Status:", status);
```

> [!NOTE]
>
> * No request body required
> * Returns a simple status response confirming API availability
> * Useful for monitoring and connectivity verification
> * No authentication required (but include API key if available)

> **[Async Response](/api-reference/utilities/get_async_response)**
>
> Retrieve results from async operations

> **[Chat Completions](/api-reference/utilities/create_chat)**
>
> Generate AI chat responses with video context


# Utilities
Source: https://docs.videodb.io/api-reference/utilities/index

System utilities and async operations

System utilities, health checks, and async operation tracking.

## Endpoints

> **[Health Check](/api-reference/utilities/health_check)**
>
> Check API status

> **[Async Response](/api-reference/utilities/get_async_response)**
>
> Poll async operation status

> **[Chat](/api-reference/utilities/create_chat)**
>
> Interactive chat interface


# Create Video Clip
Source: https://docs.videodb.io/api-reference/videos/advanced/create_video_clip

POST /video/{video_id}/clip
Extract a clip from a video with optional transcoding

Generate video clips automatically from a text prompt using AI-powered content understanding.

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Generate clip using prompt
const result = await video.clip(
  'Show all the key moments',
  'highlights',
  'basic'
);

// Result contains clip information with stream URL
console.log(`Clip created: ${result.streamUrl}`);
```

> [!NOTE]
>
> * Uses AI to identify and extract relevant segments based on prompt
> * Content type guides the type of clips to extract (e.g., "highlights", "summaries")
> * Returns SearchResult object with clip stream URL
> * Model name allows selecting different AI models for clip generation
> * Clipped video maintains original quality and format


# Execute Video Workflow
Source: https://docs.videodb.io/api-reference/videos/advanced/create_video_workflow

POST /video/{video_id}/workflow/
Execute complex multi-step video processing workflows

Apply complex video processing workflows such as adding subtitles, overlays, or other transformations.

```python Python theme={null}
import videodb
from videodb import SubtitleStyle

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Add subtitles to video
subtitle_style = SubtitleStyle(
    background_color="#000000",
    text_color="#FFFFFF"
)
stream_url = video.add_subtitle(subtitle_style)

print(f"Video with subtitles: {stream_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Add subtitles to video
const streamUrl = await video.addSubtitle({
  backgroundColor: '#000000',
  textColor: '#FFFFFF'
});

console.log(`Video with subtitles: ${streamUrl}`);
```

> [!NOTE]
>
> * Supports multiple workflow types (subtitles currently implemented)
> * Workflows modify video stream without changing original
> * Subtitle style customization includes colors, positioning, and fonts
> * Returns new stream URL with applied workflow
> * Original video remains unchanged


# Get Reframe Status
Source: https://docs.videodb.io/api-reference/videos/advanced/get_reframe_status

GET /video/{video_id}/reframe/{reframe_id}
Check the status of a video reframe job

Check the status of an ongoing reframe job. Returns the current status and result data when complete.

> [!NOTE]
>
> * Status values: `processing`, `done`, `failed`
> * Poll this endpoint to track progress, or use `callback_url` when starting the reframe job
> * When status is `done`, the response includes the reframed video data

> **[Reframe Video](/api-reference/videos/advanced/reframe_video)**
>
> Start a new reframe job


# Advanced Operations
Source: https://docs.videodb.io/api-reference/videos/advanced/index

Complex video processing workflows

Advanced video processing operations for clipping and workflows.

## Endpoints

> **[Clip Video](/api-reference/videos/advanced/create_video_clip)**
>
> Create video clips

> **[Run Workflow](/api-reference/videos/advanced/create_video_workflow)**
>
> Execute processing workflow

> **[Reframe Video](/api-reference/videos/advanced/reframe_video)**
>
> Reframe to different aspect ratio

> **[Get Reframe Status](/api-reference/videos/advanced/get_reframe_status)**
>
> Check reframe job progress


# Reframe Video
Source: https://docs.videodb.io/api-reference/videos/advanced/reframe_video

POST /video/{video_id}/reframe
Reframe a video to a different aspect ratio

Reframe a video to a different aspect ratio (e.g., 16:9 to 9:16 for vertical content). The system intelligently tracks subjects to keep them in frame.

> [!NOTE]
>
> * Reframing is asynchronous; use the [Get Reframe Status endpoint](/api-reference/videos/advanced/get_reframe_status) to track progress
> * `target` accepts `"vertical"`, `"square"`, `"landscape"`, or a custom `{"width": number, "height": number}` object
> * `mode` accepts `"simple"` or `"smart"`
> * Use `callback_url` for webhook notification when reframing completes

> **[Get Reframe Status](/api-reference/videos/advanced/get_reframe_status)**
>
> Check reframe job progress

> **[Aspect Ratio Guide](/pages/act/programmable-editing/aspect-ratio-control)**
>
> Learn about aspect ratio control


# Delete scene collection
Source: https://docs.videodb.io/api-reference/videos/delete-scene-collection

/openapi.yaml delete /video/{video_id}/scenes/{scene_collection_id}/



# Delete video index
Source: https://docs.videodb.io/api-reference/videos/delete-video-index

/openapi.yaml post /video/{video_id}/index/delete



# Delete Video
Source: https://docs.videodb.io/api-reference/videos/delete_video

DELETE /video/{video_id}
Delete a video and its metadata from VideoDB

Permanently delete a video from VideoDB. This removes all associated metadata, indexes, and transcripts.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Get video
video = coll.get_videos()[0]

# Delete the video
video.delete()

print("Video deleted successfully")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Get video
const videos = await coll.getVideos();
const video = videos[0];

// Delete the video
await video.delete();

console.log('Video deleted successfully');
```

> [!WARNING]
>
> * This operation is irreversible
> * Deletes all video data including: video file, metadata, transcript, indexes, thumbnails, and generated streams
> * Associated scene collections and indexes are also removed
> * The video ID cannot be reused


# Delete Video Storage
Source: https://docs.videodb.io/api-reference/videos/delete_video_storage

DELETE /video/{video_id}/storage/
Permanently delete the video file from storage

Delete only the video file from storage while keeping the video metadata, indexes, and transcripts. This reduces storage costs when you no longer need the original file.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Get video
video = coll.get_videos()[0]

# Remove storage while keeping metadata
video.remove_storage()

print("Video storage deleted (metadata retained)")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Get video
const videos = await coll.getVideos();
const video = videos[0];

// Remove storage while keeping metadata
await video.removeStorage();

console.log('Video storage deleted (metadata retained)');
```

> [!WARNING]
>
> * This operation is irreversible and cannot restore the video file
> * After removal, you cannot generate new streams or thumbnails from the video
> * Existing generated streams and thumbnails remain available
> * Video metadata, indexes, and transcripts are preserved
> * Used for cost optimization when original file is no longer needed


# Get reframe job status
Source: https://docs.videodb.io/api-reference/videos/get-reframe-job-status

/openapi.yaml get /video/{video_id}/reframe/{reframe_id}



# Get scene collection details
Source: https://docs.videodb.io/api-reference/videos/get-scene-collection-details

/openapi.yaml get /video/{video_id}/scenes/{scene_collection_id}/



# Get Video Details
Source: https://docs.videodb.io/api-reference/videos/get_video

GET /video/{video_id}
Retrieve detailed metadata for a specific video

Retrieve complete metadata for a specific video including duration, transcript status, thumbnail, and stream URLs.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Get video from collection
video = coll.get_videos()[0]

print(f"Video ID: {video.id}")
print(f"Name: {video.name}")
print(f"Duration: {video.length} seconds")
print(f"Stream URL: {video.stream_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Get video from collection
const videos = await coll.getVideos();
const video = videos[0];

console.log(`Video ID: ${video.id}`);
console.log(`Name: ${video.name}`);
console.log(`Duration: ${video.length} seconds`);
console.log(`Stream URL: ${video.streamUrl}`);
```

> [!NOTE]
>
> * Video IDs start with the prefix `v-`
> * Each video maintains its own stream URL for playback
> * Video metadata includes length in seconds, name, description, and thumbnail URL
> * Videos belong to a collection and inherit collection-level access controls


# Videos
Source: https://docs.videodb.io/api-reference/videos/index

Core video operations and management

Core video management endpoints for listing, retrieving, and deleting videos.

## Endpoints

> **[List Videos](/api-reference/videos/list_videos)**
>
> Get all videos

> **[Get Video](/api-reference/videos/get_video)**
>
> Retrieve video details

> **[Delete Video](/api-reference/videos/delete_video)**
>
> Remove a video

> **[Delete Video Storage](/api-reference/videos/delete_video_storage)**
>
> Remove video from storage

> **[Update Video](/api-reference/videos/update_video)**
>
> Update video metadata


# Create Scene Index
Source: https://docs.videodb.io/api-reference/videos/indexing/create_scene_index

POST /video/{video_id}/index/scene/
Create a visual search index by analyzing video frames

Create an AI-powered index of scenes in the video with descriptions, enabling visual search and scene understanding.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Index scenes for visual search
scene_index_id = video.index_scenes()

print(f"Scene index created: {scene_index_id}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Index scenes for visual search
const sceneIndexId = await video.indexScenes();

console.log(`Scene index created: ${sceneIndexId}`);
```

> [!NOTE]
>
> * Analyzes video frames and creates AI descriptions for scenes
> * Supports shot-based and time-based extraction strategies
> * Can use custom prompts to guide scene description
> * Generates visual embeddings for semantic search
> * Returns scene index ID for later retrieval and searching
> * Extraction configuration controls frame sampling and scene detection


# Create Spoken Word Index
Source: https://docs.videodb.io/api-reference/videos/indexing/create_video_index

POST /video/{video_id}/index/
Create a semantic search index from video transcription

Create a semantic search index for spoken words in the video, enabling intelligent search across video transcripts.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Index spoken words for semantic search
video.index_spoken_words()

print("Indexing complete. Video is now searchable.")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Index spoken words for semantic search
await video.indexSpokenWords();

console.log('Indexing complete. Video is now searchable.');
```

> [!NOTE]
>
> * Automatically generates transcript if not already created
> * Creates semantic embeddings for all spoken words
> * Enables powerful search capabilities on video content
> * Indexing is asynchronous; progress can be tracked with callbacks
> * Index remains in place even if video storage is deleted
> * Use language\_code parameter for non-English videos (optional)


# Delete Scene Index
Source: https://docs.videodb.io/api-reference/videos/indexing/delete_scene_index

DELETE /video/{video_id}/index/scene/{scene_index_id}
Delete a scene index from a video

Remove a scene index from a video, freeing up resources used by that index.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Delete scene index
video.delete_scene_index("scene_index_id")

print("Scene index deleted")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Delete scene index
await video.deleteSceneIndex('scene_index_id');

console.log('Scene index deleted');
```

> [!WARNING]
>
> * Deleting a scene index is irreversible
> * All scene records and visual embeddings are removed
> * You can create a new index to replace it if needed
> * Existing scene collections are not affected by deleting the index


# Delete Video Index
Source: https://docs.videodb.io/api-reference/videos/indexing/delete_video_index

POST /video/{video_id}/index/delete
Delete an existing video index

Delete an existing index from a video. You can delete spoken word indexes, scene indexes, or all indexes at once.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

> [!NOTE]
>
> * `index_type` can be `spoken_word`, `scene`, or `all`
> * This operation is irreversible; you will need to re-index the video to search it again
> * Use `model_name` to target a specific model's index when deleting scene indexes

> **[Create Index](/api-reference/videos/indexing/create_video_index)**
>
> Index video content

> **[Get Index](/api-reference/videos/indexing/get_video_index)**
>
> Retrieve index details


# Get Scene Index Details
Source: https://docs.videodb.io/api-reference/videos/indexing/get_scene_index

GET /video/{video_id}/index/scene/{scene_index_id}
Retrieve details of a specific scene index

Retrieve all scene records from a specific scene index including descriptions and visual embeddings.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get scene index records
scene_records = video.get_scene_index("scene_index_id")

for record in scene_records:
    print(f"Scene: {record['scene_id']}")
    print(f"Description: {record['description']}")
    print(f"Time: {record['start']}s - {record['end']}s")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get scene index records
const sceneRecords = await video.getSceneIndex('scene_index_id');

for (const record of sceneRecords) {
  console.log(`Scene: ${record.sceneId}`);
  console.log(`Description: ${record.description}`);
  console.log(`Time: ${record.start}s - ${record.end}s`);
}
```

> [!NOTE]
>
> * Returns list of scene records with AI-generated descriptions
> * Each record includes start/end timestamps and visual embeddings
> * Scene IDs follow the pattern `sc-` prefix
> * Can be used to search for specific scenes by description
> * Useful for creating visual navigation interfaces


# Get Video Index
Source: https://docs.videodb.io/api-reference/videos/indexing/get_video_index

GET /video/{video_id}/index/
Retrieve the status of a video index

Retrieve metadata and status of a video index.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`video.get_index()`, `video.list_indexes()`, or `video.delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

> [!NOTE]
>
> * `index_type` is required and can be `spoken_word` or `scene`
> * Returns index metadata and status for the requested type
> * Requires index to be created first with create-index endpoint
> * `engine` is optional and defaults to `default`


# Indexing & Search
Source: https://docs.videodb.io/api-reference/videos/indexing/index

Index videos and search across content

These are supported legacy singular `/video/.../index` endpoints. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

## Endpoints

> **[Create Index](/api-reference/videos/indexing/create_video_index)**
>
> Index spoken words

> **[Get Index](/api-reference/videos/indexing/get_video_index)**
>
> Retrieve index details

> **[Create Scene Index](/api-reference/videos/indexing/create_scene_index)**
>
> Index visual content

> **[List Scene Indexes](/api-reference/videos/indexing/list_scene_indexes)**
>
> Get all scene indexes

> **[Get Scene Index](/api-reference/videos/indexing/get_scene_index)**
>
> Retrieve scene index

> **[Delete Scene Index](/api-reference/videos/indexing/delete_scene_index)**
>
> Remove scene index

> **[Search](/api-reference/videos/indexing/search_video_index)**
>
> Search indexed content

> **[Delete Index](/api-reference/videos/indexing/delete_video_index)**
>
> Delete a video index


# List Scene Indexes
Source: https://docs.videodb.io/api-reference/videos/indexing/list_scene_indexes

GET /video/{video_id}/index/scene/
Retrieve all scene indexes for visual search

Retrieve all scene indexes created for a video.

> [!NOTE]
>
> This is a supported legacy singular `/video/.../index` endpoint. For current work, use [Understand](/pages/understand/indexing-pipelines/understanding-artifacts) → [`video.index()`](/pages/understand/indexing-pipelines/create-an-index), then [`get_index()`, `list_indexes()`, or `delete_index()`](/pages/understand/indexing-pipelines/view-indexes).

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# List all scene indexes
indexes = video.list_scene_index()

for index in indexes:
    print(f"Index ID: {index['scene_index_id']}")
    print(f"Name: {index.get('name', 'Unnamed')}")
    print(f"Status: {index.get('status')}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// List all scene indexes
const indexes = await video.listSceneIndex();

for (const index of indexes) {
  console.log(`Index ID: ${index.sceneIndexId}`);
  console.log(`Name: ${index.name || 'Unnamed'}`);
  console.log(`Status: ${index.status}`);
}
```

> [!NOTE]
>
> * Returns all scene indexes associated with a video
> * Each index contains metadata including ID, name, and creation status
> * Scene indexes allow different indexing strategies for the same video
> * Use returned IDs to retrieve specific index details or delete indexes


# Search Video
Source: https://docs.videodb.io/api-reference/videos/indexing/search_video_index

POST /video/{video_id}/search/
Search video content by spoken words or visual scenes

Search for specific content within a video using semantic search on transcripts or visual indexes.

> [!NOTE]
>
> This is a supported legacy `POST /video/{video_id}/search/` endpoint. For current retrieval, use [Search V2 direct methods](/pages/understand/search-and-retrieval/natural-language-query): `semantic_search()`, `query()`, or `aggregate()`.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Search a legacy index
results = video.legacy_search("customer reviews discussion")
shots = results.get_shots()

print(f"Found {len(shots)} matches")
for shot in shots:
    print(f"Time: {shot.start:.2f}s - {shot.end:.2f}s")
    print(f"Text: {shot.text}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Search a legacy index
const results = await video.legacySearch('customer reviews discussion');
const shots = results.getShots();

console.log(`Found ${shots.length} matches`);
for (const shot of shots) {
  console.log(`Time: ${shot.start.toFixed(2)}s - ${shot.end.toFixed(2)}s`);
  console.log(`Text: ${shot.text}`);
}
```

> [!NOTE]
>
> * Requires index to be created first (create-index endpoint)
> * Returns matched segments with timestamps and relevance scores
> * Supports semantic search on transcribed text
> * Can filter results by score threshold and result count
> * Search results include exact timestamp ranges for video navigation


# List Videos
Source: https://docs.videodb.io/api-reference/videos/list_videos

GET /video/
Retrieve a paginated list of all videos in a collection

Retrieve all videos from a collection with full metadata for each video including duration, thumbnails, and stream URLs.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()

# Get all videos
videos = coll.get_videos()

for video in videos:
    print(f"Video: {video.name} ({video.id})")
    print(f"Duration: {video.length}s")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();

// Get all videos
const videos = await coll.getVideos();

for (const video of videos) {
  console.log(`Video: ${video.name} (${video.id})`);
  console.log(`Duration: ${video.length}s`);
}
```

> [!NOTE]
>
> * Returns all videos within a specific collection
> * Each video object contains complete metadata (ID, name, duration, stream URL, thumbnail)
> * Videos are organized at the collection level
> * Video list includes only videos belonging to the authenticated user's collections


# Reframe video to different aspect ratio
Source: https://docs.videodb.io/api-reference/videos/reframe-video-to-different-aspect-ratio

/openapi.yaml post /video/{video_id}/reframe



# Delete Scene Collection
Source: https://docs.videodb.io/api-reference/videos/scenes/delete_scene_collection

DELETE /video/{video_id}/scenes/{scene_collection_id}/
Delete a scene collection from a video

Delete a specific scene collection from a video. This removes all scene data within the collection.

> [!NOTE]
>
> * This operation is irreversible
> * Scene collection IDs start with `sc-`
> * Re-extract scenes to recreate the collection

> **[Get Scene Collection](/api-reference/videos/scenes/get_scene_collection)**
>
> Retrieve scene collection details

> **[Extract Scenes](/api-reference/videos/scenes/extract_video_scenes)**
>
> Extract new scenes from video


# Describe Video Frame
Source: https://docs.videodb.io/api-reference/videos/scenes/describe_video_frame

POST /video/{video_id}/frame/{frame_id}/describe/
Generate AI description of a specific video frame

Generate an AI-powered description of a specific video frame's visual content.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get scene collection
scene_collection = video.get_scene_collection("scene_collection_id")

# Get a frame from a scene
scene = scene_collection.scenes[0]
frame = scene.frames[0]

# Describe the frame
description = frame.describe()

print(f"Frame description: {description}")
print(f"Frame time: {frame.frame_time}s")

# Describe with custom prompt
custom_desc = frame.describe(prompt="List all objects visible")

# Describe with custom model configuration
description = frame.describe(
    prompt="Describe this frame in detail",
    model_name="pro",
    model_config={"temperature": 0.5}
)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get scene collection
const sceneCollection = await video.getSceneCollection('scene_collection_id');

// Get a frame from a scene
const scene = sceneCollection.scenes[0];
const frame = scene.frames[0];

// Describe the frame
const description = await frame.describe();

console.log(`Frame description: ${description}`);
console.log(`Frame time: ${frame.frameTime}s`);

// Describe with custom prompt
const customDesc = await frame.describe('List all objects visible');

// Describe with custom model configuration
const detailed = await frame.describe(
    'Describe this frame in detail',
    'pro',
    { temperature: 0.5 }
);
```

> [!NOTE]
>
> * Generates detailed visual description using AI vision models
> * Can use custom prompts to focus description on specific aspects
> * Pass `model_name` separately from `model_config`
> * Overwrites existing frame description
> * Frame IDs start with `fr-` prefix
> * Each frame includes timestamp (frame\_time) in the video
> * Useful for generating captions or accessibility descriptions


# Describe Video Scene
Source: https://docs.videodb.io/api-reference/videos/scenes/describe_video_scene

POST /video/{video_id}/scene/{scene_id}/describe/
Generate AI description of a video scene

Generate an AI-powered description of a video scene's visual content.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get scene collection
scene_collection = video.get_scene_collection("scene_collection_id")

# Describe a specific scene
scene = scene_collection.scenes[0]
description = scene.describe()

print(f"Scene description: {description}")

# Describe with custom prompt
custom_desc = scene.describe(prompt="Focus on people and their actions")

# Describe with custom model configuration
description = scene.describe(
    prompt="Describe this scene in detail",
    model_name="pro",
    model_config={"temperature": 0.5}
)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get scene collection
const sceneCollection = await video.getSceneCollection('scene_collection_id');

// Describe a specific scene
const scene = sceneCollection.scenes[0];
const description = await scene.describe();

console.log(`Scene description: ${description}`);

// Describe with custom prompt
const customDesc = await scene.describe('Focus on people and their actions');

// Describe with custom model configuration
const detailed = await scene.describe(
    'Describe this scene in detail',
    'pro',
    { temperature: 0.5 }
);
```

> [!NOTE]
>
> * Generates description using AI vision models
> * Can use custom prompts to guide description focus
> * Pass `model_name` separately; use the optional `model_config` dict for settings such as `temperature` (0.0-1.0)
> * Overwrites existing scene description
> * Requires scene to be part of extracted scene collection
> * Scene IDs start with `sc-` prefix


# Extract Video Scenes
Source: https://docs.videodb.io/api-reference/videos/scenes/extract_video_scenes

POST /video/{video_id}/scenes/
Extract and analyze scenes from video content

Extract scenes from a video using shot detection or time-based segmentation, with automatic frame sampling and descriptions.

```python Python theme={null}
import videodb
from videodb import SceneExtractionType

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Extract scenes using shot-based detection (default)
scene_collection = video.extract_scenes()

for scene in scene_collection.scenes:
    print(f"Scene: {scene.id}")
    print(f"Time: {scene.start}s - {scene.end}s")
    print(f"Description: {scene.description}")

# Extract with time-based segmentation (every 10 seconds)
config = {"time": 10, "frame_count": 2}
scenes = video.extract_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config=config
)
```

```javascript Node.js theme={null}
import { connect, SceneExtractionType } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Extract scenes using shot-based detection (default)
const sceneCollection = await video.extractScenes();

for (const scene of sceneCollection.scenes) {
  console.log(`Scene: ${scene.id}`);
  console.log(`Time: ${scene.start}s - ${scene.end}s`);
}

// Extract with time-based segmentation (every 10 seconds)
const scenes = await video.extractScenes({
  extractionType: SceneExtractionType.timeBased,
  extractionConfig: { time: 10, frame_count: 2 }
});
```

> [!NOTE]
>
> * Two extraction types: shot-based (detects camera cuts) and time-based (fixed intervals)
> * Shot-based default threshold is 20 (lower = more sensitive to changes)
> * Time-based allows specifying interval (in seconds) and frames per segment
> * Returns SceneCollection with Scene objects including frames and metadata
> * Extracted frames can be used for visual search and thumbnail generation


# Get Scene Collection
Source: https://docs.videodb.io/api-reference/videos/scenes/get_scene_collection

GET /video/{video_id}/scenes/{scene_collection_id}/
Retrieve details of a specific scene collection

Retrieve the details and scenes within a specific scene collection, including scene timestamps and descriptions.

> [!NOTE]
>
> * Use `offset` and `limit` parameters for pagination
> * Scene collection IDs start with `sc-`
> * Each scene includes start/end timestamps and optional descriptions

> **[Update Scene Collection](/api-reference/videos/scenes/update_scene_collection)**
>
> Modify scene data

> **[Extract Scenes](/api-reference/videos/scenes/extract_video_scenes)**
>
> Extract scenes from video


# Get Video Scenes
Source: https://docs.videodb.io/api-reference/videos/scenes/get_video_scenes

GET /video/{video_id}/scenes/
Retrieve detected scenes from video analysis

Retrieve a previously extracted scene collection by its ID.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get a specific scene collection
scene_collection = video.get_scene_collection("scene_collection_id")

print(f"Scenes in collection: {len(scene_collection.scenes)}")
for scene in scene_collection.scenes:
    print(f"Scene {scene.id}: {scene.start}s - {scene.end}s")
    for frame in scene.frames:
        print(f"  Frame {frame.id} at {frame.frame_time}s")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get a specific scene collection
const sceneCollection = await video.getSceneCollection('scene_collection_id');

console.log(`Scenes: ${sceneCollection.scenes.length}`);
for (const scene of sceneCollection.scenes) {
  console.log(`Scene ${scene.id}: ${scene.start}s - ${scene.end}s`);
  for (const frame of scene.frames) {
    console.log(`  Frame ${frame.id} at ${frame.frameTime}s`);
  }
}
```

> [!NOTE]
>
> * Scene collection IDs follow the pattern `sc-` prefix
> * Returns SceneCollection containing multiple Scene objects
> * Each scene includes all extracted frames with URLs
> * Frames are ordered and can be displayed as galleries
> * Use scene IDs or frame IDs to get specific elements


# Scene & Frame Analysis
Source: https://docs.videodb.io/api-reference/videos/scenes/index

Extract and analyze video scenes and frames

Extract scenes and analyze frames using computer vision.

## Endpoints

> **[Extract Scenes](/api-reference/videos/scenes/extract_video_scenes)**
>
> Extract scenes from video

> **[Get Scenes](/api-reference/videos/scenes/get_video_scenes)**
>
> Retrieve scene data

> **[Describe Scene](/api-reference/videos/scenes/describe_video_scene)**
>
> Generate scene description

> **[Describe Frame](/api-reference/videos/scenes/describe_video_frame)**
>
> Analyze specific frame

> **[Get Scene Collection](/api-reference/videos/scenes/get_scene_collection)**
>
> Retrieve scene collection

> **[Update Scene Collection](/api-reference/videos/scenes/update_scene_collection)**
>
> Modify scene collection

> **[Delete Scene Collection](/api-reference/videos/scenes/delete_scene_collection)**
>
> Remove scene collection


# Update Scene Collection
Source: https://docs.videodb.io/api-reference/videos/scenes/update_scene_collection

PATCH /video/{video_id}/scenes/{scene_collection_id}/
Update scenes within a scene collection

Update the scenes within an existing scene collection. Use this to modify scene boundaries or descriptions.

> [!NOTE]
>
> * The `scenes` array is required and replaces the existing scenes in the collection
> * Each scene object should include the scene boundaries and optional description

> **[Get Scene Collection](/api-reference/videos/scenes/get_scene_collection)**
>
> Retrieve scene collection details

> **[Delete Scene Collection](/api-reference/videos/scenes/delete_scene_collection)**
>
> Remove a scene collection


# Create Custom Thumbnail
Source: https://docs.videodb.io/api-reference/videos/streaming/create_video_thumbnail

POST /video/{video_id}/thumbnail/
Generate a custom thumbnail from a specific video frame

Generate a custom thumbnail image from a specific timestamp in the video.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Generate thumbnail from specific timestamp (in seconds)
thumbnail = video.generate_thumbnail(time=5.0)

# thumbnail is an Image object
print(f"Thumbnail URL: {thumbnail.url}")
print(f"Thumbnail ID: {thumbnail.id}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Generate thumbnail from specific timestamp (in seconds)
const thumbnail = await video.generateThumbnail(5.0);

// thumbnail is an Image object
console.log(`Thumbnail URL: ${thumbnail.url}`);
console.log(`Thumbnail ID: ${thumbnail.id}`);
```

> [!NOTE]
>
> * Time parameter is in seconds, can be a float (e.g., 2.5 for 2.5 seconds into video)
> * Returns an Image object with URL and metadata
> * Generated thumbnails are stored and can be retrieved later
> * Time value should be within video duration (0 to video.length)
> * Multiple thumbnails can be generated at different timestamps


# Generate Video Stream
Source: https://docs.videodb.io/api-reference/videos/streaming/generate_video_stream

POST /video/{video_id}/stream/
Generate a streamable URL for video playback with optional timeline customization

Generate a streamable URL for the entire video or create a custom stream from specific timeline segments.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get full stream URL (returns existing if available)
stream_url = video.generate_stream()

# Create stream from custom timeline (in seconds)
# Format: [(start, end), (start, end), ...]
timeline = [(0, 30), (60, 90)]
custom_stream_url = video.generate_stream(timeline=timeline)

print(f"Stream URL: {stream_url}")
print(f"Custom stream: {custom_stream_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get full stream URL (returns existing if available)
const streamUrl = await video.generateStream();

// Create stream from custom timeline (in seconds)
// Format: [[start, end], [start, end], ...]
const timeline = [[0, 30], [60, 90]];
const customStreamUrl = await video.generateStream(timeline);

console.log(`Stream URL: ${streamUrl}`);
console.log(`Custom stream: ${customStreamUrl}`);
```

> [!NOTE]
>
> * Returns a pre-existing stream URL if available (no re-processing needed)
> * Timeline allows creating videos with multiple segments automatically stitched
> * Timeline format: list of (start\_time, end\_time) tuples in seconds
> * Stream URLs are HLS format and work with most video players
> * Stream URLs remain valid as long as the video is stored


# Get Video Thumbnail
Source: https://docs.videodb.io/api-reference/videos/streaming/get_video_thumbnail

GET /video/{video_id}/thumbnail/
Retrieve the default thumbnail image for a video

Retrieve the default thumbnail URL for a video. This is automatically generated when the video is uploaded.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get default thumbnail URL (no time parameter)
thumbnail_url = video.generate_thumbnail()

print(f"Thumbnail URL: {thumbnail_url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get default thumbnail URL (no time parameter)
const thumbnailUrl = await video.generateThumbnail();

console.log(`Thumbnail URL: ${thumbnailUrl}`);
```

> [!NOTE]
>
> * Returns thumbnail as a URL string (not an Image object)
> * Thumbnail is auto-generated from the video at upload time
> * Thumbnail URL can be used directly in img tags for web display
> * If stored in video.thumbnail\_url, this method returns cached version
> * For custom thumbnails at specific timestamps, use create-thumbnail endpoint


# Streaming & Playback
Source: https://docs.videodb.io/api-reference/videos/streaming/index

Generate streams and manage thumbnails

Generate streaming URLs and manage video thumbnails.

## Endpoints

> **[Create Thumbnail](/api-reference/videos/streaming/create_video_thumbnail)**
>
> Generate a thumbnail

> **[List Thumbnails](/api-reference/videos/streaming/list_video_thumbnails)**
>
> Get all thumbnails

> **[Get Thumbnail](/api-reference/videos/streaming/get_video_thumbnail)**
>
> Retrieve thumbnail details

> **[Generate Stream](/api-reference/videos/streaming/generate_video_stream)**
>
> Create streaming URL


# List Video Thumbnails
Source: https://docs.videodb.io/api-reference/videos/streaming/list_video_thumbnails

GET /video/{video_id}/thumbnails/
Retrieve all custom thumbnails created for a video

Retrieve all custom thumbnails that have been generated for a video at different timestamps.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get all thumbnails
thumbnails = video.get_thumbnails()

for thumbnail in thumbnails:
    print(f"Thumbnail ID: {thumbnail.id}")
    print(f"URL: {thumbnail.url}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get all thumbnails
const thumbnails = await video.getThumbnails();

for (const thumbnail of thumbnails) {
  console.log(`Thumbnail ID: ${thumbnail.id}`);
  console.log(`URL: ${thumbnail.url}`);
}
```

> [!NOTE]
>
> * Returns list of Image objects for each generated custom thumbnail
> * Only includes custom thumbnails created with create-thumbnail endpoint
> * Each Image object contains ID, URL, and metadata
> * Default auto-generated thumbnail is not included in this list
> * Use to manage multiple thumbnails created at different video timestamps


# Create Video Transcription
Source: https://docs.videodb.io/api-reference/videos/transcription/create_video_transcription

POST /video/{video_id}/transcription/
Generate a transcription for a video with speech recognition and speaker diarization

Generate a complete transcription of the video's audio content using automatic speech recognition.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Generate transcript
result = video.generate_transcript()

if isinstance(result, dict) and result.get('success'):
    print("Transcript generated successfully")
else:
    print("Transcript generation started or already exists")

# Generate transcript with language hint
video.generate_transcript(language_code="en")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Generate transcript
const result = await video.generateTranscript();

console.log(result);

// Generate transcript with language hint
await video.generateTranscript(false, 'en');
```

> [!NOTE]
>
> * Generates timestamped transcript with word-level timing information
> * Includes automatic speaker diarization when available
> * Returns success confirmation; retrieve actual transcript with get-transcription
> * Processing time depends on video duration
> * Transcript is cached after first generation (use force: true to regenerate)
> * Optional `language_code` parameter hints the transcription engine about the spoken language for better accuracy


# Get Video Transcription
Source: https://docs.videodb.io/api-reference/videos/transcription/get_video_transcription

GET /video/{video_id}/transcription/
Retrieve the transcription for a video

Retrieve the timestamped transcription for a video. Supports multiple segmentation types and time ranges.

```python Python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
video = coll.get_videos()[0]

# Get full transcript with word-level timestamps
transcript = video.get_transcript()

for segment in transcript:
    print(f"{segment['start']:.2f}s - {segment['text']}")

# Get plain text transcript
text = video.get_transcript_text()
print(f"Full text: {text}")

# Segment by sentence instead of word
transcript = video.get_transcript(segmenter="sentence")

# Group every 10 words
transcript = video.get_transcript(segmenter="word", length=10)

# Fixed-duration audio segments (e.g., 5-second chunks)
transcript = video.get_transcript(segmenter="time", length=5)

# Get transcript for a specific time range
transcript = video.get_transcript(start=10, end=60)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect({ apiKey: 'your_api_key' });
const coll = await conn.getCollection();
const videos = await coll.getVideos();
const video = videos[0];

// Get full transcript with timestamps
const transcript = await video.getTranscript();

for (const word of transcript.wordTimestamps) {
  console.log(`${word.start.toFixed(2)}s - ${word.word}`);
}

// Get plain text transcript
const text = await video.getTranscriptText();
console.log(`Full text: ${text}`);

// Segment by sentence instead of word
const sentenceTranscript = await video.getTranscript(0, video.length, 'sentence');

// Group every 10 words
const wordGroups = await video.getTranscript(0, video.length, 'word', 10);

// Fixed-duration audio segments
const timeTranscript = await video.getTranscript(0, video.length, 'time', 5);

// Get transcript for a specific time range
const rangeTranscript = await video.getTranscript(10, 60);
```

### Parameters

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `segmenter` | str | `"word"` | How to split the transcript: `"word"`, `"sentence"`, or `"time"` |
| `length` | int | `1` | Seconds when `segmenter` is `time` |
| `start` | float | — | Start time in seconds (must be >= 0) |
| `end` | float | — | End time in seconds (must be >= start) |
| `force` | bool | `false` | Force re-fetch from server, bypassing cache |

> [!NOTE]
>
> * Python returns a list of segment dicts; Node.js returns a `Transcript` object with segments in `wordTimestamps`
> * The raw API response wraps this as `data.word_timestamps` plus `data.text`
> * Transcription segmentation is audio/text-based and is independent of visual scene segmentation
> * Invalid segmenter values raise a `ValueError`
> * Negative `start`/`end` or `start > end` raise a `ValueError`
> * Generates transcript automatically if not already created

### Raw response shape

When completed, the server returns transcript text and timestamped entries:

```json theme={null}
{
  "success": true,
  "status": "completed",
  "data": {
    "text": "I need to leave now.",
    "word_timestamps": [
      {"start": 0.4, "end": 0.8, "text": "I"},
      {"start": 0.8, "end": 1.2, "text": "need"},
      {"start": 1.2, "end": 1.6, "text": "to"},
      {"start": 1.6, "end": 2.1, "text": "leave"},
      {"start": 2.1, "end": 2.8, "text": "now."}
    ]
  }
}
```

Silence ranges may be included with `text: "-"`.


# Transcription
Source: https://docs.videodb.io/api-reference/videos/transcription/index

Generate and retrieve video transcripts

Create and retrieve video transcriptions.

## Endpoints

> **[Create Transcription](/api-reference/videos/transcription/create_video_transcription)**
>
> Generate transcript

> **[Get Transcription](/api-reference/videos/transcription/get_video_transcription)**
>
> Retrieve transcript


# Update scene collection
Source: https://docs.videodb.io/api-reference/videos/update-scene-collection

/openapi.yaml patch /video/{video_id}/scenes/{scene_collection_id}/



# Update video
Source: https://docs.videodb.io/api-reference/videos/update-video

/openapi.yaml patch /video/{video_id}



# Update Video
Source: https://docs.videodb.io/api-reference/videos/update_video

PATCH /video/{video_id}
Update video metadata such as name and description

Update metadata for a specific video, including its name and description.

> [!NOTE]
>
> * Video IDs start with the prefix `m-`
> * Only provided fields are updated; omitted fields remain unchanged


# Bloom
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/bloom

Open-source, local-first screen recorder with AI processing - record, upload to VideoDB, and query with natural language

[Bloom - Local-First Screen Recorder](https://www.youtube.com/embed/RjIYRNRTlx4)

> **[Bloom on GitHub](https://github.com/video-db/bloom)**
>
> Complete source code, installation guide, and configuration

## What Is It?

Your local-first screen recorder, built for AI workflows.

Bloom records on your machine (no lock-in), automatically uploads to VideoDB for transcription and indexing, then lets you search any moment with natural language. It's Loom meets agent-ready infrastructure.

> [!TIP]
>
> **The Power**: Your recordings become queryable data. Search "when did I mention the deadline?" or "show me the deployment steps" — get timestamped clips with context.

## When to Use Bloom

Bloom shines in scenarios where you need recordings to become searchable data:

* **Sales demos**: Record product demos once, search forever. "Show me when we explained the pricing model to enterprise clients last month."
* **Bug reproduction**: Capture exact steps that trigger issues, then let teammates search "when did the error first appear?"
* **Tutorial learning**: Following along with video courses? Record your implementation so AI assistants can reference both the tutorial and your work.
* **Async team updates**: Record your screen while explaining complex work. Teammates can search the recording instead of watching the whole thing.
* **Personal knowledge base**: Capture important calls and workshops. Search months later when you need to remember "what did we decide about the API design?"

## Why You Need This

**Local-First**

### No Lock-In

Your files stay on your machine:

* Record locally, no forced cloud upload
* Pay only for AI processing (when you need it)
* Export and own your recordings
* No subscription, no storage limits

**AI-Ready**

### Make Recordings Searchable

Transform video into queryable data:

* Automatic transcription of spoken words
* Visual embeddings for scene understanding
* Metadata extraction from content
* Natural language search across all recordings

**Agent Integration**

### Built for Agents

Access recordings from AI workflows:

* Query via VideoDB API
* Compatible with Claude Code and agent frameworks
* Structured data output for automation
* Shareable links for team access

## How It Works

**1. Record Locally**

Capture screen, microphone, system audio, and camera with a floating control bar. Multi-monitor support lets you choose which display to record.

**2. Upload to VideoDB (Optional but Powerful)**

As you record, Bloom can sync chunks to VideoDB Cloud in real-time. Why? Processing video on your local machine would drain battery and CPU. By streaming to VideoDB, the heavy lifting—transcription, visual indexing, embedding generation—happens server-side while you keep working. You keep the local file, VideoDB keeps the intelligence.

**3. Process with AI**

VideoDB indexes spoken words, generates transcripts, and creates subtitled streams. Visual embeddings make every frame searchable.

**4. Query Anywhere**

Search your recordings with natural language. "Show me when I explained the architecture" returns timestamped clips with full context.

**5. Share or Integrate**

One-click shareable links, VideoDB Chat for questions, or API access for agent frameworks. Download MP4 files anytime.

## Key Features

> **Recording & Capture**
>
> **Complete Desktop Capture**
>
> * Screen recording with multi-monitor support
> * Microphone and system audio capture
> * Draggable camera bubble overlay
> * Display picker to choose which screen
> * Global keyboard shortcut: `Cmd+Shift+R`
> * Always-on-top floating control bar

> **Library Management**
>
> **Organize Your Recordings**
>
> * Browse all recordings with search
> * Sort by date (newest/oldest)
> * Rename recordings in-place
> * Delete with confirmation
> * Download recordings as MP4
> * Keyboard navigation (arrow keys, delete key)

> **AI Processing**
>
> **Automatic Intelligence**
>
> * Transcription of spoken words via VideoDB
> * Subtitle generation for video playback
> * HLS streaming with in-app player
> * Visual indexing for scene search
> * Metadata extraction from content

> **Sharing & Integration**
>
> **Connect with Workflows**
>
> * One-click shareable links for any recording
> * VideoDB Chat to query recordings with AI
> * Download MP4 files for offline use
> * API access for agent frameworks
> * Integration with Claude Code and other tools

> **Developer Experience**
>
> **Open Source & Customizable**
>
> * Full UI source code available (MIT license)
> * System tray integration with status indicator
> * Light/dark theme support
> * Permission management for mic, screen, camera
> * Local SQLite database for metadata

## Tech Stack

Built for performance and reliability:

> **Electron 39.7**
>
> Desktop application shell

> **Node.js 18+**
>
> JavaScript runtime

> **VideoDB SDK 0.2.4**
>
> Recording and AI processing

> **SQLite (sql.js)**
>
> Local metadata storage

> **HLS.js**
>
> In-app video playback

> **Universal Binary**
>
> Apple Silicon + Intel support

## Getting Started

> [!NOTE]
>
> **Prerequisites**
>
> * macOS 12+ or Windows 10+ (Linux support coming soon)
> * [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) (free tier available)

**1. Install Bloom**

**Automated installation** (recommended):

```bash theme={null}
curl -fsSL https://artifacts.videodb.io/bloom/install | bash
```

**Manual installation:**

* **Apple Silicon (M1/M2/M3/M4)**: [bloom-2.2.0-arm64.dmg](https://artifacts.videodb.io/bloom/bloom-2.2.0-arm64.dmg)
* **Apple Intel**: [bloom-2.2.0-x64.dmg](https://artifacts.videodb.io/bloom/bloom-2.2.0-x64.dmg)
* **Windows**: [bloom-2.2.0-x64.exe](https://artifacts.videodb.io/bloom/bloom-2.2.0-x64.exe)

For manual install: mount the DMG and drag Bloom to Applications, then run:

```bash theme={null}
xattr -cr /Applications/Bloom.app
```

**2. Launch and Register**

1. Launch Bloom from Applications or Spotlight
2. Enter your VideoDB API key ([get one free](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs))
3. Grant system permissions when prompted

**3. Grant Permissions**

Bloom needs access to:

* **Microphone** - For voice recording
* **Screen Recording** - For screen capture
* **Camera** (optional) - For camera bubble overlay

Grant these in **System Settings > Privacy & Security**.

**4. Start Recording**

1. Press `Cmd+Shift+R` or click the Bloom icon in system tray
2. Choose your display from the picker
3. Toggle camera, mic, and system audio as needed
4. Click **Start Recording**

> [!WARNING]
>
> **macOS Permissions Required**
>
> Grant in **System Settings > Privacy & Security**:
>
> * Microphone (for voice recording)
> * Screen Recording (for screen capture)
> * Camera (for camera overlay, optional)

## Configuration

All features are configurable through the Bloom interface:

| Feature | Customizable |
| - | - |
| Recording sources | Enable/disable mic, system audio, camera |
| Display selection | Choose which screen to record |
| Camera position | Drag camera bubble anywhere on screen |
| Keyboard shortcut | Default: `Cmd+Shift+R` |
| Theme | Light or dark mode |
| Storage location | Local database at `~/Library/Application Support/bloom/` |

## Privacy & Data

> [!TIP]
>
> **Local Recording** - Files saved on your machine first, cloud upload is optional

> [!TIP]
>
> **No Lock-In** - Download recordings anytime, export as MP4

> [!TIP]
>
> **Encrypted Storage** - API keys encrypted, credentials protected

> [!TIP]
>
> **User Control** - Delete recordings locally and from cloud independently

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/bloom)**
>
> Detailed installation instructions, troubleshooting guide, and development setup

## Related Tutorials

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> AI coding assistant with screen and audio context

> **[Focusd](/examples-and-tutorials/ai-copilots/focusd)**
>
> AI-powered productivity tracking with automatic insights

> **[Call.md](/examples-and-tutorials/ai-copilots/call-md)**
>
> Real-time AI meeting assistant with live intelligence


# Call.md
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/call-md

Turn meetings into live agent loops - real-time AI meeting assistant with dual-channel transcription and intelligent automation

[Call.md - AI Meeting Assistant](https://www.youtube.com/embed/n24NPxYofRc)

> **[Call.md on GitHub](https://github.com/video-db/call.md)**
>
> Complete source code, installation guide, and configuration

## What Is It?

Your AI meeting assistant, live during every call.

Most meeting tools transcribe after the fact. Call.md is different: it understands your meeting as it happens, giving you AI-powered suggestions, conversation metrics, and automatic tool triggers in real-time. It's like having an AI copilot who watches your meeting, tracks engagement, and whispers helpful context right when you need it.

After the call ends, it generates three-part summaries (overview, key points, action items) and can automatically update your CRM or project management tools through workflow webhooks.

If you've used Otter.ai or Fireflies.ai, think of Call.md as the same idea but built for agent automation—not just passive transcription, but active intelligence during and after meetings.

> [!TIP]
>
> **The Power**: During calls, AI generates contextual suggestions, monitors conversation balance, and automatically triggers your MCP tools when information needs arise — all in real-time.

## Why You Need This

**Live Intelligence**

### During the Meeting

Get real-time assistance while you talk:

* AI-generated suggestions (things to say, questions to ask)
* Conversation metrics (talk ratio, pace, questions asked)
* MCP tools triggered automatically from conversation context
* Coaching nudges when conversation needs steering

**Meeting Intelligence**

### Post-Meeting Analysis

Comprehensive summaries after every call:

* Three-part AI summary (overview, key points, action items)
* Full transcript with dual-channel separation
* Conversation metrics and engagement statistics
* Markdown export with complete meeting intelligence

**Agent Integration**

### MCP Automation

Connect your agent ecosystem:

* Auto-triggers MCP tools during conversations
* Results appear inline during meetings
* Workflow webhooks to n8n, Zapier, CRMs
* Agent-ready structured output

## How It Works

**1. Dual-Channel Transcription**

Captures your mic (labeled "you") and system audio (labeled "them") separately. This separation powers all downstream intelligence.

> [!TIP]
>
> **Why Dual-Channel Matters**: Separating "you" (microphone) from "them" (system audio) isn't just for attribution. It enables Call.md to track conversation balance, detect when one person is dominating, measure your speaking pace independently, and generate insights like "You asked 3 questions but the client asked 0—they might not be engaged." Single-channel transcription can't do this.

**1. Live Assist**

Every 20 seconds, AI analyzes recent transcript and generates contextual suggestions: things to say, questions to ask.

**2. Conversation Metrics**

Real-time tracking of talk ratio, speaking pace (WPM), questions asked, and monologue detection. No LLM required — pure statistics.

**3. MCP Auto-Triggering**

Intent detector scans conversation for information needs (active or passive). When detected, automatically calls relevant MCP tools and displays results inline.

**4. Post-Meeting Intelligence**

When the meeting ends, generates three parallel summaries: narrative overview, key points by topic, and action items. Sends to workflow automation platforms.

## Key Features

> **Recording & Transcription**
>
> **Real-Time Speech-to-Text**
>
> * Separate channels for you and them
> * Live transcription powered by VideoDB
> * Recording history with full transcripts
> * Screen, mic, and system audio capture

> **Live Assist**
>
> **AI-Generated Suggestions**
>
> * Contextual things to say
> * Questions to ask
> * Updates every 20 seconds
> * Based on recent conversation context

> **Conversation Metrics**
>
> **Track Engagement**
>
> * Talk ratio (you vs them)
> * Speaking pace and question count
> * Engagement score
> * Monologue detection (45s threshold)

> **MCP Integration**
>
> **Model Context Protocol**
>
> * Auto-triggers tools from conversation
> * Runs every 20 seconds
> * Max 3 tool calls per run
> * Results display inline during meetings
> * Supports stdio and HTTP servers

> **MCP Auto-Triggering Example**
>
> **How it works in practice:**
>
> During a sales call, the client mentions "Can you send me pricing for the enterprise plan?" The MCP intent detector recognizes this as an information need, automatically calls your CRM tool to fetch the pricing doc, and displays it inline. You see the result immediately and can reference it without breaking flow.
>
> Or: A customer says "I'm seeing error code 502." The MCP agent searches your knowledge base tool, finds relevant docs, and shows them to you in real-time—before you even finish taking notes.
>
> This happens automatically every 20 seconds based on conversation context. You configure which MCP tools are available; the agent decides when to call them.

> **Post-Meeting Summaries**
>
> **Three-Part AI Analysis**
>
> * Short overview (3-5 sentence narrative)
> * Key points by topic (attributed to participants)
> * Action items (3-10 concrete next steps)
> * Generated in parallel for speed

> **Workflow Webhooks**
>
> **Automation Integration**
>
> * Auto-send to n8n, Zapier, CRMs
> * Triggered when meeting ends
> * Structured data payload
> * Agent-ready output format

> **Meeting Preparation**
>
> **Setup Wizard**
>
> * AI-generated probing questions
> * Dynamic discussion checklist
> * Google Calendar integration
> * Sync upcoming meetings

> **Bookmarking**
>
> **Mark Important Moments**
>
> * Quick bookmark during calls
> * Review later with context
> * Share with team

## Tech Stack

Built for performance and reliability:

> **Electron 34**
>
> Desktop application shell

> **React 19 + TypeScript 5.8**
>
> Modern UI with full type safety

> **tRPC 11**
>
> Type-safe API layer

> **Drizzle + SQLite**
>
> Local offline-first storage

> **VideoDB SDK (0.2.4)**
>
> Recording and transcription

> **MCP SDK (1.0.0)**
>
> Model Context Protocol integration

> **OpenAI SDK (6.19.0)**
>
> LLM calls via VideoDB API

> **Tailwind + shadcn/ui**
>
> Beautiful, modern interface

## Getting Started

> [!NOTE]
>
> **Prerequisites**
>
> * macOS 12+ (Monterey or later) or Windows 10+
> * [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) (free tier available)

**1. Install Call.md**

**macOS** (Apple Silicon & Intel):

```bash theme={null}
curl -fsSL https://artifacts.videodb.io/call.md/install | bash
```

*Currently available for macOS and Windows — Linux support coming soon*

**2. Launch and Register**

1. Launch Call.md from Applications or Spotlight
2. Enter your VideoDB API key ([get one free](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs))
3. Grant system permissions when prompted

**3. Configure Recording**

Configure preferences:

* Enable microphone capture
* Enable system audio capture
* Select screen to record
* Optionally connect Google Calendar

**4. Start Your First Meeting**

1. Click "New Meeting" from home screen
2. Optionally run Meeting Setup wizard
3. Click "Start Recording"
4. Watch live transcription and intelligence

> [!WARNING]
>
> **macOS Permissions Required**
>
> Grant in **System Preferences > Privacy & Security**:
>
> * Microphone (for voice recording)
> * Screen Recording (for screen capture)

## Configuration

All features are configurable through Settings:

| Feature | Customizable |
| - | - |
| Live Assist | Enable/disable, configure timing |
| Conversation Metrics | Set thresholds for talk ratio alerts |
| MCP Servers | Add stdio/HTTP servers, manage connections |
| Workflow Webhooks | Configure n8n, Zapier, or custom endpoints |
| Google Calendar | Connect/disconnect calendar sync |

### MCP Server Setup

Connect MCP servers in **Settings → MCP Servers**:

1. Click **Add Server**
2. Choose transport: **stdio** (local) or **http** (remote)
3. Configure connection details
4. Click **Connect**

The MCP agent runs automatically during meetings, detects information needs from conversation, and triggers relevant tools. Results appear inline in the **MCP Results** panel.

## Privacy & Data

> [!TIP]
>
> **Local Database** - All data stored in SQLite at `~/Library/Application Support/call-md/`

> [!TIP]
>
> **Secure Storage** - API keys encrypted, credentials protected

> [!TIP]
>
> **User Control** - Delete recordings anytime, export transcripts

> [!TIP]
>
> **Transcription via VideoDB** - AI features require internet connectivity

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/call.md)**
>
> Detailed installation instructions, configuration options, and troubleshooting guide

## Related Tutorials

> **[Bloom](/examples-and-tutorials/ai-copilots/bloom)**
>
> Screen recorder with AI search for async video documentation

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> AI coding assistant with real-time screen and audio context

> **[Focusd](/examples-and-tutorials/ai-copilots/focusd)**
>
> AI-powered productivity tracking with automatic time insights


# Chess Lens
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/chess-lens

Real-time chess coaching from a clearly visible board, with instant engine tips and AI explanations in a floating desktop overlay

> **[Chess Lens on GitHub](https://github.com/video-db/chess-lens)**
>
> Complete source code, installers, and troubleshooting

## What Is It?

Chess Lens is a real-time chess coach that works directly from your screen.

It watches a visible chess board, reconstructs the position in FEN notation, and shows the best move and evaluation in a floating overlay. A plain-language coaching explanation and drill question follow in the background, so you get a fast engine tip first and deeper guidance moments later.

Because Chess Lens reads the board from screen capture, it can work with chess.com, lichess, ChessBase, or another chess client that displays a clear board.

> [!TIP]
>
> **The Experience**: Keep playing in your preferred chess app while a draggable, always-on-top coach updates as the board position changes.

## What You Get

**Instant Tips**

### See the Engine Move First

Chess Lens shows the best move and evaluation as soon as the chess engine responds, without waiting for the longer coaching explanation.

**AI Coaching**

### Understand the Position

A plain-language explanation and drill question arrive in the background and update the overlay when ready.

**Game History**

### Revisit Past Sessions

Chess Lens keeps a local history of sessions. When analysis is available, you can review the recording from VideoDB, coaching tips, key moments, win probability, and the post-game summary.

## How It Works

**1. Capture the Board**

Chess Lens starts a VideoDB capture session and watches the chess board shown on your screen.

**2. Recognize the Position**

A vision model converts the visible board into FEN notation, a square-by-square representation used by chess software. Both white and black perspectives are supported.

**3. Calculate the Best Move**

The chess engine evaluates the position and returns a best move and score for the overlay.

**4. Add Coaching Context**

An AI-generated explanation and drill question are prepared in the background, then added to the existing engine tip.

> [!NOTE]
>
> Analysis refreshes when a new board position is confirmed. If the board is hidden or cannot be read for several frames, the overlay asks you to bring it back into view.

## Key Features

> **Works Across Chess Apps**
>
> Chess Lens reads the board from the screen instead of depending on a specific chess website or client integration.

> **Automatic Board Detection**
>
> Detects a clearly visible board from either the white or black perspective and updates when the position changes.

> **Floating Coaching Overlay**
>
> A transparent, draggable, always-on-top window shows the board, current tip, coaching explanation, drill question, chat, and no-board state.

> **Session Review**
>
> Browse past games and review available recordings, move history, win probability, key moments, coaching tips, and AI-generated post-session recaps.

> **Local-First History**
>
> Session history and analysis metadata are stored in a local SQLite database. Screen recordings are captured through VideoDB, and model requests are processed outside the device.

## Getting Started

> [!NOTE]
>
> **Prerequisites**
>
> * macOS 12+ or 64-bit Windows 10+
> * A [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
> * Screen Recording permission on macOS
> * Microphone permission only if you enable audio recording or transcription

**macOS**

```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/video-db/chess-lens/main/scripts/install.sh | bash
```

Start Chess Lens from Applications or Spotlight after installation.

**Windows**

Run in PowerShell:

```powershell theme={null}
irm https://raw.githubusercontent.com/video-db/chess-lens/main/scripts/install.ps1 | iex
```

Start Chess Lens from the Start Menu after installation.

**From Source**

Requires Node.js 18+:

```bash theme={null}
git clone https://github.com/video-db/chess-lens.git
cd chess-lens
npm install
npm run dev
```

**1. Add Your Details**

On first launch, enter your name and VideoDB API key. You can get a key from [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs).

**2. Grant Screen Access**

On macOS, allow Screen Recording in **System Settings > Privacy & Security** so Chess Lens can see the board.

**3. Open a Chess Board**

Start a game in your preferred chess website or desktop client. Keep the complete board visible and large enough to read.

**4. Follow the Overlay**

Chess Lens detects position changes and updates the floating coach with the latest engine tip and explanation.

## If the Board Is Not Detected

* Make sure the full board is visible and not covered by dialogs or other windows.
* Bring the chess tab or app back into focus.
* Increase the board size if the pieces and squares are difficult to read.
* On macOS, confirm that Screen Recording permission is enabled.
* Allow a few seconds for the first tip when a new position appears.

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/chess-lens)**
>
> Installation, development commands, and detailed troubleshooting

## Related Tutorials

> **[Focusd](/examples-and-tutorials/ai-copilots/focusd)**
>
> Turn desktop activity into useful summaries and work insights

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> Give coding agents real-time screen and audio context

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Learn how to build screen-aware desktop applications


# Focusd - Productivity and Personal Work Memory
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/focusd

AI-powered desktop app that records your screen, understands what you're doing, and builds your personal work memory

<video title="Focusd - Productivity and Personal Work Memory" />

> **[Focusd on GitHub](https://github.com/video-db/focusd)**
>
> Complete source code, installation guide, and demo download

## What Is It?

Your personal productivity oracle.

You know that sinking feeling at 5 PM when you wonder "where did my day go?" Or when you bill clients and can't remember what you worked on Tuesday morning? Or when you want to optimize your workflow but have no data about where time actually goes?

Focusd solves this by running invisibly all day, watching your screen and understanding context—not just "you used Chrome," but "what you were reading in Chrome and which project it relates to." No manual timers. No app-switching. No logging. Just automatic, intelligent tracking that tells you where your time went, what you accomplished, and how to improve tomorrow.

> [!TIP]
>
> **The Insight**: "You spent 2.3 hours on the authentication refactor, switching between VSCode (67%) and documentation (33%). You were blocked for 23 minutes waiting for builds. Consider optimizing your build pipeline."

> **Platform Support**: macOS and Windows (Linux support coming soon)

## Why It's Different

**Zero Manual Work**

### Automatic Everything

Forget manual time tracking. Focusd:

* Records continuously in background
* Understands context from your screen
* Detects when you switch projects
* Identifies productive vs idle time
* Generates summaries automatically

**AI Understanding**

### Smart Analysis

Not just "you used Chrome for 3 hours."

Focusd knows:

* *What* you were reading in Chrome
* *Which* project you were working on
* *Why* you switched contexts
* *How* productive each session was

**Actionable Insights**

### Real Improvements

Daily recaps include:

* Session summaries with context
* App usage breakdown by project
* Productivity patterns identified
* Specific suggestions to improve

## The 5-Layer Pipeline

Focusd transforms raw screen captures into structured insights through intelligent summarization:

**1. L0: Raw Events**

Screen indexed every \~3 seconds. Extracts app names, page titles, visible content.

**2. L1: Activity Segments**

Events grouped into time-based chunks (e.g., 5-minute windows).

**3. L2: Micro-Summaries**

Each segment summarized by LLM: *what you did, which app, productive or not*.

**4. L3: Session Summaries**

Micro-summaries roll up into session overviews with app stats and project breakdown.

**5. L4: Daily Summary**

Everything consolidates into a headline, highlights, and actionable suggestions.

> [!NOTE]
>
> **Why 5 Layers?**
>
> This hierarchical approach balances cost, accuracy, and latency. Layer 0 captures everything (cheap, fast). Layers 1-2 group and summarize in small batches (moderate cost). Layers 3-4 synthesize larger contexts (higher cost but fewer calls).
>
> The result: near-instant activity tracking at the bottom, actionable insights at the top, and a reasonable API bill. Without this structure, running LLM calls on every screen capture would be prohibitively expensive.

## Key Features

> **Live Activity Timeline**
>
> Real-time feed of what you're doing, updated every few seconds. See your work unfold as it happens.

> **🤖 AI Session Summaries**
>
> Automatic overviews of each work session:
>
> * What you worked on
> * Which apps you used
> * Projects and time breakdown
> * Productivity assessment

> **🔍 Drill Down Analysis**
>
> Select any time range for detailed breakdown:
>
> * App usage percentages
> * Project time distribution
> * Context from screen captures

> **📈 Dashboard Analytics**
>
> Visual insights:
>
> * Total tracked time
> * Productive time percentage
> * Top applications used
> * Project distribution

> **Daily Recap**
>
> End-of-day report with:
>
> * Headline summary
> * Session highlights
> * Productivity insights
> * Tomorrow's suggestions

> **🕐 History Browser**
>
> Browse past days with full summaries and activity data. Understand your weekly and monthly patterns.

## Who It's For

> **Freelancers**
>
> Track client projects automatically. Generate accurate time reports without manual logging.

> **Developers**
>
> Understand where coding time goes. Identify interruptions, context switches, and optimize your flow.

> **Knowledge Workers**
>
> See how much time you spend in meetings vs deep work. Find patterns in your most productive hours.

> **Remote Workers**
>
> Stay accountable without micromanagement. Get insights into your work patterns and share summaries with your team.

## Tech Stack

Built with modern desktop technologies for performance and reliability:

> **Electron + React**
>
> Desktop app shell with React UI

> **TypeScript**
>
> Type-safe throughout

> **SQLite**
>
> Local data storage for privacy

> **VideoDB SDK**
>
> Screen capture and AI indexing

> **OpenAI**
>
> LLM for summarization pipeline

> **Recharts**
>
> Dashboard visualizations

## Getting Started

> [!NOTE]
>
> **Prerequisites**
>
> * macOS 12+ (Apple Silicon or Intel) or Windows 10+
> * Node.js 18+
> * [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)

**1. Clone and Install**

```bash theme={null}
git clone https://github.com/video-db/focusd.git
cd focusd
npm install
```

**2. Configure API Key**

Copy `.env.sample` to `.env` and add your VideoDB API key.

**3. Run Development Mode**

Run `npm run dev` to start the app in development mode.

**4. Grant Permissions**

macOS will prompt for Screen Recording permission. Grant it in System Settings.

**5. Start Tracking**

Hit the record button. Wait a few minutes, then check your live timeline and dashboard!

> [!WARNING]
>
> **System Permissions**
>
> Required in **System Settings > Privacy & Security**:
>
> * Screen Recording (mandatory)
> * System Audio Recording (optional, for audio context)

## Configuration

All settings adjustable from the Settings page:

| Setting | Default | Purpose |
| - | - | - |
| `segment_flush_mins` | 5 min | How often events group into segments |
| `micro_summary_mins` | 10 min | Frequency of segment summarization |
| `session_summary_mins` | 30 min | Session-level summary generation |
| `idle_threshold_mins` | 5 min | Inactivity threshold before pausing |

Tweak these to balance between insight granularity and performance.

## Privacy First

> [!TIP]
>
> **Local Storage** - All data stored in SQLite on your machine (`~/Library/Application Support/VideoDB Focusd/`)

> [!TIP]
>
> **Encrypted Keys** - API keys stored in macOS Keychain via Electron's safeStorage

> [!TIP]
>
> **No Cloud Storage** - Summaries generated, then stored locally. Screen captures processed and discarded.

> [!TIP]
>
> **Full Control** - Delete all data anytime: `rm -rf ~/Library/Application\ Support/VideoDB\ Focusd/`

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/focusd)**
>
> Detailed installation instructions, configuration guide, and troubleshooting help

## Related Tutorials

> **[Bloom](/examples-and-tutorials/ai-copilots/bloom)**
>
> Screen recorder for creating searchable video documentation

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> Transform Claude Code into a context-aware AI coding assistant

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Learn how to build screen capture apps with VideoDB


# Overview
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/index

Plug video understanding into your agents — meetings, recordings, tutorials, cameras, and desktop perception

## The Context Transfer Problem

You are surrounded by videos — meetings, recordings, tutorials, channels, cameras, and even your desktop. The real bottleneck isn't processing power or model capability. It's **context transfer**.

Your agents are blind to the visual world you navigate every day. They can't see your Zoom calls, understand your Loom recordings, or follow along as you watch a tutorial. VideoDB changes that.

## Built for Agents

VideoDB is built from the ground up for agents. Your AI assistants can now acquire context from:

* **Meetings** — Understand discussions, decisions, and action items
* **Recordings** — Search and reference Loom-style videos, demos, and walkthroughs
* **Desktop perception** — See what you see in real-time
* **Tutorials** — Watch and learn alongside you
* **Cameras** — Monitor physical environments and events

Real-time understanding of your environment changes how you act in the real world.

## What's Possible

Imagine **Claude Code** with complete understanding of your meetings and Loom recordings. Start a sharing session and watch a tutorial together — your agent follows along, ready to help implement what you're learning.

Imagine **OpenClaw-style agents** recording demos of any app, capturing their autonomous workflows for review and debugging.

Imagine **research agents** navigating the internet, producing informative video summaries that cut through the noise — saving you from misinformation and clickbait.

## Install VideoDB Skills

Add video capabilities to your agent in seconds:

```bash theme={null}
npx skills add video-db/skills
```

Then run the setup command in your agent:

```
/videodb setup
```

The setup guides you through obtaining a VideoDB API key (includes \$20 free credits, no credit card required), installing the SDK, and verifying the connection.

> **[View Skills Repository](https://github.com/video-db/skills)**
>
> Complete installation guide and skill documentation

## Example Workflows

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> AI coding assistant with real-time screen and audio context

> **[OpenClaw Monitoring](/examples-and-tutorials/ai-copilots/openclaw-monitoring)**
>
> CCTV for AI agents — monitor, record, and search autonomous agent activity

> **[Call.md](/examples-and-tutorials/ai-copilots/call-md)**
>
> Real-time AI meeting assistant with dual-channel transcription

> **[Bloom](/examples-and-tutorials/ai-copilots/bloom)**
>
> Local-first screen recorder with AI processing and natural language search

> **[Focusd](/examples-and-tutorials/ai-copilots/focusd)**
>
> Productivity tracking with automatic session summaries

> **[Record & Replay](/examples-and-tutorials/ai-copilots/open-record-replay)**
>
> Turn a demonstrated desktop workflow into a reusable agent skill

> **[Chess Lens](/examples-and-tutorials/ai-copilots/chess-lens)**
>
> Real-time chess coaching from a clearly visible board on your screen

## Architecture

![Capture SDK Architecture showing two-runtime design with backend API key and desktop client token](https://mintcdn.com/videodb/DVB1BMGyP7Y3j1JP/assets/capture-sdk/capture-basic-architecture.jpeg?fit=max\&auto=format\&n=DVB1BMGyP7Y3j1JP\&q=85\&s=269c8a1ce4919ca48fe2f9d1b6ee6f45)

API keys never touch the desktop. Tokens are short-lived. Enterprise-ready security.

## Related Documentation

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Full architecture and capabilities

> **[Real-Time Context](/pages/ingest/capture-sdks/realtime-context)**
>
> How real-time AI context works

> **[Privacy Controls](/pages/ingest/capture-sdks/privacy-controls)**
>
> User consent and data protection

> **[Storage & Search](/pages/ingest/capture-sdks/storage-and-search)**
>
> Persist and search captured content

> **[MCP Server](/pages/mcp/index)**
>
> Connect VideoDB to Claude and other agents

> **[Director Framework](/pages/build-with-agents/director/index)**
>
> Build custom AI agents with VideoDB


# Record & Replay
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/open-record-replay

Record a desktop workflow once and compile it into a reusable skill that an AI agent can follow later

<video title="VideoDB Record & Replay" />

> **[Record & Replay on GitHub](https://github.com/video-db/open-record-replay)**
>
> Complete source code, installation guide, and troubleshooting

## What Is It?

Record & Replay turns a workflow you perform on your desktop into instructions an AI agent can reuse. It supports macOS and Windows.

You demonstrate the task once while the app records the desktop interactions. It then combines the action log with optional visual context from VideoDB and compiles the result into a structured, human-readable agent skill.

The result is not a fixed macro. The generated skill describes the task, starting state, inputs, actions, and checks an agent needs to repeat it. Values such as search terms, dates, choices, and file paths can become inputs instead of being copied as fixed values.

> [!TIP]
>
> **The Idea**: Show the workflow once, let Record & Replay turn the demonstration into a reusable skill, then give that skill to your agent for future runs.

## Why Use Record & Replay?

**Teach by Showing**

### Demonstrate the Real Workflow

Perform the task in the app you already use. Record & Replay captures desktop accessibility events and can include screen video as visual reference.

**Create Reusable Skills**

### Move Beyond One-Off Recordings

The compiler turns the demonstration into `SKILL.md` and `SKILL.json` files with steps, expected inputs, verification guidance, and an execution strategy.

**Flexible Capture**

### Keep Recording Without Video

Native accessibility hooks capture workflows on macOS and Windows. If full screen capture is unavailable, Record & Replay can still compile from accessibility events alone.

## How It Works

**1. Start a Recording**

Ask your agent to start recording a named workflow. A short lead-in can give you time to switch from the agent to the app where the task will happen.

**2. Demonstrate the Task**

Complete the workflow yourself. Native accessibility hooks capture the actions and interface metadata. When screen capture is available, VideoDB adds visual context.

**3. Compile the Skill**

The compiler combines the event log and available visual context, identifies reusable inputs, and generates structured and human-readable skill files.

**4. Use It with an Agent**

By default, the generated `SKILL.md` is installed in `~/.codex/skills/`. You can point the installer at another global skills directory for agents that consume the same skill format.

> [!NOTE]
>
> **Human-in-the-loop recording**: You perform the demonstration yourself. Record & Replay creates the skill, but it does not include a standalone replay engine—the agent that consumes the skill performs the workflow later.

## What the Skill Contains

> **Workflow Steps**
>
> A readable sequence of actions that explains how to complete the demonstrated workflow.

> **Reusable Inputs**
>
> Values such as search queries, dates, dropdown choices, and file paths can be represented as inputs rather than copied as fixed values.

> **Verification Guidance**
>
> The generated skill includes checks and guidance that help the consuming agent confirm whether the workflow succeeded.

> **Execution Guidance**
>
> Each structured skill records a preferred execution strategy, such as browser, desktop app, terminal, file system, or a hybrid workflow.

## Getting Started

> [!NOTE]
>
> **Prerequisites**
>
> * Python 3.10+
> * [uv](https://docs.astral.sh/uv/) package manager
> * A [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)

**1. Clone and Install**

```bash theme={null}
git clone https://github.com/video-db/open-record-replay.git
cd open-record-replay
uv sync
```

**2. Add Your API Key**

Create a `.env` file in the project root:

```env theme={null}
VIDEODB_API_KEY=your_VIDEODB_API_KEY
```

**3. Configure Your MCP Client**

Add the server to your MCP client configuration:

```json theme={null}
{
  "mcpServers": {
    "videodb-record-replay": {
      "command": "uv",
      "args": ["run", "python", "server.py"],
      "cwd": "/path/to/open-record-replay"
    }
  }
}
```

**4. Restart the Client**

Restart your MCP client so the Record & Replay tools and skill resources become available.

## Record Your First Workflow

Recording is guided by your agent:

1. Ask the agent to record a named workflow.
2. Switch to the target app and perform the task.
3. Return to the MCP client and ask the agent to stop recording.
4. Ask it to compile the recording into a skill.
5. Confirm the global `SKILL.md` path reported by the agent.

> [!TIP]
>
> Use a short lead-in before the workflow begins and trim the end of the recording if you need time to switch back to your MCP client. This keeps navigation to and from the client out of the generated instructions.

## Platform Notes

> **macOS**
>
> Screen Recording, Microphone, Accessibility, and Input Monitoring permissions may be required. Run the permission helper before recording:
>
> ```bash theme={null}
> uv run python scripts/smoke_macos_hook.py --prompt-permissions
> ```

> **Windows**
>
> Record & Replay uses Windows UI Automation. No additional setup is required beyond the standard installation.

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/open-record-replay)**
>
> Tool reference, skill output details, platform setup, and troubleshooting

## Related Tutorials

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> Give coding agents real-time screen and audio context

> **[OpenClaw Monitoring](/examples-and-tutorials/ai-copilots/openclaw-monitoring)**
>
> Record, inspect, and search autonomous agent activity

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Learn how VideoDB desktop capture works


# OpenClaw Monitoring
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/openclaw-monitoring

CCTV for AI agents — record every session, watch runs live, and search agent activity with natural language

[OpenClaw Monitoring - CCTV for AI Agents](https://www.youtube.com/embed/dnGWaFeo2FA)

> **[OpenClaw Monitoring on GitHub](https://github.com/video-db/openclaw-monitoring)**
>
> Complete source code, installation guide, and configuration examples

## What Is It?

Your AI agent just did something on a remote server. Do you know what?

Right now, most people running agents are doing this: Send task → Wait → Get "Success" in Slack → Hope for the best. That's not monitoring. That's faith.

And faith breaks down fast. Your agent could have:

* Gotten stuck on a CAPTCHA and retried 47 times
* Navigated to the wrong site and filled forms with test data
* Completed the task correctly but missed a validation step
* Encountered an error you'll never see because logs don't capture visual context

**VideoDB Monitoring** turns your OpenClaw agent into an observable, auditable worker. Every run becomes a live stream (watch in real-time), a replayable recording (shareable URL, not a dead video file), and a searchable archive (find "when did it open the spreadsheet?"). Think of it as a dashcam for your AI agent, or CCTV for computer-use agents.

> [!NOTE]
>
> **Platform Support**: OpenClaw Monitoring is platform-agnostic—it runs on any system that supports Node.js 18+ and TypeScript. Works on macOS, Windows, and Linux.

## Why Teams Need It

**Full Visibility**

### See Everything

* **Live stream** — watch your agent work in real-time
* **Replayable recordings** — shareable URL, not a dead video file
* **Full session capture** — screen and audio recorded continuously
* Catch issues before they become incidents

**Searchable History**

### Find Any Moment

* **Searchable moments** — find "when did it open the spreadsheet?"
* AI-generated summaries of agent sessions
* Timestamped results with auto-generated clips
* Full transcripts of audio content

**Compliance Ready**

### Audit Everything

* Complete visual audit trail of agent actions
* **Webhook alerts** — get notified when something looks off
* Replay failures to see exactly where things went wrong
* Meet regulatory requirements with searchable evidence

> [!WARNING]
>
> **Why Logs Aren't Enough**
>
> Traditional logging captures what your code does. But autonomous agents interact with visual interfaces—browsers, desktop apps, forms—where the "ground truth" is what's on screen, not what's in a log file.
>
> You can log "clicked button #submit" but you can't log "the submit button was disabled because the CAPTCHA wasn't solved yet" unless you see the screen. Visual monitoring captures this reality, giving you the same view the agent had when it made decisions.

## How It Works

**1. Screen Capture**

A monitor daemon (`monitor.ts`) runs alongside your OpenClaw agent, continuously capturing the screen. Recordings stream directly to VideoDB.

**2. On-Demand Indexing**

Indexing is separate from recording — you only pay for it when needed. When triggered, VideoDB processes recordings with:

* **Visual indexing** — scene descriptions, active apps, URLs, and error detection
* **Transcript** — speech-to-text from system audio
* **Audio indexing** — semantic indexing of audio content

**3. Search & Summarize**

Query agent history using natural language. Get timestamped results, auto-generated clips, and session summaries. The agent can even use its own recordings — "summarize what you did in the last 2 hours."

**4. Two Setup Options**

**Option 1**: Install as an OpenClaw skill — the agent handles recording, indexing, and search automatically.
**Option 2**: Use VideoDB's indexing APIs directly for post-recording analysis, summaries, and search.

## CLI Commands

The `videodb.ts` CLI tool provides these commands:

| Command | What It Does |
| - | - |
| `start-indexing` | Start all indexing (visual + audio + transcript) |
| `stop-indexing` | Stop all indexing |
| `start-visual-index` | Start visual scene indexing |
| `stop-visual-index` | Stop visual scene indexing |
| `start-audio-index` | Start audio indexing |
| `stop-audio-index` | Stop audio indexing |
| `start-transcript` | Start transcript extraction |
| `stop-transcript` | Stop transcript extraction |
| `search` | Natural language search across indexed recordings |
| `summary` | Generate an AI summary of agent activity |
| `transcript` | Get full transcript of audio content |
| `stream` | Get a playable stream URL for a specific time range |

## Real-World Scenarios

> **Debugging Failed Runs**
>
> An agent task failed overnight. Instead of digging through logs, search "when did the error first appear" and get a video clip of exactly what happened — the commands run, the error messages, and the state of the screen.

> **Compliance Auditing**
>
> Need proof of what your AI agents did and when? OpenClaw Monitoring provides timestamped, searchable visual records that serve as a full audit trail of agent actions.

> **Security Monitoring**
>
> Catch agents going off-script or accessing unexpected domains. Webhook alerts notify you when something looks off, giving you video evidence of unexpected behavior.

> **QA & Testing**
>
> Review agent workflows before pushing to production. Record test sessions, then search for edge cases, errors, or unexpected behaviors. Share specific clips with your team.

> **Agent Self-Reporting**
>
> Your agent can use its own recordings: "Summarize what you did in the last 2 hours", "Make a highlight video of today's work and post it to YouTube", or "Find the moment when you encountered the error."

## Tech Stack

> **TypeScript**
>
> Monitor daemon and CLI tooling (`monitor.ts`, `videodb.ts`)

> **Python + Flask**
>
> Backend server for session management and Cloudflare tunneling

> **VideoDB SDK**
>
> Media streaming, AI indexing, and semantic search

> **OpenClaw**
>
> Autonomous agent framework with skill system

## Getting Started

> [!NOTE]
>
> **Prerequisites**
>
> * [OpenClaw](https://openclaw.ai/) installed and running
> * Node.js 18+
> * [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)

**1. Install the Skill**

Point your OpenClaw agent at the repo:

```text theme={null}
please install https://github.com/video-db/openclaw-monitoring skill and set it up
```

Or install manually:

```bash theme={null}
git clone https://github.com/video-db/openclaw-monitoring.git
mkdir -p ~/.openclaw/workspace/skills/videodb-monitoring
cp -r openclaw-monitoring/videodb-monitoring-skill/* ~/.openclaw/workspace/skills/videodb-monitoring/
cd ~/.openclaw/workspace/skills/videodb-monitoring
npm install
```

**2. Set Your API Key**

```bash theme={null}
openclaw config set skills.entries.videodb-monitoring.env.VIDEODB_API_KEY 'sk-xxx'
```

Get your API key at [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs).

**3. Start the Monitor**

```bash theme={null}
cd ~/.openclaw/workspace/skills/videodb-monitoring
nohup npx tsx monitor.ts > ~/.videodb/logs/monitor.log 2>&1 & disown
openclaw gateway restart
```

**4. Use It**

Ask your agent:

* "Do X on the browser and send me the recording"
* "What did I do in the last hour?"
* "Find when I opened the spreadsheet"

> [!WARNING]
>
> **Cost Note**: VideoDB ingestion is billed at **\$0.084 / hour** of captured content. Indexing is on-demand — you only pay for it when you trigger it. See the [Capture SDK overview](/pages/ingest/capture-sdks/overview) for details.

## Try Without Setup

Skip installation and try indexing against a hosted live OpenClaw session at [matrix.videodb.io](https://matrix.videodb.io):

```bash theme={null}
git clone https://github.com/video-db/openclaw-monitoring.git
cd openclaw-monitoring
echo "VIDEO_DB_API_KEY=your_api_key_here" > .env
uv run try_without_setup.py
```

This connects to the public agent's live streams, starts indexing, and prints events to your terminal. Press `Ctrl+C` to stop and search the indexed content.

## Privacy & Security

> [!TIP]
>
> **API Key Security** — Keys stored locally via OpenClaw config, never transmitted except to VideoDB

> [!TIP]
>
> **On-Demand Indexing** — AI processing runs only when you trigger it, keeping costs controlled

> [!TIP]
>
> **Visual Audit Trail** — Complete recordings provide verifiable evidence of agent activity

> [!TIP]
>
> **Secure Tunneling** — Cloudflare Tunnel for webhook delivery without exposing ports

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/openclaw-monitoring)**
>
> Full installation instructions, advanced setup, and troubleshooting

## Related Tutorials

> **[Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> AI coding assistant with real-time screen and audio context

> **[FocusD](/examples-and-tutorials/ai-copilots/focusd)**
>
> AI-powered productivity tracking with automatic time insights


# Pair Programmer
Source: https://docs.videodb.io/examples-and-tutorials/ai-copilots/pair-programmer

Turn your coding agent into a screen aware, voice aware, context rich collaborator

[Pair Programmer - Screen Aware AI Coding Assistant](https://www.youtube.com/embed/dIvZoZr3DyM)

> **[Pair Programmer on GitHub](https://github.com/video-db/pair-programmer)**
>
> Complete source code, installation guide, and configuration examples

## What Is It?

Pair Programmer is an **agentic skill** that gives your AI coding assistant real time perception.

It captures:

* **Screen** for visual context like terminals, editors, browser tabs, errors, and UI state
* **Microphone** for your spoken intent, ideas, and debugging notes
* **System audio** for tutorials, meetings, demos, and anything else your computer is playing

Once captured, that context becomes searchable.

So instead of re explaining what was on screen, copy pasting logs, or summarizing a 20 minute debugging session, you can ask:

* *What was I doing when the auth flow broke?*
* *What did I say about the database migration?*
* *Show me what was on screen when the test failed*
* *What happened in the last 10 minutes?*

> [!TIP]
>
> **The Missing Piece**: This is the missing perception layer for coding agents. Works with Claude Code, Cursor, Codex, and other skill compatible agents.

**Why this changes everything**: Most coding agents operate in a text-only world. They can read your files and write code, but they can't see your terminal output, your browser's error messages, your Figma mockups, or hear you explaining the problem out loud. That means you spend half your time copy-pasting context, describing what's on screen, or re-explaining what you already said 5 minutes ago.

Pair Programmer closes this gap. It gives your agent the same sensory context you have—screen, mic, system audio—making collaboration feel natural instead of fragmented.

## Why This Is Useful

**Context Aware**

### Stay Grounded

Most coding agents can write code. Very few can stay grounded in the same context as you.

Pair Programmer helps your agent stay on the same page by giving it access to what you saw, what you said, and what your machine was playing.

**Natural Search**

### Ask in Plain Language

Search your session in natural language:

* "What was I working on when I mentioned the auth bug?"
* "What did I say in the last 5 minutes?"
* "Show me what was on screen when the test failed"

No more copy-pasting or repeated explanations.

**Real-Time Recording**

### Continuous Capture

A lightweight overlay shows recording status, active channels, and elapsed time.

Record your screen, mic, and system audio in real time, then search what happened when you need it.

## Use Cases

Pair Programmer is perfect for:

* **Debugging sessions** — Track what you tried and where it went wrong
* **Tutorial driven development** — Build while following video tutorials
* **Bug reproduction** — Capture exact steps that triggered the issue
* **Meeting follow ups** — Search conversations and screen activity
* **Architecture walkthroughs** — Review code with full context
* **Voice first coding workflows** — Speak your thoughts and code together

## Getting Started with Pair Programmer

The most common workflow:

1. **Start recording when you begin work** — just run `/pair-programmer record` and choose your screen. Let it capture in the background.

2. **Work normally** — code, debug, browse Stack Overflow, watch tutorials. Don't think about the recording.

3. **Ask for context when you need it** — stuck on a bug? Run `/pair-programmer search "when did the build error first appear?"` Your agent sees the exact moment with full terminal output and code context.

4. **Let your agent act on spoken instructions** — said "refactor this function to use async/await" into your mic 5 minutes ago? Run `/pair-programmer act` and your agent will do it, using your own words as the spec.

5. **Stop recording when done** — run `/pair-programmer stop`. All context is saved and searchable.

Over time, you'll develop your own patterns—maybe you record only during debugging sessions, or maybe you keep it running all day for complete work memory.

## Installation

> [!NOTE]
>
> **Prerequisites**
>
> * Node.js 18+
> * macOS 12+ or Windows 10+
> * [VideoDB API key](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) (free, no credit card required)

If you have an older version installed, remove it first before upgrading.

**Option 1: NPX**

### Install with npx (Recommended)

```bash theme={null}
npx skills add video-db/pair-programmer
```

**Option 2: Marketplace**

### Install from marketplace

```bash theme={null}
/plugin marketplace add video-db/pair-programmer
/plugin install pair-programmer
```

## Setup

**1. Get API Key**

Get a free VideoDB API key from [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)

No credit card required.

**2. Set API Key**

Export your API key in your shell:

```bash theme={null}
export VIDEO_DB_API_KEY=your-key
```

Or add it to a `.env` file in your project root

**3. Run Setup**

Install dependencies and complete local setup:

```bash theme={null}
/pair-programmer setup
```

## Quick Start

**1. Start Recording**

Start recording your screen, mic, and system audio:

```bash theme={null}
/pair-programmer record
```

A source picker will open so you can choose what to capture. Once recording starts, a lightweight overlay shows recording status, active channels, and elapsed time.

**2. Work Normally**

Continue your coding session. Pair Programmer captures everything in the background.

**3. Search Your Session**

Search your session in natural language:

```bash theme={null}
/pair-programmer search "what was I working on when I mentioned the auth bug?"
```

```bash theme={null}
/pair-programmer search "what did I say in the last 5 minutes?"
```

```bash theme={null}
/pair-programmer search "show me what was on screen when the test failed"
```

**4. Get Summary**

Get a summary of recent activity:

```bash theme={null}
/pair-programmer what-happened
```

**5. Stop Recording**

Stop recording when you're done:

```bash theme={null}
/pair-programmer stop
```

## Commands

| Command | Description |
| - | - |
| `/pair-programmer record` | Start recording and open the source picker |
| `/pair-programmer stop` | Stop the active recording |
| `/pair-programmer search "<query>"` | Search screen, mic, and audio context using natural language |
| `/pair-programmer what-happened` | Summarize recent activity |
| `/pair-programmer setup` | Install dependencies and complete local setup |
| `/pair-programmer config` | Update indexing and recording settings |

## Real-World Examples

> **Debugging Complex Issues**
>
> You're chasing a bug across multiple files and terminals. Instead of documenting every step, just keep coding. Later, run:
>
> ```bash theme={null}
> /pair-programmer search "what was on screen when the test failed"
> ```
>
> Get instant context about terminal output, error messages, and which files you had open.

> **Learning From Tutorials**
>
> Following a video tutorial while coding? Pair Programmer captures both the tutorial (system audio) and your implementation (screen).
>
> ```bash theme={null}
> /pair-programmer search "build me the project from the video I was just watching"
> ```
>
> Your agent sees what was on screen and heard what was being said in the tutorial.

> **Pair Programming Sessions**
>
> In a meeting discussing code? Pair Programmer captures your screen and the conversation.
>
> ```bash theme={null}
> /pair-programmer what-happened
> ```
>
> Get a summary of what was discussed, what code was reviewed, and action items.

> **Voice-Driven Development**
>
> Speaking your thoughts while coding? Your microphone captures your debugging notes and ideas.
>
> ```bash theme={null}
> /pair-programmer search "what did I say about the database migration?"
> ```
>
> Find moments where you verbally explained your thinking.

## How It Works

Pair Programmer uses VideoDB's Capture SDK to:

1. **Record** — Continuously capture screen, microphone, and system audio
2. **Process** — Stream to VideoDB for real-time AI indexing
3. **Search** — Query across all captured context with natural language
4. **Retrieve** — Get timestamped results with relevant clips

All context is searchable in real-time, giving your coding agent full perception of your workflow.

***

> **[Complete Setup Guide on GitHub](https://github.com/video-db/pair-programmer)**
>
> Detailed installation instructions, troubleshooting tips, and configuration examples

## Related Tutorials

> **[Bloom](/examples-and-tutorials/ai-copilots/bloom)**
>
> Local-first screen recorder with AI-ready search and indexing

> **[Focusd Productivity Tracker](/examples-and-tutorials/ai-copilots/focusd)**
>
> AI-powered productivity tracking with automatic time insights

> **[Call.md](/examples-and-tutorials/ai-copilots/call-md)**
>
> Real-time AI meeting assistant with live coaching


# AI-Generated Ads
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/ai-ad-films

Generate professional product advertisements from raw footage

> **Automatically Creating a professional quality advertisement from product videography B-Roll**

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Voiceover_Ad_Films.ipynb)

## Overview

Creating professional product advertisements typically requires a team: copywriters for the script, voice actors for the narration, and editors to stitch it all together.

**VideoDB** streamlines this into a single, automated workflow.

In this tutorial, we will:

1. **Upload** raw product footage (a jewelry shoot).
2. **Analyze** the visual content automatically.
3. **Generate** a professional ad script based on the visuals.
4. **Synthesize** a high-quality voiceover.
5. **Publish** the final commercial.

## Setup

### Installing VideoDB

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Key

> [!NOTE]
>
> You only need your VideoDB API Key. Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

## Implementation

### Step 1: Connect to VideoDB

Establish a connection to your VideoDB project.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

// Connect to VideoDB
const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Product Footage

We'll upload a raw video clip of a jewelry product shoot from YouTube.

```python Python theme={null}
# Upload a video by URL
video = coll.upload(url='https://www.youtube.com/watch?v=2DcAMbmmYNM')
```

```javascript Node.js theme={null}
// Upload a video by URL
const video = await coll.uploadURL({ url: 'https://www.youtube.com/watch?v=2DcAMbmmYNM' });
```

### Step 3: Analyze Visuals

To write a relevant script, we first need to understand what's in the video. The analyzer creates detailed timestamped descriptions of the visual content.

```python Python theme={null}
print("Analyzing visual scenes... this might take a moment.")

scene_understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {
                "prompt": "Describe the scene in detail, focusing on the product, craftsmanship, lighting, camera movement, and mood.",
                "schema": {
                    "scene_description": "string",
                },
            },
        },
    ],
)
scene_understanding.wait_until_complete()

scene_analyzer = scene_understanding.get_analyzer("scene")
```

```javascript Node.js theme={null}
const sceneUnderstanding = await video.understand([
    {
        type: "vlm",
        name: "scene",
        config: {
            prompt: "Describe the scene in detail, focusing on the product, craftsmanship, lighting, camera movement, and mood.",
            schema: {
                scene_description: "string",
            },
        },
    },
]);
await sceneUnderstanding.waitUntilComplete();

const sceneAnalyzer = await sceneUnderstanding.getAnalyzer("scene");
```

Let's view the descriptions of the first three scenes from the video.

```python Python theme={null}
import json

scene_output = scene_analyzer.get_output()

scenes = [
    {
        "start": scene["start"],
        "end": scene["end"],
        "description": scene["data"]["scene_description"],
    }
    for scene in scene_output["scenes"]
]

print(json.dumps(scenes[:3], indent=2))
```

```javascript Node.js theme={null}
const sceneOutput = await sceneAnalyzer.getOutput();
const scenes = sceneOutput.scenes.map((scene) => ({
    start: scene.start,
    end: scene.end,
    description: scene.data.scene_description,
}));

console.log(JSON.stringify(scenes.slice(0, 3), null, 2));
```

### Step 4: Generate Ad Script

Now we use VideoDB's text generation to write the commercial. We feed the visual descriptions into the prompt to ensure the script perfectly matches the mood of the footage.

```python Python theme={null}
# Construct a prompt with the scene context
scene_context = "\n".join([f"- {scene['description']}" for scene in scenes])

prompt = f"""
Here is a visual description of a jewelry product video:
{scene_context}

Write a short, elegant, and luxurious voiceover script for this video advertisement.
- Tone: Sophisticated, calming, premium.
- Length: Short (approx 20 seconds of speech).
- Content: Focus on beauty, craftsmanship, and elegance.
- Format: Return ONLY the raw narration text.
"""

# Generate script
text_response = coll.generate_text(
    prompt=prompt,
    model_name="pro")

ad_script = text_response["output"]

print("--- Generated Ad Script ---")
print(ad_script)
```

```javascript Node.js theme={null}
// Construct a prompt with the scene context
const sceneContext = scenes.map(scene => `- ${scene.description}`).join('\n');

const prompt = `
Here is a visual description of a jewelry product video:
${sceneContext}

Write a short, elegant, and luxurious voiceover script for this video advertisement.
- Tone: Sophisticated, calming, premium.
- Length: Short (approx 20 seconds of speech).
- Content: Focus on beauty, craftsmanship, and elegance.
- Format: Return ONLY the raw narration text.
`;

// Generate script
const adScriptResult = await coll.generateText(prompt, "pro", "text");
const adScript = typeof adScriptResult === "string"
    ? adScriptResult
    : adScriptResult.output;

if (typeof adScript !== "string") {
    throw new Error("Expected a text response for the ad script.");
}

console.log("--- Generated Ad Script ---");
console.log(adScript);
```

### Step 5: Generate Voiceover

We'll turn the script into audio.

```python Python theme={null}
# Generate speech directly as a VideoDB Audio Asset
audio = coll.generate_voice(
    text=ad_script,
    voice_name="Default")
```

```javascript Node.js theme={null}
// Generate speech directly as a VideoDB Audio Asset
import { Audio, GenerationJob } from 'videodb';

const audioCandidate = await coll.generateVoice(
    adScript,
    "Default"
);
const audio = audioCandidate instanceof GenerationJob
    ? await audioCandidate.wait()
    : audioCandidate;

if (!(audio instanceof Audio)) {
    throw new Error("Expected generated voice audio.");
}
```

### Step 6: Compose the Advertisement

We'll overlay the generated voiceover onto the original video using the Timeline editor.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset, AudioAsset

# Create a timeline
timeline = Timeline(conn)

# 1. Video Track (Background)
video_track = Track()
video_asset = VideoAsset(id=video.id)
video_clip = Clip(asset=video_asset, duration=float(video.length))
video_track.add_clip(0, video_clip)
timeline.add_track(video_track)

# 2. Audio Track (Voiceover Overlay)
audio_track = Track()
audio_asset = AudioAsset(id=audio.id)
audio_clip = Clip(asset=audio_asset, duration=float(audio.length))
audio_track.add_clip(0, audio_clip)
timeline.add_track(audio_track)
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset, EditorAudioAsset } from 'videodb';

// Create a timeline
const timeline = new EditorTimeline(conn);

// 1. Video Track (Background)
const videoTrack = new Track();
const videoAsset = new EditorVideoAsset({ id: video.id });
const videoClip = new Clip({ asset: videoAsset, duration: parseFloat(video.length) });
videoTrack.addClip(0, videoClip);
timeline.addTrack(videoTrack);

// 2. Audio Track (Voiceover Overlay)
const audioTrack = new Track();
const audioAsset = new EditorAudioAsset({ id: audio.id });
const audioClip = new Clip({ asset: audioAsset, duration: parseFloat(audio.length) });
audioTrack.addClip(0, audioClip);
timeline.addTrack(audioTrack);
```

### Step 7: Review and Share

Generate the stream URL to watch your AI-created commercial.

```python Python theme={null}
from videodb import play_stream

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

**Output:**

[AI-Generated Product Advertisement](https://www.youtube.com/embed/NPvxczplmW0)

### Conclusion

You have successfully automated the production of a product advertisement.

By replacing multiple external tools with VideoDB's unified SDK, you can now build scalable video generation engines that turn raw footage into polished content automatically.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Voiceover_Ad_Films.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Generate complete videos from scripts without camera

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional narration to silent footage


# AI Storyteller for Kids
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/ai-storyteller-kids

Generate complete animated educational videos for kids with AI scripts, voiceovers, and captions

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/ai_storyteller_for_kids.ipynb)

## The Idea

Ever wished you could create fun, educational videos for kids with just a single topic? Tell the system "explain the solar system" and get back a complete animated video with:

* Kid-friendly voiceover narration
* Fun background music
* Loopable animated backgrounds
* Timed subject videos (cartoons of planets, stars, etc.)
* Animated captions that kids can follow along

This workflow takes a topic and magically transforms it into a complete learning video. All powered by **VideoDB's Editor SDK** — pure automation magic.

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import os
import math
import videodb
from videodb import play_stream
from videodb.editor import (
    Timeline, Track, Clip,
    VideoAsset, AudioAsset, TextAsset, CaptionAsset,
    Fit, Position, Offset, Transition,
    Font, Border,
    CaptionAnimation, CaptionAlignment, CaptionBorderStyle,
    FontStyling, Positioning, BorderAndShadow,
)

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Generate Script and Title

```python theme={null}
import json

# Define the learning topic
topic = "Explain the solar system"

script_prompt = f"""You are a friendly children's educational content writer.

Write a fun, simple explanation about: "{topic}"

Requirements:
- Target audience: 5-year-old children
- Length: 250 words
- Use simple words and short sentences
- Make it fun and engaging

Return a JSON with:
- "title": A fun, catchy title for the video (max 5 words)
- "script": The complete narration script

Example format:
{{"title": "The Amazing Sun!", "script": "Hey friends! Today we're going to learn about..."}}
"""

script_response = coll.generate_text(
    prompt=script_prompt,
    model_name="pro",
    response_type="json"
)

# Handle nested output structure
if 'output' in script_response and isinstance(script_response['output'], dict):
    video_title = script_response["output"]["title"]
    video_script = script_response["output"]["script"]
else:
    video_title = script_response["title"]
    video_script = script_response["script"]
```

### Step 2: Generate Background Prompts

```python theme={null}
bg_prompt = f"""Based on this children's educational video topic: "{topic}"

Create prompts for background media.

Return JSON with:
- "background_video_prompt": Detailed prompt for generating background video
- "music_prompt": Prompt for generating background music

Example: {{"background_video_prompt": "...", "music_prompt": "..."}}
"""

bg_response = coll.generate_text(
    prompt=bg_prompt,
    model_name="pro",
    response_type="json"
)

# Handle nested output structure
if isinstance(bg_response, dict) and "output" in bg_response:
    background_video_prompt = bg_response["output"]["background_video_prompt"]
    music_prompt = bg_response["output"]["music_prompt"]
else:
    background_video_prompt = bg_response["background_video_prompt"]
    music_prompt = bg_response["music_prompt"]
```

### Step 3: Generate All Assets

```python theme={null}
# Generate voiceover from script
voiceover = coll.generate_voice(
    text=video_script,
    voice_name="Default"
)

# Generate background music (loopable)
bg_music = coll.generate_music(
    prompt=music_prompt,
    duration=10
)

# Generate background video (loopable)
bg_video = coll.generate_video(
    prompt=background_video_prompt,
    duration=5
)
```

### Step 4: Generate Timed Transcript

```python theme={null}
# Generate and fetch timestamped transcript
print("📝 Generating transcript with timestamps...")

voiceover.generate_transcript()
transcript = voiceover.get_transcript(segmenter=videodb.Segmenter.time, length=5)

import base64

def format_srt_time(seconds):
    total_ms = int(round(float(seconds) * 1000))
    hours, remainder = divmod(total_ms, 3_600_000)
    minutes, remainder = divmod(remainder, 60_000)
    secs, milliseconds = divmod(remainder, 1_000)
    return f"{hours:02}:{minutes:02}:{secs:02},{milliseconds:03}"

srt = "\n".join(
    f"{number}\n"
    f"{format_srt_time(segment['start'])} --> {format_srt_time(segment['end'])}\n"
    f"{segment['text']}\n"
    for number, segment in enumerate(transcript, start=1)
)
srt_src = base64.b64encode(srt.encode("utf-8")).decode("utf-8")

# Calculate total video duration (intro + voiceover + outro)
voiceover_length_float = float(voiceover.length)
intro_duration = 5
outro_duration = 5
total_duration = intro_duration + voiceover_length_float + outro_duration

print(f"✅ Transcript ready! {len(transcript)} segments")
print(f"\n⏱️ Video breakdown:")
print(f"   - Intro: {intro_duration}s")
print(f"   - Voiceover: {voiceover_length_float}s")
print(f"   - Outro: {outro_duration}s")
print(f"   - Total: {total_duration}s")

import json
print(json.dumps(transcript, indent=2))
```

### Step 5: Configure Reusable Captions

```python theme={null}
# Configure reusable captions from the timed transcript
caption_asset = CaptionAsset(
    src=srt_src,
    animation=CaptionAnimation.karaoke,
    primary_color="&H00FFFFFF",
    secondary_color="&H0000FFFF",
    back_color="&H80000000",
    position=Positioning(
        alignment=CaptionAlignment.bottom_center,
        margin_l=20,
        margin_r=20,
        margin_v=30,
    ),
    font=FontStyling(
        name="Clear Sans",
        size=52,
        bold=True,
    ),
    border=BorderAndShadow(
        style=CaptionBorderStyle.outline_and_shadow,
        outline=4,
        outline_color="&H00000000",
        shadow=2,
    ),
)

print("✅ Reusable caption style ready!")
```

### Step 6: Build the Complete Video

```python theme={null}
# Use the actual length of each background asset for looping
timeline = Timeline(conn)
timeline.background = "#0a0a1a"
timeline.resolution = "1280x720"

bg_video_actual_length = bg_video.length
video_loops = math.ceil(total_duration / bg_video_actual_length)
music_loops = math.ceil(total_duration / 10)

# Track 1: Looped background video
bg_video_track = Track()
for i in range(video_loops):
    start = i * bg_video_actual_length
    remaining = total_duration - start
    dur = min(bg_video_actual_length, remaining)

    if dur <= 0:
        break

    bg_video_track.add_clip(
        start=start,
        clip=Clip(
            asset=VideoAsset(id=bg_video.id, volume=0),
            duration=dur,
            fit=Fit.cover,
            opacity=0.7,
        ),
    )

# Track 2: Looped background music
bg_music_track = Track()
for i in range(music_loops):
    start = i * 10
    remaining = total_duration - start
    dur = min(10, remaining)
    bg_music_track.add_clip(
        start=start,
        clip=Clip(asset=AudioAsset(id=bg_music.id, volume=0.5), duration=dur),
    )

# Track 3: Voiceover and captions (start after intro)
vo_track = Track()
vo_track.add_clip(
    start=intro_duration,
    clip=Clip(
        asset=AudioAsset(id=voiceover.id, volume=1.0),
        duration=voiceover_length_float,
    ),
)
vo_track.add_clip(
    start=intro_duration,
    clip=Clip(asset=caption_asset, duration=voiceover_length_float),
)

# Track 4: Title card
title_track = Track()
title_track.add_clip(
    start=0,
    clip=Clip(
        asset=TextAsset(
            text=video_title,
            font=Font(family="Quicksand", size=96, color="#FFD700"),
            border=Border(color="#000000", width=6),
        ),
        duration=intro_duration,
        position=Position.center,
        transition=Transition(in_="fade", out="fade", duration=0.8),
    ),
)

# Track 5: Outro card
outro_track = Track()
outro_track.add_clip(
    start=intro_duration + voiceover_length_float,
    clip=Clip(
        asset=TextAsset(
            text="Happy Learning!",
            font=Font(family="Quicksand", size=88, color="#90EE90"),
            border=Border(color="#000000", width=6),
        ),
        duration=outro_duration,
        position=Position.center,
        transition=Transition(in_="fade", out="fade", duration=0.8),
    ),
)

timeline.add_track(bg_video_track)
timeline.add_track(bg_music_track)
timeline.add_track(vo_track)
timeline.add_track(title_track)
timeline.add_track(outro_track)

print("✅ Timeline built!")
stream = timeline.generate_stream()
play_stream(stream)
```

### Step 7: Generate Subject Video Prompts

```python theme={null}
# Group the timed transcript into pairs for a visual subject prompt per pair
transcript_pairs = []
for i in range(0, len(transcript), 2):
    pair = {
        "start": float(transcript[i]["start"]),
        "end": float(transcript[i + 1]["end"]) if i + 1 < len(transcript) else float(transcript[i]["end"]),
        "text": transcript[i]["text"] + " " + (transcript[i + 1]["text"] if i + 1 < len(transcript) else ""),
    }
    transcript_pairs.append(pair)

subject_prompt = f"""You are creating visual prompts for a children's educational video about: "{topic}"

For each PAIR of narration lines below, create ONE video generation prompt that captures the essence of both lines.

Transcript pairs:
{json.dumps(transcript_pairs, indent=2)}

Requirements for each video prompt:
- Style: Cartoonish, whimsical, colorful, kid-friendly
- NO text in the videos
- NO complex actions
- Focus on a SINGLE subject that represents both lines
- Think fun and simple — like a storybook illustration come to life
- Slight motion is okay (spinning, floating, twinkling, bouncing)

Return a JSON object with a "videos" array containing ONE item per PAIR, in the SAME ORDER:
{{
    "videos": [
        {{"subject": "<what it shows>", "prompt": "<video generation prompt for pair 1>"}},
        {{"subject": "<what it shows>", "prompt": "<video generation prompt for pair 2>"}},
        ...
    ]
}}

You MUST return exactly {len(transcript_pairs)} items, one for each pair.
"""

subject_response = coll.generate_text(
    prompt=subject_prompt,
    model_name="pro",
    response_type="json",
)

video_prompts = subject_response["output"]["videos"]
if not isinstance(video_prompts, list) or len(video_prompts) != len(transcript_pairs):
    raise ValueError("Expected exactly one video prompt per transcript pair.")

subject_moments = []
for i, pair in enumerate(transcript_pairs):
    prompt_data = video_prompts[i]
    subject_moments.append(
        {
            "start": pair["start"],
            "end": pair["end"],
            "text": pair["text"],
            "subject": prompt_data.get("subject", f"Scene {i + 1}"),
            "prompt": prompt_data["prompt"],
        }
    )

print(f"✅ Generated {len(subject_moments)} video prompts for {len(transcript)} transcript lines!")
```

### Step 8: Generate Subject Videos

```python theme={null}
subject_videos = []

for moment in subject_moments:
    subject_video = coll.generate_video(
        prompt=moment["prompt"],
        duration=5,
    )
    subject_videos.append(
        {
            "video": subject_video,
            "start": moment["start"],
            "end": moment["end"],
            "subject": moment["subject"],
        }
    )

print(f"🎉 All {len(subject_videos)} subject videos generated!")
```

### Step 9: Render the Final Video

```python theme={null}
final_timeline = Timeline(conn)
final_timeline.background = "#0a0a1a"
final_timeline.resolution = "1280x720"

# Track 1: Looped background video
bg_video_track = Track()
for i in range(math.ceil(total_duration / bg_video.length)):
    start = i * bg_video.length
    dur = min(bg_video.length, total_duration - start)
    if dur <= 0:
        break
    bg_video_track.add_clip(
        start=start,
        clip=Clip(
            asset=VideoAsset(id=bg_video.id, volume=0),
            duration=dur,
            fit=Fit.cover,
            opacity=0.7,
        ),
    )

# Track 2: Looped background music
bg_music_track = Track()
for i in range(math.ceil(total_duration / 10)):
    start = i * 10
    dur = min(10, total_duration - start)
    bg_music_track.add_clip(
        start=start,
        clip=Clip(asset=AudioAsset(id=bg_music.id, volume=0.5), duration=dur),
    )

# Track 3: Voiceover and captions
vo_track = Track()
vo_track.add_clip(
    start=intro_duration,
    clip=Clip(
        asset=AudioAsset(id=voiceover.id, volume=1.0),
        duration=voiceover_length_float,
    ),
)
vo_track.add_clip(
    start=intro_duration,
    clip=Clip(asset=caption_asset, duration=voiceover_length_float),
)

# Track 4: Subject videos mapped to their paired transcript timestamps
subject_track = Track()
sorted_subjects = sorted(subject_videos, key=lambda item: float(item["start"]))

for i, subject in enumerate(sorted_subjects):
    clip_start = intro_duration + subject["start"]
    clip_duration = max(1, subject["end"] - subject["start"])

    if i < len(sorted_subjects) - 1:
        next_start = intro_duration + sorted_subjects[i + 1]["start"]
        clip_duration = min(clip_duration, next_start - clip_start)

    voiceover_end = intro_duration + voiceover_length_float
    clip_duration = min(clip_duration, voiceover_end - clip_start)
    if clip_duration <= 0:
        continue

    subject_video_length = subject["video"].length
    for loop in range(math.ceil(clip_duration / subject_video_length)):
        loop_start = clip_start + (loop * subject_video_length)
        loop_duration = min(subject_video_length, clip_duration - (loop * subject_video_length))
        if loop_duration <= 0:
            break
        subject_track.add_clip(
            start=loop_start,
            clip=Clip(
                asset=VideoAsset(id=subject["video"].id, volume=0),
                duration=loop_duration,
                fit=Fit.contain,
                scale=0.75,
                position=Position.top,
                offset=Offset(x=0, y=0.05),
                opacity=0.95,
            ),
        )

# Track 5: Title card
title_track = Track()
title_track.add_clip(
    start=0,
    clip=Clip(
        asset=TextAsset(
            text=video_title,
            font=Font(family="Quicksand", size=96, color="#FFD700"),
            border=Border(color="#000000", width=6),
        ),
        duration=intro_duration,
        position=Position.center,
        transition=Transition(in_="fade", out="fade", duration=0.8),
    ),
)

# Track 6: Outro card
outro_track = Track()
outro_track.add_clip(
    start=intro_duration + voiceover_length_float,
    clip=Clip(
        asset=TextAsset(
            text="Happy Learning!",
            font=Font(family="Quicksand", size=88, color="#90EE90"),
            border=Border(color="#000000", width=6),
        ),
        duration=outro_duration,
        position=Position.center,
        transition=Transition(in_="fade", out="fade", duration=0.8),
    ),
)

final_timeline.add_track(bg_video_track)
final_timeline.add_track(bg_music_track)
final_timeline.add_track(vo_track)
final_timeline.add_track(subject_track)
final_timeline.add_track(title_track)
final_timeline.add_track(outro_track)

final_stream = final_timeline.generate_stream()
play_stream(final_stream)
```

## What You Get

A complete educational video with:

* Kid-friendly AI-written script
* Natural voiceover narration (kids' voice)
* Loopable animated background
* Timed subject videos (cartoon illustrations)
* Synchronized animated captions
* Upbeat background music
* Professional intro and outro cards

Here's the final educational video:

[AI Storyteller Educational Video for Kids](https://console.videodb.io/player?url=https://d1zudc7ewmc6ey.cloudfront.net/v1/9c1816f6-8375-4834-a60a-cfd4a4889735.m3u8)

## Perfect For

* **Home Learning** - Parents teaching kids at home
* **Educational Channels** - YouTube Kids content creators
* **School Supplements** - Teachers augmenting classroom lessons
* **Language Learning** - Kids' language learning videos
* **Skill Development** - Tutorial videos for children

## What Topics Can You Cover?

* The Solar System
* Ocean Animals
* Ancient Civilizations
* How Photosynthesis Works
* The Water Cycle
* Math Basics
* History Stories
* Science Experiments
* Geography
* And anything else!

## The Result

What used to take weeks of scriptwriting, voiceover recording, animation, and editing now takes minutes.

Teachers and parents can create engaging, professional educational content instantly. Kids get to learn with animated characters, fun music, and engaging visuals. Everyone wins.

**Pure learning magic — powered by VideoDB.**

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/ai_storyteller_for_kids.ipynb)**
>
> Open the complete implementation with advanced subject video generation, caption styling, and content customization.

## Related Tutorials

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Generate complete videos from scripts

> **[Text-to-Video](/examples-and-tutorials/content-factory/text-prompts)**
>
> Create video directly from text prompts


# Video Dubbing
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/dubbing

Dub videos into multiple languages with AI voice synthesis

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Dubbing%20-%20Replace%20Soundtrack%20with%20New%20Audio.ipynb)

## Overview

VideoDB makes video dubbing incredibly simple with AI-powered translation and voice synthesis. With just **one function call**, you can dub your videos into multiple languages while preserving the original speaking style and timing.

No need for complex audio editing, timeline manipulation, or third-party tools. VideoDB's `coll.dub_video()` handles everything automatically.

## Prerequisites

> [!NOTE]
>
> Ensure you have VideoDB installed in your environment and an API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). The first 50 uploads are free with no credit card required!

```bash theme={null}
!pip install videodb
```

## Connect to VideoDB

Connect to VideoDB using your API key:

```python Python theme={null}
import videodb

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();

console.log("Connected to VideoDB");
```

## Upload Your Video

Upload the video you want to dub. For this example, we'll use an English video:

```python Python theme={null}
# Upload the original video (English)
video_url = "https://www.youtube.com/watch?v=0e3GPea1Tyg"
video = coll.upload(url=video_url)

print(f"Video uploaded: {video.id}")
```

```javascript Node.js theme={null}
// Upload the original video (English)
const videoUrl = "https://www.youtube.com/watch?v=0e3GPea1Tyg";
const video = await coll.uploadURL({ url: videoUrl });

console.log(`Video uploaded: ${video.id}`);
```

### Preview the Original

```python Python theme={null}
video.play()
```

## Dub Your Video

> [!TIP]
>
> Here's where the magic happens! Dub your video into any supported language with a single function call:

```python Python theme={null}
# Dub the video into Hindi
dubbed_video = coll.dub_video(
    video_id=video.id,
    language_code="hi"  # Hindi
)

print(f"Video dubbed successfully: {dubbed_video.id}")
```

```javascript Node.js theme={null}
// Dub the video into Hindi
const dubbedVideo = await coll.dubVideo(
    video.id,
    "hi"  // Hindi
);

console.log(`Video dubbed successfully: ${dubbedVideo.id}`);
```

That's it! VideoDB automatically dubbed your video.

## View the Dubbed Video

Generate and play your dubbed video:

```python Python theme={null}
dubbed_video.play()
```

[Dubbed Video Example](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/5dd7e078-0ecb-4b23-89bd-968e557c98a6.m3u8)

## Supported Languages

VideoDB supports dubbing into many languages. Simply change the `language_code` parameter:

| Language | Code | Language | Code |
| - | - | - | - |
| Spanish | `es` | Japanese | `ja` |
| French | `fr` | Korean | `ko` |
| German | `de` | Chinese (Simplified) | `zh` |
| Italian | `it` | Arabic | `ar` |
| Portuguese | `pt` | Russian | `ru` |
| Hindi | `hi` | … and more! | |

### Example: Dub into Spanish

```python Python theme={null}
# Dub the same video into Spanish
spanish_dubbed = coll.dub_video(
    video_id=video.id,
    language_code="es"  # Spanish
)

print(f"Spanish version created: {spanish_dubbed.id}")

# Play the Spanish version
spanish_dubbed.play()
```

## Use Cases

> **Content Localization**
>
> Reach global audiences by dubbing your content into multiple languages

> **Educational Content**
>
> Make learning materials accessible in students' native languages

> **Marketing Videos**
>
> Create localized versions of promotional content

> **Entertainment**
>
> Dub movies, shows, or vlogs for international viewers

> **Accessibility**
>
> Provide dubbed versions for audiences who prefer audio in their native language

## Wrap-up

With VideoDB's `coll.dub_video()`, video dubbing is as simple as querying a database. No complex audio editing, no timeline manipulation - just one function call to create professional multilingual content.

> [!TIP]
>
> Start dubbing your videos today and reach audiences worldwide!

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Dubbing%20-%20Replace%20Soundtrack%20with%20New%20Audio.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional narration to silent footage with AI voice synthesis

> **[Translation & Dubbing Guide](/pages/act/generative-media/translation-and-dubbing)**
>
> Deep dive into VideoDB's dubbing architecture and language support


# Faceless Video Creator
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/faceless-video-creator

Build complete faceless videos with AI-generated scripts, voiceovers, and multi-layer composition

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/faceless_video.ipynb)

## The Concept

Faceless videos are everywhere—TikTok, YouTube Shorts, Instagram Reels. They combine engaging visuals with voiceover narration and captions, but never show a person on camera. Think gaming clips with commentary, stock footage with educational content, or animated explainers.

The problem: Creating faceless videos requires scripting, voiceover recording, audio mixing, and video editing—all separate tools and skills.

What if you could generate it all programmatically from just a topic?

## What You'll Build

In this guide, you'll build a complete faceless video pipeline using VideoDB Editor. You'll:

* Generate an engaging script from a topic
* Convert that script to natural voiceover
* Layer it with background visuals and music
* Compile everything into a finished video

All powered by **VideoDB's Editor SDK** — pure automation magic.

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Upload Background Assets

```python theme={null}
# Upload background video (muted - we'll use for visuals only)
background_video = coll.upload(url='https://www.youtube.com/watch?v=VL1CHvsUSNo')
```

### Step 2: Generate AI Script

```python theme={null}
# Define your video topic
video_topic = "How AI is changing the gaming industry"

# Create prompt for script generation
script_prompt = f"""You are a GenZ content creator writing a script for a faceless video about: "{video_topic}"

Your task: Write an engaging, fun, fast-paced voiceover script.

Style Guidelines:
- Conversational and energetic tone (like you're talking to a friend)
- Use short, punchy sentences
- Include hooks and interesting facts
- Keep it engaging and easy to follow
- No intros like "Hey guys" or "In this video" - jump straight into the content
- Length: 300-400 words (about 2 minutes when spoken)

Critical: Return ONLY the script text. No titles, no commentary, no explanations. Just the pure voiceover script."""

# Generate script using AI
script_response = coll.generate_text(
    prompt=script_prompt,
    response_type="text",
    model_name="pro"
)

script = script_response["output"]
```

### Step 3: Generate Voiceover from Script

```python theme={null}
# Generate AI voiceover from script
voiceover_audio = coll.generate_voice(
    text=script,
    voice_name="Default"
)
```

### Step 4: Build Multi-Layer Timeline

Create the composition with background video, voiceover, and music:

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset, AudioAsset
from videodb import MediaType

# Upload background music as audio
background_music = coll.upload(
    url='https://www.youtube.com/watch?v=kkoIpjQ16YY',
    media_type=MediaType.audio
)

# Initialize timeline
timeline = Timeline(conn)
timeline.background = "#000000"  # Black background

# Track 1: Background video (muted)
video_clip = Clip(
    asset=VideoAsset(
        id=background_video.id,
        start=3,
        volume=0  # Muted - we only want visuals
    ),
    duration=float(voiceover_audio.length)
)

video_track = Track()
video_track.add_clip(0, video_clip)
timeline.add_track(video_track)
```

### Step 5: Add Voiceover Track

```python theme={null}
# Track 2: Voiceover (full volume)
voiceover_clip = Clip(
    asset=AudioAsset(
        id=voiceover_audio.id,
        start=0,
        volume=1.0  # Full volume for clear narration
    ),
    duration = float(voiceover_audio.length)
)

voiceover_track = Track()
voiceover_track.add_clip(0, voiceover_clip)
timeline.add_track(voiceover_track)
```

### Step 6: Add Background Music

```python theme={null}
# Track 3: Background music (low volume)
music_clip = Clip(
    asset=AudioAsset(
        id=background_music.id,
        start=0,
        volume=0.15  # Low volume so it doesn't overpower voiceover
    ),
    duration= float(voiceover_audio.length)
)

music_track = Track()
music_track.add_clip(0, music_clip)
timeline.add_track(music_track)
```

### Step 7: Render as Vertical Video

```python theme={null}
# Set vertical resolution for shorts/reels
timeline.resolution = "608x1080"

# Generate stream
vertical_stream_url = timeline.generate_stream()
```

## What You Get

A complete faceless video with:

* AI-generated engaging script
* Natural voiceover narration
* Background visuals (your choice)
* Ambient background music
* Proper audio mixing (voiceover prioritized)
* Vertical format ready for social media

Here's the final rendered video:

[Faceless Video Output](https://console.videodb.io/player?url=https://play.videodb.io/v1/21d1f186-4158-4bf5-900d-b630d18501b6.m3u8)

## Perfect Use Cases

* **Educational Content** - Explainers, how-tos, tutorials
* **Gaming Commentary** - Gameplay footage with voiceover analysis
* **News/Updates** - Topic-driven news videos
* **Product Reviews** - B-roll with narrated reviews
* **Storytelling** - Narrative content over visuals

## The Result

With this system, you can:

* Generate new faceless videos in minutes
* Scale content production without hiring narrators
* Maintain consistent voiceover quality across all videos
* Focus on visual storytelling rather than on-camera performance

No faces. No cameras. Just compelling content powered by AI.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/faceless_video.ipynb)**
>
> Open the complete implementation with advanced audio mixing, timing optimization, and caption generation.

## Related Tutorials

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional AI narration to silent footage

> **[TikTok Lyric Video](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Create engaging lyric videos with animated text overlays


# Overview
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/index

Automate video creation at scale - dubbing, voiceovers, AI-generated content, and more

## Content Factory

**Produce video content at scale without manual editing.** From AI voiceovers to multi-language dubbing, automate your video production pipeline.

### When to Use This

* You need to localize content into 10+ languages
* You want to create faceless YouTube/TikTok content at scale
* You're building an AI ad generator for e-commerce
* You need professional narration without hiring voice actors

### What You'll Build

> **[Faceless Videos](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Create complete videos from scripts - no camera needed

> **[AI Ad Films](/examples-and-tutorials/content-factory/ai-ad-films)**
>
> Generate product ads from text descriptions

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Auto-sync lyrics to music with animations

> **[Multi-Language Dubbing](/examples-and-tutorials/content-factory/dubbing)**
>
> Dub videos into any language with AI voices

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional narration to silent footage

> **[Trailer Narration](/examples-and-tutorials/content-factory/trailer-narration)**
>
> Generate cinematic trailer voices

> **[Voice Cloning](/examples-and-tutorials/content-factory/voice-cloning)**
>
> Clone voices for consistent brand audio

> **[Text-to-Video](/examples-and-tutorials/content-factory/text-prompts)**
>
> Generate video from text prompts

> **[Kids Storyteller](/examples-and-tutorials/content-factory/ai-storyteller-kids)**
>
> Create animated educational stories

> **[Year in Frames](/examples-and-tutorials/content-factory/year-in-frames)**
>
> Turn photo collections into video montages

## How It Works

```
Script/Text → AI Generation (Voice, Video, Images) → Timeline Composition → Stream/Export
```

## Related Documentation

> **[Generative Media](/pages/act/generative-media/index)**
>
> AI-powered audio and video generation

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> How programmatic video composition works

> **[Translation & Dubbing](/pages/act/generative-media/translation-and-dubbing)**
>
> Multi-language content localization

> **[Streams & Exports](/pages/act/output-and-delivery/streams-and-exports)**
>
> Output your generated content

## Explore Use Cases by Category

### Video Generation & Creation

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Generate complete videos from scripts

> **[AI Ad Films](/examples-and-tutorials/content-factory/ai-ad-films)**
>
> Auto-generate product ads from text

> **[Text-to-Video](/examples-and-tutorials/content-factory/text-prompts)**
>
> Create video directly from prompts

> **[Kids Storyteller](/examples-and-tutorials/content-factory/ai-storyteller-kids)**
>
> Generate animated educational stories

### Audio & Voice

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional narration automatically

> **[Multi-Language Dubbing](/examples-and-tutorials/content-factory/dubbing)**
>
> Dub videos into any language

> **[Voice Cloning](/examples-and-tutorials/content-factory/voice-cloning)**
>
> Clone voices for consistent audio

> **[Trailer Narration](/examples-and-tutorials/content-factory/trailer-narration)**
>
> Generate cinematic trailer voices

### Content Compilation

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Auto-sync lyrics with animations

> **[Year in Frames](/examples-and-tutorials/content-factory/year-in-frames)**
>
> Create video montages from photo collections


# Text to Video
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/text-prompts

Create video storyboards and content from text descriptions

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/GenAI_Storyboard.ipynb)

## Overview

Creating video storyboards for app user flows is traditionally a laborious process involving scriptwriting, recording voiceovers, designing frames, and editing them together.

**VideoDB** automates this entire pipeline.

In this tutorial, we will build a **Storyboard Generator Tool**.

1. **Input:** You provide an app name and a list of user steps.
2. **Process:** VideoDB's AI agents generate:
   * Step-by-step narration scripts (Text Gen)
   * Professional voiceovers (Voice Gen)
   * Concept art for each screen (Image Gen)
3. **Output:** A fully compiled video walkthrough with visual overlays and synced audio.

No external tools or complex integrations required.

[Generate Automated Video Outputs with Text Prompts](https://www.youtube.com/embed/de-5Y8xKp3Y)

## Setup

### Installing VideoDB

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

> [!NOTE]
>
> You only need your VideoDB API Key. Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

## Implementation

### Step 1: Connect to VideoDB

Connect to VideoDB using your API key to establish a session.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Set up the primary text inputs

While building an app, these input fields will be exposed to your users and this input will then become the foundation for the rest of this workflow.

For the purpose of this tutorial, we are using the sample use case of a user requesting a storyboard for their meditation app via the storyboarding tool that we're building.

```python Python theme={null}
# Define Your App Concept
app_description = "A meditation app for busy people with anxiety."

# Define the User Flow
raw_steps = [
    "Set up profile",
    "Select preference for theme & music",
    "Set meditation session timing",
    "Start the session"
]
```

```javascript Node.js theme={null}
// Define Your App Concept
const appDescription = "A meditation app for busy people with anxiety.";

// Define the User Flow
const rawSteps = [
    "Set up profile",
    "Select preference for theme & music",
    "Set meditation session timing",
    "Start the session"
];
```

### 🕹️ Step 3: Generate Assets

We will now iterate through each step of the user journey. For every step, we use VideoDB to:

1. **Write a Script:** Generate a short, conversational script based on the step name.
2. **Create Visuals:** Generate a sketch-style illustration of the user action.
3. **Synthesize Voice:** Turn the script into audio.

We store the resulting Asset IDs directly, skipping any manual file management.

```python Python theme={null}
import json

storyboard_assets = []

for i, step_name in enumerate(raw_steps):
    # 1. Generate Script using Text Generation
    # We ask for a short sentence.
    script_prompt = f"""
    Write a single conversational sentence for a video narration explaining the step: '{step_name}'
    for an app described as: '{app_description}'.
    Keep it encouraging and brief.
    """

    # Generate text
    text_response = coll.generate_text(prompt=script_prompt, model_name="pro")
    script_text = text_response["output"]

    # 2. Generate Voiceover
    audio_asset = coll.generate_voice(
        text=script_text,
        voice_name="Aria")

    # 3. Generate Image
    # We create a consistent art style prompt
    image_prompt = f"""
    A minimal, stippling black ballpoint pen illustration of a user interface or scene representing: '{step_name}'.
    Context: {app_description}.
    Clean white background, professional storyboard style.
    """

    image_asset = coll.generate_image(
        prompt=image_prompt)

    # Store everything we need for the timeline
    storyboard_assets.append({
        "step_name": step_name,
        "audio_id": audio_asset.id,
        "image_id": image_asset.id,
        "duration": float(audio_asset.length)
    })
```

```javascript Node.js theme={null}
const storyboardAssets = [];

for (const stepName of rawSteps) {
    // 1. Generate Script using Text Generation
    const scriptPrompt = `
    Write a single conversational sentence for a video narration explaining the step: '${stepName}'
    for an app described as: '${appDescription}'.
    Keep it encouraging and brief.
    `;

    // Generate text
    const textResponse = await coll.generateText(
        scriptPrompt,
        "pro"
    );
    const scriptText = textResponse.output;

    // 2. Generate Voiceover
    const audioAsset = await coll.generateVoice(
        scriptText,
        "Aria"
    );

    // 3. Generate Image
    const imagePrompt = `
    A minimal, stippling black ballpoint pen illustration of a user interface or scene representing: '${stepName}'.
    Context: ${appDescription}.
    Clean white background, professional storyboard style.
    `;

    const imageAsset = await coll.generateImage(imagePrompt);

    // Store everything we need for the timeline
    storyboardAssets.push({
        stepName,
        audioId: audioAsset.id,
        imageId: imageAsset.id,
        duration: parseFloat(audioAsset.length)
    });
}
```

### Step 4: Create the Timeline

Now we assemble the video. We will use:

* **Background:** A generic looping video to serve as a canvas.
* **Image Track:** The AI-generated sketches overlaid on the center.
* **Audio Track:** The generated voiceovers sequenced one after another.
* **Text Track:** A label at the bottom showing the current step name.

### Background Track

We use a stock video as a dynamic background.

```python Python theme={null}
coll = conn.get_collection()
base_vid = coll.upload(url="https://www.youtube.com/watch?v=4dW1ybhA5bM")
```

```javascript Node.js theme={null}
const coll = await conn.getCollection();
const baseVid = await coll.uploadURL({
    url: "https://www.youtube.com/watch?v=4dW1ybhA5bM"
});
```

```python Python theme={null}
from videodb.editor import (
    Timeline, Track, Clip,
    VideoAsset, ImageAsset, AudioAsset, TextAsset,
    Font, Background, Alignment, HorizontalAlignment, VerticalAlignment, Position)

# Initialize Timeline
timeline = Timeline(conn)

# Calculate total duration
total_duration = sum(item['duration'] for item in storyboard_assets)

# Create main track loop
main_track = Track()
video_asset = VideoAsset(id=base_vid.id)
video_clip = Clip(asset=video_asset, duration=total_duration)
main_track.add_clip(0, video_clip)
timeline.add_track(main_track)

# Setup Overlay Tracks
image_track = Track()
audio_track = Track()
text_track = Track()

current_time = 0

# Assemble the Sequence
for asset in storyboard_assets:
    duration = asset['duration']

    # A. Visual: The AI Sketch (Centered)
    image_clip = Clip(
        asset=ImageAsset(id=asset['image_id']),
        duration=duration,
        position=Position.center,)
    image_track.add_clip(current_time, image_clip)

    # B. Audio: The Voiceover
    audio_clip = Clip(
        asset=AudioAsset(id=asset['audio_id']),
        duration=duration)
    audio_track.add_clip(current_time, audio_clip)

    # C. Text: The Step Name Label
    text_clip = Clip(
        asset=TextAsset(
            text=asset['step_name'],
            font=Font(family="League Spartan", size=36, color="#FFFAFA"),
            background=Background(color="#FF4500", border_width=10, opacity=1.0),
            alignment=Alignment(horizontal=HorizontalAlignment.center, vertical=VerticalAlignment.bottom),),
        duration=duration,
        position=Position.bottom)
    text_track.add_clip(current_time, text_clip)

    # Advance the seeker
    current_time += duration

# Add all tracks to timeline
timeline.add_track(image_track)
timeline.add_track(audio_track)
timeline.add_track(text_track)
```

```javascript Node.js theme={null}
import {
  EditorTimeline, Track, Clip,
  EditorVideoAsset, EditorImageAsset, EditorAudioAsset, EditorTextAsset,
  Font, Background, Alignment, HorizontalAlignment, VerticalAlignment, Position
} from 'videodb';

// Initialize Timeline
const timeline = new EditorTimeline(conn);

// Calculate total duration
const totalDuration = storyboardAssets.reduce((sum, item) => sum + item.duration, 0);

// Create main track loop
const mainTrack = new Track();
const videoAsset = new EditorVideoAsset({ id: baseVid.id });
const videoClip = new Clip({ asset: videoAsset, duration: totalDuration });
mainTrack.addClip(0, videoClip);
timeline.addTrack(mainTrack);

// Setup Overlay Tracks
const imageTrack = new Track();
const audioTrack = new Track();
const textTrack = new Track();

let currentTime = 0;

// Assemble the Sequence
for (const asset of storyboardAssets) {
    const duration = asset.duration;

    // A. Visual: The AI Sketch (Centered)
    const imageClip = new Clip({
        asset: new EditorImageAsset({ id: asset.imageId }),
        duration,
        position: Position.center
    });
    imageTrack.addClip(currentTime, imageClip);

    // B. Audio: The Voiceover
    const audioClip = new Clip({
        asset: new EditorAudioAsset({ id: asset.audioId }),
        duration
    });
    audioTrack.addClip(currentTime, audioClip);

    // C. Text: The Step Name Label
    const textClip = new Clip({
        asset: new EditorTextAsset({
            text: asset.stepName,
            font: new Font({
                family: "League Spartan",
                size: 36,
                color: "#FFFAFA"
            }),
            background: new Background({
                color: "#FF4500",
                borderWidth: 10,
                opacity: 1.0
            }),
            alignment: new Alignment({
                horizontal: HorizontalAlignment.center,
                vertical: VerticalAlignment.bottom
            })
        }),
        duration,
        position: Position.bottom
    });
    textTrack.addClip(currentTime, textClip);

    // Advance the seeker
    currentTime += duration;
}

// Add all tracks to timeline
timeline.addTrack(imageTrack);
timeline.addTrack(audioTrack);
timeline.addTrack(textTrack);
```

### Step 5: Watch the Storyboard

Generate the stream and view your automated video creation.

```python Python theme={null}
from videodb import play_stream

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

[Storyboard Video Output](https://www.youtube.com/embed/20iako1LzGo)

### Conclusion

You have successfully built a **Generative AI Storyboard Tool** in under 50 lines of logic.

You can now expand this to generate marketing videos, tutorials, or dynamic social media content instantly.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/GenAI_Storyboard.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Generate complete videos from scripts

> **[AI Ad Films](/examples-and-tutorials/content-factory/ai-ad-films)**
>
> Auto-generate product ads from text descriptions


# TikTok Style Lyric Video Creator
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/tiktok-lyric-video

Transform full-length music videos into viral 15-60 second clips with AI-generated backgrounds and synced lyrics

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/tiktok_style_lyric_video_creator.ipynb)

## The Challenge

You have a beautiful music video, but it's 4 minutes long. Social media demands are different — TikTok, Reels, and Shorts want 15-60 second vertical clips that stop people mid-scroll.

Manually identifying the best segment, extracting it, generating backgrounds, adding lyrics, and exporting for vertical format is tedious and time-consuming.

What if AI could do all of that for you?

## What You'll Build

Turn any music video into viral-ready vertical clips optimized for social media. This system automates the complete workflow:

* AI identifies the catchiest segments (chorus with build-up)
* Generates vertical 9:16 backgrounds for mobile viewing
* Creates mobile-optimized captions with high contrast
* Adds pre-roll, CTA, and post-roll for professional polish
* Outputs 15-60 second clips ready to upload

All powered by **VideoDB's Editor SDK** — built for social media from the ground up.

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Upload Your Music Video

```python theme={null}
# Upload from URL
video = coll.upload(url="https://www.youtube.com/watch?v=gtnD0eCcCC8")
```

### Step 2: Create a Timed Transcript Artifact

Create a timed transcript artifact with server-provided word text and timestamp ranges. These timed words support lyric planning while keeping timing aligned with the source video.

```python theme={null}
print("Creating timed transcript artifact... this might take a moment.")

transcript_understanding = video.understand(
    analyzers=[
        {
            "type": "spoken_words",
            "name": "transcript",
        },
    ],
)
transcript_understanding.wait_until_complete()

transcript_analyzer = transcript_understanding.get_analyzer("transcript")
```

### Prepare Timed Transcript Words

Use the transcript artifact to prepare timed words for lyric alignment and plain transcript text for song-structure analysis.

```python theme={null}
from math import isfinite

transcript_output = transcript_analyzer.get_output()
raw_entries = []
transcript_parts = []
source_order = 0

for scene in transcript_output.get("scenes", []):
    data = scene.get("data", {})

    scene_text = data.get("text")
    if isinstance(scene_text, str) and scene_text.strip():
        transcript_parts.append(scene_text.strip())

    for word in data.get("words", []):
        text = word.get("text")
        normalized_text = text.strip() if isinstance(text, str) else ""
        silence_marker = normalized_text.strip("[]()<>").strip().casefold()
        start = word.get("start")
        end = word.get("end")

        if (
            normalized_text
            and silence_marker not in {"-", "silence"}
            and isinstance(start, (int, float))
            and not isinstance(start, bool)
            and isinstance(end, (int, float))
            and not isinstance(end, bool)
            and isfinite(start)
            and isfinite(end)
            and 0 <= start < end
        ):
            raw_entries.append(
                {
                    "start": start,
                    "end": end,
                    "text": normalized_text,
                    "source_order": source_order,
                }
            )

        source_order += 1

raw_entries.sort(
    key=lambda entry: (entry["start"], entry["end"], entry["source_order"])
)

transcript_timed = [
    {
        "id": f"entry_{index:04d}",
        "start": entry["start"],
        "end": entry["end"],
        "text": entry["text"],
    }
    for index, entry in enumerate(raw_entries, start=1)
]

transcript_entries_by_id = {entry["id"]: entry for entry in transcript_timed}
transcript_entry_positions = {
    entry["id"]: index for index, entry in enumerate(transcript_timed)
}

transcript_text = "\n".join(transcript_parts)
if not transcript_text:
    transcript_text = " ".join(entry["text"] for entry in transcript_timed)

print(f"Prepared {len(transcript_timed)} timed transcript words.")
print(transcript_timed[:3])
print(transcript_text)
```

### Step 3: Identify the Best Segment

The AI analyzes the transcript to find the catchiest chorus with proper build-up and wind-down:

```python theme={null}
import json

# Define visual aesthetic
user_request = "Calm and serene vibe. Ocean, beach, sunsets and peace"

prompt = f"""
You are a viral social media content strategist specializing in short-form vertical video. Your task is to identify the most engaging, catchy, and shareable segments from a music video to create TikTok/Reels/Shorts content.

# INPUT DATA

1. Full Transcript Text (may contain ASR errors/missing punctuation):
{transcript_text}

2. Local Transcript Entries (each entry has id/start/end/text; use IDs to select contiguous lyric ranges):
{json.dumps(transcript_timed, ensure_ascii=False)}

3. Video Metadata:
   - Name: {video.name}
   - Duration: {video.length} seconds
   - User Style Request: {user_request}

# SEGMENT SELECTION CRITERIA

Identify **1-3 high-quality segments** (prioritize quality over quantity). Each segment should:

## What Makes a Segment "Catchy"?

**PRIMARY FOCUS: Chorus/Hook with Proper Framing**

Segments MUST be built around the chorus with these components:
1. **Build-up (Pre-Chorus/Verse End)**: 2-4 lines leading into the chorus that create anticipation
2. **The Chorus/Hook**: The main catchy, repetitive, memorable section
3. **Wind-down (Post-Chorus)**: 2-4 lines after the chorus that provide resolution

The segment structure should be: **Ramp Up → Peak (Chorus) → Ramp Down**

Additional qualities that enhance a chorus segment:
- **Emotional Peak**: Most intense, emotionally charged moment (climax, drop, powerful vocals)
- **Quotable Lines**: Lyrics that are relatable, funny, profound, or highly shareable
- **Viral Potential**: Lyrics that could inspire trends, dances, duets, or memes
- **Energy Shift**: Dramatic beat drop, tempo change, or dynamic transition in/around the chorus
- **Memorable Moment**: Distinctive vocal run, ad-lib, or production element that makes the chorus special

**Non-Negotiable Rules:**
- The chorus is the centerpiece - always include it
- Never start directly on the first chorus line - include the approach
- Never end directly after the last chorus line - include the exit
- Build-up and wind-down are MANDATORY for professional feel

## Segment Requirements
- **Compactness**: Prefer naturally concise selections suitable for short-form viewing
- **Focus on Chorus**: Segments should CENTER around the chorus/hook - this is the primary target
- **Build-up REQUIRED**: MUST include the build-up/lead-in before the chorus (last 2-4 lines of pre-chorus or verse)
- **Wind-down REQUIRED**: MUST include the wind-down/resolution after the chorus (first 2-4 lines after chorus ends)
- **No Abrupt Starts**: Never start directly on the main chorus line - include runway space before it
- **No Abrupt Ends**: Never cut immediately when the chorus ends - include graceful exit
- **Ramp Up & Ramp Down**: The segment should feel like a complete emotional arc with natural entry and exit
- **Completeness**: Each segment should feel like a complete mini-story with beginning, climax (chorus), and resolution
- **Standalone Quality**: Segment must work independently and feel professionally edited, not chopped
- **Quality Over Quantity**: If only 1 truly excellent segment exists, return only that one

## Lyric Processing

### Text Handling
- Lightly fix obvious ASR errors ONLY when strongly implied by context
- Do NOT invent new lyrics beyond what is clearly present
- Merge words into natural phrases suitable for vertical video display
- Target 3-8 words per line (shorter than full videos due to vertical format)
- Split long lines for mobile readability

### Transcript Selection
- Select whole-word, contiguous ranges by `first_id` and `last_id`.
- Do not return text, numeric timestamps, stanza bounds, segment bounds, or duration.
- All rendered text and timing are derived locally from the selected IDs.

# IMAGE GENERATION REQUIREMENTS

For EACH stanza within EACH segment, create an "image_prompt" following these rules:

## Short-Form Specific Considerations
- **Vertical aspect ratio**: 9:16 (portrait orientation for mobile)
- **Mobile-first design**: Images should look compelling on small phone screens
- **Attention-grabbing**: Short-form content needs more visual impact than full videos
- **Text readability**: Even more critical in vertical format with limited screen space

## Independence & Self-Containment
- Each prompt must be FULLY SELF-CONTAINED with complete scene descriptions
- NEVER use references like "previous image", "same as before", "similar to", or "continue from"
- Each prompt must work standalone if generated in isolation

## Content & Style
- Honor the user's style request: {user_request}
- Reflect the mood, theme, and emotional tone of the current stanza
- Create visual progression within each segment
- **CRITICAL**: Prompts must NEVER include text, lyrics, words, letters, names, or any written language in the generated image

## Composition for Vertical Video Text Overlay
- **Vertical framing**: Design for 9:16 portrait orientation
- Keep **center vertical strip** relatively clear for text overlay
- Text typically appears in middle-to-upper-middle area on mobile
- Avoid busy details in the central vertical zone
- Can have more detail at top/bottom edges where text is less likely
- Use balanced, mobile-optimized compositions
- Images should be visually striking but NOT overly distracting

## Visual Consistency Within Each Segment
Each segment should have internal visual consistency:
- **Color palette**: Harmonious colors within the segment
- **Artistic style**: Consistent style throughout the segment
- **Mood/atmosphere**: Unified emotional tone
- **Composition approach**: Similar framing strategy

## Prompt Structure
Each image_prompt should specify:
1. Scene/subject matter relevant to the stanza
2. Emotional mood and atmosphere matching the segment's energy
3. Lighting conditions and color tone
4. Artistic style (aligned with user_request)
5. **Vertical composition notes** (e.g., "portrait orientation, centered vertical negative space, detailed top and bottom thirds")
6. Mobile-optimized visual impact

# FONT COLOR SELECTION

For EACH stanza, select a "font_color" ensuring optimal mobile readability:

## Available Colors
Choose from this list ONLY:
- White: #FFFFFF (for dark backgrounds)
- Black: #000000 (for light backgrounds)
- Yellow: #FFFF00 (for dark backgrounds, extremely high visibility on mobile)
- Dark Blue: #00008B (for light backgrounds)
- Dark Green: #006400 (for light backgrounds)

## Selection Guidelines
- **Light/bright backgrounds** → Black (#000000), Dark Blue (#00008B), or Dark Green (#006400)
- **Dark/dim backgrounds** → White (#FFFFFF) or Yellow (#FFFF00)
- **Mobile priority**: Ensure MAXIMUM CONTRAST for small screen readability
- Consider that users often view in bright outdoor lighting or dim rooms
- Yellow (#FFFF00) works exceptionally well for high-energy viral content on dark backgrounds

# SEGMENT METADATA

For each segment, provide:

## segment_title
- A catchy, descriptive title for the segment (3-6 words)
- Should hint at what makes this segment special
- Examples: "Explosive Chorus Drop", "Emotional Bridge Moment", "Viral Hook Section"

# OUTPUT FORMAT

Return ONLY a valid JSON array with this exact structure:

[
  {{
    "segment_title": "string (catchy 3-6 word title)",
    "why_catchy": "string (1-2 sentence explanation of what makes this segment viral-worthy)",
    "stanzas": [
      {{
        "image_prompt": "string (detailed, self-contained, vertical 9:16 image generation prompt)",
        "font_color": "string (hex code from approved list)",
        "lines": [
          {{
            "first_id": "entry_0001",
            "last_id": "entry_0004"
          }}
        ]
      }}
    ]
  }}
]

# CRITICAL REMINDERS

1. **CHORUS-CENTERED STRUCTURE**: Every segment must be built around a chorus with proper build-up and wind-down
2. **NO ABRUPT STARTS**: Always include 2-4 lines BEFORE the chorus begins (pre-chorus/verse ending)
3. **NO ABRUPT ENDS**: Always include 2-4 lines AFTER the chorus ends (post-chorus beginning)
4. **PROFESSIONAL RAMP UP/DOWN**: The segment should feel like a complete emotional journey, not a choppy cut
5. **Quality over quantity**: 1 amazing segment > 3 mediocre ones
6. **Each image_prompt is independent**: No cross-references between prompts
7. **Vertical format**: All image prompts must specify 9:16 portrait orientation
8. **No text in images**: Never include lyrics, words, or letters in image generation prompts
9. **ID ranges only**: Return contiguous first_id/last_id ranges; local code derives all text and timing
10. **Mobile-first**: Optimize everything for small screen viewing
11. **Honor style request**: All creative decisions must align with: {user_request}
12. Each text line cannot exceed beyond 20 characters [ Highly Important ]

Output the JSON array only, no additional text.
"""

stanzas = coll.generate_text(
    prompt=prompt,
    model_name="pro",
    response_type="json",
)

print(stanzas)

if not isinstance(stanzas, dict) or not isinstance(stanzas.get("output"), list):
    raise ValueError("Expected a JSON segment plan.")

segments = []
for planned_segment in stanzas["output"]:
    if not isinstance(planned_segment, dict) or not isinstance(planned_segment.get("stanzas"), list):
        raise ValueError("Each segment must contain stanzas.")
    rehydrated_stanzas = []
    for planned_stanza in planned_segment["stanzas"]:
        if not isinstance(planned_stanza, dict) or not isinstance(planned_stanza.get("lines"), list):
            raise ValueError("Each stanza must contain ID-based lines.")
        lines = []
        for planned_line in planned_stanza["lines"]:
            if not isinstance(planned_line, dict):
                raise ValueError("Each lyric line must be an object.")
            first_position = transcript_entry_positions[planned_line["first_id"]]
            last_position = transcript_entry_positions[planned_line["last_id"]]
            if first_position > last_position:
                raise ValueError("Lyric IDs must form a contiguous forward range.")
            for entry in transcript_timed[first_position:last_position + 1]:
                lines.append({"text": entry["text"], "start": entry["start"], "end": entry["end"]})
        if lines:
            rehydrated_stanzas.append({
                "stanza_start": lines[0]["start"],
                "stanza_end": lines[-1]["end"],
                "image_prompt": planned_stanza["image_prompt"],
                "font_color": planned_stanza["font_color"],
                "lines": lines,
            })
    if rehydrated_stanzas:
        segments.append({
            "segment_title": planned_segment["segment_title"],
            "start_time": rehydrated_stanzas[0]["stanza_start"],
            "end_time": rehydrated_stanzas[-1]["stanza_end"],
            "duration": rehydrated_stanzas[-1]["stanza_end"] - rehydrated_stanzas[0]["stanza_start"],
            "why_catchy": planned_segment["why_catchy"],
            "stanzas": rehydrated_stanzas,
        })
```

### Step 4: Generate Vertical Background Images with AI

Create AI-generated background images for each lyric stanza:

```python theme={null}
from concurrent.futures import ThreadPoolExecutor, as_completed

# Helper function to optimize segment stanzas
def optimize_segment_stanzas(stanzas):
    if not stanzas:
        return []
    return [
        {**stanza, "lines": [dict(line) for line in stanza["lines"]]}
        for stanza in stanzas
    ]

# Prepare image generation tasks
image_tasks = []
for seg_idx, segment in enumerate(segments):
    segment["optimized_stanzas"] = optimize_segment_stanzas(segment["stanzas"])

    for stan_idx, s in enumerate(segment["optimized_stanzas"]):
        image_tasks.append({
            "seg_idx": seg_idx,
            "stan_idx": stan_idx,
            "prompt": s["image_prompt"]
        })

def generate_worker(task):
    img = coll.generate_image(prompt=task["prompt"], aspect_ratio="9:16")
    return task["seg_idx"], task["stan_idx"], img.id

# Generate images in parallel
MAX_WORKERS = 10
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
    futures = [executor.submit(generate_worker, task) for task in image_tasks]

    for future in as_completed(futures):
        seg_i, stan_i, img_id = future.result()
        segments[seg_i]["optimized_stanzas"][stan_i]["image_id"] = img_id

```

### Step 5: Build Multi-Layer Timeline

Create a timeline with audio, backgrounds, and animated lyrics:

```python theme={null}
import math
from videodb.editor import (
    VideoAsset, ImageAsset, TextAsset, Font,
    Clip, Track, Timeline, Transition,
    Position, Fit, Offset, Alignment, HorizontalAlignment, VerticalAlignment, Border
)

def calculate_y_offsets(num_lines, line_height=45, timeline_height=1080):
    offsets = []
    start_pixel_offset = -((num_lines - 1) * line_height) / 2
    half_height = timeline_height / 2
    for i in range(num_lines):
        pixel_y = start_pixel_offset + (i * line_height)
        offsets.append(pixel_y / half_height)
    return offsets

# Timeline configuration
TIMELINE_WIDTH = 608
TIMELINE_HEIGHT = 1080
PRE_ROLL = 5.0
POST_ROLL = 5.0
CTA_DURATION = 5.0

# Render each segment
for idx, segment in enumerate(segments):
    # Calculate timing
    actual_seg_start = segment["start_time"]
    actual_seg_end = segment["end_time"]

    timeline_start_og = max(0, actual_seg_start - PRE_ROLL)
    intro_duration = actual_seg_start - timeline_start_og
    total_audio_duration = (actual_seg_end - timeline_start_og) + CTA_DURATION + POST_ROLL

    stanza_items = segment["optimized_stanzas"]

    # Initialize timeline
    timeline = Timeline(conn)
    timeline.resolution = f"{TIMELINE_WIDTH}x{TIMELINE_HEIGHT}"
    timeline.background = "#000000"

    # Layer 1: Audio Track (opacity=0 for audio only)
    audio_track = Track(z_index=0)
    audio_track.add_clip(
        start=0.0,
        clip=Clip(
            asset=VideoAsset(id=video.id, start=timeline_start_og, volume=1.0),
            duration=total_audio_duration,
            fit=Fit.crop,
            position=Position.center,
            opacity=0.0,
            transition=Transition(in_="fade", out="fade", duration=5.0)
        ),
    )
    timeline.add_track(audio_track)

    # Layer 2: Background Images Track
    images_track = Track(z_index=1)
    for i, s in enumerate(stanza_items):
        local_start = s["stanza_start"] - timeline_start_og
        local_end = s["stanza_end"] - timeline_start_og

        if i == 0:
            local_start = 0.0
            trans_in = Transition(in_="fade", duration=intro_duration)
        else:
            trans_in = Transition(in_="fade", out="fade", duration=0.35)

        duration = max(0.1, local_end - local_start)

        images_track.add_clip(
            start=local_start,
            clip=Clip(
                asset=ImageAsset(id=s["image_id"]),
                duration=duration,
                fit=Fit.crop,
                position=Position.center,
                transition=trans_in
            ),
        )
    timeline.add_track(images_track)

    # Layer 3: Lyrics Track
    lyrics_track = Track(z_index=2)
    LINE_HEIGHT = 45
    lyrics_border = Border(color="#000000", width=1.5)

    for s in stanza_items:
        lines = s.get("lines", [])
        y_offsets = calculate_y_offsets(len(lines), LINE_HEIGHT, TIMELINE_HEIGHT)
        local_stanza_end = s["stanza_end"] - timeline_start_og

        for l_idx, line in enumerate(lines):
            local_line_start = line["start"] - timeline_start_og
            line_duration = max(0.1, local_stanza_end - local_line_start)

            text_asset = TextAsset(
                text=line["text"],
                font=Font(family="Bebas Neue", size=48, color="#FFFFFF"),
                border=lyrics_border,
                alignment=Alignment(horizontal=HorizontalAlignment.center, vertical=VerticalAlignment.center)
            )

            lyrics_track.add_clip(
                start=local_line_start,
                clip=Clip(
                    asset=text_asset,
                    duration=line_duration,
                    position=Position.center,
                    offset=Offset(x=0, y=y_offsets[l_idx]),
                    transition=Transition(in_="fade", duration=0.5)
                )
            )

    # Layer 4: CTA (Call to Action)
    cta_start_time = (actual_seg_end - timeline_start_og)
    cta_lines = ["Visit the channel", "for the full video"]
    cta_y_offsets = calculate_y_offsets(len(cta_lines), line_height=60, timeline_height=TIMELINE_HEIGHT)

    for c_idx, cta_text in enumerate(cta_lines):
        cta_asset = TextAsset(
            text=cta_text,
            font=Font(family="Bebas Neue", size=42, color="#FFFFFF"),
            border=lyrics_border,
            alignment=Alignment(horizontal=HorizontalAlignment.center, vertical=VerticalAlignment.center)
        )

        lyrics_track.add_clip(
            start=cta_start_time,
            clip=Clip(
                asset=cta_asset,
                duration=CTA_DURATION,
                position=Position.center,
                offset=Offset(x=0, y=cta_y_offsets[c_idx]),
                transition=Transition(in_="fade", duration=0.5)
            )
        )

    timeline.add_track(lyrics_track)

    # Generate final stream
    stream_url = timeline.generate_stream()
```

## What You Get

A professional-looking short-form video ready for social media:

* Perfectly timed to the music
* AI-generated matching visuals
* Synced lyrics on screen
* Professional pre-roll and CTA
* Vertical format (608x1080)
* Ready to upload to TikTok, Reels, or Shorts

Here's the final rendered video:

[TikTok Style Lyric Video Output](https://console.videodb.io/player?url=https://d1zudc7ewmc6ey.cloudfront.net/v1/72fc1788-4b5d-4606-ba55-aa7cf0d0a60e.m3u8)

## The Result

With this system, you can:

* Process music videos in minutes instead of hours
* Generate unlimited vertical clips from one source video
* Maintain consistency across all clips with the same visual style
* Stay on top of social media trends with fast production

No more manual editing. Just music, lyrics, AI-generated visuals, and viral potential.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/tiktok_style_lyric_video_creator.ipynb)**
>
> Open the complete implementation with parallel processing, custom effects, and advanced styling options.

## Related Tutorials

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Build complete faceless videos with AI scripts and voiceovers

> **[Year in Frames](/examples-and-tutorials/content-factory/year-in-frames)**
>
> Create personalized year recap videos from photo collections


# Trailer Narration
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/trailer-narration

Create dramatic trailers with automated narration

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Automated_Trailer_Voiceover.ipynb)

## Introduction

Narration is the heartbeat of trailers, injecting excitement and intrigue into every frame. With **VideoDB**, adding narration becomes a seamless, creative process.

In this tutorial, we will:

1. **Analyze** a movie trailer to understand its scenes.
2. **Generate** a dramatic script using VideoDB's text generation.
3. **Synthesize** a deep, trailer-style voiceover using VideoDB's voice generation.
4. **Edit** the narration into specific time slots to match the video's pacing.
5. **Overlay** a movie poster at the end.

All using a single SDK.

Here's an example of weaving a thrilling storyline from a reel of unrelated, but valuable cinematic shots:

[YouTube video player](https://www.youtube.com/embed/sldlSP_aSmQ)

## Setup

### Installing VideoDB

```
!pip install videodb
```

### API Keys

> [!NOTE]
>
> You only need your VideoDB API Key. Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

## Step 1: Connect to VideoDB

Establish a connection to your VideoDB project.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

// Connect to VideoDB
const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

## Step 2: Upload the Trailer

We'll upload a sample movie trailer (Chase) to VideoDB. This creates the base video asset we will edit.

```python Python theme={null}
video = coll.upload(url='https://www.youtube.com/watch?v=WQmGwmc-XUY')
```

```javascript Node.js theme={null}
const video = await coll.uploadURL({ url: 'https://www.youtube.com/watch?v=WQmGwmc-XUY' });
```

## Step 3: Analyze 10-Second Scenes

Divide the trailer into 10-second visual segments so each narration line can respond to what is on screen and fit inside its own time slot.

```python Python theme={null}
print("Analyzing trailer scenes... this might take a moment.")

scene_understanding = video.understand(
    segmentation={"type": "time", "seconds": 10},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {
                "prompt": "Describe the key action, mood, and story beat in this trailer segment.",
                "schema": {
                    "description": "string",
                },
            },
        },
    ],
)
scene_understanding.wait_until_complete()

scene_analyzer = scene_understanding.get_analyzer("scene")
```

```javascript Node.js theme={null}
const sceneUnderstanding = await video.understand(
    [
        {
            type: "vlm",
            name: "scene",
            config: {
                prompt: "Describe the key action, mood, and story beat in this trailer segment.",
                schema: {
                    description: "string",
                },
            },
        },
    ],
    { segmentation: { type: "time", seconds: 10 } }
);
await sceneUnderstanding.waitUntilComplete();

const sceneAnalyzer = await sceneUnderstanding.getAnalyzer("scene");
```

Let's view the description of the first scene from the video.

```python Python theme={null}
scene_output = scene_analyzer.get_output()

video_scenes = [
    {
        "start": scene["start"],
        "end": scene["end"],
        "description": scene["data"]["description"],
    }
    for scene in scene_output["scenes"]
]

import json
print(json.dumps(video_scenes[0], indent=2))
```

```javascript Node.js theme={null}
const sceneOutput = await sceneAnalyzer.getOutput();
const videoScenes = sceneOutput.scenes.map((scene) => ({
    start: scene.start,
    end: scene.end,
    description: scene.data.description,
}));

console.log(JSON.stringify(videoScenes[0], null, 2));
```

## Step 4: Generate Narration Script

We'll use VideoDB's `generate_text` to write a dramatic script. We feed the scene descriptions into the prompt to ensure the narration matches the visual action.

```python Python theme={null}
scene_context = json.dumps(video_scenes, indent=2)

prompt = f"""
Craft a gripping, dramatic movie-trailer narration that weaves a compelling story through these timed visual segments:
{scene_context}

Requirements:
- Style: Intense, gritty, like a blockbuster action movie trailer.
- Return exactly {len(video_scenes)} voiceovers, one per segment, in the same order.
- Each voiceover must be a single impactful sentence of 10 to 12 words, building a continuous, dramatic narrative.
- Crucially, do not simply describe the on-screen action; instead, interpret and advance the overarching plot dramatically.
- Do not include stage directions, timestamps, labels, or sound cues.

Return only JSON in this exact structure:
{{
  "voiceovers": [
    {{"text": "Short narration for segment 1"}},
    {{"text": "Short narration for segment 2"}}
  ]
}}
"""

script_response = coll.generate_text(
    prompt=prompt,
    model_name="pro",
    response_type="json",
)

voiceover_lines = script_response["output"]["voiceovers"]
narration_plan = [
    {
        "start": scene["start"],
        "end": scene["end"],
        "text": voiceover_lines[index]["text"],
    }
    for index, scene in enumerate(video_scenes)
]

print("--- Narration Plan ---")
for line in narration_plan:
    print(f"{line['start']:.1f}s: {line['text']}")
```

```javascript Node.js theme={null}
const sceneContext = JSON.stringify(videoScenes, null, 2);

const prompt = `
Craft a gripping, dramatic movie-trailer narration that weaves a compelling story through these timed visual segments:
${sceneContext}

Requirements:
- Style: Intense, gritty, like a blockbuster action movie trailer.
- Return exactly ${videoScenes.length} voiceovers, one per segment, in the same order.
- Each voiceover must be a single impactful sentence of 10 to 12 words, building a continuous, dramatic narrative.
- Crucially, do not simply describe the on-screen action; instead, interpret and advance the overarching plot dramatically.
- Do not include stage directions, timestamps, labels, or sound cues.

Return only JSON in this exact structure:
{
  "voiceovers": [
    {"text": "Short narration for segment 1"},
    {"text": "Short narration for segment 2"}
  ]
}
`;

const scriptResult = await coll.generateText(
    prompt,
    "pro",
    "json"
);

const scriptOutput = typeof scriptResult === "string"
    ? scriptResult
    : scriptResult.output;
const scriptResponse = typeof scriptOutput === "string"
    ? JSON.parse(scriptOutput)
    : scriptOutput;
const voiceoverLines = scriptResponse?.voiceovers;

if (
    !Array.isArray(voiceoverLines)
    || voiceoverLines.length !== videoScenes.length
    || !voiceoverLines.every((line) => line && typeof line.text === "string")
) {
    throw new Error("Expected voiceovers in the generated narration plan.");
}
const narrationPlan = videoScenes.map((scene, index) => ({
    start: scene.start,
    end: scene.end,
    text: voiceoverLines[index].text,
}));

console.log("--- Narration Plan ---");
narrationPlan.forEach((line) => console.log(`${line.start.toFixed(1)}s: ${line.text}`));
```

You can refine the narration script prompt to ensure synchronization with the scene descriptions, optimizing the storytelling experience.

## Step 5: Generate Voiceover Audio

Now we synthesize the audio.

```python Python theme={null}
voiceovers = []

for line in narration_plan:
    audio = coll.generate_voice(
        text=line["text"],
        voice_name="Brian",
    )
    voiceovers.append({**line, "audio": audio})
    print(
        f"Generated {float(audio.length):.1f}s voiceover "
        f"for {line['start']:.1f}s: {audio.id}"
    )
```

```javascript Node.js theme={null}
import { Audio, GenerationJob } from 'videodb';

const voiceovers = [];

for (const line of narrationPlan) {
    const audioCandidate = await coll.generateVoice(
        line.text,
        "Brian"
    );
    const audio = audioCandidate instanceof GenerationJob
        ? await audioCandidate.wait()
        : audioCandidate;

    if (!(audio instanceof Audio)) {
        throw new Error("Expected generated voice audio.");
    }

    voiceovers.push({ ...line, audio });
    console.log(`Generated ${Number(audio.length).toFixed(1)}s voiceover for ${line.start.toFixed(1)}s: ${audio.id}`);
}
```

## Step 6: Edit the Timeline

Now we combine the video and audio.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset, AudioAsset

timeline = Timeline(conn)

# --- 1. Main Video Track ---
main_track = Track()
video_asset = VideoAsset(id=video.id, volume=0.5)
video_clip = Clip(asset=video_asset, duration=float(video.length))
main_track.add_clip(0, video_clip)
timeline.add_track(main_track)

# --- 2. Scene-Aware Narration Track ---
audio_track = Track()

for voiceover in voiceovers:
    audio = voiceover["audio"]
    segment_duration = float(voiceover["end"] - voiceover["start"])
    clip_duration = min(float(audio.length), segment_duration)

    audio_asset = AudioAsset(id=audio.id, start=0, volume=2.0)
    audio_clip = Clip(asset=audio_asset, duration=clip_duration)
    audio_track.add_clip(float(voiceover["start"]), audio_clip)

timeline.add_track(audio_track)
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset, EditorAudioAsset } from 'videodb';

const timeline = new EditorTimeline(conn);

// --- 1. Main Video Track ---
const mainTrack = new Track();
const videoAsset = new EditorVideoAsset({ id: video.id, volume: 0.5 });
const videoClip = new Clip({ asset: videoAsset, duration: parseFloat(video.length) });
mainTrack.addClip(0, videoClip);
timeline.addTrack(mainTrack);

// --- 2. Scene-Aware Narration Track ---
const audioTrack = new Track();

for (const voiceover of voiceovers) {
    const segmentDuration = Number(voiceover.end - voiceover.start);
    const clipDuration = Math.min(Number(voiceover.audio.length), segmentDuration);
    const audioAsset = new EditorAudioAsset({ id: voiceover.audio.id, start: 0, volume: 2.0 });
    const audioClip = new Clip({ asset: audioAsset, duration: clipDuration });
    audioTrack.addClip(Number(voiceover.start), audioClip);
}

timeline.addTrack(audioTrack);
```

## Step 7: Review and Share

Preview the trailer with the integrated narration to ensure it aligns with your vision. Once satisfied, share the trailer with others to experience the enhanced storytelling.

```python Python theme={null}
from videodb import play_stream

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

## Bonus - Add Movie Poster

Let's add a "Coming Soon" style movie poster at the very end of the trailer. We'll upload an image URL and overlay it on the video.

```python Python theme={null}
from videodb import MediaType

# Upload movie poster
poster_url = "https://img.freepik.com/free-photo/darkly-atmospheric-retail-environment-rendering_23-2151153755.jpg"
image = coll.upload(url=poster_url, media_type=MediaType.image)
```

```javascript Node.js theme={null}
import { MediaType } from 'videodb';

// Upload movie poster
const posterUrl = "https://img.freepik.com/free-photo/darkly-atmospheric-retail-environment-rendering_23-2151153755.jpg";
const image = await coll.uploadURL({ url: posterUrl, mediaType: MediaType.image });
```

```python Python theme={null}
from videodb.editor import ImageAsset

# Create an overlay track for the poster
image_track = Track()

# Show poster at the 10 seconds of the trailer
image_asset = ImageAsset(id=image.id)
image_clip = Clip(
    asset=image_asset,
    duration=10.0,
    fit="contain")

image_track.add_clip(float(video.length) - 10, image_clip)
timeline.add_track(image_track)

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
import { EditorImageAsset } from 'videodb';

// Create an overlay track for the poster
const imageTrack = new Track();

// Show poster at the 10 seconds of the trailer
const imageAsset = new EditorImageAsset({ id: image.id });
const imageClip = new Clip({
    asset: imageAsset,
    duration: 10.0,
    fit: "contain"
});

imageTrack.addClip(parseFloat(video.length) - 10, imageClip);
timeline.addTrack(imageTrack);

const finalStreamUrl = await timeline.generateStream();
console.log(finalStreamUrl);
```

## Conclusion

You’ve successfully built a sophisticated video editing workflow:

* **Analysis:** Automated scene understanding.
* **Generation:** AI Scripting and Voice synthesis.
* **Composition:** Non-linear editing with multi-track audio and image overlays.

[Enhanced Trailer with Narration](https://www.youtube.com/embed/KMqtcMzOmW8)

Here's another interesting experiment to generate automatic voiceovers for silent footages. In this example, we've added the classic David Attenborough styled documentary narration to this footage of the underwater world. Check out the complete tutorial [here](/examples-and-tutorials/content-factory/voiceovers).

[David Attenborough Style Documentary Narration](https://www.youtube.com/embed/gsU14KgORgg)

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Automated_Trailer_Voiceover.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional narration to silent footage

> **[Voice Cloning](/examples-and-tutorials/content-factory/voice-cloning)**
>
> Clone voices for consistent brand audio


# Voice Cloning
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/voice-cloning

Replace audio with cloned voices for consistent narration

Imagine watching or sharing your videos, but instead of the original low-quality audio, you hear your voice rendered in crystal-clear, studio-quality sound.

A cloned voice can breathe new life into your videos and in this blog, I'll show you how **to build a Voice Replacement Agent in under 1 hour** using Director.

> **[VideoDB Director GitHub](https://github.com/video-db/Director)**
>
> Explore the complete Director framework and see real-world agent implementations

## Game Plan for Voice Replacement Agent

The steps are fairly simple:

1. User uploads voice samples (to clone) and a video (whose voice needs to be replaced).
2. Clone the voice from the provided samples.
3. Extract the transcript from the video.
4. Generate an audio clip in the cloned voice, narrating the transcript.
5. Overlay the new audio onto the video, replacing the original voice.

## Core Architecture

The Voice Replacement Agent is built on Director's extensible framework, leveraging its session management and state tracking capabilities while adding specialised voice replacement functionality. This is how the flow from the user's input to the video output will look like.

![Voice cloning agent workflow showing user input to video output flow](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/director/voice-cloning.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=29d7c6ab24b54583a646be3e3612d3b4)

> [!NOTE]
>
> The following parameters will be needed for the agent. To learn more about the importance of an agent's parameters, check out the Director documentation.

| Parameter | Type of parameter | Description |
| - | - | - |
| `sample_video` | object | An object containing the `video_id` and start and end time for the sample length |
| `name_of_voice` | string | Name to assign to the cloned voice |
| `description` | string | Description of how the voice sounds (e.g., "elderly" or "child-like", the accent etc.) |
| `is_authorized_to_clone_voice` | boolean | Flag indicating if the user is authorized to clone the voice |
| `video_ids` | list | List of Unique IDs given to the videos stored in VideoDB |
| `collection_id` | string | Collection ID for storing the generated audio |

> [!TIP]
>
> The `collection_id` parameter refers to the ID of a VideoDB collection where all your videos are stored and the generated (synthesised) audio file can be stored for future use. To learn more, explore VideoDB Collections documentation.

**From ElevenLabs:**

You'll need the following two key methods from the ElevenLabs SDK:

1. `clone`: Used to clone a voice based on the provided audio samples.
2. `generate`: Generates synthesised audio from text using the cloned voice.

You will need to implement these above methods in the VideoDB Director for the same.

## Setup

### VideoDB and Director Setup

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Install the latest SDK

```
pip install videodb
```

> **[Setup Director Locally](/pages/build-with-agents/director/setup-director)**
>
> Complete installation and configuration guide for Director framework

## Create ElevenLabs methods

For the voice cloning feature, the `clone` and `generate` methods given by ElevenLabs need to be implemented. For this, you can access the existing ElevenLabs tool present in the `/backend/director/tools` [folder](https://github.com/video-db/Director/tree/main/backend/director/tools). Let's go to the ElevenLabs tool in the `elevenlabs.py`.

Define the required methods in the `ElevenLabsTool`.

For cloning, create the `clone_audio` method

```python Python theme={null}
def clone_audio(self, audio_files: list[str], name_of_voice, description):
        voice = self.client.clone(
            name=name_of_voice,
            files=audio_files,
            description=description)
        return voice
```

```javascript Node.js theme={null}
// Method inside your class:
cloneAudio(audioFiles, nameOfVoice, description) {
    const voice = this.client.clone({
        name: nameOfVoice,
        files: audioFiles,
        description: description
    });
    return voice;
}
```

And for generating the audio, create the `synthesise_text` method

```python Python theme={null}
def synthesise_text(self, voice:Voice, text_to_synthesis:str):
        audio = self.client.generate(text=text_to_synthesis, voice=voice, model="eleven_multilingual_v2")
        return audio
```

```javascript Node.js theme={null}
// Method inside your class:
synthesiseText(voice, textToSynthesis) {
    const audio = this.client.generate({
        text: textToSynthesis,
        voice: voice,
        model: "eleven_multilingual_v2"
    });
    return audio;
}
```

## Building the Agent

### 1. Import the required components

Create a `voice_replacement.py` file inside `backend/director/agents` [folder](https://github.com/video-db/Director/tree/main/backend/director/agents) and add all our imports to it. They include:

```python Python theme={null}
import logging
import os
import requests
import uuid

# For using FFmpeg
import subprocess

# Base agent functionality
from director.agents.base import BaseAgent, AgentResponse, AgentStatus

# Session management and messaging
from director.core.session import Session, MsgStatus, TextContent, VideoContent, VideoData

# Integration tools
from director.tools.elevenlabs import ElevenLabsTool
from director.tools.videodb_tool import VideoDBTool

# path for downloads
from director.constants import DOWNLOADS_PATH

# For the Director timeline overlay
from videodb.timeline import Timeline
from videodb.asset import VideoAsset, AudioAsset
```

### 2. Define the parameters for the agent

Referring to the **Game Plan**, create a JSON schema for these parameters.

```python Python theme={null}
VOICE_REPLACEMENT_AGENT_PARAMETERS = {
    "type": "object",
    "properties": {
        "sample_video": {
            "type": "object",
            "properties": {
                "video_id": {
                    "type": "string",
                    "description": "The video id from which the 1 and a half minute sample audio has to be taken",
                }, 
                "start_time": {
                    "type": "number",
                    "description": "The start time from where the 1 and a half minute. start time is given in seconds that is 1 minute 37 seconds is 97",
                    "default": 0
                },
                "end_time": {
                    "type": "number",
                    "description": "The end time is where the extracted audio sample must end. Make sure that the end time is farther than start time and should only be 1 to 2 minutes farther than start_time. end time is given in seconds. for example 1 minute 37 seconds is 97",
                    "default": 90
                }

            }
        },
        "name_of_voice" : {
            "type": "string",
            "description": "The name to give to the voice. This can be the user's name",
        },
        "description" : {
            "type": "string",
            "description": "Description about how the voice sounds like. For example: This is a sounds of an old person, the voice is child like and cute etc."
        },
        "is_authorized_to_clone_voice": {
            "type": "boolean",
            "description": "This is a flag to check if the user is authorised to clone the voice or not. If the user has explicitly mentioned that they are authorised to clone the voice, then the flag is TRUE else FALSE. Make sure to confirm that the user is authorised or not. If not specified explicitly or not specified at all, the flag should be FALSE"
        },
        "cloned_voice_id": {
            "type": "string",
            "description": "This is the ID of the voice which is present if the user has already cloned a voice before. The cloned_voice_id can be taken from the previous results of cloning if the audio URL is not changed"
        },
        "collection_id": {
            "type": "string",
            "description": "the ID of the collection to store the output audio file",
        },
        "video_ids": {
            "type": "array",
            "items": {
                "type": "string"
            },
            "description": "A list of IDs of videos which needs to be overlayed"
        }
    },
    "required": ["sample_video", "is_authorized_to_clone_voice", "collection_id", "video_ids", "name_of_voice"],
}

```

### 3. Implement Agent Class

We will now create the agent class. The parameters set here ( `self.agent_name`, `self.description` and the `self.parameters`) determine how the agent interacts with the reasoning engine.

```python Python theme={null}
class VoiceReplacementAgent(BaseAgent):
    def __init__(self, session: Session, **kwargs):
        self.agent_name = "voice_replacement"
        self.description = "This agent is used to clone the voice of the given by the user and overlay it on top of all the videos given. The user must be authorised to clone the voice. This agent can handle multiple agents at once"
        self.parameters = VOICE_REPLACEMENT_AGENT_PARAMETERS
        super().__init__(session=session, **kwargs)
```

### 4. Implement the core logic of the voice replacement agent

**1. Declare a `run` method**

We will need to implement a `run` method in the agent’s class. This is the heart of the agent as this is the method that runs when the agent is called.

In the `run` method, define the required parameters which will be used to implement the agent.

```python Python theme={null}
def run(
    self,
    sample_video,
    video_ids: list[str],
    name_of_voice: str,
    is_authorized_to_clone_voice: bool,
    collection_id: str,
    description="",
    cloned_voice_id=None,
    *args,
    **kwargs) -> AgentResponse:
    """Main run method - implementation shown in sections below"""
    pass
```

**2. Check Authorisation**: If the user isn't authorised, return an error response

```python Python theme={null}
# Inside the run method:
try:
    if not is_authorised_to_clone_voice:
        return AgentResponse(
            status=AgentStatus.ERROR,
            message="Not authorised to clone the voice")
    # ... rest of implementation continues below
except Exception as e:
    pass  # See error handling section later
```

**3. Initialize Tools**: Check for the ElevenLabs API key and initialize the required tools.

```python Python theme={null}
ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY")
if not ELEVENLABS_API_KEY:
  raise Exception("Elevenlabs API key not present in .env")

audio_gen_tool = ElevenLabsTool(api_key=ELEVENLABS_API_KEY)
self.videodb_tool = VideoDBTool(collection_id=collection_id)
```

**4. Save the audio files locally:** From the video stored in VideoDB, we will need to extract the required audio file. This can be broken into the following steps:

1. Generate a stream of the video for the sample based on start and end time of sample audio
2. Download the video based on the stream
3. Extract the audio file from the video

For generating the stream, we will use the existing `get_video_stream` method from the `VideoDBTool` :

```python Python theme={null}
self.output_message.actions.append("Getting the sample video's stream")
self.output_message.push_update()

stream_url = self.videodb_tool.generate_video_stream(sample_video["video_id"], [(sample_video.get("start_time", 0),sample_video.get("end_time", 90))]) 
```

For downloading the video, first we will get the download link of the above generated stream

```python Python theme={null}
self.output_message.actions.append("Getting the sample video's download URL")
self.output_message.push_update()

download_response = self.videodb_tool.download(stream_url)

if download_response.get("status") == "done":
    download_url = download_response.get("download_url")
else:
    raise Exception("Couldn't find the video download URL")
```

Now, we will write a `_download_video_file` method which will download the video via the `download_url` and give the path where it is saved

```python Python theme={null}
def _download_video_file(self, video_url: str) -> str:
    os.makedirs(DOWNLOADS_PATH, exist_ok=True)

    try:
        response = requests.get(video_url, stream=True)
        response.raise_for_status()

        if not response.headers.get('Content-Type', '').startswith('video'):
            raise ValueError(f"The URL does not point to a video file: {video_url}")

        download_file_name = f"video_download_{uuid.uuid4()}.mp4"
        local_path = os.path.join(DOWNLOADS_PATH, download_file_name)

        with open(local_path, 'wb') as file:
            for chunk in response.iter_content(chunk_size=65536):
                print("Wrote chunk")
                file.write(chunk)

        print("Download finished")
        return local_path

    except Exception as e:
        print(f"Failed to download {video_url}: {e}")
        return None

```

Now, let’s use the method to get the `video_path`

```python Python theme={null}
self.output_message.actions.append("Downloading the sample video")
self.output_message.push_update()

video_path = self._download_video_file(download_url)

if not video_path:
    raise Exception("Couldn't fetch the video for sampling")
```

We will now extract the audio from a video using VideoDB. For this, we will need an `extract_audio_from_video` method.

Start by defining a method `_extract_audio_from_video`, which takes a video file path as input and extracts audio from it.

```python Python theme={null}
def _extract_audio_from_video(self, video_path: str):
    try:
        audio_uploaded = self.videodb_tool.upload(video_path, source_type="file_path", media_type="audio")
        audio = self.videodb_tool.get_audio(audio_id=audio_uploaded["id"])
        audio_path = self._download_audio_file(audio["url"])
        return audio_path

    except Exception as e:
        logger.error(f"Failed to download audio: {e}")
        return None

```

Now, modify the existing `get_audio` method inside `tools/videodb-tool.py` [file](https://github.com/video-db/Director/blob/main/backend/director/tools/videodb_tool.py) to include a `url` field in the returned object. This field provides a direct link to the extracted audio file.

```python Python theme={null}
def get_audio(self, audio_id):
    """Get an audio by ID."""
    audio = self.collection.get_audio(audio_id)
    return {
        "id": audio.id,
        "name": audio.name,
        "collection_id": audio.collection_id,
        "length": audio.length,
        "url": audio.generate_url()
    }
```

Create the `_download_audio_file` method which will take the URL

```python Python theme={null}
    def _download_audio_file(self, audio_url: str) -> str:

        os.makedirs(DOWNLOADS_PATH, exist_ok=True)

        try:
            response = requests.get(audio_url, stream=True)
            response.raise_for_status()

            download_file_name = f"audio_url_{str(uuid.uuid4())}.mp3"
            local_path = os.path.join(DOWNLOADS_PATH, download_file_name)

            with open(local_path, 'wb') as file:
                for chunk in response.iter_content(chunk_size=65536):
                    file.write(chunk)

            return local_path

        except Exception as e:
            logger.error(f"Failed to download {audio_url}: {e}")
            return None

```

Finally, let’s use this method to get the audio file from it

```python Python theme={null}
sample_audio = self._extract_audio_from_video(video_path)

if not sample_audio:
    return AgentResponse(status=AgentStatus.ERROR, message="Couldn't process the sample audios")

```

**5. Clone the Voice**: Call ElevenLabs' `clone_audio` method to create the cloned voice.

```python Python theme={null}
self.output_message.actions.append("Cloning the voice")
self.output_message.push_update()
voice = audio_gen_tool.clone_audio(audio_files=[sample_audio], name_of_voice=name_of_voice, description=description)

if not voice:
  return AgentResponse(
   status=AgentStatus.ERROR, 
   message="Failed to generate the voice clone")
```

**6. Start processing all videos:** We have generated the cloned voice and we can now create overlays for the videos. We will start processing the videos for each `video_id` present in the `video_ids` list

```python Python theme={null}
for video_id in video_ids:
    video = self.videodb_tool.get_video(video_id=video_id)
```

**7. Extract transcript from video**: For the generation of audio for the video, we will need to extract the transcript from the video. For this, we will make a method in the agent which will take the video\_id and return the transcript

```python Python theme={null}
def _get_transcript(self, video_id):
    self.output_message.actions.append("Retrieving video transcript..")
    self.output_message.push_update()
    try:
        return self.videodb_tool.get_transcript(video_id)
    except Exception:
        self.output_message.actions.append(
            "Transcript unavailable. Indexing spoken content.")
        self.output_message.push_update()
        self.videodb_tool.index_spoken_words(video_id)
    return self.videodb_tool.get_transcript(video_id) 
```

And now, you can get the transcript from the video in the `run` method.

```python Python theme={null}
text_to_synthesis = self._get_transcript(video_id=video_id)
```

**8. Synthesise Text**: Use ElevenLabs' `synthesise_text` method to generate audio from the input text

```python Python theme={null}
self.output_message.actions.append(f"Synthesising {video["name"]}'s transcript in cloned voice")
self.output_message.push_update()

synthesised_audio = audio_gen_tool.synthesis_text(voice=voice, text_to_synthesis=text_to_synthesis)
            
if not synthesised_audio:
  return AgentResponse(
    status=AgentStatus.ERROR, 
    message="Failed to generate the voice clone")
```

> [!TIP]
>
> To communicate the steps that the agent is taking, you can simply use the `self.output_message.actions` and the `self.output_message.push_update` methods to send the updates to the client. This will allow you to communicate with the user about what the agent is achieving at a particular time.

**9. Save the Audio File**: Store the generated audio file locally

```python Python theme={null}
output_file_name = f"audio_clone_voice_output_{str(uuid.uuid4())}.mp3"
output_path = f"{DOWNLOADS_PATH}/{output_file_name}"

with open(output_path, "wb") as f:
  for chunk in synthesised_audio:
    if chunk:
      f.write(chunk)

self.output_message.actions.append(
  f"Generated audio saved at <i>{output_path}</i>")
self.output_message.push_update()
```

**10. Upload to VideoDB**: We will upload the generated audio file to VideoDB and retrieve its unique audio ID.

```python Python theme={null}
media = self.videodb_tool.upload(  
    output_path, source_type="file_path", media_type="audio")

self.output_message.actions.append(
  f"Uploaded generated audio to VideoDB with Audio ID {media['id']}")
```

**11. Overlay audio on to the video:** We will use VideoDB's timeline feature to overlay the cloned voice onto the video.

> [!TIP]
>
> To know more about timeline and audio overlays, you can visit the Audio overlay + Video + Timeline documentation.

For this, make a method for adding the overlay using the `video_id` and `audio_id` which returns a stream link so that we can stream the video.

```python Python theme={null}
def _generate_overlay(self, video_id, audio_id):
    if self.timeline is None:
        return None
    
    self.timeline.add_inline(VideoAsset(video_id))
    self.timeline.add_overlay(start=0, asset=AudioAsset(audio_id))
    
    stream = self.timeline.generate_stream()
    return stream
```

Now, in the `run` method, before adding the overlay to the video, we will use `VideoContent` to display the video output in the Director’s UI to show that adding the audio overlay is in progress.

```python Python theme={null}
video_content = VideoContent(
    status=MsgStatus.progress,
    status_message="Adding cloned voice to the video",
    agent_name=self.agent_name)

self.output_message.content.append(video_content)
self.output_message.push_update()
```

We will now add the overlay and pass the generated stream link to the `video_content` so that we can watch it in the Director’s UI.

```python Python theme={null}
self.timeline = self.videodb_tool.get_and_set_timeline()

stream_url = self._generate_overlay(video_id, audio_id=audio['id'])

video_content.video = VideoData(stream_url=stream_url)
video_content.status = MsgStatus.success
video_content.status_message = "Here is your video with the cloned voice"

self.output_message.push_update()

```

**12. Publish the updates and return an `AgentResponse` to end the agent's process**

Once all the videos are processed, we will use the `publish` method to save the messages as a final step.

```
self.output_message.publish()
```

Also, you can send certain information such as the `cloned_voice_id` and `audio_id` as a response so that any subsequent chat requests will be able to use them to generate further responses.

```python Python theme={null}
return AgentResponse(
  status=AgentStatus.SUCCESS,
  message=f"Agent {self.name} completed successfully.",
  data={
    "cloned_voice_id": voice.voice_id,
    "audio_id": media["id"]
    })
```

**13. Implement Error Handling**

Ensure robust error handling to manage failures gracefully.

```python Python theme={null}
# Exception handler for the try block above:
try:
    pass  # See implementation in previous sections
except Exception as e:
    logger.exception(f"Error in {self.agent_name}")
    text_content = TextContent(
        agent_name=self.agent_name,
        status=MsgStatus.error,
        status_message="Failed to generate the transformed videos")
    self.output_message.content.append(text_content)
    self.output_message.push_update()
    self.output_message.publish()
    error_message = f"Agent failed with error {e}"
    return AgentResponse(status=AgentStatus.ERROR, message=error_message)
```

### 5. Register the Agent

To use the agent, go inside the `backend/director/handler.py` [file](https://github.com/video-db/Director/blob/main/backend/director/handler.py) and import the agent.

```python Python theme={null}
from director.agents.voice_replacement import VoiceReplacementAgent 
```

And add the agent in the `self.agents` list

```python Python theme={null}
class ChatHandler:
    def __init__(self, db, **kwargs):
        self.db = db

        # Register the agents here
        self.agents = [
            ...,
            VoiceReplacementAgent
        ]
```

And that is it! It was this straightforward to write an agent. Now you can try out this agent in the Director locally and explore the cloning capabilities of the ElevenLabs’ cloning feature and seamless video overlay feature from VideoDB to breathe new life into your videos!

## Using the agent

Simply go to the frontend at [http://localhost:8080](http://localhost:8080) and refresh the page to see the agent available in the options for use.

## Conclusion

The Voice Transformation Agent demonstrates the power and flexibility of Director's agent framework. By leveraging VideoDB's video overlay feature and robust database system and ElevenLabs' voice cloning capabilities, we've created a secure, scalable solution for your voice cloning and video overlay needs.

**Key takeaways:**

* Simple integration with Director's framework
* Robust error handling and security measures
* Scalable architecture for audio processing
* Seamless way of adding audio overlays on videos
* Ethical considerations built into the design

Creating an agent in VideoDB Director is incredibly easy, allowing you to build powerful and customised solutions quickly.

## Resources

## Related Tutorials

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add professional narration to silent footage

> **[Multi-Language Dubbing](/examples-and-tutorials/content-factory/dubbing)**
>
> Dub videos into any language with AI voices

Ready to build your own AI agents? Join our community and start exploring what's possible with Director!

> **[Director Documentation](https://docs.director.videodb.io/index.html)**
>
> Comprehensive guides and API references for Director framework

> **[Sample Agents Repository](https://github.com/video-db/Director/tree/main/backend/director/agents)**
>
> Real-world examples and implementations of custom agents

> **[Discord Community](https://discord.com/invite/py9P639jGz)**
>
> Connect with other agent developers and get support

> [!NOTE]
>
> Need help? Reach out to our community or [open an issue on GitHub](https://github.com/video-db). Happy building!


# AI Voiceovers
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/voiceovers

Add professional narration to silent footage

> **Case: Automatically Creating Voiceover for Silent Footage of the Underwater World**

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/AI_Voiceover.ipynb)

## Overview

Voiceovers are the secret sauce that turns silent footage into captivating stories. They add depth, emotion, and excitement, elevating the viewing experience.

Traditionally, this workflow required stitching together multiple tools: one for script writing (LLM), one for voice generation (TTS), and another for video editing.

**VideoDB** simplifies this by bringing everything under one roof. In this tutorial, we will:

1. **Upload** a silent video.
2. **Analyze** the video to understand its visual content.
3. **Generate** a narration script using VideoDB's text generation.
4. **Generate** a professional AI voiceover using VideoDB's voice generation.
5. **Merge** them instantly into a final video.

## Setup

### Installing VideoDB

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

> [!NOTE]
>
> You only need your VideoDB API Key. Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

## Implementation

### Step 1: Connect to VideoDB

Connect to VideoDB using your API key to establish a session.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

// Connect to VideoDB
const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Video

We'll upload the silent underwater footage directly from YouTube.

```python Python theme={null}
# Upload a video by URL
video = coll.upload(url='https://youtu.be/RcRjY5kzia8')
```

```javascript Node.js theme={null}
// Upload a video by URL
const video = await coll.uploadURL({ url: 'https://youtu.be/RcRjY5kzia8' });
```

### Step 3: Analyze Visuals

We create visual scene descriptions of the video to use as context when writing the voiceover script.

```python Python theme={null}
print("Analyzing visuals... this might take a moment.")
visual_understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {
                "schema": {
                    "description": "string",
                },
            },
        },
    ],
)
visual_understanding.wait_until_complete()
```

```javascript Node.js theme={null}
const visualUnderstanding = await video.understand([
    {
        type: "vlm",
        name: "scene",
        config: {
            schema: {
                description: "string",
            },
        },
    },
]);
await visualUnderstanding.waitUntilComplete();
```

Let's view the description of the first scene from the video.

```python Python theme={null}
scene_analyzer = visual_understanding.get_analyzer("scene")

scene_output = scene_analyzer.get_output()

video_scenes = [
    {
        "start": scene["start"],
        "end": scene["end"],
        "description": scene["data"]["description"],
    }
    for scene in scene_output["scenes"]
]

import json
print(json.dumps(video_scenes[0], indent=2))
```

```javascript Node.js theme={null}
const sceneAnalyzer = await visualUnderstanding.getAnalyzer("scene");
const sceneOutput = await sceneAnalyzer.getOutput();
const videoScenes = sceneOutput.scenes.map((scene) => ({
    start: scene.start,
    end: scene.end,
    description: scene.data.description,
}));

console.log(JSON.stringify(videoScenes[0], null, 2));
```

### Step 4: Generate Script

Now, we use VideoDB's `generate_text` method to write a voiceover script based on the scene descriptions we just retrieved.

```python Python theme={null}
# Construct a prompt with the scene context
scene_context = "\n".join([f"- {scene['description']}" for scene in video_scenes])

prompt = f"""
Here is a visual description of a video about the underwater world:
{scene_context}

Based on this, write a short, engaging voiceover script in the style of a nature documentary narrator (like David Attenborough).
Keep it synced to the flow of the visuals described.
Return ONLY the raw text of the narration, no stage directions or titles.
"""

# Generate the script using VideoDB
script_response = coll.generate_text(
    prompt=prompt,
    model_name="pro")

script_text = script_response["output"]

print("--- Generated Script ---")
print(script_text)
```

```javascript Node.js theme={null}
// Construct a prompt with the scene context
const sceneContext = videoScenes.map(scene => `- ${scene.description}`).join('\n');

const prompt = `
Here is a visual description of a video about the underwater world:
${sceneContext}

Based on this, write a short, engaging voiceover script in the style of a nature documentary narrator (like David Attenborough).
Keep it synced to the flow of the visuals described.
Return ONLY the raw text of the narration, no stage directions or titles.
`;

// Generate the script using VideoDB
const scriptResult = await coll.generateText(
    prompt,
    "pro",
    "text"
);
const scriptText = typeof scriptResult === "string"
    ? scriptResult
    : scriptResult.output;

if (typeof scriptText !== "string") {
    throw new Error("Expected a text response for the voiceover script.");
}

console.log("--- Generated Script ---");
console.log(scriptText);
```

### Step 5: Generate Voiceover Audio

We can now turn that text into speech using `generate_voice`. This returns an Audio object directly, so we don't need to save or upload files manually.

```python Python theme={null}
# Generate speech directly as a VideoDB Audio Asset
audio = coll.generate_voice(
    text=script_text,
    voice_name="Default")

print(f"Generated Audio Asset ID: {audio.id}")
```

```javascript Node.js theme={null}
// Generate speech directly as a VideoDB Audio Asset
import { Audio, GenerationJob } from 'videodb';

const audioCandidate = await coll.generateVoice(
    scriptText,
    "Default"
);
const audio = audioCandidate instanceof GenerationJob
    ? await audioCandidate.wait()
    : audioCandidate;

if (!(audio instanceof Audio)) {
    throw new Error("Expected generated voice audio.");
}

console.log(`Generated Audio Asset ID: ${audio.id}`);
```

### Step 6: Compose the Video

We have the video and the generated voiceover. Now we merge them using the Timeline Editor.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset, AudioAsset

# Create a timeline
timeline = Timeline(conn)

# 1. Create a Video Track
video_track = Track()
video_asset = VideoAsset(id=video.id)
# Add the video clip
video_clip = Clip(asset=video_asset, duration=float(video.length))
video_track.add_clip(0, video_clip)

# 2. Create an Audio Track for the voiceover
audio_track = Track()
# Use the audio object we generated in Step 5
audio_asset = AudioAsset(id=audio.id)
audio_clip = Clip(asset=audio_asset, duration=float(audio.length))
audio_track.add_clip(0, audio_clip)

# Add tracks to timeline
timeline.add_track(video_track)
timeline.add_track(audio_track)
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset, EditorAudioAsset } from 'videodb';

// Create a timeline
const timeline = new EditorTimeline(conn);

// 1. Create a Video Track
const videoTrack = new Track();
const videoAsset = new EditorVideoAsset({ id: video.id });
// Add the video clip
const videoClip = new Clip({ asset: videoAsset, duration: parseFloat(video.length) });
videoTrack.addClip(0, videoClip);

// 2. Create an Audio Track for the voiceover
const audioTrack = new Track();
// Use the audio object we generated in Step 5
const audioAsset = new EditorAudioAsset({ id: audio.id });
const audioClip = new Clip({ asset: audioAsset, duration: parseFloat(audio.length) });
audioTrack.addClip(0, audioClip);

// Add tracks to timeline
timeline.addTrack(videoTrack);
timeline.addTrack(audioTrack);
```

### Step 7: Review and Share

Generate the final stream URL and watch your AI-narrated video!

```python Python theme={null}
from videodb import play_stream

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

**Output:**

[AI-Narrated Underwater Video](https://www.youtube.com/embed/gsU14KgORgg)

## Conclusion

Congratulations! You have successfully automated the process of creating custom and personalized voiceovers based on a simple prompt and raw video footage using VideoDB.

By leveraging advanced AI technologies, you can enhance the storytelling and immersive experience of your video content. Experiment with different prompts and scene analysis techniques to further improve the quality and accuracy of the voiceovers. Enjoy creating captivating narratives with AI-powered voiceovers using VideoDB!

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/AI_Voiceover.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Video Dubbing](/examples-and-tutorials/content-factory/dubbing)**
>
> Dub videos into multiple languages with AI voice synthesis

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Build complete faceless videos with AI scripts, voiceovers, and multi-layer composition


# Annual Video Statistics Recap
Source: https://docs.videodb.io/examples-and-tutorials/content-factory/year-in-frames

Transform video analytics into a cinematic recap video with AI-powered visualizations

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/2025_in_frames.ipynb)

## The Idea

You've uploaded hundreds of videos this year. You've run thousands of searches. Your automation has processed countless scenes. But how do you celebrate and share those numbers in a way that actually captivates people?

Traditional dashboards and spreadsheets are boring. What if you could transform your video analytics into a shareable cinematic recap video?

## What You'll Build

Turn your video analytics into a shareable recap video. This system creates a cinematic journey through your video stats:

* Total minutes uploaded with growth metrics
* Search activity visualization
* Clips generated via automation
* Scenes and frames analyzed
* All set to music with dynamic video grids and text overlays

All powered by **VideoDB's Editor SDK** — turning data into a visual story through code.

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import json
import random
import videodb
from videodb import MediaType, play_stream

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Calculate Your Video Metrics

```python theme={null}
# Get all videos from your collection
videos = coll.get_videos()

TOTAL_SECONDS = 0
for v in videos:
  TOTAL_SECONDS += v.length

# Core metrics
TOTAL_MINUTES = round(TOTAL_SECONDS / 60, 1)
TOTAL_SEARCHES = 2573
CLIPS_GENERATED = 574
SCENES_ANALYZED = 248
FRAMES_ANALYZED = 3200

# Previous year comparison
PREV_YEAR_MINUTES = 6400

# Time conversions
TOTAL_HOURS = round(TOTAL_MINUTES / 60, 1)
TOTAL_DAYS = round(TOTAL_HOURS / 24, 1)

# Growth percentage
MINUTES_GROWTH = round(((TOTAL_MINUTES - PREV_YEAR_MINUTES) / PREV_YEAR_MINUTES) * 100)
```

### Step 2: Upload Background Music

```python theme={null}
# Upload background music as audio
audio = coll.upload("https://youtu.be/vIEi8AoP_Fw?si=AUVUt6Komqtt7rWI", media_type="audio")
```

### Step 3: Create Timeline with 5 Sections

Initialize the timeline and set up base configuration:

```python theme={null}
from videodb.editor import (
    Timeline, Track, Clip, VideoAsset, AudioAsset, TextAsset,
    Offset, Transition, Font, Filter,
)

# Use subset of videos
videos = videos[20:40]

# Create timeline
timeline = Timeline(conn)
timeline.background = "#E85E00"
track = Track()
video_track = Track()

# Add background music
b_music = Clip(
    asset=AudioAsset(id=audio.id, start=0, volume=1),
    duration=28,
)
track.add_clip(4, b_music)
```

### Step 4: Build Intro Section (0-5s)

Create a dynamic mosaic effect with 60 video clips:

```python theme={null}
# Intro text
intro_text = Clip(
    asset=TextAsset(
        text="2025 was a blur. Let's index it.",
        font=Font(size=60, family="Roboto Bold"),
    ),
    duration=5,
    transition=Transition(in_="fade"),
)
track.add_clip(1, intro_text)

# Create 60 video clips with varying scales and random positioning
for i in range(60):
    scale_value = 0.5 + (i * 0.025)  # Scale: 0.5 to 0.975
    start_time = random.randint(0, 4)
    volume = 0.5
    if start_time == 3:
        volume = 0.3
    elif start_time == 4:
        volume = 0.3

    video = random.choice(videos)

    video_clip = Clip(
        asset=VideoAsset(
            id=video.id,
            start=random.randint(10, int(video.length - 10)),
            volume=volume,
        ),
        duration=1,
        scale=scale_value,
        fit=None,
        offset=Offset(x=random.uniform(-0.5, 0.5), y=random.uniform(-0.5, 0.5)),
    )

    video_track.add_clip(start=start_time, clip=video_clip)
```

### Step 5: Build Upload Metrics Section (6-13s)

Display total minutes uploaded with zoom-out effect:

```python theme={null}
# Section 1 text
section1_text = Clip(
    asset=TextAsset(
        text=f"You uploaded {TOTAL_MINUTES:,} minutes\nof video content.\n\nThat's {TOTAL_HOURS} hours.\nOr {TOTAL_DAYS} full days of footage.\n\n↑ {MINUTES_GROWTH}% from 2024",
        font=Font(size=48, family="default"),
    ),
    duration=7,
)
track.add_clip(6, section1_text)

# Zoom out effect with greyscale filter
for i in range(20):
    scale_value = 0.5 + (i * 0.0263)  # Scale: 0.5 to 1.0
    start_time = 6 + (i * 0.175)

    video = random.choice(videos)
    video_start = random.randint(10, max(11, int(video.length - 10)))

    video_clip = Clip(
        asset=VideoAsset(
            id=video.id,
            start=video_start,
            volume=0.15,
        ),
        duration=1,
        scale=scale_value,
        fit="crop",
        filter=Filter.greyscale,
    )

    video_track.add_clip(start=start_time, clip=video_clip)
```

### Step 6: Build Search Analytics Section (13-19s)

Create 3 rows of horizontal scanning video clips:

```python theme={null}
# Section 2 text
section2_text = Clip(
    asset=TextAsset(
        text=f"{TOTAL_SEARCHES:,} questions asked.\n\nYour videos aren't just stored—\nthey're understood.",
        font=Font(size=58, family="default"),
    ),
    duration=6,
)
track.add_clip(13, section2_text)

# Create horizontal scanning effect with 3 rows
rows = [
    {"y": -0.5, "num_clips": 20, "duration": 0.25, "scale": 0.35},  # Top row
    {"y": 0, "num_clips": 15, "duration": 0.33, "scale": 0.45},     # Middle row
    {"y": 0.5, "num_clips": 12, "duration": 0.42, "scale": 0.35},   # Bottom row
]

clip_idx = 0
for row in rows:
    for i in range(row["num_clips"]):
        x_pos = -0.9 + (i * 0.1)
        video = videos[clip_idx % len(videos)]

        video_start = 30 + (clip_idx % 10)
        if video_start > video.length - 10:
            video_start = 15

        video_clip = Clip(
            asset=VideoAsset(
                id=video.id,
                start=video_start,
                volume=0.15,
            ),
            duration=row["duration"],
            scale=row["scale"],
            fit="crop",
            offset=Offset(x=x_pos, y=row["y"]),
        )
        video_track.add_clip(start=12 + (i * row["duration"]), clip=video_clip)
        clip_idx += 1
```

### Step 7: Build Automation Stats Section (19-25s)

Large background clips with cycling effect:

```python theme={null}
# Section 3 text
section3_text = Clip(
    asset=TextAsset(
        text=f"{CLIPS_GENERATED:,} clips\ngenerated via code.\n\nAutomation at its finest.",
        font=Font(size=62, family="default"),
    ),
    duration=6,
)
track.add_clip(19, section3_text)

# Cycling background effect
for i in range(3):
    video = videos[(i + 5) % len(videos)]
    video_start = 20 + (i * 15)
    if video_start > video.length - 10:
        video_start = 10

    bg_clip = Clip(
        asset=VideoAsset(
            id=video.id,
            start=video_start,
            volume=0.05,
        ),
        duration=2,
        scale=1.2,
        fit="crop",
        filter=Filter.greyscale,
        opacity=0.2,
        z_index=0,
    )
    video_track.add_clip(start=19 + (i * 2), clip=bg_clip)
```

### Step 8: Build Intelligence Metrics Section (25-31s)

3x3 grid with scanning animation:

```python theme={null}
# Section 4 text
section4_text = Clip(
    asset=TextAsset(
        text=f"{SCENES_ANALYZED:,} scenes analyzed\n{FRAMES_ANALYZED:,} Frames Analyzed\n\nPure video intelligence.\nPowered by you.",
        font=Font(size=52, color="#FFFFFF", family="Arial Bold"),
    ),
    duration=5,
)
track.add_clip(25, section4_text)

# 3x3 grid with scanning effect
grid_positions = [
    (-0.4, -0.4), (0, -0.4), (0.4, -0.4),  # Top row
    (-0.4, 0), (0, 0), (0.4, 0),           # Middle row
    (-0.4, 0.4), (0, 0.4), (0.4, 0.4),     # Bottom row
]

for idx, (x_pos, y_pos) in enumerate(grid_positions):
    video = videos[idx % len(videos)]

    video_start = 50 + (idx % 10)
    if video_start > video.length - 10:
        video_start = 20

    video_clip = Clip(
        asset=VideoAsset(
            id=video.id,
            start=video_start,
            volume=0.1,
        ),
        duration=0.6,
        scale=0.28,
        fit="crop",
        offset=Offset(x=x_pos, y=y_pos),
        transition=Transition(in_="fade", duration=0.15),
        filter=Filter.contrast if idx % 2 == 0 else Filter.boost,
        opacity=0.9,
    )

    video_track.add_clip(start=24.5 + (idx * 0.08), clip=video_clip)

# Add cascading columns
num_columns = 2
clips_per_column = 8
for col in range(num_columns):
    x_pos = -0.3 + (col * 0.6)

    for row in range(clips_per_column):
        y_pos = -0.7 + (row * 0.23)
        clip_idx = col * clips_per_column + row

        video = videos[clip_idx % len(videos)]

        video_start = 40 + (clip_idx % 8)
        if video_start > video.length - 10:
            video_start = 25

        video_clip = Clip(
            asset=VideoAsset(
                id=video.id,
                start=video_start,
                volume=0.15,
            ),
            duration=0.3,
            scale=0.3,
            fit="crop",
            offset=Offset(x=x_pos, y=y_pos),
        )
        video_track.add_clip(start=25.7 + (row * 0.3) + (col * 0.1), clip=video_clip)
```

### Step 9: Add Outro and Render

```python theme={null}
# Outro text
outro_text = Clip(
    asset=TextAsset(
        text="Ready for 2026? Build it.",
        font=Font(size=60, family="Roboto Bold")
    ),
    duration=2,
)
track.add_clip(31, outro_text)

# Add tracks to timeline
timeline.add_track(video_track)
timeline.add_track(track)

# Generate stream
stream_url = timeline.generate_stream()
```

## What You Get

A professional video that:

* Visualizes your video metrics in real-time
* Uses 100+ of your actual videos in the composition
* Creates dynamic animations and transitions
* Pairs everything with music
* Tells the story of your year in video data

Perfect for:

* Marketing your platform
* Celebrating milestones with users
* Annual reports
* Social media recap posts
* Team celebrations

Here's the final rendered recap video:

[Annual Video Statistics Recap](https://console.videodb.io/player?url=https://d1zudc7ewmc6ey.cloudfront.net/v1/a0f7b1cd-26b1-4c1b-b2d8-deaef0be5524.m3u8)

## The Result

With this system, you transform dry analytics into engaging visual stories. Your users see not just numbers, but a cinematic journey through what you've accomplished together.

It's data visualization meets creative storytelling — all powered by code.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/2025_in_frames.ipynb)**
>
> Open the complete implementation with procedural animation logic, timing calculations, and advanced effects.

## Related Tutorials

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Generate complete videos from scripts

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Auto-sync lyrics to music with animations


# Examples & Tutorials
Source: https://docs.videodb.io/examples-and-tutorials/index

Build real applications with VideoDB - from agentic workflows to video search and understanding

## Run Notebooks

Jump straight into runnable Colab notebooks and GitHub repos. Whether you're getting started or diving deep, each link takes you directly to working code.

> **Agentic Workflows**
>
> * [Pair Programmer](https://github.com/video-db/pair-programmer) - Screen-aware coding assistant
> * [OpenClaw Monitoring](https://github.com/video-db/openclaw-monitoring) - CCTV for AI agents
> * [Call.md](https://github.com/video-db/call.md) - AI meeting assistant
> * [Bloom](https://github.com/video-db/bloom) - Local-first screen recorder
> * [Focusd](https://github.com/video-db/focusd) - Productivity tracking
> * [Record & Replay](https://github.com/video-db/open-record-replay) - Turn demonstrated desktop workflows into reusable agent skills
> * [Chess Lens](https://github.com/video-db/chess-lens) - Real-time chess coaching from a clearly visible board

> **Video Search**
>
> * [Keyword Search](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Keyword_Search_1.ipynb) - Search by spoken words
> * [Multimodal Search](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/quickstart/Multimodal_Quickstart.ipynb) - Text, image & semantic search
> * [Character Clips](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Create%20Clip%20Faces.ipynb) - Extract character clips
> * [Conference Slides](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/conference_slide_scraper.ipynb) - Extract presentation slides

> **Live Intelligence**
>
> * [Intrusion Detection](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Intrusion_Detection.ipynb) - Property security
> * [Baby Monitoring](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Baby_Crib_Monitoring.ipynb) - AI baby monitoring
> * [Road Monitoring](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Road_Monitoring.ipynb) - Traffic analysis
> * [Flash Flood Detection](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Flash_Flood_Detection.ipynb) - Early flood warnings
> * [Cricket Highlights](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Cricket_Match_Monitoring.ipynb) - Live sports highlights
> * [Multicam Quickstart](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/quickstart/Multicam_Quickstart.ipynb) - Four-camera live monitoring
> * [Basketball Analytics](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/multicam/Multicam_Basketball_Analysis.ipynb) - Multi-camera sports

> **Content Creation**
>
> * [Faceless Videos](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/faceless_video.ipynb) - No-camera content
> * [AI Dubbing](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Dubbing%20-%20Replace%20Soundtrack%20with%20New%20Audio.ipynb) - Multi-language dubbing
> * [Voiceovers](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/AI_Voiceover.ipynb) - Natural narration
> * [Trailer Narration](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Automated_Trailer_Voiceover.ipynb) - Epic voiceovers
> * [TikTok Lyrics](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/tiktok_style_lyric_video_creator.ipynb) - TikTok-style videos
> * [Text to Video](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/GenAI_Storyboard.ipynb) - Generate from prompts

> **Programmatic Editing**
>
> * [Intro/Outro](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Intro_Outro_Inline.ipynb) - Branded bookends
> * [Brand Elements](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Adding_Brand_Elements.ipynb) - Logos & watermarks
> * [Audio Overlay](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Audio_Overlay.ipynb) - Background music
> * [Dynamic Ads](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/master/examples/Insert%20Inline%20Video.ipynb) - Personalized ads
> * [Dynamic Streams](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Programmatic_Streams.ipynb) - Adaptive streams

> **Safety & Compliance**
>
> * [Profanity Beep](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Beep%20Curse%20Words.ipynb) - Auto-beep profanity
> * [Content Moderation](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Content_Moderation.ipynb) - Remove inappropriate content
> * [Copyright Detection](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/automated_video_copyright_detection.ipynb) - Identify copyrighted material

Star the [VideoDB Cookbook](https://github.com/video-db/videodb-cookbook) and subscribe to [YouTube](https://www.youtube.com/@video_db) for the latest tutorials.

***

## What Do You Want to Build?

> **[Agentic Workflows](/examples-and-tutorials/ai-copilots/index)**
>
> **Plug video into your agents.** Give your AI context from meetings, recordings, desktop, and cameras. Build pair programmers, agent monitoring, and personal work memory.

> **[Video Search and Understanding](/examples-and-tutorials/video-rag/index)**
>
> **Find and understand videos.** Turn hours of video into searchable knowledge. Query meetings, lectures, or archives with natural language and get timestamped, playable results.

> **[Live Intelligence](/examples-and-tutorials/live-intelligence/index)**
>
> **Monitor cameras & streams.** Real-time AI detection for security, safety, traffic, and sports. Get instant alerts when events happen.

> **[Content Factory](/examples-and-tutorials/content-factory/index)**
>
> **Create videos at scale.** Automate production with AI dubbing, voiceovers, faceless videos, and text-to-video generation. No manual editing required.

> **[Programmatic Editing](/examples-and-tutorials/programmatic-editing/index)**
>
> **Edit video with code.** Add intros, overlays, and dynamic elements using the Timeline API. Generate multiple versions from one source.

> **[Safety & Compliance](/examples-and-tutorials/safety-compliance/index)**
>
> **Moderate & protect.** Detect profanity, filter inappropriate content, and identify copyright violations automatically.


# Baby Crib Monitoring with AI
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/baby-crib-monitoring

AI-powered baby safety monitoring system that detects escape attempts and sends real-time alerts

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Baby_Crib_Monitoring.ipynb)

## Storytime: Why This Matters

Meet **Vidit** and **Meghna** — a young couple juggling demanding jobs and household responsibilities. After a long, exhausting day, all they hope for is a peaceful night's sleep. But their energetic little one has other plans.

Their child, once safely tucked into his crib, has recently discovered how to climb out. While the parents sleep, unaware, the baby risks injury by wandering unsupervised at night. How can they keep him safe without losing their much-needed rest?

## Enter VideoDB RTStream

**VideoDB** offers the perfect solution for this problem. Using **RTStream**, we can let AI continuously monitor a live video feed, index scenes, detect specific events like **baby attempting to climb out of the crib**, and instantly send alerts to the parents when something risky happens.

In this guide, **Vidit and Meghna install an IP camera near the crib** and use **VideoDB RTStream** to power an AI monitoring system. As soon as the baby makes a move to climb out, AI detects it, triggers an event, and fires a real-time alert so the parents can step in.

## What You'll Learn

By the end of this guide, you'll learn how to:

* Connect a live RTSP video stream to VideoDB
* Continuously analyze video scenes using AI-generated natural language descriptions
* Detect specific events like *"baby escaping crib"*
* Trigger real-time alerts on such events

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Connect to the RTSP Stream

Connect to the live video stream of the crib using its RTSP URL. In this demo, the stream is running at `rtsp://samples.rts.videodb.io:8554/crib`.

```python theme={null}
rtsp_url = "rtsp://samples.rts.videodb.io:8554/crib"
crib_stream = coll.connect_rtstream(
    name="Baby Crib Monitor",
    url=rtsp_url,
    store=True,
)
```

### Step 2: Index Scenes with AI Descriptions

Create a real-time scene index that periodically analyzes the video and generates natural language descriptions of what's happening in the crib. The AI model watches for activity such as the baby moving, sitting, or attempting to climb out.

```python theme={null}
crib_understanding = crib_stream.understand(
    segmentation={"type": "time", "window": "10s"},
    analyzers=[{
        "type": "vlm",
        "name": "scene",
        "sampling": {"frame_count": 1},
        "config": {"prompt": "Describe the activity of the baby kept inside a baby crib. Notice if baby climbs out or attempts to escape."},
    }],
    store=True,
)
scene_output = crib_understanding.outputs.get("scene")

crib_scene_index = crib_stream.index(
    source=scene_output,
    name="Baby_Crib_Index",
    use_for=["semantic"],
)
```

`segmentation.window` and `sampling.frame_count` define how frequently the AI analyzes the stream:

* `window: "10s"` - Analyze every 10 seconds
* `frame_count: 1` - Extract 1 frame per analysis window

### Step 3: Define an Event for Baby Escape

Create an event in VideoDB to detect when the AI spots the baby attempting to climb out.

```python theme={null}
event_id = conn.create_event(
    event_prompt="Detect if the baby is trying to escape or climbing out of the crib.",
    label="baby_escape",
)
```

The event acts as a filter - when the AI's scene description matches this prompt, it triggers the event.

### Step 4: Attach an Alert for Real-Time Notifications

Link a real-time alert to this event, which will notify the parents instantly through a webhook.

```python theme={null}
webhook_url = "https://your-webhook-url.com"

alert_id = crib_scene_index.create_alert(
    event_id,
    callback_url=webhook_url,
)
```

## What You Receive

When a baby escape attempt is detected, your webhook receives a detailed alert payload:

```json theme={null}
{
  "event_id": "event-3adc40d26d6fed0d",
  "label": "baby_escape",
  "confidence": 0.95,
  "explanation": "The baby is actively trying to climb out of the crib by holding onto the top rail and attempting to pull itself up, which indicates an escape attempt.",
  "timestamp": "2025-05-28T23:36:39.979133+00:00",
  "start_time": "2025-05-29T05:06:36.612197+05:30",
  "end_time": "2025-05-29T05:06:46.612197+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748475396000000-1748475407000000.m3u8"
}
```

## Wrapping Up: Peace of Mind

With this system in place, Vidit and Meghna can finally sleep peacefully, knowing their child is being safely monitored through AI-driven surveillance.

But this is just one story. What if the same system could:

* Monitor an elderly parent at home — detecting falls or prolonged inactivity?
* Watch over a pet while the family is away, alerting them if it leaves a safe zone?
* Notify parents when a toddler approaches dangerous areas like staircases or kitchen counters?

The possibilities of real-time video intelligence at home are endless.

**What would you monitor next?**

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Baby_Crib_Monitoring.ipynb)**
>
> Open the complete implementation with additional features like WebSocket connections, audio indexing, and helper functions for stream visualization.

## Related Tutorials

> **[Property Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Intelligent security system with tiered threat detection

> **[Flash Flood Early Warning](/examples-and-tutorials/live-intelligence/flash-flood-detection)**
>
> Natural disaster monitoring with emergency alerts


# Live Cricket Highlight Detection
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/cricket-match-monitoring

AI-powered real-time detection of cricket highlights - sixes, fours, wickets, and catches

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Cricket_Match_Monitoring.ipynb)

## The Story

Tonight is the ICC World Cup finals between India and Pakistan. Millions are watching the match live, and the competition to post match highlights — sixes, wickets, and spectacular catches — on social media is fiercer than ever.

In the usual workflow, someone watches the match, waits for a moment to happen, then clips the video manually and uploads it online — often several minutes too late.

But we have a smarter way. What if AI could monitor the match for you, detect key moments in real-time, and instantly send alerts when something exciting happens — giving you a headstart on posting highlights while everyone else scrambles?

## What You'll Build

With VideoDB RTStream, you can build a system that:

* Monitors live cricket feeds continuously
* Detects key moments: sixes, fours, catches, and wickets
* Sends real-time alerts when highlights happen
* Provides instant video clips for social media sharing

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Connect to the Cricket Stream

```python theme={null}
rtsp_url = "rtsp://samples.rts.videodb.io:8554/cricket"
cricket_stream = coll.connect_rtstream(
    name="Cricket Match Stream",
    url=rtsp_url,
    store=True,
)
```

### Step 2: Understand Cricket Scenes

Continuous understanding uses seven-second windows and samples seven frames per window to capture fast-paced cricket action.

```python theme={null}
cricket_understanding = cricket_stream.understand(
    segmentation={"type": "time", "window": "7s"},
    analyzers=[{
        "type": "vlm",
        "name": "cricket_scene",
        "sampling": {"frame_count": 7},
        "config": {"prompt": "Identify and mention when a batsman hits a SIX (ball flying over the boundary rope), a FOUR (ball crosses boundary rope after bouncing), a CATCH OUT (fielder catches the ball mid-air before it touches ground) or a WICKET (when the wicket stumps are put down by the ball)."},
    }],
    store=True,
)
cricket_scene_output = cricket_understanding.outputs.get("cricket_scene")

cricket_scene_index = cricket_stream.index(
    source=cricket_scene_output,
    name="Cricket_Match_Scene_Index",
    use_for=["semantic"],
)
```

The higher `frame_count: 7` captures the rapid action of cricket, ensuring key moments aren't missed.

### Step 3: Define Four Highlight Events

Create events for each type of highlight that matches your indexing prompt:

```python theme={null}
# Six Hit
six_event_id = conn.create_event(
    event_prompt="Detect when a batsman hits a SIX - ball flying over the boundary rope.",
    label="six_hit",
)

# Four Hit
four_event_id = conn.create_event(
    event_prompt="Detect when a batsman hits a FOUR - ball crosses boundary rope after bouncing.",
    label="four_hit",
)

# Catch Out
catch_event_id = conn.create_event(
    event_prompt="Detect when a fielder takes a CATCH OUT - catching the ball mid-air.",
    label="catch_out",
)

# Wicket
wicket_event_id = conn.create_event(
    event_prompt="Detect when a WICKET occurs - wicket stumps are put down.",
    label="wicket",
)
```

### Step 4: Attach Alerts for Each Highlight

Create alerts for all four events on the same webhook:

```python theme={null}
webhook_url = "https://your-webhook-url.com"

# Attach alerts for each event type
six_alert_id = cricket_scene_index.create_alert(six_event_id, callback_url=webhook_url)
four_alert_id = cricket_scene_index.create_alert(four_event_id, callback_url=webhook_url)
catch_alert_id = cricket_scene_index.create_alert(catch_event_id, callback_url=webhook_url)
wicket_alert_id = cricket_scene_index.create_alert(wicket_event_id, callback_url=webhook_url)
```

All alerts go to the same webhook, with the `label` field indicating which highlight was detected.

## What You Receive

When a highlight is detected, your webhook receives an alert with the exact moment and video clip:

```json theme={null}
{
  "event_id": "event-3bfdd25d9239861b",
  "label": "four_hit",
  "confidence": 0.95,
  "explanation": "The ball is crossing the boundary after bouncing, indicating a FOUR has been scored.",
  "timestamp": "2025-05-29T00:11:09.256447+00:00",
  "start_time": "2025-05-29T05:40:32.544547+05:30",
  "end_time": "2025-05-29T05:40:39.730362+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1748477432000000-1748477440000000.m3u8"
}
```

## The Result

With this setup in place, broadcasters and content creators no longer have to wait, clip, and scramble. They can stay ahead of the crowd, instantly catching and sharing match-defining moments as they happen — turning every six, four, wicket, and catch into social media gold within seconds.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Cricket_Match_Monitoring.ipynb)**
>
> Open the complete implementation with additional features and helper functions for match analysis.

## Related Tutorials

> **[Multi-Camera Basketball Analytics](/examples-and-tutorials/live-intelligence/multicam-basketball-analysis)**
>
> Real-time sports analysis across multiple synchronized camera angles

> **[Multi-Camera Public Surveillance](/examples-and-tutorials/live-intelligence/multicam-public-surveillance)**
>
> Enterprise-scale multi-camera monitoring and event detection


# Flash Flood Early Warning System
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/flash-flood-detection

AI-powered real-time detection of flash floods for emergency early warning

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Flash_Flood_Detection.ipynb)

## The Story

The stunning Arizona deserts, known for their dry riverbeds and scenic beauty, hide a deadly risk. During the summer monsoon, sudden torrential rains can trigger flash floods in these seemingly harmless dry zones — with little or no warning.

Conventional alert systems relying on rain gauges or weather satellites often fail to deliver timely, location-specific warnings. By the time a danger alert is sent, it might already be too late.

But we have a smarter way. With VideoDB RTStream, we can install real-time cameras near flood-prone areas and let AI continuously monitor the visuals. As soon as the AI detects signs of a flash flood — like a sudden surge of water through dry land — it can instantly send alerts, giving local authorities and tourists precious moments to act.

## What You'll Build

With VideoDB RTStream, you can build a two-layer monitoring system that:

* Continuously monitors dry riverbeds and surrounding areas
* Detects flash floods immediately upon occurrence
* Identifies heavy rainfall as early warning signals
* Alerts for people in distress requiring rescue
* Sends real-time notifications to emergency services

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Connect to the Flood Monitoring Stream

```python theme={null}
rtsp_url = "rtsp://samples.rts.videodb.io:8554/floods"
flood_stream = coll.connect_rtstream(
    name="Flood Detection Stream",
    url=rtsp_url,
    store=True,
)
```

### Step 2: Create Primary Index - Flash Flood Detection

Create the primary scene index focused on immediate flood detection with more frequent analysis:

```python theme={null}
flood_understanding = flood_stream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[{
        "type": "vlm",
        "name": "flood_scene",
        "sampling": {"frame_count": 3},
        "config": {"prompt": "Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene."},
    }],
    store=True,
)
flood_scene_output = flood_understanding.outputs.get("flood_scene")

flood_scene_index = flood_stream.index(
    source=flood_scene_output,
    name="Flash_Flood_Detection_Index",
    use_for=["semantic"],
)
```

The `frame_count: 3` captures 3 frames every 5 seconds for detailed water movement analysis.

### Step 3: Define the Flash Flood Event

Create the first-stage event:

```python theme={null}
# Flash Flood Event
flood_event_id = conn.create_event(
    event_prompt="Detect a flash flood - sudden water surge across the riverbed.",
    label="flash_flood",
)
```

### Step 4: Attach the Flash Flood Alert

```python theme={null}
import asyncio

webhook_url = "https://your-webhook-url.com"
ws_wrapper = conn.connect_websocket()
ws = await ws_wrapper.connect()

flood_alert_id = flood_scene_index.create_alert(
    flood_event_id,
    callback_url=webhook_url,
    ws_connection_id=ws.connection_id,
)
```

### Finish the First Monitoring Stage

Disable its alert, stop its index and source understanding, and close its WebSocket before starting the independent riverbed-monitoring lesson.

```python theme={null}
flood_scene_index.disable_alert(flood_alert_id)
flood_scene_index.stop()
flood_understanding.stop()
await ws_wrapper.close()
```

### Step 5: Create the Riverbed Monitoring Index

This independent stage analyzes five-second windows with three frames for rainfall and rescue signals.

```python theme={null}
riverbed_monitoring_understanding = flood_stream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[{
        "type": "vlm",
        "name": "riverbed_scene",
        "sampling": {"frame_count": 3},
        "config": {"prompt": "Monitor the dry riverbed and surrounding area. If any person is detected, output 'Person detected, rescue needed'. Else if rainfall is detected, output 'Rainfall detected'. Otherwise, output 'No significant events detected.'"},
    }],
    store=True,
)
riverbed_scene_output = riverbed_monitoring_understanding.outputs.get("riverbed_scene")

early_warning_index = flood_stream.index(
    source=riverbed_scene_output,
    name="Riverbed_Monitoring_Index",
    use_for=["semantic"],
)
```

### Step 6: Attach Riverbed Alerts

```python theme={null}
rainfall_event_id = conn.create_event(
    event_prompt="Detect heavy rainfall - potential precursor to flooding.",
    label="heavy_rainfall",
)
rescue_event_id = conn.create_event(
    event_prompt="Detect a person in distress or stuck - rescue needed.",
    label="person_rescue_needed",
)

ws_wrapper = conn.connect_websocket()
ws = await ws_wrapper.connect()
rainfall_alert_id = early_warning_index.create_alert(
    rainfall_event_id,
    callback_url=webhook_url,
    ws_connection_id=ws.connection_id,
)
rescue_alert_id = early_warning_index.create_alert(
    rescue_event_id,
    callback_url=webhook_url,
    ws_connection_id=ws.connection_id,
)
```

## Alert Payloads

### Flash Flood Alert (Critical Priority)

When a flood is detected, the system sends an immediate alert with the exact location and video:

```json theme={null}
{
  "event_id": "event-flood-detection",
  "label": "flash_flood",
  "confidence": 0.95,
  "explanation": "Flash flood detected! Water is flowing rapidly and forcefully through the riverbed. The water appears muddy, carrying sediment and debris. Multiple people visible on rocky banks.",
  "timestamp": "2025-05-29T07:17:51.123456+00:00",
  "start_time": "2025-05-29T07:17:51.000000+05:30",
  "end_time": "2025-05-29T07:17:56.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748475471000000-1748475476000000.m3u8"
}
```

### Heavy Rainfall Alert (Warning Priority)

Early warning alerts help authorities respond proactively:

```json theme={null}
{
  "event_id": "event-rainfall-warning",
  "label": "heavy_rainfall",
  "confidence": 0.85,
  "explanation": "Heavy rainfall detected in the monitoring area. Conditions favorable for flash flooding.",
  "timestamp": "2025-05-29T07:10:00.000000+00:00",
  "start_time": "2025-05-29T07:10:00.000000+05:30",
  "end_time": "2025-05-29T07:10:15.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748475000000000-1748475015000000.m3u8"
}
```

## The Result

With this system in place, communities, tourists, and local authorities in Arizona's desert regions can receive immediate alerts when a dangerous flash flood occurs — gaining critical seconds to take cover, clear routes, or initiate rescues.

The two-layer system ensures:

* **Real-time flood detection** for immediate response
* **Early rainfall warnings** for proactive preparation
* **Rescue alerts** for people in distress

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Flash_Flood_Detection.ipynb)**
>
> Open the complete implementation with advanced monitoring features and configuration options.

## Related Tutorials

> **[Baby Crib Monitoring with AI](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> Real-time safety monitoring for infants with instant alerts

> **[Property Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Intelligent security system with tiered threat detection


# Overview
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/index

Real-time AI monitoring for cameras and streams - detect events, trigger alerts, and take action

## Live Intelligence

**Turn live camera feeds into actionable intelligence.** Detect events in real-time and trigger automated responses.

### When to Use This

* You have security cameras and need intrusion detection
* You want automated alerts for safety violations
* You're building sports highlight detection from live streams
* You need real-time monitoring with AI understanding

### What You'll Build

#### Safety & Security

> **[Baby Monitor AI](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> Detect when baby tries to climb out of crib

> **[Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Real-time alerts when unauthorized access detected

> **[Flash Flood Detection](/examples-and-tutorials/live-intelligence/flash-flood-detection)**
>
> Early warning system for rising water levels

#### Traffic & Roads

> **[Traffic Violations](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Detect no helmet, wrong-side driving, red light violations

> **[Road Monitoring](/examples-and-tutorials/live-intelligence/road-monitoring)**
>
> Analyze road conditions and traffic patterns

> **[Dashcam Analysis](/examples-and-tutorials/live-intelligence/roadcam-monitoring)**
>
> Process dashcam footage for incidents

#### Sports & Events

> **[Cricket Highlights](/examples-and-tutorials/live-intelligence/cricket-match-monitoring)**
>
> Auto-detect sixes, wickets, catches in real-time

> **[Multi-Cam Basketball](/examples-and-tutorials/live-intelligence/multicam-basketball-analysis)**
>
> Analyze games from multiple camera angles

> **[Multi-Cam Surveillance](/examples-and-tutorials/live-intelligence/multicam-public-surveillance)**
>
> Enterprise-scale multi-camera monitoring

> **[Multicam Quickstart](/examples-and-tutorials/live-intelligence/multicam-quickstart)**
>
> Connect four cameras, detect events, and create same-interval multi-angle evidence

#### Advanced AI

> **[TwelveLabs Integration](/examples-and-tutorials/live-intelligence/twelvelabs-integration)**
>
> Enhanced AI understanding with TwelveLabs Pegasus model

## How It Works

```
RTSP Stream → Continuous VLM Understanding → Scene Output Index → Event Detection → Webhook Alert → Your Action
```

Every alert includes:

* **Timestamp** - When the event occurred
* **Event type** - What was detected
* **Video clip** - Playable evidence
* **Confidence score** - How certain the detection is

## Related Documentation

> **[RTSP Ingest](/pages/ingest/live-streams/rtsp-ingest)**
>
> Connect live camera streams

> **[Event Detection Patterns](/pages/act/live-action/event-detection-patterns)**
>
> Design effective detection rules

> **[Alerts & Callbacks](/pages/act/live-action/alerts-and-callbacks)**
>
> Configure webhook notifications

> **[Stream Lifecycle](/pages/ingest/live-streams/stream-lifecycle)**
>
> Manage long-running streams

## Explore Use Cases by Category

### Safety & Security

> **[Baby Crib Monitoring](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> Real-time infant safety monitoring with escape detection

> **[Property Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Tiered threat detection with escalating alerts

> **[Flash Flood Early Warning](/examples-and-tutorials/live-intelligence/flash-flood-detection)**
>
> Natural disaster detection and emergency alerts

### Traffic & Road Safety

> **[Traffic Violations Detection](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Helmet violations, red lights, and unsafe driving detection

> **[Road Monitoring System](/examples-and-tutorials/live-intelligence/road-monitoring)**
>
> Accident, violation, and congestion detection

> **[Dashcam Analysis](/examples-and-tutorials/live-intelligence/roadcam-monitoring)**
>
> Automated traffic violation detection and reporting

### Sports & Entertainment

> **[Cricket Highlight Detection](/examples-and-tutorials/live-intelligence/cricket-match-monitoring)**
>
> Real-time sixes, wickets, and catch detection

> **[Multi-Camera Basketball](/examples-and-tutorials/live-intelligence/multicam-basketball-analysis)**
>
> Synchronized multi-angle sports analysis

### Enterprise Monitoring

> **[Public Safety Surveillance](/examples-and-tutorials/live-intelligence/multicam-public-surveillance)**
>
> 7-camera enterprise-scale monitoring system

> **[Advanced AI Integration](/examples-and-tutorials/live-intelligence/twelvelabs-integration)**
>
> TwelveLabs Pegasus for enhanced video understanding


# Intelligent Property Intrusion Detection
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/intrusion-detection

AI-powered property security system with tiered threat detection - loitering, attempts, and entry

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Intrusion_Detection.ipynb)

## The Story

Have you ever felt anxious leaving your home, shop, or property unattended? What if someone's lurking around your property? What if someone's trying the door or peeking through windows?

Sure — you could install IP cameras, but who has the time to watch them 24/7?

Good news — you don't have to anymore. With VideoDB RTStream, you can build a smart, AI-powered property surveillance system that actively monitors live video streams, detects suspicious activity, and immediately sends alerts for escalating security breaches — all without human supervision.

## What You'll Build

With VideoDB RTStream, you can build a tiered security system that:

* Monitors your property continuously
* Classifies threats into three levels: loitering, door interaction, and entry
* Sends escalating alerts based on threat severity
* Provides video evidence for each detection

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Connect to Your Property Stream

```python theme={null}
rtsp_url = "rtsp://samples.rts.videodb.io:8554/intruder"
property_stream = coll.connect_rtstream(
    name="Property Security Stream",
    url=rtsp_url,
    store=True,
)
```

### Step 2: Index Scenes with Multi-Level Threat Analysis

Create a scene index that monitors for all threat levels:

```python theme={null}
intruder_understanding = property_stream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[{
        "type": "vlm",
        "name": "scene",
        "sampling": {"frame_count": 2},
        "config": {"prompt": "Monitor the area around the house closely. Detect and classify human presence around the house as either loitering, interacting with the door/lock, or entering the house; otherwise, consider the area safe."},
    }],
    store=True,
)
scene_output = intruder_understanding.outputs.get("scene")

property_scene_index = property_stream.index(
    source=scene_output,
    name="Intruder_Detection_Index",
    use_for=["semantic"],
)
```

### Step 3: Define Three-Tiered Events

Create events for each threat level:

```python theme={null}
# Level 1: Loitering (Low Threat)
loitering_event_id = conn.create_event(
    event_prompt="Detect if a person is loitering near the house perimeter.",
    label="loitering_near_property",
)

# Level 2: Door Interaction (Medium Threat)
intrusion_attempt_event_id = conn.create_event(
    event_prompt="Detect if a person is interacting with the door or visibly checking the lock.",
    label="intrusion_attempt",
)

# Level 3: Property Entry (Critical Threat)
entry_event_id = conn.create_event(
    event_prompt="Detect if a person enters the house or crosses the property boundary unlawfully.",
    label="property_entry",
)
```

### Step 4: Attach Alerts with Progressive Response

```python theme={null}
webhook_url = "https://your-webhook-url.com"

# All three alerts routed to the same webhook with different priority labels
loitering_alert_id = property_scene_index.create_alert(loitering_event_id, callback_url=webhook_url)
intrusion_alert_id = property_scene_index.create_alert(intrusion_attempt_event_id, callback_url=webhook_url)
entry_alert_id = property_scene_index.create_alert(entry_event_id, callback_url=webhook_url)
```

## Alert Examples

### Level 1: Loitering Alert (Advisory)

```json theme={null}
{
  "event_id": "event-loitering-001",
  "label": "loitering_near_property",
  "confidence": 0.88,
  "explanation": "Person detected loitering near property perimeter. Not showing signs of intent to breach.",
  "timestamp": "2025-05-27T20:24:39.123456+00:00",
  "start_time": "2025-05-27T20:24:39.000000+05:30",
  "end_time": "2025-05-27T20:24:44.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748444679000000-1748444684000000.m3u8"
}
```

### Level 2: Intrusion Attempt Alert (Warning)

```json theme={null}
{
  "event_id": "event-intrusion-attempt-001",
  "label": "intrusion_attempt",
  "confidence": 0.92,
  "explanation": "Person is directly interacting with the door, checking the lock mechanism.",
  "timestamp": "2025-05-27T20:30:15.123456+00:00",
  "start_time": "2025-05-27T20:30:15.000000+05:30",
  "end_time": "2025-05-27T20:30:20.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748445015000000-1748445020000000.m3u8"
}
```

### Level 3: Property Entry Alert (Critical)

```json theme={null}
{
  "event_id": "event-entry-breach-001",
  "label": "property_entry",
  "confidence": 0.96,
  "explanation": "Unauthorized entry detected! Person has crossed the property boundary and is entering the house.",
  "timestamp": "2025-05-27T20:35:42.123456+00:00",
  "start_time": "2025-05-27T20:35:42.000000+05:30",
  "end_time": "2025-05-27T20:35:47.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748445342000000-1748445347000000.m3u8"
}
```

## The Result

With this in place, property owners can leave home without anxiety, knowing they'll be immediately notified if anyone is:

* Loitering nearby
* Interacting with the door
* Or breaking in

The tiered alert system lets you respond appropriately:

* **Level 1 alerts** → Stay informed and monitor
* **Level 2 alerts** → Increase vigilance, contact neighbors
* **Level 3 alerts** → Contact emergency services immediately

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Intrusion_Detection.ipynb)**
>
> Open the complete implementation with advanced configuration and customization options.

## Related Tutorials

> **[Baby Crib Monitoring](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> AI-powered safety monitoring for home environments

> **[Road Safety Monitoring](/examples-and-tutorials/live-intelligence/road-monitoring)**
>
> Real-time traffic and accident detection system


# Multi-Camera Basketball Analytics
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/multicam-basketball-analysis

Real-time 3-camera basketball monitoring with synchronized multi-angle highlight detection

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/multicam/Multicam_Basketball_Analysis.ipynb)

## The Challenge

Modern sports analysis faces growing challenges in providing real-time insights. From fast-paced plays to subtle player movements, monitoring every aspect of the game is tough.

Traditional broadcasters use multiple camera angles but lack AI intelligence to automatically detect key moments and generate synchronized multi-angle highlights in real-time.

What if AI could monitor all feeds, detect key plays instantly, and generate highlights automatically?

## What You'll Build

VideoDB RTStream brings AI-powered intelligence to multi-camera sports systems. In this guide, you'll build a system that:

* Connects 3 synchronized camera feeds (main court + two baskets)
* Detects key basketball events in real-time
* Sends alerts for every highlight across all angles
* Generates synchronized multi-angle video clips for playback

## Multi-Camera Setup

```
🏀 Basketball Arena - 3 Camera System

├── CAM 1: Main Court Field (Wide Angle)
│   └── rtsp://samples.rts.videodb.io:8554/bb-cam1
│
├── CAM 2: North Basket Area Field (Close Up)
│   └── rtsp://samples.rts.videodb.io:8554/bb-cam2
│
└── CAM 3: South Basket Area Field (Close Up)
    └── rtsp://samples.rts.videodb.io:8554/bb-cam3
```

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Connect All Three Camera Feeds

```python theme={null}
# Camera URLs
cameras = {
    "main_court": "rtsp://samples.rts.videodb.io:8554/bb-cam1",
    "north_basket": "rtsp://samples.rts.videodb.io:8554/bb-cam2",
    "south_basket": "rtsp://samples.rts.videodb.io:8554/bb-cam3"
}

# Connect all streams
rtstreams = {}
for cam_name, rtsp_url in cameras.items():
    rtstreams[cam_name] = coll.connect_rtstream(
        name=f"Basketball {cam_name.replace('_', ' ').title()}",
        url=rtsp_url,
        store=True,
    )
```

### Step 2: Create Shared Indexes on All Cameras

Define one analysis prompt and create indexes on all streams:

```python theme={null}
analysis_prompt = """Analyze this basketball game footage and describe:
1. Slam dunks, three-point shots, and scoring plays
2. Fast breaks and quick transition plays
3. Fouls, physical contact, or player collisions
Be specific about player positions and court location."""

scene_understandings = {}
for cam_name, rtstream in rtstreams.items():
    understanding = rtstream.understand(
        segmentation={"type": "time", "window": "10s"},
        analyzers=[{
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": analysis_prompt},
        }],
        store=True,
    )
    scene_understandings[cam_name] = {
        "understanding": understanding,
        "output": understanding.outputs.get("scene"),
    }

scene_indexes = {}
for cam_name, rtstream in rtstreams.items():
    scene = scene_understandings[cam_name]
    index = rtstream.index(
        source=scene["output"],
        name=f"Basketball_{cam_name.upper()}",
        use_for=["semantic"],
    )
    scene_indexes[cam_name] = {
        "understanding": scene["understanding"],
        "output": scene["output"],
        "index": index,
        "index_id": index.id,
    }
```

### Step 3: Define Four Shared Events

Create four events that apply across all camera angles:

```python theme={null}
# Slam Dunk
slam_dunk_event_id = conn.create_event(
    event_prompt="Detect a slam dunk: a player forcefully puts the ball through the hoop.",
    label="slam_dunk",
)

# Three-Pointer
three_pointer_event_id = conn.create_event(
    event_prompt="Detect a successful or attempted three-point shot from beyond the arc.",
    label="three_pointer",
)

# Foul
foul_event_id = conn.create_event(
    event_prompt="Detect fouls, aggressive behavior, or rule violations.",
    label="foul",
)

# Fast Break
fast_break_event_id = conn.create_event(
    event_prompt="Detect a fast break or quick transition toward the basket.",
    label="fast_break",
)
```

### Step 4: Attach Alerts to All Cameras

```python theme={null}
webhook_url = "https://your-webhook-url.com"

# Create alerts for all events on all cameras
for cam_name, scene_index in scene_indexes.items():
    scene_index["index"].create_alert(slam_dunk_event_id, callback_url=webhook_url)
    scene_index["index"].create_alert(three_pointer_event_id, callback_url=webhook_url)
    scene_index["index"].create_alert(foul_event_id, callback_url=webhook_url)
    scene_index["index"].create_alert(fast_break_event_id, callback_url=webhook_url)
```

Now you have 12 total alerts (4 events × 3 cameras).

## Alert Example

An illustrative alert envelope for a detected play can look like this; inspect the live payload before depending on optional fields:

```json theme={null}
{
  "event_id": "event-slam-dunk",
  "label": "slam_dunk",
  "confidence": 0.95,
  "explanation": "The ball is directly above the rim and net, descending through the hoop, indicating a basket is being scored.",
  "timestamp": "2025-09-10T11:21:16.614553+00:00",
  "start_time": "2025-09-10T16:50:45.698108+05:30",
  "end_time": "2025-09-10T16:51:00.698108+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1757503245000000-1757503261000000.m3u8"
}
```

The same event can trigger on all 3 cameras. Multi-angle replay assumes the camera clocks share a time basis; align clocks before treating same-interval clips as synchronized evidence.

## Synchronized Playback

Generate synchronized clips from all cameras for the same moment:

```python theme={null}
# When you receive a play alert with timestamp, extract it
alert_timestamp = "2025-09-10T16:50:45.698108+05:30"
import datetime
epoch_time = int(datetime.datetime.fromisoformat(alert_timestamp.replace("Z", "+00:00")).timestamp())

# Generate streams from all cameras for same time window (±5 seconds)
time_window = 5
synchronized_streams = {}

for cam_name, rtstream in rtstreams.items():
    rtstream.generate_stream(
        start=epoch_time - time_window,
        end=epoch_time + time_window
    )
    synchronized_streams[cam_name] = rtstream.stream_url
```

Now you have three clips requested for the same time window from different angles.

## Advanced: Webhook Integration with ngrok

For real-time event handling, set up ngrok tunneling:

```bash theme={null}
# Terminal: Start ngrok tunnel
ngrok http 5000
# Get public URL: https://xxxx-xxxx-xxxx.ngrok.io
```

Then use this public webhook URL when creating alerts for instant event notifications.

## The Result

This demo shows how AI turns raw game feeds into actionable sports intelligence, enabling:

* **Real-time game analysis** across multiple angles
* **Instant highlight generation** with synchronized playback
* **Multi-angle replay** for coaches and analysts
* **Automated event detection** across multiple viewpoints
* **Forensic evidence** for rule reviews and disputes

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/multicam/Multicam_Basketball_Analysis.ipynb)**
>
> Open the complete implementation with webhook setup, ngrok tunneling, and data processing.

## Related Tutorials

> **[Cricket Highlights Detection](/examples-and-tutorials/live-intelligence/cricket-match-monitoring)**
>
> Auto-detect sports highlights in real-time

> **[Multi-Camera Public Surveillance](/examples-and-tutorials/live-intelligence/multicam-public-surveillance)**
>
> Enterprise-scale multi-camera monitoring system


# Multi-Camera Public Safety Surveillance
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/multicam-public-surveillance

AI-powered 4-camera surveillance system for public safety monitoring with multi-angle alerting

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/multicam/Multicam_Public_Surveillance.ipynb)

## The Challenge

Modern cities face growing challenges in ensuring public safety. From crowded metros to busy intersections, monitoring many locations in real-time is tough.

Traditional systems rely on humans watching multiple screens — costly, error-prone, and unable to correlate events across cameras.

What if AI could monitor all feeds, detect incidents instantly, and alert authorities automatically?

## What You'll Build

VideoDB RTStream brings AI-powered intelligence to multi-camera systems. In this guide, you'll build a surveillance network that:

* Connects 4 camera feeds across a public space
* Detects 3 types of security incidents simultaneously
* Monitors 12 event streams (4 cameras × 3 events)
* Generates 2×2 multi-angle evidence for forensic analysis
* Sends real-time alerts to security authorities

## 4-Camera Surveillance Network

```
🏙️ City Public Square - 4 Camera System

├── CAM 1: Plaza Overview (High Angle - East)
│   └── rtsp://samples.rts.videodb.io:8554/pub-cam1
│
├── CAM 2: Main Walkway (Close Up)
│   └── rtsp://samples.rts.videodb.io:8554/pub-cam2
│
├── CAM 3: Stairway Junction
│   └── rtsp://samples.rts.videodb.io:8554/pub-cam3
│
└── CAM 4: Central Plaza (Mid Angle)
    └── rtsp://samples.rts.videodb.io:8554/pub-cam4
```

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Define All 4 Camera Streams

```python theme={null}
cameras = {
    "plaza_overview": "rtsp://samples.rts.videodb.io:8554/pub-cam1",
    "main_walkway": "rtsp://samples.rts.videodb.io:8554/pub-cam2",
    "stairway_junction": "rtsp://samples.rts.videodb.io:8554/pub-cam3",
    "central_plaza": "rtsp://samples.rts.videodb.io:8554/pub-cam4",
}

# Connect all camera streams
rtstreams = {}
for cam_name, rtsp_url in cameras.items():
    rtstreams[cam_name] = coll.connect_rtstream(
        name=f"Surveillance: {cam_name.replace('_', ' ').title()}",
        url=rtsp_url,
        store=True,
    )
```

### Step 2: Create Shared Indexes on All Cameras

Define one comprehensive analysis prompt and create indexes on all 4 streams:

```python theme={null}
analysis_prompt = """Analyze this surveillance footage. Describe:
1. People with trolley bags, backpacks, or luggage
2. Crowd behavior (groups gathering, movement patterns)
3. Unusual objects (unattended items)
Be specific about location and appearance."""

scene_understandings = {}
for cam_name, rtstream in rtstreams.items():
    understanding = rtstream.understand(
        segmentation={"type": "time", "window": "10s"},
        analyzers=[{
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": analysis_prompt},
        }],
        store=True,
    )
    scene_understandings[cam_name] = {
        "understanding": understanding,
        "output": understanding.outputs.get("scene"),
    }

scene_indexes = {}
for cam_name, rtstream in rtstreams.items():
    scene = scene_understandings[cam_name]
    index = rtstream.index(
        source=scene["output"],
        name=f"Surveillance_{cam_name.upper()}",
        use_for=["semantic"],
    )
    scene_indexes[cam_name] = {
        "understanding": scene["understanding"],
        "output": scene["output"],
        "index": index,
        "index_id": index.id,
    }
```

### Step 3: Define 3 Shared Security Events

Create events that apply across all cameras:

```python theme={null}
# Unattended Luggage (High Priority)
luggage_event_id = conn.create_event(
    event_prompt="Detect luggage, backpacks, or packages left unattended.",
    label="unattended_luggage",
)

# Large Crowd Formation (Alert)
crowd_event_id = conn.create_event(
    event_prompt="Identify when >4 people gather quickly in a concentrated area.",
    label="large_crowd_formation",
)

# Person with Trolley (Tracking)
trolley_event_id = conn.create_event(
    event_prompt="Detect any person with trolley bag or rolling suitcase.",
    label="person_with_trolley",
)

```

### Step 4: Attach Alerts to All 4 Cameras

```python theme={null}
webhook_url = "https://your-security-center-webhook.com"

event_ids = {
    "luggage": luggage_event_id,
    "crowd": crowd_event_id,
    "trolley": trolley_event_id,
}

# Create alerts for all 3 events on all 4 cameras (12 total alerts)
for cam_name, scene_index in scene_indexes.items():
    for event_type, event_id in event_ids.items():
        scene_index["index"].create_alert(event_id, callback_url=webhook_url)
```

Now you have a comprehensive 4-camera × 3-event surveillance grid (12 alert streams).

## Illustrative Alert Envelopes

These examples show a possible live payload. Inspect the received alert before relying on optional fields.

### Unattended Luggage Alert (Critical Priority)

```json theme={null}
{
  "event_id": "event-luggage-001",
  "label": "unattended_luggage",
  "confidence": 0.96,
  "explanation": "A black backpack is detected on the plaza ground with no person visibly attached to it. Has been stationary for >1 minute.",
  "timestamp": "2025-09-10T17:02:15.811085+00:00",
  "start_time": "2025-09-10T22:31:50.886736+05:30",
  "end_time": "2025-09-10T22:32:05.886736+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1757523710000000-1757523726000000.m3u8"
}
```

### Large Crowd Formation Alert

```json theme={null}
{
  "event_id": "event-crowd-001",
  "label": "large_crowd_formation",
  "confidence": 0.92,
  "explanation": "Multiple individuals (8+) rapidly gathering in central plaza area. Unusual congregation pattern.",
  "timestamp": "2025-09-10T17:15:42.123456+00:00",
  "start_time": "2025-09-10T22:45:30.000000+05:30",
  "end_time": "2025-09-10T22:45:45.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1757524530000000-1757524545000000.m3u8"
}
```

### Person with Trolley Alert (Tracking)

```json theme={null}
{
  "event_id": "event-trolley-001",
  "label": "person_with_trolley",
  "confidence": 0.95,
  "explanation": "Female with black rolling suitcase detected moving through plaza.",
  "timestamp": "2025-09-10T17:02:15.811085+00:00",
  "start_time": "2025-09-10T22:31:50.886736+05:30",
  "end_time": "2025-09-10T22:32:05.886736+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1757523710000000-1757523726000000.m3u8"
}
```

## Synchronized Multi-Angle Evidence

When an incident is detected, you can retrieve clips from all 4 cameras for the same requested interval and arrange them in a 2×2 view:

```python theme={null}
# Extract incident timestamp from alert
incident_epoch = 1757523730  # from alert data

# Generate four clips for the same requested interval (±10 seconds)
time_window = 10
evidence_clips = {}

for cam_name, rtstream in rtstreams.items():
    rtstream.generate_stream(
        start=incident_epoch - time_window,
        end=incident_epoch + time_window
    )
    evidence_clips[cam_name] = rtstream.stream_url

# Compose the four HLS clips in a 2×2 layout for review.
```

Use clock-aligned camera sources before treating clips from the same requested interval as synchronized evidence.

## The Result

This demo shows how AI turns raw surveillance feeds into actionable intelligence for:

* **Public safety monitoring** across wide areas
* **Crowd management** and anomaly detection
* **Security threat detection** in real-time
* **Forensic investigation** with multi-angle evidence
* **Real-time incident response** with 2×2 multi-angle playback

Key capabilities:

* **12 Event Streams** - Coverage of 4 camera angles with 3 security events each
* **Sub-Second Alerts** - Instant notification when incidents occur
* **Multi-Angle Evidence** - Synchronized clips from all cameras for investigations
* **Scalability** - Pattern can extend to 10, 20, or 100+ cameras
* **24/7 Monitoring** - Fully automated, no human fatigue

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/multicam/Multicam_Public_Surveillance.ipynb)**
>
> Open the complete implementation with webhook setup, ngrok integration, and forensic analysis.

## Related Tutorials

> **[Property Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Intelligent security system with tiered threat detection

> **[Multi-Camera Basketball Analytics](/examples-and-tutorials/live-intelligence/multicam-basketball-analysis)**
>
> Real-time sports analysis with synchronized playback


# Multicam Quickstart
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/multicam-quickstart

Connect four live cameras, receive WebSocket alerts, and create synchronized multi-angle evidence

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/quickstart/Multicam_Quickstart.ipynb)

## What You'll Build

This quickstart covers the reusable multicam pattern behind VideoDB's live-intelligence examples:

* Connect and store four RTSP feeds
* Run an independent visual index for every camera
* Detect three events on every feed
* Receive and attribute alerts over one WebSocket
* Retrieve the detected interval from all cameras
* Compose a synchronized 2×2 video with the Timeline Editor

The sample uses four public-plaza feeds and detects people with trolley bags, crowd formation, and unattended luggage. Replace the URLs and prompts to adapt the pattern to your own cameras.

## Setup

```bash theme={null}
pip install videodb
```

```python theme={null}
import videodb

conn = videodb.connect(api_key="your_api_key")
coll = conn.get_collection()
```

## 1. Connect the Camera Feeds

```python theme={null}
CAMERA_CONFIG = {
    "cam1": ("Plaza Overview", "rtsp://samples.rts.videodb.io:8554/pub-cam1"),
    "cam2": ("Main Walkway", "rtsp://samples.rts.videodb.io:8554/pub-cam2"),
    "cam3": ("Stairway Junction", "rtsp://samples.rts.videodb.io:8554/pub-cam3"),
    "cam4": ("Central Plaza", "rtsp://samples.rts.videodb.io:8554/pub-cam4"),
}

streams = {}
for camera_id, (_, rtsp_url) in CAMERA_CONFIG.items():
    streams[camera_id] = coll.connect_rtstream(
        name=f"Surveillance_{camera_id}",
        url=rtsp_url,
        store=True,
    )
```

> [!NOTE]
>
> Keep `store=True` enabled. Multicam replay depends on requesting the same historical time range from every camera.

## 2. Index Every Camera

```python theme={null}
analysis_prompt = """Analyze this surveillance footage. Describe:
1. People with trolley bags, backpacks, or luggage
2. Crowd behavior and groups gathering
3. Unusual or unattended objects
Be specific about location and appearance."""

scene_understandings = {}
for camera_id, stream in streams.items():
    understanding = stream.understand(
        segmentation={"type": "time", "window": "10s"},
        analyzers=[{"type": "vlm", "name": "scene",
                    "sampling": {"frame_count": 1},
                    "config": {"prompt": analysis_prompt}}],
        store=True,
    )
    scene_understandings[camera_id] = {
        "understanding": understanding,
        "output": understanding.outputs.get("scene"),
    }

scene_indexes = {}
for camera_id, stream in streams.items():
    scene = scene_understandings[camera_id]
    index = stream.index(
        source=scene["output"],
        name=f"Surveillance_{camera_id}_Index",
        use_for=["semantic"],
    )
    scene_indexes[camera_id] = {
        "understanding": scene["understanding"],
        "output": scene["output"],
        "index": index,
        "index_id": index.id,
    }
```

## 3. Create Events and Alerts

```python theme={null}
EVENTS_CONFIG = [
    ("person_with_trolley", "Detect any person with a trolley bag or rolling suitcase."),
    ("large_crowd_formation", "Identify 5+ people gathering quickly."),
    ("unattended_luggage", "Detect luggage left unattended for 1+ minute."),
]

events = {
    label: conn.create_event(event_prompt=prompt, label=label)
    for label, prompt in EVENTS_CONFIG
}
```

Open one WebSocket and attach every camera-event combination to it:

```python theme={null}
import asyncio

ws_wrapper = conn.connect_websocket()
ws = await ws_wrapper.connect()

alerts = {}
for camera_id, scene in scene_indexes.items():
    alerts[camera_id] = {}
    for label, event_id in events.items():
        alerts[camera_id][label] = scene["index"].create_alert(
            event_id,
            callback_url="https://example.com",
            ws_connection_id=ws.connection_id,
        )
```

This creates 12 alerts: four cameras multiplied by three event types.

## 4. Receive Alerts from Every Camera

```python theme={null}
stream_to_camera = {stream.id: camera_id for camera_id, stream in streams.items()}
received_alerts = []

async def listen_for_alerts(timeout=30):
    try:
        async with asyncio.timeout(timeout):
            async for message in ws.receive():
                if message.get("channel") == "alert":
                    received_alerts.append(message)
                    camera_id = stream_to_camera.get(message.get("rtstream_id"))
                    data = message.get("data", {})
                    print(camera_id, data.get("label"), data.get("confidence"))
    except asyncio.TimeoutError:
        print(f"Received {len(received_alerts)} alert(s)")

await listen_for_alerts()
```

An alert's top-level `rtstream_id` identifies the camera. The observed `data` envelope can include the event label, confidence, explanation, detected `start` and `end` timestamps, and a clip URL; inspect live messages because optional fields are delivery-dependent.

## 5. Retrieve a Synchronized Time Window

This uses the same Unix timestamp range across every stream, so the camera clocks must already be aligned for frame-level synchronization.

```python theme={null}
stream_urls = {}

selected = next(
    (
        alert for alert in received_alerts
        if isinstance(alert, dict)
        and isinstance(alert.get("data"), dict)
        and alert["data"].get("label") == "person_with_trolley"
    ),
    None,
)

if selected:
    data = selected["data"]
    start, end = data.get("start"), data.get("end")
    if isinstance(start, (int, float)) and isinstance(end, (int, float)) and start < end:
        padding = 10
        clip_start = int(start - padding)
        clip_end = int(end + padding)
        for camera_id, stream in streams.items():
            stream.generate_stream(clip_start, clip_end)
            stream_urls[camera_id] = stream.stream_url
    else:
        print("The selected alert does not include a valid data.start/data.end interval.")
else:
    print("No trolley alerts received yet; run the listener longer or choose another event.")
```

> [!WARNING]
>
> `generate_stream()` returns a player URL. Read `stream.stream_url` after the call when you need the HLS media URL for downloading or composition.

## 6. Compose the Multi-Angle Video

The notebook completes the replay in three stages:

1. Download each HLS URL with FFmpeg.
2. Upload the four MP4 clips with `coll.upload(file_path=...)`.
3. Place each `VideoAsset(id=...)` on its own Timeline Editor track.

```python theme={null}
import subprocess

uploaded_videos = {}
if stream_urls:
    for camera_id, stream_url in stream_urls.items():
        output_file = f"{camera_id}_clip.mp4"
        subprocess.run(
            ["ffmpeg", "-y", "-i", stream_url, "-c", "copy", output_file],
            check=True,
        )
        uploaded_videos[camera_id] = coll.upload(file_path=output_file)

    final_duration = min(video.length for video in uploaded_videos.values())
else:
    print("No synchronized clips are available yet; skip composition.")
```

```python theme={null}
if uploaded_videos:
    from videodb.editor import Timeline, Track, Clip, VideoAsset, Position, Offset, Fit

    timeline = Timeline(conn)
    timeline.resolution = "1280x720"
    timeline.background = "#404040"

    layout = [
        ("cam1", Position.top_left, Offset(x=0.03, y=0.025)),
        ("cam2", Position.top_right, Offset(x=-0.03, y=0.025)),
        ("cam3", Position.bottom_left, Offset(x=0.03, y=-0.025)),
        ("cam4", Position.bottom_right, Offset(x=-0.03, y=-0.025)),
    ]

    for camera_id, position, offset in layout:
        track = Track()
        track.add_clip(0, Clip(
            asset=VideoAsset(id=uploaded_videos[camera_id].id),
            duration=final_duration,
            fit=Fit.crop,
            position=position,
            offset=offset,
            scale=0.45,
        ))
        timeline.add_track(track)

    multicam_url = timeline.generate_stream()
else:
    print("No synchronized clips are available yet; skip composition.")
```

Open the notebook for the complete FFmpeg, upload, duration matching, camera label, and playback cells.

## Cleanup

```python theme={null}
for camera_id, camera_alerts in alerts.items():
    for alert_id in camera_alerts.values():
        scene_indexes[camera_id]["index"].disable_alert(alert_id)

for scene in scene_indexes.values():
    scene["index"].stop()
    scene["understanding"].stop()

for stream in streams.values():
    stream.stop()

await ws_wrapper.close()
```

> **[Open the Multicam Quickstart](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/quickstart/Multicam_Quickstart.ipynb)**
>
> Run the complete four-camera workflow in Google Colab.

## Next Steps

> **[Public Safety Surveillance](/examples-and-tutorials/live-intelligence/multicam-public-surveillance)**
>
> Apply the pattern to a public-plaza monitoring use case

> **[Basketball Analytics](/examples-and-tutorials/live-intelligence/multicam-basketball-analysis)**
>
> Adapt multicam alerts and composition to live sports


# Multi-Use Road Monitoring System
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/road-monitoring

Integrated road safety system detecting accidents, violations, and congestion in real-time

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Road_Monitoring.ipynb)

## The Story

Road accidents happen every single day — and many lives are lost not because of the severity of the crash itself, but because victims don't receive timely aid and medical attention.

Often, there's no one around to report an incident, or bystanders take too long to respond — either from shock, panic, or the overwhelming nature of witnessing an accident.

But in those critical moments, even a few seconds can make the difference between life and death.

This is where AI can help. With VideoDB RTStream, we can deploy cameras at accident-prone locations and let AI constantly monitor live video streams. As soon as an accident occurs, AI will detect it and instantly send alerts to nearby emergency services or traffic authorities.

## What You'll Build

In this guide, we'll build a comprehensive road monitoring system that addresses three critical traffic challenges:

* **Accident Detection** - Instant alerts for vehicle collisions
* **Violation Monitoring** - Real-time detection of traffic rule breaking
* **Congestion Detection** - Early warning for traffic jams

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Part 1: Accident Detection at Toll Plaza

**Step 1: Connect to Toll Plaza Stream**

```python theme={null}
accident_stream = coll.connect_rtstream(
    name="Toll Plaza Accident Monitor",
    url="rtsp://samples.rts.videodb.io:8554/accident",
    store=True,
)
```

**Step 2: Index for Accident Detection**

```python theme={null}
accident_understanding = accident_stream.understand(
    segmentation={"type": "time", "window": "10s"},
    analyzers=[{
        "type": "vlm",
        "name": "scene",
        "sampling": {"frame_count": 2},
        "config": {"prompt": "Monitor the toll plaza road carefully. Detect if a vehicle collides, crashes, or a person falls. Describe the situation clearly if an accident occurs."},
    }],
    store=True,
)
scene_output = accident_understanding.outputs.get("scene")

accident_index = accident_stream.index(
    source=scene_output,
    name="Accident_Index",
    use_for=["semantic"],
)
```

**Step 3: Create Accident Event & Alert**

```python theme={null}
import asyncio

accident_event_id = conn.create_event(
    event_prompt="Detect if an accident or vehicle collision takes place.",
    label="road_accident",
)

webhook_url = "https://your-webhook-url.com"
ws_wrapper = conn.connect_websocket()
ws = await ws_wrapper.connect()
accident_alert_id = accident_index.create_alert(
    accident_event_id,
    callback_url=webhook_url,
    ws_connection_id=ws.connection_id,
)
```

### Finish Accident Monitoring

Disable the accident alert, stop its continuous index and source understanding, and close its WebSocket before beginning the separate toll-monitoring stage.

```python theme={null}
accident_index.disable_alert(accident_alert_id)
accident_index.stop()
accident_understanding.stop()
await ws_wrapper.close()
```

### Part 2: Traffic Violation Detection at Toll Booth

**Step 4: Create Violation Index on Same Stream**

Create a separate index on the same stream with faster analysis:

```python theme={null}
rule_understanding = accident_stream.understand(
    segmentation={"type": "time", "window": "3s"},
    analyzers=[{
        "type": "vlm",
        "name": "scene",
        "sampling": {"frame_count": 2},
        "config": {"prompt": "Monitor the toll plaza carefully. Detect if any vehicle breaks traffic rules — for example, skipping the toll booth, crossing without stopping, driving in the wrong lane, or ignoring the barrier. Describe such violations clearly."},
    }],
    store=True,
)
scene_output = rule_understanding.outputs.get("scene")

violation_index = accident_stream.index(
    source=scene_output,
    name="Toll_Rule_Violation_Index",
    use_for=["semantic"],
)
```

Continuous three-second segmentation helps capture short violations.

**Step 5: Create Violation Event & Alert**

```python theme={null}
violation_event_id = conn.create_event(
    event_prompt="Detect if a vehicle breaks traffic rules at the toll plaza.",
    label="toll_rule_violation",
)

ws_wrapper = conn.connect_websocket()
ws = await ws_wrapper.connect()
violation_alert_id = violation_index.create_alert(
    violation_event_id,
    callback_url=webhook_url,
    ws_connection_id=ws.connection_id,
)
```

### Part 3: Traffic Congestion Detection

**Step 6: Connect to Highway Stream**

```python theme={null}
congestion_stream = coll.connect_rtstream(
    name="Highway Traffic Monitor",
    url="rtsp://3.6.198.206:8554/traffic",
    store=True,
)
```

**Step 7: Index for Congestion**

```python theme={null}
traffic_jam_understanding = congestion_stream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[{
        "type": "vlm",
        "name": "scene",
        "sampling": {"frame_count": 3},
        "config": {"prompt": "Monitor the lanes of vehicles over several video frames. If the whole lane of cars consistently moves very slowly or stops, classify the situation as 'traffic congestion detected.' Otherwise, classify it as 'regular traffic flow.'"},
    }],
    store=True,
)
scene_output = traffic_jam_understanding.outputs.get("scene")

congestion_index = congestion_stream.index(
    source=scene_output,
    name="Traffic_Congestion_Index",
    use_for=["semantic"],
)
```

**Step 8: Create Congestion Event & Alert**

```python theme={null}
congestion_event_id = conn.create_event(
    event_prompt="Detect if traffic congestion or jam is forming.",
    label="traffic_congestion",
)

congestion_alert_id = congestion_index.create_alert(congestion_event_id, callback_url=webhook_url)
```

## Alert Examples

### Accident Alert (Critical)

```json theme={null}
{
  "event_id": "event-accident-001",
  "label": "road_accident",
  "confidence": 0.96,
  "explanation": "Vehicle collision detected at toll plaza. Two vehicles involved with visible impact.",
  "timestamp": "2025-05-29T10:15:33.123456+00:00",
  "start_time": "2025-05-29T10:15:33.000000+05:30",
  "end_time": "2025-05-29T10:15:43.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748485533000000-1748485543000000.m3u8"
}
```

### Violation Alert (Warning)

```json theme={null}
{
  "event_id": "event-violation-001",
  "label": "toll_rule_violation",
  "confidence": 0.90,
  "explanation": "Vehicle crossing toll plaza without stopping, ignoring barrier.",
  "timestamp": "2025-05-29T10:20:15.123456+00:00",
  "start_time": "2025-05-29T10:20:15.000000+05:30",
  "end_time": "2025-05-29T10:20:18.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748485815000000-1748485818000000.m3u8"
}
```

### Congestion Alert (Advisory)

```json theme={null}
{
  "event_id": "event-congestion-001",
  "label": "traffic_congestion",
  "confidence": 0.95,
  "explanation": "High vehicle density and slow movement observed across multiple lanes, indicating traffic congestion.",
  "timestamp": "2025-05-29T13:41:20.123456+00:00",
  "start_time": "2025-05-29T13:41:08.000000+05:30",
  "end_time": "2025-05-29T13:41:13.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748504468000000-1748504473000000.m3u8"
}
```

## The Result

With these systems in place, we built a smart AI-powered road monitoring system addressing two major challenges:

**Accident Detection at toll plazas** — instantly spotting crashes and alerting emergency services without delay.

**Traffic Congestion Detection on busy highways** — catching early signs of jams so authorities can act before things spiral.

Together, these tools show how AI video monitoring can make roads safer, traffic smoother, and emergency responses faster — all in real time.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Road_Monitoring.ipynb)**
>
> Open the complete implementation with multi-stream monitoring and advanced configuration.

## Related Tutorials

> **[Traffic Violations Detection](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Real-time detection of helmet violations, wrong-side driving, and red light violations

> **[Flash Flood Early Warning](/examples-and-tutorials/live-intelligence/flash-flood-detection)**
>
> Natural disaster monitoring with emergency alerts


# Dashcam Monitoring of Traffic
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/roadcam-monitoring

Real-time traffic enforcement system detecting violations like no helmet, red lights, and unsafe driving

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Roadcam.ipynb)

## The Challenge

Traffic violations cause preventable accidents and deaths every day. From riders without helmets to drivers using mobile phones, unsafe behavior on roads puts lives at risk.

Traditional enforcement requires traffic police to manually monitor roads — costly, limited in coverage, and often reactive rather than preventive.

What if AI could automatically detect violations in real-time and generate evidence for enforcement? This is where AI-powered dashcam monitoring comes in.

## What You'll Build

With VideoDB RTStream, you can build an automated traffic violation detection system that:

* Monitors live road feeds continuously
* Detects multiple violation types automatically
* Generates video evidence for enforcement
* Sends real-time alerts to traffic authorities
* Works 24/7 without manual supervision

## Violations Detected

The system monitors for six critical violations:

* **No Helmet** - Two-wheeler riders without proper head protection
* **Mobile Phone Use** - Drivers operating vehicles while using phones
* **Wrong Side Driving** - Vehicles traveling against traffic flow
* **Red Light Violation** - Vehicles crossing intersections during red signals
* **Triple Riding** - More than two people on a single two-wheeler
* **No Seatbelt** - Drivers or passengers without seatbelt protection

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Connect to Your Roadcam Stream

```python theme={null}
rtsp_url = "rtsp://your-dashcam-or-roadcam-url"
roadcam_stream = coll.connect_rtstream(
    name="Traffic Violation Detector",
    url=rtsp_url,
    store=True,
)
```

### Step 2: Index Scenes with Violation Analysis

Create a scene index tuned for capturing traffic violations with focused frame sampling:

```python theme={null}
violation_prompt = """
Focus on vehicles visible on the road and monitor them for the following traffic rule violations:

1. NO HELMET: Two-wheeler (motorcycle/scooter) rider or pillion not wearing a helmet
2. MOBILE PHONE USE: Driver using mobile phone while operating the vehicle
3. WRONG SIDE DRIVING: Vehicle traveling against the designated traffic flow direction
4. RED LIGHT VIOLATION: Vehicle crossing the stop line or intersection when traffic signal is red
5. TRIPLE RIDING: More than two people riding on a single two-wheeler
6. NO SEATBELT: Driver or front passenger in a four-wheeler not wearing seatbelt

Analyze the frames carefully. If you detect one or more violations from the above list, respond EXACTLY in this format:

Traffic Rule Violated
Vehicle: [vehicle type and color, e.g., "Black motorcycle", "White sedan"]
Plate Number: [license plate number if visible, otherwise "Not Visible"]
Violation: [specific violation(s) from the list above]
Description: [brief description of what you observed]

If NO violation is detected, respond ONLY with:
No Traffic Rule Violation Detected
"""

violation_understanding = roadcam_stream.understand(
    segmentation={"type": "time", "window": "7s"},
    analyzers=[{
        "type": "vlm",
        "name": "traffic_violations",
        "sampling": {"frame_count": 5},
        "config": {"prompt": violation_prompt},
    }],
    store=True,
)
traffic_violations_output = violation_understanding.outputs.get("traffic_violations")

violation_index = roadcam_stream.index(
    source=traffic_violations_output,
    name="Traffic_Violation_Index_7_5",
    use_for=["semantic"],
)
```

The `frame_count: 5` captures multiple angles of each violation for clear evidence.

### Step 3: Create Violation Event

```python theme={null}
violation_event_id = conn.create_event(
    event_prompt="""Detect when a traffic rule violation occurs, such as no helmet, mobile phone use,
                    wrong side driving, red light violation, triple riding, or no seatbelt.
                    Your explanation should clearly mention:
                    - Traffic Rule Violated
                    - Vehicle type and color
                    - License plate number if visible
                    - Violation type
                    - Brief description of what was observed""",
    label="traffic_violation",
)
```

### Step 4: Attach Alert for Automated Reporting

```python theme={null}
webhook_url = "https://your-traffic-authority-webhook.com"
violation_alert_id = violation_index.create_alert(violation_event_id, callback_url=webhook_url)
```

## Alert Example

When a violation is detected, the system sends structured evidence:

```json theme={null}
{
  "event_id": "event-violation-no-helmet",
  "label": "traffic_violation",
  "confidence": 0.95,
  "explanation": "Traffic Rule Violated: NO HELMET
                  Vehicle: Orange scooter
                  License Plate: DL 3S CW 4952
                  Violation: Both rider and pillion not wearing helmets
                  Description: The rider and pillion rider on the orange scooter are clearly not wearing helmets, which is a critical safety violation.",
  "timestamp": "2025-05-29T04:57:44.123456+00:00",
  "start_time": "2025-05-29T10:27:20.000000+05:30",
  "end_time": "2025-05-29T10:27:27.000000+05:30",
  "stream_url": "https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748475396000000-1748475407000000.m3u8"
}
```

The structured explanation makes it easy to automatically process violations for enforcement.

## Integration with Enforcement Systems

The alerts can be integrated with existing traffic enforcement workflows:

**Automated Report Generation:**

* Extract vehicle details and license plate
* Generate violation evidence video
* Create standardized enforcement notice
* Route to appropriate authority

**Data-Driven Enforcement:**

* Track violation hotspots
* Identify repeat offenders
* Optimize traffic enforcement resources
* Measure safety improvements

## The Result

This system enables automated enforcement of traffic rules, reducing need for manual traffic police monitoring and improving road safety through instant violation detection and alerts.

Key benefits:

* **24/7 Monitoring** - No breaks, always watching
* **Instant Evidence** - Video proof for every violation
* **Consistent Enforcement** - AI doesn't have bias or fatigue
* **Data Insights** - Analytics on violation patterns and hotspots
* **Deterrent Effect** - Known monitoring discourages violations

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Roadcam.ipynb)**
>
> Open the complete implementation with additional configuration and integration examples.

## Related Tutorials

> **[Traffic Violations Detection](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Real-time detection of helmet violations and traffic rule breaking

> **[Road Monitoring System](/examples-and-tutorials/live-intelligence/road-monitoring)**
>
> Multi-use road monitoring for accidents and congestion


# Traffic Violation Detection
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/traffic-violations

Detect and report real-time traffic rule violations

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Roadcam.ipynb)

## The Viral Inspiration

You've seen the post. A guy got so fed up with daily traffic chaos that he automated his helmet camera to snap violations and send them straight to the traffic police - no manual intervention needed.

> **[View the inspiration](https://x.com/the2ndfloorguy/status/2007411050984157452?s=20)**
>
> Check out the original post that sparked this idea

The internet loved it. And we thought - why not make this accessible to everyone?

With **VideoDB RTStream**, you can do exactly this. Connect your dashcam or helmet cam, let AI monitor for violations, and auto-report them. Let's see how it works.

## How It Works

```
Helmet Cam / Dashcam
        ↓
🔗 RTSP Stream → VideoDB RTStream
        ↓
🤖 AI Scene Analysis (every 5 sec, 5 frames)
        ↓
🚨 Violation Detected? → Webhook Alert
        ↓
📧 n8n Workflow → Email to Traffic Police
```

Simple pipeline. Powerful impact.

## The Setup

### 1. Connect Your Stream

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

rtsp_url = "rtsp://your-camera-stream-url"
roadcam_stream = coll.connect_rtstream(
    name="RoadCam Violation Stream",
    url=rtsp_url,
    store=True,
)
```

```javascript Node.js theme={null}
import { connect } from "videodb";

const conn = connect();
const coll = await conn.getCollection();

const rtspUrl = "rtsp://your-camera-stream-url";
const roadcamStream = await coll.connectRTStream(
    rtspUrl,
    "RoadCam Violation Stream",
    ["video"],
    30,
    true,
);
```

### 2. Create the Violation Detection Index

This is where the magic happens. We tell the AI exactly what to look for:

```python theme={null}
violation_prompt = """
Focus on vehicles visible on the road and monitor them for the following traffic rule violations:

1. NO HELMET: Two-wheeler (motorcycle/scooter) rider or pillion not wearing a helmet
2. MOBILE PHONE USE: Driver using mobile phone while operating the vehicle
3. WRONG SIDE DRIVING: Vehicle traveling against the designated traffic flow direction
4. RED LIGHT VIOLATION: Vehicle crossing the stop line or intersection when traffic signal is red
5. TRIPLE RIDING: More than two people riding on a single two-wheeler
6. NO SEATBELT: Driver or front passenger in a four-wheeler not wearing seatbelt

Analyze the frames carefully. If you detect one or more violations from the above list, respond EXACTLY in this format:

Traffic Rule Violated
Vehicle: [vehicle type and color, e.g., "Black motorcycle", "White sedan"]
Plate Number: [license plate number if visible, otherwise "Not Visible"]
Violation: [specific violation(s) from the list above]
Description: [brief description of what you observed]

If NO violation is detected, respond ONLY with:
No Traffic Rule Violation Detected
"""

violation_understanding = roadcam_stream.understand(
    segmentation={"type": "time", "window": "7s"},
    analyzers=[{
        "type": "vlm",
        "name": "traffic_violations",
        "sampling": {"frame_count": 5},
        "config": {"prompt": violation_prompt},
    }],
    store=True,
)
traffic_violations_output = violation_understanding.outputs.get("traffic_violations")

violation_index = roadcam_stream.index(
    source=traffic_violations_output,
    name="Traffic_Violation_Index_7_5",
    use_for=["semantic"],
)
```

```javascript Node.js theme={null}
const violationPrompt = `
Focus on vehicles visible on the road and monitor them for the following traffic rule violations:

1. NO HELMET: Two-wheeler (motorcycle/scooter) rider or pillion not wearing a helmet
2. MOBILE PHONE USE: Driver using mobile phone while operating the vehicle
3. WRONG SIDE DRIVING: Vehicle traveling against the designated traffic flow direction
4. RED LIGHT VIOLATION: Vehicle crossing the stop line or intersection when traffic signal is red
5. TRIPLE RIDING: More than two people riding on a single two-wheeler
6. NO SEATBELT: Driver or front passenger in a four-wheeler not wearing seatbelt

Analyze the frames carefully. If you detect one or more violations from the above list, respond exactly in this format:

Traffic Rule Violated
Vehicle: [vehicle type and color, e.g., "Black motorcycle", "White sedan"]
Plate Number: [license plate number if visible, otherwise "Not Visible"]
Violation: [specific violation(s) from the list above]
Description: [brief description of what you observed]

If no violation is detected, respond only with: No Traffic Rule Violation Detected.
`;

const violationUnderstanding = await roadcamStream.understand({
    segmentation: { type: "time", window: "7s" },
    analyzers: [{
        type: "vlm",
        name: "traffic_violations",
        sampling: { frame_count: 5 },
        config: { prompt: violationPrompt },
    }],
    store: true,
});

const violationIndex = await roadcamStream.index({
    source: {
        understanding_id: violationUnderstanding.id,
        output: "traffic_violations",
    },
    name: "Traffic_Violation_Index_7_5",
    useFor: ["semantic"],
});
```

### 3. Set Up Event & Alert

```python theme={null}
violation_event_id = conn.create_event(
    event_prompt="""
    Detect when a traffic rule violation occurs, such as no helmet, mobile phone use, wrong side driving, red light violation, triple riding, or no seatbelt.
    Your 'explanation' should not include any commentary, and should clearly mention the following things:
    Traffic Rule Violated
    Vehicle: [vehicle type and color, e.g., "Black motorcycle", "White sedan"]
    Plate Number: [license plate number if visible, otherwise "Not Visible"]
    Violation: [specific violation(s) from the list above]
    Description: [brief description of what you observed]
    """,
    label="traffic_violation",
)
violation_alert_id = violation_index.create_alert(
    event_id=violation_event_id,
    callback_url="https://your-webhook-url.com",
)
```

```javascript Node.js theme={null}
const violationEventId = await conn.createEvent(
    `Detect when a traffic rule violation occurs, such as no helmet, mobile phone use, wrong side driving, red light violation, triple riding, or no seatbelt.
    Your 'explanation' should not include any commentary, and should clearly mention the following things:
    Traffic Rule Violated
    Vehicle: [vehicle type and color, e.g., "Black motorcycle", "White sedan"]
    Plate Number: [license plate number if visible, otherwise "Not Visible"]
    Violation: [specific violation(s) from the list above]
    Description: [brief description of what you observed]`,
    "traffic_violation",
);
const violationAlertId = await violationIndex.createAlert(
    violationEventId,
    "https://your-webhook-url.com",
);
```

## What You Get

Inspect the server payload before forwarding an event to your automation:

```javascript Node.js theme={null}
app.post("/traffic-violations", (req, res) => {
    const alert = req.body?.data;

    if (req.body?.channel === "alert" && alert?.triggered) {
        console.log({
            eventId: alert.event_id,
            label: alert.label,
            confidence: alert.confidence,
            start: alert.start,
            end: alert.end,
        });
    }

    res.sendStatus(204);
});
```

Use the fields present in your event payload when formatting a report or attaching evidence.

## Automated Enforcement Pipeline: From Video to Inbox

This n8n workflow acts as the **Digital Dispatch Center**, transforming raw AI detections into professional traffic reports in real-time.

1. **Webhook Trigger**: Receives the raw event payload from the VideoDB Safety Agent immediately upon violation detection.
2. **AI Data Extraction**: A dedicated VideoDB node parses the unstructured explanation string into a structured JSON object containing the License Plate, Vehicle Description, and Violation Type.
3. **Report Formatting**: A code node generates a high-contrast, professional HTML email template that maps the AI observations to a formal report structure.
4. **Official Delivery**: The finalized report - complete with the dynamic subject line and a direct link to the video evidence - is dispatched instantly via the Gmail node to the Traffic Control Room.

![N8N workflow automation showing the steps for traffic violation detection and email delivery](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/roadcam-n8n-workflow.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=292401a689cceae02be2367379c13ba2)

No more manual reporting. Just set it and forget it.

## Email received via N8N Automation

![Email received via N8N automation showing the professional report format](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/roadcam-email.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=3179e04ac261441f9133132ae5f74be3)

## Try It Yourself

> **[Open the Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Roadcam.ipynb)**
>
> Run the complete implementation in Google Colab with step-by-step examples

n8n workflow JSON attached below, simply copy the code and paste in your N8N instance, and set up the credentials to get the automation running!

> **n8n Workflow JSON (Click to expand)**
>
> ```json theme={null}
> {
>   "nodes": [
>     {
>       "parameters": {
>         "path": "traffic-violation-webhook",
>         "options": {}
>       },
>       "type": "n8n-nodes-base.webhook",
>       "typeVersion": 2.1,
>       "position": [0, 80],
>       "id": "34895f49-bcde-43cd-9112-60bbee8147a0",
>       "name": "Webhook"
>     },
>     {
>       "parameters": {
>         "operation": "generateText",
>         "prompt": "=You are a data extraction specialist. Your task is to parse a raw traffic violation string into a strict JSON format.|INPUT: \"{{ $json.explanation }}\" TASK:Extract the following fields. If a field is missing or unclear, use \"N/A\" as the value.1. vehicle_details2. license_plate: (The alpha-numeric plate number)3. violation_type: (The specific rule broken like no helmet, mobile phone use, wrong side driving, red light violation, triple riding, or no seatbelt)4. observation: (A short, clean summary of the evidence)STRICT RULES:- Output ONLY valid JSON.- No markdown formatting, no backticks, no preamble.- If the input is empty or invalid, return an object with all values set to \"Unknown\".JSON STRUCTURE:{  \"vehicle_type\": \"\",  \"license_plate\": \"\",  \"violation_type\": \"\",  \"observation\": \"\"}",
>         "response_type": "json"
>       },
>       "type": "@videodb/n8n-nodes-videodb.videoDb",
>       "typeVersion": 1,
>       "position": [224, 80],
>       "id": "5a16a040-9ab9-4001-9281-8e6963cf5d6b",
>       "name": "VideoDB"
>     },
>     {
>       "parameters": {},
>       "type": "n8n-nodes-base.wait",
>       "typeVersion": 1.1,
>       "position": [448, 80],
>       "id": "d04dff57-2b8b-4d5f-b96f-364349623c49",
>       "name": "Wait"
>     },
>     {
>       "parameters": {
>         "url": "={{ $('VideoDB').item.json.data.output_url }}",
>         "authentication": "predefinedCredentialType",
>         "nodeCredentialType": "videoDBApi",
>         "options": {}
>       },
>       "type": "n8n-nodes-base.httpRequest",
>       "typeVersion": 4.3,
>       "position": [672, 0],
>       "id": "0048b380-50ac-4da0-a949-35d7ae0f3fe3",
>       "name": "HTTP Request"
>     },
>     {
>       "parameters": {
>         "conditions": {
>           "options": {
>             "caseSensitive": true,
>             "leftValue": "",
>             "typeValidation": "loose",
>             "version": 2
>           },
>           "conditions": [
>             {
>               "id": "0ab68efa-a4ba-4235-b3e8-6c3967880aef",
>               "leftValue": "={{ $json.status }}",
>               "rightValue": "complete",
>               "operator": {
>                 "type": "string",
>                 "operation": "equals"
>               }
>             }
>           ],
>           "combinator": "and"
>         },
>         "looseTypeValidation": true,
>         "options": {}
>       },
>       "type": "n8n-nodes-base.if",
>       "typeVersion": 2.2,
>       "position": [896, 80],
>       "id": "74645a4a-4ab0-4efe-84e9-052457fb2cb2",
>       "name": "If"
>     },
>     {
>       "parameters": {
>         "sendTo": "={{ $json.to }}",
>         "subject": "={{ $json.subject }}",
>         "message": "={{ $json.html }}",
>         "options": {}
>       },
>       "type": "n8n-nodes-base.gmail",
>       "typeVersion": 2.1,
>       "position": [1344, 80],
>       "id": "17d26012-f389-40e9-be75-dcce6640d0dc",
>       "name": "Send a message"
>     },
>     {
>       "parameters": {
>         "jsCode": "const webhookData = $('Webhook').first().json;\nconst llmResult = $input.first().json.response.data.output;\n\nconst eventDate = new Date(webhookData.start_time).toLocaleString('en-IN', {\n  timeZone: 'Asia/Kolkata',\n  dateStyle: 'long',\n  timeStyle: 'medium'\n});\n\nconst htmlBody = `<div style=\"max-width:500px;margin:20px auto;border:2px solid #000;font-family:Arial,sans-serif;color:#000;padding:0;\"><div style=\"padding:20px;border-bottom:2px solid #000;text-align:center;\"><h2 style=\"margin:0;text-transform:uppercase;\">Traffic Violation Alert</h2><p style=\"margin:5px 0 0 0;font-size:14px;\">Automated Detection Report</p></div><div style=\"padding:20px;\"><table style=\"width:100%;border-collapse:collapse;color:#000;\"><tr><td style=\"padding:10px 0;border-bottom:1px solid #eee;\"><strong>Detected Violation</strong></td><td style=\"padding:10px 0;border-bottom:1px solid #eee;text-align:right;\">${llmResult.violation_type}</td></tr><tr><td style=\"padding:10px 0;border-bottom:1px solid #eee;\"><strong>Vehicle Description</strong></td><td style=\"padding:10px 0;border-bottom:1px solid #eee;text-align:right;\">${llmResult.vehicle_type}</td></tr><tr><td style=\"padding:10px 0;border-bottom:1px solid #eee;\"><strong>License Plate</strong></td><td style=\"padding:10px 0;border-bottom:1px solid #eee;text-align:right;\"><code style=\"font-size:16px;font-weight:bold;\">${llmResult.license_plate}</code></td></tr><tr><td style=\"padding:10px 0;border-bottom:1px solid #eee;\"><strong>Incident Time</strong></td><td style=\"padding:8px 0;border-bottom:1px solid #eee;text-align:right;\">${eventDate}</td></tr></table><div style=\"margin-top:20px;padding:15px;border:1px solid #000;background-color:#fcfcfc;\"><p style=\"margin:0;font-size:14px;\"><strong>AI Observation:</strong> ${llmResult.observation}</p></div><div style=\"margin-top:30px;text-align:center;\"><a href=\"${webhookData.player_url}\" style=\"display:inline-block;padding:12px 30px;border:2px solid #000;color:#000;text-decoration:none;font-weight:bold;text-transform:uppercase;font-size:14px;\">Click to View Evidence Clip</a></div></div></div>`;\n\nreturn {\n  to: \"control.room@dtp.nic.in\",\n  subject: `Violation Alert: ${llmResult.violation_type} [${llmResult.license_plate}]`,\n  html: htmlBody\n};"
>       },
>       "type": "n8n-nodes-base.code",
>       "typeVersion": 2,
>       "position": [1120, 80],
>       "id": "342feeb0-a5b6-4e20-a4eb-e4e962539439",
>       "name": "Format Report"
>     }
>   ],
>   "connections": {
>     "Webhook": {
>       "main": [[{"node": "VideoDB","type": "main","index": 0}]]
>     },
>     "VideoDB": {
>       "main": [[{"node": "Wait","type": "main","index": 0}]]
>     },
>     "Wait": {
>       "main": [[{"node": "HTTP Request","type": "main","index": 0}]]
>     },
>     "HTTP Request": {
>       "main": [[{"node": "If","type": "main","index": 0}]]
>     },
>     "If": {
>       "main": [
>         [{"node": "Format Report","type": "main","index": 0}],
>         [{"node": "Wait","type": "main","index": 0}]
>       ]
>     },
>     "Format Report": {
>       "main": [[{"node": "Send a message","type": "main","index": 0}]]
>     }
>   }
> }
> ```

**Small cameras. Smart AI. Big change.**

**Built with [VideoDB](https://videodb.io)**

## Related Tutorials

> **[Road Monitoring System](/examples-and-tutorials/live-intelligence/road-monitoring)**
>
> Multi-use road monitoring for accidents and congestion

> **[Automated Traffic Violation Detection](/examples-and-tutorials/live-intelligence/roadcam-monitoring)**
>
> Dashcam-based violation detection and enforcement


# TwelveLabs Integration
Source: https://docs.videodb.io/examples-and-tutorials/live-intelligence/twelvelabs-integration

Real-time video understanding with advanced AI models

> **Human Monitoring is Expensive, Exhausting, and Doesn't Scale**

From baby monitors to warehouse cameras or endless CCTV feeds, many companies still depend on human eyes to monitor live video—a costly, tedious, and highly error-prone approach. Fatigue inevitably sets in, accuracy declines, and scaling up means hiring more personnel rather than enhancing systems.

But in an AI-driven era, monitoring shouldn’t be manual, miss critical details, or become a bottleneck. Imagine receiving instant alerts when packages are stolen, notifying parents the moment a baby attempts to climb out of a crib, proactively highlighting safety risks before they escalate, or doctors instantly knowing when ICU patients need immediate attention.

This is precisely the responsive monitoring **[VideoDB](https://videodb.io/real-time-video-intelligence#header)** delivers through its real-time infrastructure. VideoDB now provides first party integration with **[TwelveLabs](https://www.twelvelabs.io/)**'s advanced **Pegasus 1.2** AI model for precise frame understanding.

## Live video to Instant Action

Real-time video analysis isn’t merely a nice-to-have—it's a fundamental shift in capability.

* **Safety and Security**: Transforming reactive measures into proactive alerts, potentially saving lives during emergencies like flash floods or security breaches.
* **Enterprise Productivity**: Converting passive meeting archives into interactive, searchable knowledge repositories, vastly enhancing collaboration.
* **Content Platforms**: Automatically tagging, chaptering, and moderating content in unprecedented detail, elevating user experiences.

The opportunities are immense, yet technical barriers have historically prevented many development teams from unlocking this potential.

### Reality of Building Video AI

If you've ever attempted to build a robust video understanding system, you know the pain firsthand. You're often stuck playing the role of a systems integrator, wrestling with:

* **API Spaghetti**: Managing credentials, juggling rate limits, and navigating diverse SDKs across multiple video processing, AI modeling, and storage services.
* **Scaling Nightmares**: Each component scales independently, causing bottlenecks and inefficiencies—particularly with resource-intensive GPU workloads.
* **Latency Issues**: The delays involved in interactions between storage, AI models, and your applications undermine genuine real-time capabilities.

These challenges can quickly stall innovation, turning promising ideas into lengthy, frustrating engineering endeavors.

### Simplifying Video AI Infrastructure with VideoDB

VideoDB is a purpose-built infrastructure for AI driven video management. It provides a unified, AI-native infrastructure handling the complete lifecycle of video content—from ingestion and indexing to alert management—all via a streamlined, developer-friendly API.

Leveraging VideoDB, you can effortlessly ingest multiple real-time video streams, manage customizable indexes uniquely tailored for specific analyses, and simultaneously analyze diverse aspects of a single video stream. Additionally, VideoDB’s targeted alerting system triggers automated webhooks, ensuring rapid responses to critical events.

And now, we've equipped that powerful infrastructure with an even more powerful intelligence.

## Introducing the TwelveLabs Integration: Your Frame Understanding, Supercharged

We're excited to introduce TwelveLabs model configurations for VideoDB's VLM workflow when the selected model is available for your backend and project.

![VideoDB and TwelveLabs integration architecture](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/videodb-twelvelabs.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=8562798603071bb0a87731616b896d34)

Configure the VLM analyzer with `config.model` to use TwelveLabs' sophisticated video understanding models, such as Pegasus, within the VideoDB workflow.

## Real-World Use Cases: From Theory to Action

With VideoDB + TwelveLabs, sophisticated real-time video monitoring is just minutes away:

## Real-World Use Cases in Action

### 🌊 Flash Flood Detection

![Real-time flash flood detection example - Click to open notebook](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/flash-flood-card.png?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=c891f0ac0e7df3db0dc3aa3e28c21dcc)

Click image to open interactive notebook →

[Open](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/integrations/twelvelabs/Flash_Flood_Detection_TwelveLabs.ipynb)

Imagine a camera monitoring a dry riverbed in a flood-prone area. With TwelveLabs integrated into VideoDB, you can continuously detect critical events—like rapidly rising floodwaters—in real-time. The moment Pegasus recognizes the signs of a flash flood, VideoDB immediately triggers life-saving alerts. This isn't just video analysis; it's proactive, intelligent disaster prevention.

### 👶🏻 Baby Crib Monitoring

![Baby crib monitoring example - Click to open notebook](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/baby-monitor-card.png?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=a283ec00681466b73fb1f7b315b4f7bd)

Click image to open interactive notebook →

[Open](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/integrations/twelvelabs/Baby_Crib_Monitoring_TwelveLabs.ipynb)

After a long day, parents deserve restful, worry-free sleep. With VideoDB's real-time monitoring powered by TwelveLabs' Pegasus, you'll instantly know if your baby tries climbing out of the crib or needs immediate attention. Sleep easy, knowing VideoDB and TwelveLabs have you covered—every second of the night.

## How It Works: One Line of Code to Unlock a New World

Ready to see how simple this is? To understand a video stream with TwelveLabs' powerful Pegasus model, specify it as `config.model` for the VLM analyzer.

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
rtsp_url = "rtsp://your-stream-url"  # Replace with your camera stream.
flood_stream = coll.connect_rtstream(
    name="Flood Detection Stream",
    url=rtsp_url,
    store=True,
)

flood_understanding = flood_stream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[{
        "type": "vlm",
        "name": "flood_scene",
        "sampling": {"frame_count": 3},
        "config": {
            "prompt": "Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.",
            "model": "twelvelabs-pegasus-1.2",
        },
    }],
    store=True,
)
flood_scene_output = flood_understanding.outputs.get("flood_scene")
flood_index = flood_stream.index(
    source=flood_scene_output,
    name="Flash_Flood_Detection_Index",
    use_for=["semantic"],
)
print("Continuous index ID:", flood_index.id)
```

```javascript Node.js theme={null}
import { connect } from "videodb";

const conn = connect();
const coll = await conn.getCollection();
const rtspUrl = "rtsp://your-stream-url"; // Replace with your camera stream.
const floodStream = await coll.connectRTStream(
    rtspUrl,
    "Flood Detection Stream",
    ["video"],
    30,
    true,
);

const floodUnderstanding = await floodStream.understand({
    segmentation: { type: "time", window: "5s" },
    analyzers: [{
        type: "vlm",
        name: "flood_scene",
        sampling: { frame_count: 3 },
        config: {
            prompt: "Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.",
            model: "twelvelabs-pegasus-1.2",
        },
    }],
    store: true,
});

const floodIndex = await floodStream.index({
    source: {
        understanding_id: floodUnderstanding.id,
        output: "flood_scene",
    },
    name: "Flash_Flood_Detection_Index",
    useFor: ["semantic"],
});
console.log("Continuous index ID:", floodIndex.id);
```

When the selected model is available for your backend and project, configure it in the VLM analyzer and use it in VideoDB's indexing pipeline.

> [!NOTE]
>
> Provider routing for model names is backend-dependent; confirm the configured model is available for your project.

## Try It Now: Interactive Notebooks

Ready to build your own real-time video intelligence system? Open these interactive notebooks directly in Google Colab and start experimenting with TwelveLabs' Pegasus model:

> **[Flash Flood Detection Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/integrations/twelvelabs/Flash_Flood_Detection_TwelveLabs.ipynb)**
>
> Build a real-time flood detection system that monitors dry riverbeds and sends instant alerts

> **[Baby Crib Monitoring Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/integrations/twelvelabs/Baby_Crib_Monitoring_TwelveLabs.ipynb)**
>
> Create an AI-powered baby monitor that detects escape attempts and keeps parents informed


# Audio Overlay
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/audio-overlay

Combine multiple audio tracks with video

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Audio_Overlay.ipynb)

## Overview

Welcome to the groovy world of audio overlays with VideoDB! 🎶 In this tutorial, we're diving into the magic of adding audio overlays to your video assets. Picture this: you've got your video content all set, but it's missing that extra oomph. That's where audio overlay swoops in to save the day! With VideoDB's easy-to-use feature, you can seamlessly weave in background music, voiceovers, or funky sound effects, transforming your videos from ordinary to extraordinary. Let's crank up the volume and get ready to rock and roll!

## Setup

### Installing packages

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

Before proceeding, ensure access to [VideoDB](https://videodb.io)

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). ( Free for first 50 uploads, No credit card required)

## Steps

### Step 1: Connect to VideoDB

Connect to VideoDB to establish a session for uploading and manipulating video files. Import the necessary modules from VideoDB library to access functionalities.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Video

Upload the video to VideoDB collection. You can upload the video asset from your local device or from a YouTube URL to upload the video from its source.

```python Python theme={null}
video = coll.upload(url="https://youtu.be/e49VEpWg61M")
video.play()
```

```javascript Node.js theme={null}
const video = await coll.uploadURL({
  url: "https://youtu.be/e49VEpWg61M"
});
console.log(video.playerUrl);
```

You can upload from your local file system too by passing `file_path` in `upload()`.

Here's the video we'll be using for this tutorial:

[YouTube video player](https://www.youtube.com/embed/e49VEpWg61M)

### Step 3: Upload Audio

Upload the audio file to VideoDB collection. You can upload the audio asset from your local device or from a YouTube URL to upload the audio from its source. Make sure to mention `media_type` if you want to use audio track of a video.

```python Python theme={null}
audio = coll.upload(url="https://youtu.be/_Gd8mbQ3-mI", media_type="audio")
```

```javascript Node.js theme={null}
import { MediaType } from 'videodb';

const audio = await coll.uploadURL({
  url: "https://youtu.be/_Gd8mbQ3-mI",
  mediaType: MediaType.audio
});
```

### Step 4: Create assets

Create assets for Audio and Video using `VideoAsset` and `AudioAsset` from the Editor SDK.

```python Python theme={null}
from videodb.editor import VideoAsset, AudioAsset

video_asset = VideoAsset(id=video.id, volume=0.5) # Reducing volume of the original video

audio_asset = AudioAsset(id=audio.id)
```

```javascript Node.js theme={null}
import { EditorVideoAsset, EditorAudioAsset } from 'videodb';

const videoAsset = new EditorVideoAsset({
  id: video.id,
  volume: 0.5 // Reducing volume of the original video
});

const audioAsset = new EditorAudioAsset({
  id: audio.id
});
```

### Step 5: Create a Timeline

Create a timeline using the `Track` and `Clip` pattern. Add the video clip and audio clip to a track, then add the track to the timeline.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip

# Create a new timeline
timeline = Timeline(conn)

# Create a track with video and audio clips
track = Track()

# Add video clip (full duration)
video_clip = Clip(asset=video_asset, duration=float(video.length))
track.add_clip(0, video_clip)

# Add audio overlay clip starting at 0
audio_clip = Clip(asset=audio_asset, duration=float(video.length))
track.add_clip(0, audio_clip)

timeline.add_track(track)
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip } from 'videodb';

// Create a new timeline
const timeline = new EditorTimeline(conn);

// Create a track with video and audio clips
const track = new Track();

// Add video clip (full duration)
const videoDuration = parseFloat(video.length);
const videoClip = new Clip({ asset: videoAsset, duration: videoDuration });
track.addClip(0, videoClip);

// Add audio overlay clip starting at 0
const audioClip = new Clip({ asset: audioAsset, duration: videoDuration });
track.addClip(0, audioClip);

timeline.addTrack(track);
```

### Final Step: Review and Share

Preview the video with the integrated voiceover to ensure it functions correctly. Once satisfied, generate a stream of the video and share the link for others to view and enjoy this wholesome creation!

```python Python theme={null}
from videodb import play_stream

stream = timeline.generate_stream()
play_stream(stream)
```

```javascript Node.js theme={null}
const stream = await timeline.generateStream();
console.log(stream);
```

[Audio Overlay Video Output](https://www.youtube.com/embed/WWV_8FtIqiU)

## Conclusion

And there you have it, folks! We've unlocked the door to audio awesomeness with VideoDB's audio overlay feature. By seamlessly integrating background music, voiceovers, or sound effects into your videos, you're not just adding layers of sound – you're adding layers of engagement, emotion, and excitement!

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Audio_Overlay.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Generate professional narration with AI voice synthesis

> **[Intro/Outro](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Add polished opening and closing sequences


# Brand Elements
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/brand-elements

Add logos, text overlays, and custom styling

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Adding_Brand_Elements.ipynb)

### Introduction

Adding brand elements like logos and overlay styles elevates your video content to new levels of professionalism. This tutorial will guide you through the process of integrating logos and custom text assets, ensuring your brand shines through in every frame.

For this tutorial, we'll add Kyvos' branding to a video through image and text assets and see how those can be modified to enhance the content of raw footage. Although this is a quick example, the possibilities are endless! We look forward to seeing your experiments with these building blocks.

[YouTube video player](https://www.youtube.com/embed/ps3cNAcPEMs)

## Setup

### Installing packages

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

Before proceeding, ensure access to [VideoDB](https://videodb.io)

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). ( Free for first 50 uploads, No credit card required)

## Implementation

### Step 1: Connect to VideoDB

Begin by establishing a connection to VideoDB using your API key:

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Video and Image Assets

Begin the branding process by uploading your video and image assets (base video and logo image) to VideoDB:

```python Python theme={null}
from videodb import MediaType

# Upload Video to VideoDB
video = coll.upload(url="https://youtu.be/ps3cNAcPEMs")

# Upload Image asset for branding
image = coll.upload(url="https://raw.githubusercontent.com/video-db/videodb-cookbook-assets/main/images/examples/Kyvos_Logo.png", media_type=MediaType.image)

print(f"Video ID: {video.id}")
print(f"Image ID: {image.id}")
```

```javascript Node.js theme={null}
import { MediaType } from 'videodb';

// Upload Video to VideoDB
const video = await coll.uploadURL({ url: "https://youtu.be/ps3cNAcPEMs" });

// Upload Image asset for branding
const image = await coll.uploadURL({
  url: "https://raw.githubusercontent.com/video-db/videodb-cookbook-assets/main/images/examples/Kyvos_Logo.png",
  mediaType: MediaType.image
});

console.log(`Video ID: ${video.id}`);
console.log(`Image ID: ${image.id}`);
```

Original Video:

[YouTube video player](https://www.youtube.com/embed/ps3cNAcPEMs)

Here's the logo (image asset) we will be using for this tutorial:

![Kyvos logo example for brand elements tutorial](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/kyvos.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=2c993650d5e7d8fa3db7c8f271df5152)

### Step 3: Add Brand Elements to Video

We'll define the text asset with styling parameters and create video/image assets for use in our timeline.

The new Editor SDK uses a `Track` and `Clip` pattern for composing video. Each asset is wrapped in a `Clip` with duration and positioning controls.

```python Python theme={null}
from videodb.editor import (
    Timeline, Track, Clip,
    VideoAsset, ImageAsset, TextAsset,
    Font, Border, Shadow, Background,
    Alignment, HorizontalAlignment, VerticalAlignment, TextAlignment,
    Position, Offset, Fit)

# Create the text asset with new styling parameters
text_asset = TextAsset(
    text="Visit kyvosinsights.com today!",
    font=Font(family="PT Sans", size=38, color="#F58C29"),
    border=Border(color="#1D1C21", width=1),
    background=Background(color="#29272D", border_width=6, opacity=1.0),
    alignment=Alignment(
      horizontal=HorizontalAlignment.center,
      vertical=VerticalAlignment.top),)

# Specify the video asset (trimmed from 0 to 44 seconds)
video_asset = VideoAsset(id=video.id, start=0)
video_duration = 44

# Image asset with positioning via Clip parameters
image_asset = ImageAsset(id=image.id)
```

```javascript Node.js theme={null}
import {
  EditorTimeline, Track, Clip,
  EditorVideoAsset, EditorImageAsset, EditorTextAsset,
  Font, Border, Background,
  Alignment, HorizontalAlignment, VerticalAlignment, Position, Offset
} from 'videodb';

// Create the text asset with styling parameters
const textAsset = new EditorTextAsset({
  text: "Visit kyvosinsights.com today!",
  font: new Font({
    family: "PT Sans",
    size: 38,
    color: "#F58C29"
  }),
  border: new Border({ color: "#1D1C21", width: 1 }),
  background: new Background({
    color: "#29272D",
    borderWidth: 6,
    opacity: 1.0
  }),
  alignment: new Alignment({
    horizontal: HorizontalAlignment.center,
    vertical: VerticalAlignment.top
  })
});

// Specify the video asset (trimmed from 0 to 44 seconds)
const videoAsset = new EditorVideoAsset({ id: video.id, start: 0 });
const videoDuration = 44;

// Image asset with positioning via Clip parameters
const imageAsset = new EditorImageAsset({ id: image.id });
```

### Step 4: Add Brand Elements to Video

Now we'll bring all assets together using the `Timeline`, `Track`, and `Clip` pattern. The video goes on the main track, while overlays (logo and text) go on a separate track with their start times.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, Position

timeline = Timeline(conn)

# Create the main video track
video_track = Track()
video_clip = Clip(asset=video_asset, duration=video_duration)
video_track.add_clip(0, video_clip)
timeline.add_track(video_track)

# Create overlay track for logo & text
overlay_track = Track()

# Add logo overlay (starts at 2.5s, duration 36s)
logo_clip = Clip(
    asset=image_asset,
    duration=36,
    fit=Fit.none,
    scale=0.15,
    position=Position.top_right,
    offset=Offset(x=-0.02, y=0.02))
overlay_track.add_clip(2.5, logo_clip)

# Add text overlay (starts at 42s, duration 2s)
text_clip = Clip(
    asset=text_asset,
    duration=2,
    position=Position.top,
    offset=Offset(y=0.05))
overlay_track.add_clip(42, text_clip)

timeline.add_track(overlay_track)
```

```javascript Node.js theme={null}
const timeline = new EditorTimeline(conn);

// Create the main video track
const videoTrack = new Track();
const videoClip = new Clip({ asset: videoAsset, duration: videoDuration });
videoTrack.addClip(0, videoClip);
timeline.addTrack(videoTrack);

// Create overlay track for logo & text
const overlayTrack = new Track();

// Add logo overlay (starts at 2.5s, duration 36s)
const logoClip = new Clip({
  asset: imageAsset,
  duration: 36,
  scale: 0.15,
  position: Position.topRight,
  offset: new Offset({ x: -0.02, y: 0.02 })
});
overlayTrack.addClip(2.5, logoClip);

// Add text overlay (starts at 42s, duration 2s)
const textClip = new Clip({
  asset: textAsset,
  duration: 2,
  position: Position.top,
  offset: new Offset({ y: 0.05 })
});
overlayTrack.addClip(42, textClip);

timeline.addTrack(overlayTrack);
```

### Step 5: Review and Share

Review your branded video to ensure it aligns perfectly with your brand identity, then share it with your audience:

```python Python theme={null}
from videodb import play_stream

# Preview the branded video
stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
// Preview the branded video
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

Let's have a look at the output!

[Branded Video Output](https://play.videodb.io/v1/ce541325-9903-4aca-b786-4c85bb17820d.m3u8)

### Conclusion

Congratulations on mastering the art of branding with VideoDB! By seamlessly integrating brand elements into your videos, you've enhanced their professionalism and engagement. Experiment with different branding techniques to ensure your brand shines through in every frame.

[Branding Tutorial](https://www.youtube.com/embed/ka8x7nFChN8)

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Adding_Brand_Elements.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Intro/Outro](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Add opening and closing sequences to your videos

> **[Audio Overlay](/examples-and-tutorials/programmatic-editing/audio-overlay)**
>
> Add background music and sound effects to your compositions


# Chess Match Montage Generator
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/chess-montage

Automatically create highlight reels from long chess matches with AI-powered move detection

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/chess_montage.ipynb)

## The Challenge

A chess match can last 4+ hours. As a content creator, you want to turn that into a punchy highlight reel, but manually scrubbing through and identifying key moments is tedious.

What if AI could watch the entire match, detect every move, and automatically compile a highlight montage?

## What You'll Build

Turn a long chess match into a **punchy highlight reel** — automatically! You'll:

1. Upload chess video + background music
2. Index scenes with simple move detection
3. Extract timestamps automatically using AI
4. Build a montage with transitions and effects
5. Output a professional highlight reel

All powered by **VideoDB's Editor SDK** — no manual frame-by-frame editing needed.

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Upload Chess Video and Music

```python theme={null}
from videodb import MediaType

# Upload the chess match video
chess_video = coll.upload(url="https://www.youtube.com/watch?v=dhDe-RcoyAU")

# Upload background music for the montage
bg_music = coll.upload(
    url="https://www.youtube.com/watch?v=S19UcWdOA-I",
    media_type=MediaType.audio
)
```

### Step 2: Analyze Scenes with Structured Move Detection

Analyze the video in eight-second segments with a **strict binary prompt**. Each scene returns `move_status` as exactly `Player Moved` when a chess move is evidenced across the sampled frames, or `No Move` otherwise.

```python theme={null}
print("Analyzing scenes for move detection...")

moves_understanding = chess_video.understand(
    segmentation={"type": "time", "seconds": 8},
    analyzers=[
        {
            "type": "vlm",
            "name": "move_detection",
            "sampling": {
                "strategy": "uniform",
                "frame_count": 5,
            },
            "config": {
                "prompt": """Look at this chess scene and focus on the chess board. Your task is to detect when pieces are moved.

Set move_status to exactly "Player Moved" only when a chess move is evidenced across the sampled frames. Otherwise, set move_status to exactly "No Move" (same position, paused, talking, etc.).

Be strict. Only set move_status to "Player Moved" if you clearly see a chess piece moved.""",
                "schema": {
                    "move_status": "string",
                },
            },
        },
    ],
)
moves_understanding.wait_until_complete()

moves_analyzer = moves_understanding.get_analyzer("move_detection")
```

### View the Structured Move Detections

```python theme={null}
import json

moves_output = moves_analyzer.get_output()
moves_scenes = moves_output.get("scenes", [])
print(json.dumps(moves_scenes, indent=2))
```

### Step 3: Extract Move Timestamps

```python theme={null}
timestamps = [
    scene["start"]
    for scene in moves_scenes
    if "move_status" in scene["data"] and scene["data"]["move_status"] == "Player Moved"
]

print("✓ Move timestamps extracted!")
```

### Review the Timestamps

```python theme={null}
print(f"✓ Found {len(timestamps)} moves")
print(f"Timestamps: {timestamps}")
```

Initialize timeline and sample moves:

```python theme={null}
from videodb.editor import (
    Timeline, Track, Clip, VideoAsset, AudioAsset,
    Filter, Transition, TextAsset, Font
)

CLIP_DURATION = 5  # seconds per clip
TARGET_CLIPS = 10  # how many clips we want

# Sample timestamps evenly
total_detected = len(timestamps)
step = max(1, total_detected // TARGET_CLIPS)
sampled_timestamps = timestamps[::step][:TARGET_CLIPS]

# Initialize timeline
timeline = Timeline(conn)
timeline.background = "#000000"

# Create intro text
intro_text = TextAsset(
    text="Let the Match Begin",
    font=Font(family="Clear Sans", size=56, color="#FFFFFF"),
)

intro_clip = Clip(
    asset=intro_text,
    duration=3,
    transition=Transition(in_="fade", out="fade", duration=0.5)
)

intro_track = Track()
intro_track.add_clip(0, intro_clip)
timeline.add_track(intro_track)
```

### Step 5: Add Video Clips with Transitions

```python theme={null}
# Add video clips
video_track = Track()
timeline_position = 3

for i, start_time in enumerate(sampled_timestamps):
    clip = Clip(
        asset=VideoAsset(
            id=chess_video.id,
            start=start_time-1,
            volume=0  # Muting the original audio
        ),
        duration=CLIP_DURATION,
        filter=Filter.contrast,
        transition=Transition(in_="fade", out="fade", duration=1)
    )

    video_track.add_clip(timeline_position, clip)
    timeline_position += CLIP_DURATION

timeline.add_track(video_track)

total_duration = len(sampled_timestamps) * CLIP_DURATION + 3
```

### Step 6: Add Background Music

```python theme={null}
# Add music track
music_clip = Clip(
    asset=AudioAsset(
        id=bg_music.id,
        start=0,
        volume=0.7
    ),
    duration=total_duration
)

audio_track = Track()
audio_track.add_clip(0, music_clip)
timeline.add_track(audio_track)
```

### Step 7: Render Montage

```python theme={null}
# Generate final montage stream
stream_url = timeline.generate_stream()
```

## What You Get

A professional highlight reel with:

* AI-detected key moves
* Evenly sampled clips for pacing
* Smooth fade transitions
* Enhanced contrast for visual impact
* Background music
* Professional polish in seconds

Here's the final chess montage:

[Chess Match Highlight Montage](https://play.videodb.io/v1/e66a4b69-251b-4a83-aba9-dd4c254b2060.m3u8)

## Perfect For

* **Tournament Highlights** - Post-tournament recap videos
* **Streamer Clips** - Highlight reels for streaming communities
* **Educational Analysis** - Study videos from master games
* **Social Media** - Short-form clips from longer matches
* **Archive Content** - Transform library of matches into reels

## The Result

What took hours of manual editing now takes minutes. Your audience gets punchy, professional highlight reels. You get back your time.

No more endless scrubbing. Just AI-powered chess analysis and automatic montages.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/chess_montage.ipynb)**
>
> Open the complete implementation with advanced filtering, custom transitions, and batch processing.

## Related Tutorials

> **[Intro & Outro](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Auto-add opening and closing sequences

> **[Word Counter](/examples-and-tutorials/programmatic-editing/word-counter)**
>
> Add analytical text overlays


# Dynamic Ads
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/dynamic-ads

Insert advertisements into video streams

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/master/examples/Insert%20Inline%20Video.ipynb)

Video files are not very flexible if we want to change the flow and insert another video at a certain place. Imagine putting a video advertisement on your video stream for your customers. You don't need to edit the video file and create another one.

VideoDB simplifies it for you. It gives you power to build contextual Ad insertion and personalized Ad insertion on your videos.

![Dynamic advertisement insertion example showing ad placement in video streams](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/dynamic-advertisment.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=9a82599f84245adaca10083b3e9e2b8e)

## Setup

### Installing VideoDB in your environment

`VideoDB` is available as a [python package](https://pypi.org/project/videodb)

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### Setting Up a connection to VideoDB

> [!NOTE]
>
> To connect to VideoDB, simply create a Connection object, and connect to the collection. This can be done by either providing your VideoDB API key directly to the constructor or by setting the `VIDEO_DB_API_KEY` environment variable with your API key. Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

```python Python theme={null}
import videodb

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()

```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

## Uploading the videos to VideoDB

Let's have a base video as Sam Altman's conversation on OpenAI and AGI. We'll choose another video to insert in this 👉 (let's get IBM's Advertisement). We are going to insert the Ad video into the base video at a specific timestamp.

For this, we will need to first upload both the videos to `VideoDB`.

```python Python theme={null}
base_video_url = "https://www.youtube.com/watch?v=e1cf58VWzt8"
ad_video_url = "https://www.youtube.com/watch?v=jtwduf2lh08"

base_video = coll.upload(url=base_video_url)
ad_video = coll.upload(url=ad_video_url)
```

```javascript Node.js theme={null}
const baseVideoUrl = "https://www.youtube.com/watch?v=e1cf58VWzt8";
const adVideoUrl = "https://www.youtube.com/watch?v=jtwduf2lh08";

const baseVideo = await coll.uploadURL({ url: baseVideoUrl });
const adVideo = await coll.uploadURL({ url: adVideoUrl });
```

## Inserting Ad in our Base Video

Now that we have both videos uploaded, we'll use VideoDB's Editor SDK to create a timeline with multiple clips. We'll break the base video into segments and insert the ad at the 10-second mark:

1. **Clip 1**: Base video from 0 to 10 seconds
2. **Clip 2**: Full ad video
3. **Clip 3**: Base video continues from 10 seconds to end

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset
from videodb import play_stream

# Define timing
insert_time = 10  # Insert ad at 10 seconds
base_duration = int(base_video.length)
ad_duration = int(ad_video.length)

# Create timeline and track
timeline = Timeline(conn)
track = Track()

# Clip 1: Base video from 0 to insert_time
clip1 = Clip(
    asset=VideoAsset(id=base_video.id),
    duration=insert_time)
track.add_clip(0, clip1)

# Clip 2: Full ad video
clip2 = Clip(
    asset=VideoAsset(id=ad_video.id),
    duration=ad_duration)
track.add_clip(insert_time, clip2)

# Clip 3: Base video continues from insert_time to end
clip3 = Clip(
    asset=VideoAsset(id=base_video.id, start=insert_time),
    duration=base_duration - insert_time)
track.add_clip(insert_time + ad_duration, clip3)

# Generate stream
timeline.add_track(track)
stream_link = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset } from 'videodb';

// Define timing
const insertTime = 10;  // Insert ad at 10 seconds
const baseDuration = parseInt(baseVideo.length);
const adDuration = parseInt(adVideo.length);

// Create timeline and track
const timeline = new EditorTimeline(conn);
const track = new Track();

// Clip 1: Base video from 0 to insertTime
const clip1 = new Clip({
    asset: new EditorVideoAsset({ id: baseVideo.id }),
    duration: insertTime
});
track.addClip(0, clip1);

// Clip 2: Full ad video
const clip2 = new Clip({
    asset: new EditorVideoAsset({ id: adVideo.id }),
    duration: adDuration
});
track.addClip(insertTime, clip2);

// Clip 3: Base video continues from insertTime to end
const clip3 = new Clip({
    asset: new EditorVideoAsset({ id: baseVideo.id, start: insertTime }),
    duration: baseDuration - insertTime
});
track.addClip(insertTime + adDuration, clip3);

// Generate stream
timeline.addTrack(track);
const streamLink = await timeline.generateStream();
```

`stream_link` is a streaming link that offers instant playback capability – no rendering necessary. ⚡️

## Play Modified Video

Let's check the results:

```python Python theme={null}
from videodb import play_stream
play_stream(stream_link)
```

```javascript Node.js theme={null}
console.log(streamLink);
```

You can generate as many streaming links as needed for your use-case. This gives you power to personalize your video content for each user.

## No Limitations

The inserted video doesn't have to be solely for advertisements; it can be a disclaimer or announcement for the person watching the video. Furthermore, your stream is dynamic, enabling you to adjust the timestamp and insertion video based on your business logic.

This is an incredible power to have for any product hosting videos. Let's master video content manipulation like a pro!

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/master/examples/Insert%20Inline%20Video.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Add logos, watermarks, and graphics

> **[Dynamic Streams](/examples-and-tutorials/programmatic-editing/dynamic-streams)**
>
> Generate multiple versions from one source


# Dynamic Streams
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/dynamic-streams

Create personalized video streams with data overlays

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Programmatic_Streams.ipynb)

## Introduction

Imagine you're watching a captivating keynote session from your favorite conference, and you’re welcomed with a personalized stream just for you.

This tutorial demonstrates how to create dynamic video streams by integrating data from custom databases and external APIs. We'll use a practical example: a recording of a [Config 2023](https://www.youtube.com/watch?v=Nmv8XdFiej0) keynote session. By using VideoDB, we'll show how companies like [Figma](https://www.figma.com/files/recents-and-sharing?fuid=940498258276625180) can personalize the viewing experience for their audience, delivering a richer and more engaging experience.

We'll showcase how to:

* **Fetch data from a random user API** to represent a hypothetical viewer.
* Integrate this data into a **custom VideoDB timeline**.
* Create a **personalized stream** that dynamically displays relevant information alongside the keynote video.

This tutorial is your guide to unlocking the potential of dynamic video streams and transforming your video content with personalized experiences.

## Setup

### Installing packages

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

Before proceeding, ensure access to [VideoDB](https://videodb.io)

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). ( Free for first 50 uploads, No credit card required)

## Steps

### Step 1: Connect to VideoDB

Begin by establishing a connection to VideoDB using your API key:

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### 🗳️ Step 2: Upload Base Video

Upload and play the video to ensure it's correctly loaded. We'll be using [this video](https://www.youtube.com/watch?v=Nmv8XdFiej0) for the purpose of this tutorial.

```python Python theme={null}
# Upload and play a video from a URL
video = coll.upload(url="https://www.youtube.com/watch?v=Nmv8XdFiej0")
video.play()

# Alternatively, get a video from your VideoDB collection
# video = coll.get_video('VIDEO_ID_HERE')
# video.play()
```

```javascript Node.js theme={null}
// Upload and play a video from a URL
const video = await coll.uploadURL({ url: "https://www.youtube.com/watch?v=Nmv8XdFiej0" });
console.log(video.playerUrl);

// Alternatively, get a video from your VideoDB collection
// const video = await coll.getVideo('VIDEO_ID_HERE');
// console.log(video.playerUrl);
```

### Step 3: Fetch Data from a Random User API

This code fetches a random user's data (name and picture) from the "randomuser.me" API. You can adapt this to retrieve data from any relevant API (e.g., product data, news articles) for your use case.

```python Python theme={null}
import requests

# Make a request to the Randomizer API
response = requests.get('https://randomuser.me/api/?results=1&nat=us,ca,gb,au')
data = response.json()

# Extract relevant information
first_name = data['results'][0]['name']['first']
medium_picture = data['results'][0]['picture']['medium']
```

```javascript Node.js theme={null}
// Make a request to the Randomizer API
const response = await fetch('https://randomuser.me/api/?results=1&nat=us,ca,gb,au');
const data = await response.json();

// Extract relevant information
const firstName = data.results[0].name.first;
const mediumPicture = data.results[0].picture.medium;
```

### Step 4: Upload the image to VideoDB

* First we download the image to local storage
* Then we use the local path to upload it to VideoDB

```python Python theme={null}
import requests

# 1. Download the image locally
local_path = "my_local_image.jpg"

response = requests.get(medium_picture)
if response.status_code == 200:
    with open(local_path, 'wb') as f:
        f.write(response.content)
    print(f"Image downloaded successfully to: {local_path}")
else:
    print(f"Failed to download image. Status code: {response.status_code}")

# 2. Upload using the local file path

from videodb import play_stream, MediaType
image = coll.upload(file_path=local_path, media_type=MediaType.image)

print(f"Image uploaded to VideoDB: {image.id}")
```

```javascript Node.js theme={null}
import * as fs from 'fs';
import { MediaType } from 'videodb';

// 1. Download the image locally
const localPath = "my_local_image.jpg";

const imgResponse = await fetch(mediumPicture);
if (imgResponse.ok) {
    const buffer = Buffer.from(await imgResponse.arrayBuffer());
    fs.writeFileSync(localPath, buffer);
    console.log(`Image downloaded successfully to: ${localPath}`);
} else {
    console.log(`Failed to download image. Status code: ${imgResponse.status}`);
}

// 2. Upload using the local file path
const image = await coll.uploadFile({
    filePath: localPath,
    mediaType: MediaType.image
});

console.log(`Image uploaded to VideoDB: ${image.id}`);
```

### Step 5: Create VideoDB Assets

We create VideoDB assets for the base video, the user's name (text), and their picture (image) using the new Editor SDK. The `Font` and `Background` objects allow us to customize the appearance of text elements.

```python Python theme={null}
from videodb.editor import (
    Timeline, Track, Clip,
    VideoAsset, TextAsset, ImageAsset,
    Font, Background, Alignment, HorizontalAlignment, VerticalAlignment,
    Position, Offset, Fit)

# 1. Video Asset (Base background)
video_asset = VideoAsset(id=video.id, start=0)

# 2. Name Asset (Top)
name_asset = TextAsset(
    text=f'Hi {first_name} !',
    font=Font(family="Montserrat", size=60, color="#000000"),
    background=Background(color="#D2C11D", border_width=20, opacity=1.0),
    alignment=Alignment(horizontal=HorizontalAlignment.center, vertical=VerticalAlignment.top),)

# 3. Message Asset (Middle)
cmon_asset = TextAsset(
    text="Here are your favorite moments",
    font=Font(family="Montserrat", size=60, color="#D2C11D"),
    background=Background(color="#000000", border_width=20, opacity=1.0),
    alignment=Alignment(horizontal=HorizontalAlignment.center, vertical=VerticalAlignment.center),)

# 4. Image Asset (Bottom)
image_asset = ImageAsset(id=image.id)
```

```javascript Node.js theme={null}
import {
    EditorTimeline, Track, Clip,
    EditorVideoAsset, EditorTextAsset, EditorImageAsset,
    Font, Background, Alignment, HorizontalAlignment, VerticalAlignment,
    Position, Offset
} from 'videodb';

// 1. Video Asset (Base background)
const videoAsset = new EditorVideoAsset({ id: video.id, start: 0 });

// 2. Name Asset (Top)
const nameAsset = new EditorTextAsset({
    text: `Hi ${firstName} !`,
    font: new Font({ family: "Montserrat", size: 60, color: "#000000" }),
    background: new Background({ color: "#D2C11D", borderWidth: 20, opacity: 1.0 }),
    alignment: new Alignment({ horizontal: HorizontalAlignment.center, vertical: VerticalAlignment.top })
});

// 3. Message Asset (Middle)
const cmonAsset = new EditorTextAsset({
    text: "Here are your favorite moments",
    font: new Font({ family: "Montserrat", size: 60, color: "#D2C11D" }),
    background: new Background({ color: "#000000", borderWidth: 20, opacity: 1.0 }),
    alignment: new Alignment({ horizontal: HorizontalAlignment.center, vertical: VerticalAlignment.center })
});

// 4. Image Asset (Bottom)
const imageAsset = new EditorImageAsset({ id: image.id });
```

### ↔️ Step 6: Create the VideoDB Timeline

Using the `Track` and `Clip` pattern, we arrange and layer assets to create a dynamic video stream. The main video goes on one track, while overlays (name, message, image) go on separate tracks with their start times.

```python Python theme={null}
# Create the timeline
timeline = Timeline(conn)

# --- Track 1: Main Video ---
video_track = Track()
video_clip = Clip(asset=video_asset, duration=float(video.length))
video_track.add_clip(0, video_clip)
timeline.add_track(video_track)

# --- Track 2: Overlays ---
overlay_track = Track()

# 1. Add Name Overlay (Top)
name_clip = Clip(
    asset=name_asset,
    duration=4,
    position=Position.top,
    offset=Offset(y=0.15))
overlay_track.add_clip(5, name_clip)

# 2. Add Message Overlay (Center)
cmon_clip = Clip(
    asset=cmon_asset,
    duration=4,
    position=Position.center,)
overlay_track.add_clip(5, cmon_clip)

# 3. Add Image Overlay (Bottom)
image_clip = Clip(
    asset=image_asset,
    duration=4,
    position=Position.bottom,
    scale=2,
    fit=Fit.none,
    offset=Offset(y=-0.15))
overlay_track.add_clip(5, image_clip)

timeline.add_track(overlay_track)
```

```javascript Node.js theme={null}
// Create the timeline
const timeline = new EditorTimeline(conn);

// --- Track 1: Main Video ---
const videoTrack = new Track();
const videoClip = new Clip({ asset: videoAsset, duration: parseFloat(video.length) });
videoTrack.addClip(0, videoClip);
timeline.addTrack(videoTrack);

// --- Track 2: Overlays ---
const overlayTrack = new Track();

// 1. Add Name Overlay (Top)
const nameClip = new Clip({
    asset: nameAsset,
    duration: 4,
    position: Position.top,
    offset: new Offset({ y: 0.15 })
});
overlayTrack.addClip(5, nameClip);

// 2. Add Message Overlay (Center)
const cmonClip = new Clip({
    asset: cmonAsset,
    duration: 4,
    position: Position.center
});
overlayTrack.addClip(5, cmonClip);

// 3. Add Image Overlay (Bottom)
const imageClip = new Clip({
    asset: imageAsset,
    duration: 4,
    position: Position.bottom,
    scale: 2,
    offset: new Offset({ y: -0.15 })
});
overlayTrack.addClip(5, imageClip);

timeline.addTrack(overlayTrack);
```

### ▶️ Step 7: Generate and Play the Personalized Stream

The `generate_stream()` method creates a streamable URL for your personalized video stream. You can then use `play_stream()` to preview it in your browser.

```python Python theme={null}
from videodb import play_stream

stream_url = timeline.generate_stream()
print(stream_url)
play_stream(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

[Dynamic Video Streams Output](https://www.youtube.com/embed/wm8JUjS-TwY)

## Conclusion

This tutorial showcased how to create personalized video streams using VideoDB. By integrating data from external APIs and custom databases, you can enhance your video content, personalize user experiences, and unlock new possibilities for engagement. Explore various data sources, experiment with different integrations, and customize your video streams to suit your specific needs.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Programmatic_Streams.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Intro & Outro](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Auto-add opening and closing sequences

> **[Dynamic Ads](/examples-and-tutorials/programmatic-editing/dynamic-ads)**
>
> Insert personalized ads per viewer


# Overview
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/index

Edit and compose videos with code - no manual editing tools required

## Programmatic Editing

**Edit videos with code, not clicks.** Compose, overlay, trim, and transform videos using the Timeline API.

### When to Use This

* You need to add intros/outros to hundreds of videos
* You want to insert dynamic ads based on viewer data
* You're building a tool that adds branded elements automatically
* You need multi-version output (different aspect ratios, subtitles)

### What You'll Build

> **[Intro & Outro](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Automatically add opening/closing sequences

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Overlay logos, watermarks, and graphics

> **[Audio Overlay](/examples-and-tutorials/programmatic-editing/audio-overlay)**
>
> Add background music and sound effects

> **[Dynamic Ads](/examples-and-tutorials/programmatic-editing/dynamic-ads)**
>
> Insert personalized ads per viewer

> **[Dynamic Streams](/examples-and-tutorials/programmatic-editing/dynamic-streams)**
>
> Generate multiple versions from one source

> **[Word Counter](/examples-and-tutorials/programmatic-editing/word-counter)**
>
> Add analytical text overlays

> **[Chess Montage](/examples-and-tutorials/programmatic-editing/chess-montage)**
>
> Auto-compile highlight reels from long videos

## Timeline Architecture

![Timeline Architecture Diagram](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/timeline-architecture.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=f5bb2dfed99b8ec5aaffc483fb40d8eb)

**4 Layers:**

* **Asset** - The media file (video, audio, image)
* **Clip** - A segment with timing and parameters
* **Track** - A layer in the composition
* **Timeline** - The complete output

## How It Works

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

timeline = Timeline(conn)
track = Track()
track.add_clip(0, Clip(asset=VideoAsset(id=video.id), duration=30))
timeline.add_track(track)

stream_url = timeline.generate_stream()  # Instant, no rendering
```

## Related Documentation

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Deep dive into the 4-layer model

> **[Clip Parameters](/pages/act/programmable-editing/clip-parameters)**
>
> All available clip options

> **[Aspect Ratio Control](/pages/act/programmable-editing/aspect-ratio-control)**
>
> Handle different video dimensions

> **[Caption Asset](/pages/act/programmable-editing/caption-asset)**
>
> Add text overlays and subtitles

## Explore Use Cases by Category

### Video Enhancement

> **[Intro & Outro](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Auto-add opening and closing sequences

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Add logos, watermarks, and graphics

> **[Audio Overlay](/examples-and-tutorials/programmatic-editing/audio-overlay)**
>
> Add background music and sound effects

### Dynamic Content

> **[Dynamic Ads](/examples-and-tutorials/programmatic-editing/dynamic-ads)**
>
> Insert personalized ads per viewer

> **[Dynamic Streams](/examples-and-tutorials/programmatic-editing/dynamic-streams)**
>
> Generate multiple versions from one source

### Analysis & Compilation

> **[Word Counter](/examples-and-tutorials/programmatic-editing/word-counter)**
>
> Add analytical text overlays

> **[Chess Montage](/examples-and-tutorials/programmatic-editing/chess-montage)**
>
> Auto-compile highlight reels from videos


# Intro/Outro
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/intro-outro

Add opening and closing sequences to videos

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Intro_Outro_Inline.ipynb)

## Overview

Imagine a virtual DJ mixing deck where you can seamlessly blend multiple videos into one epic timeline. Whether you're adding flashy intros, snazzy outros, or even splicing in some behind-the-scenes footage, this feature lets you take your video content to the next level!

In this tutorial, let's dive into how you can seamlessly integrate multiple videos onto a single timeline. Users can easily enhance their video content by appending intros, outros, or supplementary segments. The workflow is straightforward and scalable.

![Intro and outro sequence example showing video composition with opening and closing segments](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/intro-outro.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=7d4c7bd064ada4040d6007bb776613f3)

## Setup

### Installing packages

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

> [!NOTE]
>
> Before proceeding, ensure access to VideoDB. Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

## Implementation

### Step 1: Connect VideoDB

Connect to VideoDB using your API key to establish a session for uploading and manipulating video files. Import the necessary modules from VideoDB library to access functionalities.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Videos

First, we upload an introductory video ("intro.mp4") and an outro video ("outro.mp4") into the collection, followed by the base video ("sugar\_craving.mp4"). This approach allows us to efficiently reuse the intro and outro videos for other projects by simply changing the base video, thereby saving time and streamlining the video creation process.

You can upload the video asset from your local device or from a YouTube URL to upload the video from its source.

```python Python theme={null}
intro = coll.upload(url="https://github.com/video-db/videodb-cookbook-assets/raw/main/videos/intro.mp4")
outro = coll.upload(url="https://github.com/video-db/videodb-cookbook-assets/raw/main/videos/outro.mp4")
base = coll.upload(url="https://github.com/video-db/videodb-cookbook-assets/raw/main/videos/sugar_craving.mp4")
```

```javascript Node.js theme={null}
const intro = await coll.uploadURL({
  url: "https://github.com/video-db/videodb-cookbook-assets/raw/main/videos/intro.mp4"
});
const outro = await coll.uploadURL({
  url: "https://github.com/video-db/videodb-cookbook-assets/raw/main/videos/outro.mp4"
});
const base = await coll.uploadURL({
  url: "https://github.com/video-db/videodb-cookbook-assets/raw/main/videos/sugar_craving.mp4"
});
```

### Step 3: Create assets

Adjust parameters for all the video assets according to your preference, such as start and end times.

```python Python theme={null}
from videodb.editor import VideoAsset

intro_asset = VideoAsset(id=intro.id, start=0)
intro_duration = 3

base_asset = VideoAsset(id=base.id, start=0)
base_duration = 90

outro_asset = VideoAsset(id=outro.id, start=0)
outro_duration = 3
```

```javascript Node.js theme={null}
import { EditorVideoAsset } from 'videodb';

const introAsset = new EditorVideoAsset({ id: intro.id, start: 0 });
const introDuration = 3;

const baseAsset = new EditorVideoAsset({ id: base.id, start: 0 });
const baseDuration = 90;

const outroAsset = new EditorVideoAsset({ id: outro.id, start: 0 });
const outroDuration = 3;
```

### Step 4: Create timeline

Create video assets using the Editor SDK. The `start` parameter in `VideoAsset` trims from the beginning, and the `duration` in `Clip` controls how long the clip plays.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip

timeline = Timeline(conn)

# Create main track
track = Track()

# Add intro clip at 0 seconds
intro_clip = Clip(asset=intro_asset, duration=intro_duration)
track.add_clip(0, intro_clip)

# Add base video clip after intro
base_clip = Clip(asset=base_asset, duration=base_duration)
track.add_clip(intro_duration, base_clip)

# Add outro clip after base video
outro_clip = Clip(asset=outro_asset, duration=outro_duration)
track.add_clip(intro_duration + base_duration, outro_clip)

timeline.add_track(track)
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip } from 'videodb';

const timeline = new EditorTimeline(conn);

// Create main track
const track = new Track();

// Add intro clip at 0 seconds
const introClip = new Clip({ asset: introAsset, duration: introDuration });
track.addClip(0, introClip);

// Add base video clip after intro
const baseClip = new Clip({ asset: baseAsset, duration: baseDuration });
track.addClip(introDuration, baseClip);

// Add outro clip after base video
const outroClip = new Clip({ asset: outroAsset, duration: outroDuration });
track.addClip(introDuration + baseDuration, outroClip);

timeline.addTrack(track);
```

### Step 5: Play the generated video stream

```python Python theme={null}
from videodb import play_stream

stream = timeline.generate_stream()
play_stream(stream)
```

```javascript Node.js theme={null}
const stream = await timeline.generateStream();
console.log(stream);
```

Preview the video to ensure it functions correctly. Once satisfied, generate a stream of the video and share the link for others to view and enjoy this wholesome creation!

**Output:**

[Intro/Outro Video Output](https://www.youtube.com/embed/4uhIMnUnvJc)

## Conclusion

You can now efficiently manipulate and assemble video elements, resulting in professional-quality compositions.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Intro_Outro_Inline.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Add logos, text overlays, and custom styling to your videos

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Learn the fundamentals of VideoDB's programmable video editing


# Word Counter
Source: https://docs.videodb.io/examples-and-tutorials/programmatic-editing/word-counter

Visualize keyword occurrences in videos

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Keyword_Search_Counter.ipynb)

### Introduction

With an endless stream of new video content on our feeds, engaging the audience with dynamic visual elements can make educational and promotional videos much more impactful. VideoDB's suite of features allows you to enhance videos with programmatic editing.

In this tutorial, we'll explore how to create a video that visually counts and displays instances of a specified word as it's spoken. We'll use VideoDB's [Keyword Search](/examples-and-tutorials/video-rag/keyword-search) to index spoken words, and then apply audio and [text overlays](/pages/act/programmable-editing/text-asset) to show a counter updating in real-time with synchronized audio cues.

## Setup

### Installing packages

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

Before proceeding, ensure access to [VideoDB](https://videodb.io) and set up

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). ( Free for first 50 uploads, No credit card required)

## Steps

### Step 1: Connect to VideoDB

Establish a session for uploading videos. Import the necessary modules from VideoDB library to access functionalities.

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Video

Upload and play the video to ensure it's correctly loaded. We'll be using [this video](https://www.youtube.com/watch?v=Js4rTM2Z1Eg) for the purpose of this tutorial.

```python Python theme={null}
video = coll.upload(url="https://www.youtube.com/watch?v=Js4rTM2Z1Eg")
video.play()
```

```javascript Node.js theme={null}
const video = await coll.uploadURL({ url: "https://www.youtube.com/watch?v=Js4rTM2Z1Eg" });
console.log(video.playerUrl);
```

### Step 3: Understand Spoken Words

Analyze the video with a named spoken-word analyzer, then collect its timed words in chronological order.

```python Python theme={null}
spoken_understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "spoken_words"}],
)
spoken_understanding.wait_until_complete()

spoken_words_analyzer = spoken_understanding.get_analyzer("spoken_words")
spoken_words_output = spoken_words_analyzer.get_output()

timed_words = [
    {
        "text": word["text"],
        "start": word["start"],
        "end": word["end"],
    }
    for scene in spoken_words_output["scenes"]
    for word in scene["data"]["words"]
]

timed_words.sort(key=lambda word: (word["start"], word["end"]))
```

```javascript Node.js theme={null}
const spokenUnderstanding = await video.understand([
    { type: "spoken_words", name: "spoken_words" },
]);
await spokenUnderstanding.waitUntilComplete();

const spokenWordsAnalyzer = await spokenUnderstanding.getAnalyzer("spoken_words");
const spokenWordsOutput = await spokenWordsAnalyzer.getOutput();
const timedWords = spokenWordsOutput.scenes.flatMap((scene) =>
    scene.data.words.map((word) => ({
        text: word.text,
        start: word.start,
        end: word.end,
    }))
);

timedWords.sort((left, right) => left.start - right.start || left.end - right.end);
```

### Step 4: Keyword Search

Normalize only case and surrounding punctuation, then keep every timed word that exactly matches `education`. The artifact timestamps remain unchanged.

```python Python theme={null}
import unicodedata

keyword = "education"


def normalize_word(text):
    normalized = text.strip().casefold()
    while normalized and unicodedata.category(normalized[0]).startswith("P"):
        normalized = normalized[1:]
    while normalized and unicodedata.category(normalized[-1]).startswith("P"):
        normalized = normalized[:-1]
    return normalized


normalized_keyword = normalize_word(keyword)
keyword_occurrences = [
    word for word in timed_words if normalize_word(word["text"]) == normalized_keyword
]

keyword_occurrences
```

```javascript Node.js theme={null}
const keyword = "education";

const normalizeWord = (text) => text
    .trim()
    .toLowerCase()
    .replace(/^\p{P}+/u, "")
    .replace(/\p{P}+$/u, "");

const normalizedKeyword = normalizeWord(keyword);
const keywordOccurrences = timedWords.filter(
    (word) => normalizeWord(word.text) === normalizedKeyword
);
```

### Step 5: Setup Timeline and Audio

Initialize the timeline and prepare an audio asset to use for each word occurrence.

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, AudioAsset, VideoAsset, TextAsset
from videodb.editor import Font, Background, Alignment, HorizontalAlignment, VerticalAlignment, Position, Offset
from videodb import MediaType

timeline = Timeline(conn)

# Upload the twink sound effect
audio = coll.upload(url="https://github.com/video-db/videodb-cookbook-assets/raw/main/audios/twink.mp3", media_type=MediaType.audio)
```

```javascript Node.js theme={null}
import {
    EditorTimeline, Track, Clip, EditorAudioAsset, EditorVideoAsset, EditorTextAsset,
    Font, Background, Alignment, HorizontalAlignment, VerticalAlignment, Position, Offset
} from 'videodb';
import { MediaType } from 'videodb';

const timeline = new EditorTimeline(conn);

// Upload the twink sound effect
const audio = await coll.uploadURL({
    url: "https://github.com/video-db/videodb-cookbook-assets/raw/main/audios/twink.mp3",
    mediaType: MediaType.audio
});
```

### Step 6: Overlay Text and Audio

Add text and audio overlays at each instance where the word is spoken using the `Track` and `Clip` pattern.

Note: Adding the 'padding' is an optional step. It helps in adding a little more context to the exact instance identified, thus resulting in a better compiled output.

```python Python theme={null}
video_duration = min(300, int(video.length))  # First 5 minutes only
audio_offset = 1  # Delay audio/text update by 1 second for better sync

# Create timeline and tracks
timeline = Timeline(conn)
video_track = Track()
text_track = Track()
audio_track = Track()

# Add video clip (first 5 minutes)
video_clip = Clip(
    asset=VideoAsset(id=video.id, start=0),
    duration=video_duration)
video_track.add_clip(0, video_clip)

# Filter keyword occurrences within our duration
occurrences_in_range = [
    word
    for word in keyword_occurrences
    if int(word["start"]) + audio_offset < video_duration
]

# Add text overlays that update at each word occurrence
for i, occurrence in enumerate(occurrences_in_range):
    trigger_time = int(occurrence["start"]) + audio_offset

    # Initial "Count-0" from start until first word
    if i == 0 and trigger_time > 0:
        text_asset = TextAsset(
            text="Count-0",
            font=Font(family="Do Hyeon", size=72, color="#000100"),
            background=Background(color="#F702A4", opacity=1.0),
            alignment=Alignment(horizontal=HorizontalAlignment.right, vertical=VerticalAlignment.top),)
        text_clip = Clip(asset=text_asset, duration=trigger_time,
          position=Position.top_right, offset=Offset(x=-0.05, y=0.05))
        text_track.add_clip(0, text_clip)

    # Duration until next word or end of video
    if i + 1 < len(occurrences_in_range):
        next_trigger = int(occurrences_in_range[i + 1]["start"]) + audio_offset
    else:
        next_trigger = video_duration

    text_dur = next_trigger - trigger_time

    # Text overlay with updated count
    text_asset = TextAsset(
        text=f"Count-{i + 1}",
        font=Font(family="Do Hyeon", size=72, color="#000100"),
        background=Background(color="#F702A4", opacity=1.0),
        alignment=Alignment(horizontal=HorizontalAlignment.right, vertical=VerticalAlignment.top),)
    text_clip = Clip(asset=text_asset, duration=text_dur, position=Position.top_right, offset=Offset(x=-0.05, y=0.05))
    text_track.add_clip(trigger_time, text_clip)

    # Audio cue at same trigger time
    if trigger_time < video_duration - 2:
        audio_clip = Clip(asset=AudioAsset(id=audio.id), duration=1)
        audio_track.add_clip(trigger_time, audio_clip)

# Add all tracks to timeline
timeline.add_track(video_track)
timeline.add_track(text_track)
timeline.add_track(audio_track)
```

```javascript Node.js theme={null}
const videoDuration = Math.min(300, parseInt(video.length));  // First 5 minutes only
const audioOffset = 1;  // Delay audio/text update by 1 second for better sync

// Create timeline and tracks
const timeline = new EditorTimeline(conn);
const videoTrack = new Track();
const textTrack = new Track();
const audioTrack = new Track();

// Add video clip (first 5 minutes)
const videoClip = new Clip({
    asset: new EditorVideoAsset({ id: video.id, start: 0 }),
    duration: videoDuration
});
videoTrack.addClip(0, videoClip);

// Filter keyword occurrences within our duration
const occurrencesInRange = keywordOccurrences.filter(
    (word) => parseInt(word.start) + audioOffset < videoDuration
);

// Add text overlays that update at each word occurrence
occurrencesInRange.forEach((occurrence, i) => {
    const triggerTime = parseInt(occurrence.start) + audioOffset;

    // Initial "Count-0" from start until first word
    if (i === 0 && triggerTime > 0) {
        const textAsset = new EditorTextAsset({
            text: "Count-0",
            font: new Font({ family: "Do Hyeon", size: 72, color: "#000100" }),
            background: new Background({ color: "#F702A4", opacity: 1.0 }),
            alignment: new Alignment({ horizontal: HorizontalAlignment.right, vertical: VerticalAlignment.top })
        });
        const textClip = new Clip({
            asset: textAsset,
            duration: triggerTime,
            position: Position.topRight,
            offset: new Offset({ x: -0.05, y: 0.05 })
        });
        textTrack.addClip(0, textClip);
    }

    // Duration until next word or end of video
    const nextTrigger = i + 1 < occurrencesInRange.length
        ? parseInt(occurrencesInRange[i + 1].start) + audioOffset
        : videoDuration;

    const textDur = nextTrigger - triggerTime;

    // Text overlay with updated count
    const textAsset = new EditorTextAsset({
        text: `Count-${i + 1}`,
        font: new Font({ family: "Do Hyeon", size: 72, color: "#000100" }),
        background: new Background({ color: "#F702A4", opacity: 1.0 }),
        alignment: new Alignment({ horizontal: HorizontalAlignment.right, vertical: VerticalAlignment.top })
    });
    const textClip = new Clip({
        asset: textAsset,
        duration: textDur,
        position: Position.topRight,
        offset: new Offset({ x: -0.05, y: 0.05 })
    });
    textTrack.addClip(triggerTime, textClip);

    // Audio cue at same trigger time
    if (triggerTime < videoDuration - 2) {
        const audioClip = new Clip({
            asset: new EditorAudioAsset({ id: audio.id }),
            duration: 1
        });
        audioTrack.addClip(triggerTime, audioClip);
    }
});

// Add all tracks to timeline
timeline.addTrack(videoTrack);
timeline.addTrack(textTrack);
timeline.addTrack(audioTrack);
```

### Step 7: Generate and Play the Stream

Finally, generate a streaming URL for your edited video and play it.

```python Python theme={null}
from videodb import play_stream

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

Here's a preview of showing occurrence of the word **Education**.

[Word Counter Video Output](https://www.youtube.com/embed/riw1SZSlsU4)

### Conclusion

This tutorial showcases VideoDB’s capabilities to create a video that programmatically counts and displays the frequency of a specific keyword spoken throughout the video. This method can be adapted for various applications where dynamic text overlays add significant value to video content.

### Tips and Tricks

* Use different text styles and positions based on your video's theme.
* Add background sounds or effects to enhance the viewer's experience.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Keyword_Search_Counter.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Add logos, watermarks, and graphics

> **[Audio Overlay](/examples-and-tutorials/programmatic-editing/audio-overlay)**
>
> Add background music and sound effects


# Profanity Detection
Source: https://docs.videodb.io/examples-and-tutorials/safety-compliance/beep-profanity

Detect and censor curse words with audio overlays

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Beep%20Curse%20Words.ipynb)

### Overview

VideoDB's [Timeline Architecture](/pages/act/programmable-editing/timeline-architecture) makes it easy to personalize content to meet users' requirements. If users prefer not to include curse words in their content, VideoDB allows for these words to be either removed or replaced with a sound overlay such as a beep sound.

This task, typically complex for video editors, can be accomplished with just **a few lines of code** using VideoDB.

This technique can also serve as a valuable **Content Moderation** component for any social content platform, ensuring that content meets the preferences and standards of its audience.

![Example of inappropriate content detection and filtering](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/inappropriate-content.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=bb40e1caae7ecf8ab5d1838c849ce8e5)

Let's dive in!

## Prerequisites

### Install Dependencies

```bash theme={null}
pip install videodb
```

> [!NOTE]
>
> You'll also need a VideoDB API\_KEY, which can be obtained from the VideoDB console.

## Connect to VideoDB

Connect to VideoDB using your API key. This establishes a session for uploading and manipulating video and audio files:

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
```

##

## Source Content

For this tutorial, let's take the Joe Rogan clip, where he is trying to trick Siri into using curse words 🤣

```python Python theme={null}
from videodb import play_stream

# Joe rogan video clip
coll = conn.get_collection()
video = coll.upload(url='https://www.youtube.com/watch?v=7MV6tUCUd-c')

# watch the original video
o_stream = video.generate_stream()
play_stream(o_stream)
```

```javascript Node.js theme={null}
// Joe rogan video clip
const coll = await conn.getCollection();
const video = await coll.uploadURL({ url: 'https://www.youtube.com/watch?v=7MV6tUCUd-c' });

// watch the original video
const oStream = await video.generateStream();
console.log(oStream);
```

[Original Video](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/53db55a5-8fb1-44a0-b8c2-62cc1b4be532.m3u8)

## Index the video

Find out the curse words with the spoken Index.

```python Python theme={null}
transcript_understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
transcript_understanding.wait_until_complete()
transcript_analyzer = transcript_understanding.get_analyzer("transcript")
```

```javascript Node.js theme={null}
const transcriptUnderstanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await transcriptUnderstanding.waitUntilComplete();
const transcriptAnalyzer = await transcriptUnderstanding.getAnalyzer("transcript");
```

### Create Beep Asset

We have a sample beep sound in this folder, `beep.wav`. For those looking to add a more playful or unique touch, replacing the beep with alternative sound effects, such as a quack or any other sound, can make the content more engaging and fun.

```python Python theme={null}
# Import Editor SDK components
from videodb.editor import VideoAsset, AudioAsset, Timeline, Track, Clip

# upload beep sound - This is just a sample, you can replace it with quack or any other sound effect.
beep = coll.upload(file_path="beep.wav")

# Create audio asset from beep sound
beep_asset = AudioAsset(id=beep.id)
```

```javascript Node.js theme={null}
import { EditorAudioAsset } from 'videodb';

// upload beep sound - This is just a sample, you can replace it with quack or any other sound effect.
const coll = await conn.getCollection();
const beep = await coll.uploadFile({
  filePath: "beep.wav"
});

// Create audio asset from beep sound
const beepAsset = new EditorAudioAsset({ id: beep.id });
```

## Moderation

> [!TIP]
>
> Generate a moderation policy from the transcript, then apply the beep overlay only to the resulting timed matches.

## Get Transcript

Retrieve the transcript from the indexed video to analyze each word:

```python Python theme={null}
import math

transcript_output = transcript_analyzer.get_output()
transcript = []
for scene in transcript_output.get("scenes", []):
    for word in scene.get("data", {}).get("words", []):
        start, end = word.get("start"), word.get("end")
        if (
            word.get("text")
            and isinstance(start, (int, float)) and not isinstance(start, bool)
            and isinstance(end, (int, float)) and not isinstance(end, bool)
            and math.isfinite(start) and math.isfinite(end) and end > start
        ):
            transcript.append({"text": word["text"], "start": start, "end": end})
transcript.sort(key=lambda word: (word["start"], word["end"]))
print(" ".join(entry["text"] for entry in transcript))
```

```javascript Node.js theme={null}
const transcriptOutput = await transcriptAnalyzer.getOutput();
const transcript = transcriptOutput.scenes.flatMap((scene) =>
  (scene.data.words || [])
    .filter((word) =>
      word.text &&
      typeof word.start === "number" &&
      typeof word.end === "number" &&
      Number.isFinite(word.start) &&
      Number.isFinite(word.end) &&
      word.end > word.start
    )
    .map((word) => ({ text: word.text, start: word.start, end: word.end }))
).sort((left, right) => left.start - right.start || left.end - right.end);
console.log(transcript.map((entry) => entry.text).join(" "));
```

```python Python theme={null}
spoken_text = " ".join(
    word["text"]
    for scene in transcript_output["scenes"]
    for word in scene.get("data", {}).get("words", [])
    if word.get("text")
)
moderation_response = coll.generate_text(
    prompt=f"""
Review the spoken transcript below and identify words that should be censored
with a beep because they contain profanity, abusive language, or explicit insults.

Return lowercase base forms so variations such as "fucking" can match "fuck".
Only include terms that occur in this transcript.

Return this JSON structure:
{{"curse_words": ["word", "another_word"]}}

Transcript:
{spoken_text}
""",
    model_name="pro",
    response_type="json",
)
curse_words_list = [word.lower() for word in moderation_response["output"]["curse_words"]]
```

```javascript Node.js theme={null}
const spokenText = transcript
  .map((entry) => entry.text)
  .join(" ");
const moderationResult = await coll.generateText(`
Review the spoken transcript below and identify words that should be censored
with a beep because they contain profanity, abusive language, or explicit insults.

Return lowercase base forms so variations such as "fucking" can match "fuck".
Only include terms that occur in this transcript.

Return this JSON structure:
{"curse_words": ["word", "another_word"]}

Transcript:
${spokenText}
`, "pro", "json");
const moderationOutput = typeof moderationResult === "string"
  ? moderationResult
  : moderationResult.output;
const moderationResponse = typeof moderationOutput === "string"
  ? JSON.parse(moderationOutput)
  : moderationOutput;
const curseWords = moderationResponse?.curse_words;

if (!Array.isArray(curseWords) || !curseWords.every((word) => typeof word === "string")) {
  throw new Error("Expected curse_words in the moderation response.");
}

const curseWordsList = curseWords.map((word) => word.toLowerCase());
```

## Finding the Curse Words

We'll use a few NLP techniques to identify all variations of any offensive words, eliminating the need to manually find and include each form. Additionally, by analyzing the transcript, you can gain insights into how these sounds are transcribed, acknowledging the possibility of errors.

```python Python theme={null}
#install spacy
!pip -q install spacy

#install dataset english core
!python -m spacy download en_core_web_sm

# load the english corpus
import spacy
import re
nlp = spacy.load("en_core_web_sm")

def get_root_word(word):
    """
    This function convert each word into its root word
    """
    try:
        #clean punctuations
        cleaned_word = re.sub(r'[^\w\s]', '', word)

        # Process the sentence
        doc = nlp(cleaned_word)

        # Lemmatize the word
        lemmatized_word = [token.lemma_ for token in doc][0]  # Assuming single word input

        return lemmatized_word
    except Exception as e:
        print(f"some issue with lemma for the word {word}")
        return word
```

```javascript Node.js theme={null}
// Install natural: npm install natural
import natural from 'natural';

const stemmer = natural.PorterStemmer;

// Pre-stem the curse words list so comparisons work correctly
const stemmedCurseWords = curseWordsList.map(w => stemmer.stem(w));

function getRootWord(word) {
    /**
     * This function converts each word into its root word (stem)
     */
    try {
        // Clean punctuations
        const cleanedWord = word.replace(/[^\w\s]/g, '');

        // Stem the word
        const stemmedWord = stemmer.stem(cleanedWord);

        return stemmedWord;
    } catch (e) {
        console.log(`some issue with stemming for the word ${word}`);
        return word;
    }
}
```

### Find Timed Words to Censor and Create a Fresh Timeline

First we will identify the timestamps to beep, and then let's create a timeline using the `Track` and `Clip` pattern. Add the video clip to the main track, then loop through the transcript to add beep overlays wherever curse words are detected.

```python Python theme={null}
# 1. Filter and prepare curse metadata
padding = 0.15
curse_intervals = [
    {
        'word': w.get('text'),
        'start': max(0.0, float(w['start']) - padding),
        'end': min(float(video.length), float(w['end']) + padding),
        'raw_start': float(w['start']),
        'raw_end': float(w['end'])
    }
    for w in transcript
    if w.get('text') != '-' and get_root_word(w.get('text')) in curse_words_list
]

# 2. Building the Timeline

from videodb.editor import Timeline, Track, VideoAsset, AudioAsset, Clip

timeline = Timeline(conn)
video_track = Track()
beep_track = Track()
current_time = 0.0

print(f"{'WORD':<15} | {'START':<8} | {'END':<8} | {'DURATION'}")
print("-" * 50)

for interval in curse_intervals:
    # A. Clean segment
    if interval['start'] > current_time:
        clean_dur = interval['start'] - current_time
        video_track.add_clip(current_time, Clip(asset=VideoAsset(id=video.id, start=current_time), duration=clean_dur))

    # B. Muted segment
    mute_dur = interval['end'] - interval['start']
    video_track.add_clip(interval['start'], Clip(asset=VideoAsset(id=video.id, start=interval['start'], volume=0.0), duration=mute_dur))

    # C. Beep overlay
    beep_dur = interval['raw_end'] - interval['raw_start']
    beep_track.add_clip(interval['raw_start'], Clip(asset=AudioAsset(id=beep.id, start=0, volume=2.0), duration=min(beep_dur, float(beep.length))))

    # D. Professional Print Message
    print(f"{interval['word']:<15} | {interval['raw_start']:<8.2f} | {interval['raw_end']:<8.2f} | {beep_dur:.2f}s")

    current_time = interval['end']

# E. Final clean segment
if current_time < float(video.length):
    video_track.add_clip(current_time, Clip(asset=VideoAsset(id=video.id, start=current_time), duration=float(video.length) - current_time))

timeline.add_track(video_track)
timeline.add_track(beep_track)

stream_url = timeline.generate_stream()
print(f"\nProcessing complete. Stream URL: {stream_url}")
```

```javascript Node.js theme={null}
// 1. Filter and prepare curse metadata
const padding = 0.15;
const curseIntervals = transcript
    .filter(w => w.text !== '-' && stemmedCurseWords.includes(getRootWord(w.text)))
    .map(w => ({
        word: w.text,
        start: Math.max(0.0, parseFloat(w.start) - padding),
        end: Math.min(parseFloat(video.length), parseFloat(w.end) + padding),
        rawStart: parseFloat(w.start),
        rawEnd: parseFloat(w.end)
    }));

// 2. Building the Timeline

import { EditorTimeline, Track, EditorVideoAsset, EditorAudioAsset, Clip } from 'videodb';

const timeline = new EditorTimeline(conn);
const videoTrack = new Track();
const beepTrack = new Track();
let currentTime = 0.0;

console.log(`${'WORD'.padEnd(15)} | ${'START'.padEnd(8)} | ${'END'.padEnd(8)} | DURATION`);
console.log('-'.repeat(50));

for (const interval of curseIntervals) {
    // A. Clean segment
    if (interval.start > currentTime) {
        const cleanDur = interval.start - currentTime;
        videoTrack.addClip(currentTime, new Clip({
            asset: new EditorVideoAsset({ id: video.id, start: currentTime }),
            duration: cleanDur
        }));
    }

    // B. Muted segment
    const muteDur = interval.end - interval.start;
    videoTrack.addClip(interval.start, new Clip({
        asset: new EditorVideoAsset({ id: video.id, start: interval.start, volume: 0.0 }),
        duration: muteDur
    }));

    // C. Beep overlay
    const beepDur = interval.rawEnd - interval.rawStart;
    beepTrack.addClip(interval.rawStart, new Clip({
        asset: new EditorAudioAsset({ id: beep.id, start: 0, volume: 2.0 }),
        duration: Math.min(beepDur, parseFloat(beep.length))
    }));

    // D. Professional Print Message
    console.log(`${interval.word.padEnd(15)} | ${interval.rawStart.toFixed(2).padEnd(8)} | ${interval.rawEnd.toFixed(2).padEnd(8)} | ${beepDur.toFixed(2)}s`);

    currentTime = interval.end;
}

// E. Final clean segment
if (currentTime < parseFloat(video.length)) {
    videoTrack.addClip(currentTime, new Clip({
        asset: new EditorVideoAsset({ id: video.id, start: currentTime }),
        duration: parseFloat(video.length) - currentTime
    }));
}

timeline.addTrack(videoTrack);
timeline.addTrack(beepTrack);

const streamUrl = await timeline.generateStream();
console.log(`\nProcessing complete. Stream URL: ${streamUrl}`);
```

### Review and Share Your Moderated Video

Finally, watch and share your new stream:

```python Python theme={null}
from videodb import play_stream
play_stream(stream_url)
```

```javascript Node.js theme={null}
console.log(streamUrl);
```

[Beep-Censored Video](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/a49074ce-e024-4d0e-96ef-8a2ab67c5fd3.m3u8)

## The Real Power of Programmable Streams

If you have videos pre-uploaded and indexed, running this beep pipeline is **real-time**. So, based on your users' choices or your platform's policy, you can use information from spoken content to automatically moderate.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Beep%20Curse%20Words.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Remove Unwanted Content](/examples-and-tutorials/safety-compliance/remove-content)**
>
> Remove inappropriate sections from videos entirely

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Learn how programmable streams power real-time moderation


# AI Video Copyright Detection
Source: https://docs.videodb.io/examples-and-tutorials/safety-compliance/copyright-detection

AI-assisted visual-similarity screening with timestamped review clips

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/automated_video_copyright_detection.ipynb)

## The Problem

Reviewing a growing rights-controlled catalog for visually similar reuse candidates does not scale with manual checks alone.

This guide shows how to screen suspect videos with semantic retrieval and prepare timestamped side-by-side clips for human review. Similarity is a triage signal; it does not establish copying, ownership, licensing status, infringement, or grounds for an automated takedown.

## What You'll Build

Build a review workflow that:

* Indexes rights-controlled portfolio videos as searchable visual descriptions
* Screens suspect-video scenes for visually similar candidates
* Generates separate side-by-side comparison clips for human review
* Groups temporally adjacent candidates as review context
* Creates reports with retrieval relevance scores

All powered by **VideoDB's Editor SDK** and semantic search.

## Setup

### Install Dependencies

```bash theme={null}
pip install videodb
```

### Connect to VideoDB

```python theme={null}
import videodb

# Connect to VideoDB
api_key = "your_api_key"
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

## Implementation

### Step 1: Index Your Portfolio Videos

```python theme={null}
portfolio_videos = [
    coll.upload(url="https://example.com/original-video-1.mp4"),
    coll.upload(url="https://example.com/original-video-2.mp4"),
]
portfolio_indexes = {}
for video in portfolio_videos:
    index_name = f"portfolio_visual_{video.id}"
    index = next((item for item in video.list_indexes() if item.name == index_name), None)
    if index is None:
        understanding = video.understand(
            segmentation={"type": "shot", "threshold": 20},
            analyzers=[{"type": "vlm", "name": "portfolio_scene", "config": {
                "prompt": "Describe the visual content in detail: people, objects, actions, setting, colors, composition, camera angle, lighting, and any visible text or logos.",
                "schema": {"description": "string"}}}],
        )
        understanding.wait_until_complete()
        analyzer = understanding.get_analyzer("portfolio_scene")
        index = video.index(
            name=index_name,
            source=analyzer,
            use_for=["semantic"],
            fields={"semantic": ["description"]},
        )
    index.wait_until_complete()
    portfolio_indexes[video.id] = index.index_id
```

### Step 2: Upload Suspect Video for Analysis

```python theme={null}
suspect_video = coll.upload(url="https://example.com/suspect-video.mp4")
suspect_understanding = suspect_video.understand(
    segmentation={"type": "shot", "threshold": 20},
    analyzers=[{"type": "vlm", "name": "portfolio_scene", "config": {
        "prompt": "Describe the visual content in detail: people, objects, actions, setting, colors, composition, camera angle, lighting, and any visible text or logos.",
        "schema": {"description": "string"}}}],
)
suspect_understanding.wait_until_complete()
suspect_analyzer = suspect_understanding.get_analyzer("portfolio_scene")
```

### Step 3: Perform Similarity Comparison

```python theme={null}
import math


def finite_scene_records(analyzer):
    records = []
    for scene in analyzer.get_output().get("scenes", []):
        data = scene.get("data")
        start, end = scene.get("start"), scene.get("end")
        description = data.get("description") if isinstance(data, dict) else None
        if (
            isinstance(start, (int, float)) and not isinstance(start, bool)
            and isinstance(end, (int, float)) and not isinstance(end, bool)
            and math.isfinite(start) and math.isfinite(end) and end > start
            and isinstance(description, str) and description.strip()
        ):
            records.append({"start": start, "end": end, "description": description.strip()})
    return records


def similarity_candidates(suspect_video, suspect_analyzer, portfolio_indexes, relevance_threshold=0.70):
    matches = []
    for scene in finite_scene_records(suspect_analyzer):
        results = coll.semantic_search(
            query=scene["description"],
            index_ids=list(portfolio_indexes.values()),
            top_k=1,
        )
        for shot in results.get_shots():
            score, start, end = shot.search_score, shot.start, shot.end
            if (
                isinstance(score, (int, float)) and not isinstance(score, bool)
                and isinstance(start, (int, float)) and not isinstance(start, bool)
                and isinstance(end, (int, float)) and not isinstance(end, bool)
                and math.isfinite(score) and math.isfinite(start) and math.isfinite(end)
                and end > start and score >= relevance_threshold
            ):
                matches.append({
                    "suspect_video_id": suspect_video.id,
                    "suspect_start": scene["start"],
                    "suspect_end": scene["end"],
                    "portfolio_video_id": shot.video_id,
                    "portfolio_start": start,
                    "portfolio_end": end,
                    "similarity": float(score),
                })
    return sorted(matches, key=lambda item: item["similarity"], reverse=True)


matches = similarity_candidates(
    suspect_video,
    suspect_analyzer,
    portfolio_indexes,
    relevance_threshold=0.70,
)
```

### Step 4: Group Sequential Review Candidates

```python theme={null}
from collections import defaultdict


def monotonic_candidate_sequences(matches, max_gap=10.0, min_length=3):
    sequences = []
    by_portfolio_video = defaultdict(list)
    for match in matches:
        by_portfolio_video[match["portfolio_video_id"]].append(match)

    for portfolio_video_id, candidates in by_portfolio_video.items():
        ordered = sorted(candidates, key=lambda item: (item["suspect_start"], item["portfolio_start"]))
        sequence = []
        for candidate in ordered:
            if not sequence:
                sequence = [candidate]
                continue
            previous = sequence[-1]
            suspect_gap = candidate["suspect_start"] - previous["suspect_end"]
            portfolio_gap = candidate["portfolio_start"] - previous["portfolio_end"]
            if (
                candidate["suspect_start"] > previous["suspect_start"]
                and candidate["portfolio_start"] > previous["portfolio_start"]
                and 0 <= suspect_gap <= max_gap
                and 0 <= portfolio_gap <= max_gap
            ):
                sequence.append(candidate)
            else:
                if len(sequence) >= min_length:
                    sequences.append({"portfolio_video_id": portfolio_video_id, "matches": sequence})
                sequence = [candidate]
        if len(sequence) >= min_length:
            sequences.append({"portfolio_video_id": portfolio_video_id, "matches": sequence})
    return sequences


sequences = monotonic_candidate_sequences(matches)
```

### Step 5: Generate Side-by-Side Review Clips

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset, Position, Fit

review_clips = []
for match in matches[:5]:
    # Each candidate receives a separate side-by-side review timeline.
    timeline = Timeline(conn)
    track = Track()
    duration = min(
        match["portfolio_end"] - match["portfolio_start"],
        match["suspect_end"] - match["suspect_start"],
    )

    # Portfolio video (left side)
    portfolio_asset = VideoAsset(
        id=match["portfolio_video_id"],
        start=match["portfolio_start"]
    )
    portfolio_clip = Clip(
        asset=portfolio_asset,
        duration=duration,
        position=Position.left,
        fit=Fit.crop,
        scale=0.5
    )
    track.add_clip(0, portfolio_clip)

    # Suspect video (right side)
    suspect_asset = VideoAsset(
        id=suspect_video.id,
        start=match["suspect_start"]
    )
    suspect_clip = Clip(
        asset=suspect_asset,
        duration=duration,
        position=Position.right,
        fit=Fit.crop,
        scale=0.5
    )
    track.add_clip(0, suspect_clip)

    timeline.add_track(track)
    review_clips.append(
        {
            "portfolio_video_id": match["portfolio_video_id"],
            "similarity": match["similarity"],
            "stream_url": timeline.generate_stream(),
        }
    )
```

### Step 6: Generate a Review Report

```python theme={null}
# Create a review report. Similarity scores are not legal conclusions.
report = {
    "suspect_video_id": suspect_video.id,
    "relevance_threshold": 0.70,
    "candidate_count": len(matches),
    "monotonic_candidate_groups": len(sequences),
    "review_clips": review_clips,
    "review_note": "AI-assisted screening only; human rights and legal review are required.",
}
```

## What You Get

A review workflow with:

* Scene-by-scene visual descriptions for rights-controlled reference videos
* Semantic similarity candidates for human review
* Monotonic, bounded-gap candidate groups for temporal context
* Separate side-by-side comparison clips
* Retrieval relevance scores and an explicit human-review note

Here is an example side-by-side review clip:

[Copyright Detection Review Video](https://console.videodb.io/player?url=https://play.videodb.io/v1/a779fe53-bcc7-457e-a04f-4d818c031b26.m3u8)

## How It Works

1. **Portfolio Indexing** - Convert rights-controlled reference videos into searchable scene descriptions
2. **Suspect Upload** - Upload a video for screening
3. **Similarity Scan** - Retrieve visually similar candidates for review
4. **Sequential Detection** - Group monotonic candidates with bounded timestamp gaps
5. **Review Clip Generation** - Create side-by-side comparison clips
6. **Human Review** - Assess ownership, licensing, and legal questions outside this workflow

## Similarity Thresholds

* **0.70+** = Candidate included for review
* **Below 0.70** = Excluded by this example's relevance cutoff

Tune the cutoff for review workload and false positives; semantic relevance does not determine copying or infringement.

## The Result

With this system, you can:

* Screen candidate videos against a rights-controlled portfolio
* Review timestamped, visually similar moments
* Generate side-by-side review clips for investigators
* Prioritize human review with transparent relevance scores

Use the results as AI-assisted screening, not as a legal conclusion or automated enforcement decision.

> **[Explore the Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/creative/automated_video_copyright_detection.ipynb)**
>
> Open the complete AI-assisted review workflow with batch processing and portfolio management.

## Related Tutorials

> **[Profanity Beeper](/examples-and-tutorials/safety-compliance/beep-profanity)**
>
> Auto-detect and beep curse words in audio

> **[Content Removal](/examples-and-tutorials/safety-compliance/remove-content)**
>
> Skip inappropriate visual content in streams


# Overview
Source: https://docs.videodb.io/examples-and-tutorials/safety-compliance/index

Moderate content, detect violations, and ensure compliance with automated video analysis

## Safety & Compliance

**Protect your platform and users.** Automatically detect, filter, and handle problematic content.

### When to Use This

* You need to create family-friendly versions of content
* You want to detect copyright violations before publishing
* You're building a platform that requires content moderation
* You need to filter inappropriate content in real-time

### What You'll Build

> **[Profanity Beeper](/examples-and-tutorials/safety-compliance/beep-profanity)**
>
> Auto-detect and beep curse words in audio

> **[Content Removal](/examples-and-tutorials/safety-compliance/remove-content)**
>
> Skip inappropriate visual content in streams

> **[Copyright Detection](/examples-and-tutorials/safety-compliance/copyright-detection)**
>
> Identify copyrighted material with timestamps

## How It Works

### Audio Moderation

```
Video → Transcript → NLP Detection → Beep Overlay → Clean Stream
```

### Visual Moderation

```
Video → Scene Analysis → Content Classification → Safe Timeline → Family-Friendly Stream
```

### Copyright Protection

```
Video → Content Fingerprinting → Database Match → Violation Report
```

## Key Capabilities

| Feature | What It Does |
| - | - |
| **Spoken Word Detection** | Find specific words/phrases in audio |
| **Visual Content Analysis** | Detect inappropriate scenes using AI |
| **Audio Replacement** | Overlay beeps without re-encoding |
| **Timeline Skipping** | Generate streams that skip flagged sections |
| **Timestamp Reports** | Know exactly where violations occur |

## Related Documentation

> **[Event Detection](/pages/act/live-action/event-detection-patterns)**
>
> Build detection rules for violations

> **[Privacy Controls](/pages/ingest/capture-sdks/privacy-controls)**
>
> User consent and data protection

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> How content skipping works

> **[Natural Language Query](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Search for specific content patterns


# AI-Powered Content Moderation
Source: https://docs.videodb.io/examples-and-tutorials/safety-compliance/remove-content

Moderate video content with structured visual understanding and no external APIs required

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Content_Moderation.ipynb)

## The Idea

Content moderation can be complex, often requiring multiple tools, manual timestamp extraction, and intricate integration work. Setting up these pipelines involves managing credentials, parsing responses, and stitching everything together.

VideoDB simplifies this into a **"Prompt-and-Filter"** workflow using structured video understanding. The analyzer emits a structured `moderation_label` and `reason` for each timed segment, which the index can query and aggregate.

The innovation is simple: instead of generic video descriptions, we give the AI a **strict moderation role** with a small structured schema. This turns unstructured video into timed data that can be filtered and counted.

Want stricter moderation? **Update the prompt**. Need different criteria? Change a few lines. It's content moderation reimagined for the prompt engineering era.

## Setup

### Install Dependencies

```python Python theme={null}
pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### Connect to VideoDB

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). Free for first 50 uploads, no credit card required.

```python Python theme={null}
from videodb import connect

conn = connect(api_key="YOUR_API_KEY")
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

## Implementation

### Step 1: Upload Video

We'll use a Breaking Bad clip with mixed content to test the moderation workflow.

```python Python theme={null}
# Upload video from YouTube
video = coll.upload(url='https://www.youtube.com/watch?v=Xa7UaHgOGfM')
print(f"Uploaded Video ID: {video.id}")

# Preview the video
video.play()
```

```javascript Node.js theme={null}
// Upload video from YouTube
const video = await coll.uploadURL({ url: 'https://www.youtube.com/watch?v=Xa7UaHgOGfM' });
console.log(`Uploaded Video ID: ${video.id}`);

// Get the player URL
const playerUrl = video.playerUrl;
console.log(playerUrl);
```

### Step 2: Understand Content with a Moderator Prompt

This is the core innovation. We give the AI a strict role as a Content Moderator with a schema containing `moderation_label` and `reason`. The prompt instructs the AI to analyze visual content for specific inappropriate elements and record the decision in those fields.

This structured labeling transforms unstructured video into searchable, filterable data.

```python Python theme={null}
moderation_prompt = """
You are a Content Moderator. Analyze the visual content for inappropriate elements:
1. Violence (fighting, hitting, shooting)
2. Weapons (guns, knives)
3. Blood or gore
4. Drug use
5. Sexual content

Set moderation_label to exactly CONTENT_UNSAFE if any of these are present.
Otherwise, set moderation_label to exactly CONTENT_SAFE.
Set reason to a brief explanation of the decision.
"""
moderation_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[{
        "type": "vlm",
        "name": "moderation",
        "sampling": {"strategy": "uniform", "frame_count": 3},
        "config": {"prompt": moderation_prompt, "schema": {
            "moderation_label": "string", "reason": "string"}},
    }],
)
moderation_understanding.wait_until_complete()
moderation_analyzer = moderation_understanding.get_analyzer("moderation")
moderation_index = video.index(
    name="content_moderation",
    source=moderation_analyzer,
    use_for=["query", "aggregate"],
    fields={"filter": ["moderation_label"], "aggregate": ["moderation_label"]},
)
moderation_index.wait_until_complete()
```

```javascript Node.js theme={null}
const moderationPrompt = `
You are a Content Moderator. Analyze visual content for violence, weapons,
blood or gore, drug use, and sexual content.
Set moderation_label to exactly CONTENT_UNSAFE if any are present; otherwise
set it to exactly CONTENT_SAFE. Set reason to a brief explanation.
`;

const moderationUnderstanding = await video.understand(
  [{
    type: "vlm",
    name: "moderation",
    sampling: { strategy: "uniform", frame_count: 3 },
    config: {
      prompt: moderationPrompt,
      schema: { moderation_label: "string", reason: "string" },
    },
  }],
  { segmentation: { type: "time", seconds: 5 } },
);
await moderationUnderstanding.waitUntilComplete();
const moderationAnalyzer = await moderationUnderstanding.getAnalyzer("moderation");
const moderationIndex = await video.index(moderationAnalyzer, {
  name: "content_moderation",
  useFor: ["query", "aggregate"],
  fields: { filter: ["moderation_label"], aggregate: ["moderation_label"] },
});
await moderationIndex.waitUntilComplete();
```

**Why this works:** The structured `moderation_label` field supports exact filtering, while `reason` preserves the explanation for each timed decision.

### Step 3: Review Moderation Output (Optional)

Want to see what the AI detected? Inspect the analyzer output to understand each timed moderation decision.

```python Python theme={null}
moderation_output = moderation_analyzer.get_output()
for scene in moderation_output["scenes"][:5]:
    data = scene.get("data", {})
    print(scene["start"], scene["end"], data.get("moderation_label"), data.get("reason"))
```

```javascript Node.js theme={null}
const moderationOutput = await moderationAnalyzer.getOutput();
for (const scene of moderationOutput.scenes.slice(0, 5)) {
  console.log(scene.start, scene.end, scene.data?.moderation_label, scene.data?.reason);
}
```

**Sample output:**

```
0.0 5.005 CONTENT_SAFE Title cards with a smoky background.
5.005 10.01 CONTENT_SAFE Two men in indoor settings with no inappropriate elements.
20.02 25.025 CONTENT_UNSAFE Implied physical confrontation and aggressive interaction.
```

### Step 4: Filter for Safe Content

Now the magic happens. Aggregate the structured labels for a moderation summary, then query the timed records that are safe to include.

```python Python theme={null}
moderation_counts = video.aggregate(
    index_name=moderation_index.name,
    group_by="moderation_label",
    metric="count",
    limit=10,
)
safe_results = video.query(
    index_name=moderation_index.name,
    filter=[{"field": "moderation_label", "op": "==", "value": "CONTENT_SAFE"}],
    limit=100,
    sort=[("start", "asc")],
)
safe_shots = safe_results.get_shots()
print(f"Found {len(safe_shots)} safe segments")
```

```javascript Node.js theme={null}
const moderationCounts = await video.aggregate({
  indexName: moderationIndex.name,
  groupBy: "moderation_label",
  metric: "count",
  limit: 10,
});
const safeResults = await video.query({
  indexName: moderationIndex.name,
  filter: [{ field: "moderation_label", op: "==", value: "CONTENT_SAFE" }],
  limit: 100,
  sort: [["start", "asc"]],
});
console.log(`Found ${safeResults.getShots().length} safe segments`);
```

### Step 5: Play the Clean Version

Compile the filtered results into a stream for playback.

```python Python theme={null}
clean_stream_url = safe_results.compile()
print("Stream URL:", clean_stream_url)
safe_results.play()
```

```javascript Node.js theme={null}
const cleanStreamUrl = await safeResults.compile();
console.log("Stream URL:", cleanStreamUrl);
await safeResults.play();
```

Here's the result - a clean version with all inappropriate content removed:

[AI-Powered Content Moderation Result](https://stream.videodb.io/v3/published/manifests/9ff92994-de07-46b2-b2db-c652e22d2b5c.m3u8)

## What You Get

* No external APIs or credentials required
* Full control over moderation criteria through prompts
* Instant filtering without video re-encoding
* Granular 5-second scene analysis
* Real-time playback of cleaned content
* Customizable: change prompt to adjust moderation standards instantly

## Perfect For

* Educational platforms serving minor audiences
* Family-friendly streaming services
* Corporate training content libraries
* Social media platforms with content policies
* Broadcasting companies creating TV-safe edits
* User-generated content platforms with safety requirements

## The Result

What used to require multiple integrations, manual timestamp extraction, and complex video editing pipelines now works with prompt engineering. Change the moderation criteria by updating the prompt and running the moderation understanding again.

Pure simplicity powered by VideoDB's structured video understanding.

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Content_Moderation.ipynb)**
>
> Open the complete implementation in Google Colab with detailed explanations and working code.

## Related Tutorials

> **[Profanity Detection](/examples-and-tutorials/safety-compliance/beep-profanity)**
>
> Detect and censor curse words with audio overlays

> **[Keyword Search](/examples-and-tutorials/video-rag/keyword-search)**
>
> Find and extract specific content from your videos


# Case Study: NFL Game Analysis
Source: https://docs.videodb.io/examples-and-tutorials/video-rag/case-study-nfl

Learn how VideoDB reduces VLM hallucinations by 80% and cuts costs by 70% through play-by-play segmentation for NFL game footage analysis.

Vision Language Models (VLMs) shine in controlled benchmarks, yet they stumble in real-world, event-dense footage such as an NFL game. **VideoDB bridges that gap** by letting developers slice video at the right semantic boundaries, combine external stats, and run multi-tier visual/LLM pipelines that cut hallucinations by >80 % while costing up to **70 % less** than a naïve “1 fps into Gemini” workflow. If you need fast, cheap, accurate visual reasoning, you need more than a big VLM—you need a video-native AI Infrastructure.

In this blog, we'll explore different basic to advanced approaches for [NFL game footage](https://www.youtube.com/watch?v=pA_xAsb5hbA). We want this writeup to guide you into solving real world scenarios in video understanding. To know how each method performed, we focused on the following key aspects:

| Column 1 | Column 2 |
| - | - |
| **Hallucination** | Frequency of incorrect or irrelevant information produced by the VLM. |
| **Temporal Context** | How accurately the VLM maintains correct chronological relationships within the video. |
| **Performance on Granular Queries** | The VLM's effectiveness in accurately responding to detailed and specific queries. |
| **VideoDB Involvement** | The extent to which VideoDB's capabilities were leveraged to enhance VLM performance |

## 1. The Naive Gemini Approach

![Naive approach showing sending entire NFL game footage directly to Gemini VLM](https://mintcdn.com/videodb/CuGZng-6F2fSOYMm/assets/visual-analysis/native-approach.webp?fit=max\&auto=format\&n=CuGZng-6F2fSOYMm\&q=85\&s=79229ea4a7d4a81105ab2df1b7659e9c)

Initially, we tried directly inputting complete NFL game footage into Gemini, expecting robust results based on benchmark promises.

**Observations**

| Evaluation metric | Observation | Notes |
| - | - | - |
| **Hallucination** | **68.1 %** | Frequent irrelevant predictions (68.1%). |
| **Temporal Context** | **Bloated** | Model often lost critical event continuity. |
| **Performance on Granular Queries** | **Moderate** | Struggled significantly. |
| **VideoDB Involvement** | **Low** | |

The model frequently produced incorrect or imaginary events (hallucinations), and misclassifications were common. Furthermore, it often completely missed or overlooked critical events, revealing significant weaknesses in accuracy and contextual comprehension.

*This visualization shows a timeline of the first 20 minutes of the game. The top row in grey indicates the ground truth (actual events from the game), while the bottom row displays predictions made by VLM. Each predicted label is color coded: green indicates the model correctly matched the ground truth, and red indicates a mismatch. This side by side comparison helps assess the VLM’s performance. For this visualization, we have limited the scope to running plays, catches, touchdowns, and punts.*

**Known VLM Limitations (Why “just send it to Gemini” falls short)**

* **Finite context windows** – even a 1 M-token window can’t hold one NFL quarter at 30 fps.
* **Image-tile token explosion** – every 1080p frame is split into \~4–9 tiles (≈ 1–4 k tokens) before the model “sees” it.
* **Weak event reasoning** – current VLMs reason per-frame, not per-*play*; they miss temporal causality (e.g., *“Was the QB still behind the line when he released?”*).
* **Cost scales linearly** with frames, so 30 fps steals wallets fast.

## 2. Uniform-Length Chunks (Possible with VideoDB)

![Uniform-length chunks approach showing video divided into fixed-size segments](https://mintcdn.com/videodb/CuGZng-6F2fSOYMm/assets/visual-analysis/uniform-length-chunks.webp?fit=max\&auto=format\&n=CuGZng-6F2fSOYMm\&q=85\&s=9953b000460d91485b544061fcd69398)

We chopped footage into fixed 2s / 5s / 10s clips via VideoDB's scene index API. The hypothesis was straightforward: shorter, consistently sized segments might simplify the VLM's task, potentially enhancing accuracy.

**Code snippet for uniform chunk segmentation:**

```python theme={null}
from videodb import SceneExtractionType

conn = videodb.connect(api_key="YOUR_API_KEY")
collection = conn.get_collection()

video = collection.upload(url="https://www.youtube.com/...")

# Extract uniform scenes every n seconds
scene_collection = video.extract_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 5, "num_frames": 150},  # Example: 5 seconds per chunk
)

print(f"Scene Collection ID: {scene_collection.id}")
```

The goal of testing these configurations was to understand how varying segment lengths could impact model accuracy and output clarity.

**Observations**

| Evaluation metric | Observation |
| - | - |
| **Hallucination** | **74.2 %** |
| **Temporal Context** | **Insufficient** |
| **Performance on Granular Queries** | **Moderate** |
| **VideoDB Involvement** | **Moderate** |

**Arbitrary clip boundaries lose context**

* Important actions (e.g., QB throw) get split across two clips, so the model can’t see the full play and mis-judges legality or outcome.
* Same problem pops up for catches, interceptions, and other decisive moments.

**Higher hallucination rate**

* Model starts “imagining” passes and catches that never happened, simply because it lacks enough temporal evidence in a single clip.
* Resulting event timeline is noisy and bloated with false positives.

**Goldilocks problem with clip length**

* Too long → information overload and confusion.
* Too short → not enough context.
* Neither extreme works; we need a balanced segmentation window.

## 3. Play-by-Play Segmentation (Advanced Pipeline with VideoDB)

![Play-by-play segmentation approach showing semantic alignment with actual game plays using external stats](https://mintcdn.com/videodb/CuGZng-6F2fSOYMm/assets/visual-analysis/play-by-play-segmentation.webp?fit=max\&auto=format\&n=CuGZng-6F2fSOYMm\&q=85\&s=8d81c62196f9fe99ea4af5a1608ec533)

During our analysis, we found that detailed statistical reports for major sports games are typically publicly available. These reports offer extensive information ranging from basic team formations and player lineups to precise, event-specific details such as passes, touchdowns, interceptions, and catches. Most importantly for our analysis, these reports include exact timestamps marking the start and end of each play, making them ideal for accurate segmentation.

Reliable sources for such detailed play-by-play data include official NFL Scores pages, where users can select specific seasons, weeks, and games to access comprehensive statistics. For our specific analysis, we referred directly to the official game summary PDF.

However, we encountered a significant practical issue: the timestamps provided by these reports reflected the official game clock, not the actual video timestamps. To segment the video correctly, we needed to align these official game timestamps accurately with the video's runtime.

### 3.1 Aligning Game-Time with Video-Time

To solve this, we utilized the consistent visual feature found in all NFL broadcasts—the on-screen scoreboard. This scoreboard continuously displays vital game information, including scores, current quarter, down and yardage, and crucially, the game clock itself. By extracting this information, we could precisely map the game's official timestamps to corresponding points in the video.

### How We Achieved This

* **OCR-based Timestamp Extraction:** We processed the video using an Optical Character Recognition (OCR) model to detect and extract visible game times from the scoreboard throughout the video.
* **Frame Sampling Optimization:** To optimize efficiency, we sampled just one frame per second (1 fps) for OCR processing. This significantly reduced computational load without compromising the accuracy of the extracted timestamps.
* **Timestamp Mapping Creation:** The OCR results provided an exact correlation between the official game timestamps and the actual runtime of the video. Using this mapping, we segmented the video accurately into individual play-by-play events.

```python theme={null}
from videodb import SceneExtractionType

# 1 frame per second
scene_collection = video.extract_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 1, "select_frames": ["first"]} 
)

print(f"Scene Collection ID: {scene_collection.id}")

# Perform OCR with structured prompt
scene_ocr_results = {}
for scene in scene_collection.scenes:
    for frame in scene.frames:
        structured_description = frame.describe(
            """Perform OCR to extract the information from the scorebar present.
              on the bottom of the image. Output in a structured JSON format: 
              <team1 name>: <team1 score>, <team2 name>: <team2 score>, 
              quarter_number: <quarter number>, game_clock: <time left>."""                                                                                    )

# Save the OCR result with scene start time as the key
scene_ocr_results[scene.start] = structured_description

print(scene_ocr_results)

# Example Output:
{
    1.0: {"Browns": 0, "Raiders": 0, "quarter_number": 1, "game_clock": "15:00"}
    2.0: {"Browns": 0, "Raiders": 0, "quarter_number": 1, "game_clock": "14:59"}
    ...
}
```

### Integrating Play-by-Play Segmentation with VideoDB

VideoDB effectively supports this customized, non-uniform segmentation approach. We imported our precise timestamp mappings directly into VideoDB, creating accurate and detailed scene indexes seamlessly:

```python theme={null}
# code snippet for detecting catches from the game footage

# Step 1: Use the stats PDF to filter all play timestamps (game clock) where a catch occurred into `catch_play_scenes` as a list of (start, end) for plays with catches

# Step 2: Map game clocks to video timestamps using OCR outputs
catch_details = []

#iterate over each play
for play_scene in catch_play_scenes:
    start_time, end_time = play_scene
    
    #create a new scene object of the play
    scene = Scene(
        video_id=video.id,
        start=start_time,
        end=end_time
    )

    # Call describe with structured prompt to get catch_details
    result = scene.describe(
        """"This clip contains a catch. 
              Extract: type of catch, 
              position of the player who caught the ball, 
              and whether it was an interception. 
              Return a structured JSON with keys
            {catch_type,player_position,interception}."""
    )
    catch_details.append({
        "scene_start": start_time,
        "scene_end": end_time,
        "details": result
    })


print(catch_details)

# Example Output:
[
    {
        "scene_start": 120.0,
        "scene_end": 165.0,
        "details": {
            "catch_type": "over the shoulder",
            "player_position": "centre of the field",
            "interception": False
        }
    },
 ...
]

# Do the same for other concepts ( other than catches ) 
```

By adopting this precise segmentation strategy, VideoDB enhanced accuracy, dramatically reduced misclassifications, and simplified complex visual analysis tasks, providing unparalleled insights into detailed sports event analysis.

**Observations**

| Evaluation metric | Observation |
| - | - |
| Hallucination | **11.4%** |
| Temporal Context | **perfect** |
| Performance on Granular Queries | **High** |
| VideoDB Involvement | **High** |

## Approach Comparison

| Evaluation Metric | Naïve Whole-Video | Uniform Chunks | Play-by-Play |
| - | - | - | - |
| Hallucination | 68.1 % | 74.2 % | **11.4 %** |
| Temporal Context | Poor | Insufficient | **Perfect** |
| Granular Queries | Moderate | Moderate | **High** |
| VideoDB Use | Low | Moderate | **High** |

### Key Takeaways

1. **Define Key Sports Concepts:** Clearly outline and specify each concept required for analysis. For example:
   * Catch (Yes/No)
   * Running Play (Yes/No)
   * Scoring Event (Yes/No)

2. **Check Availability of Statistical Data:** Determine if these concepts can be reliably extracted from existing statistical data:
   * If statistical data is available: use it to isolate specific plays
   * If statistical data is not available: directly use the Vision Language Model (VLM) for visual extraction

3. **Extract Relevant Plays Using Statistical Data:** Use accurate statistical information to isolate relevant video scenes using VideoDB Timeline. Record timestamps and relevant metadata for these Scenes.

4. **Visual Analysis with VideoDB indexing:** Pass the extracted scenes into the VLM to gather detailed visual insights (e.g., identifying catch types like "overhead" and positions like "near sidelines").

5. **Clearly Structure the Output Data:** Organize the extracted visual information into structured data for clarity and ease of querying. For instance:

```json theme={null}
[
{
  "play_start_time" : 12,
  "play_end_time" : 52,
  "details": {
    "catch": true,
    "type": "overhead",
    "position": "near sidelines",
    "interception" : false,
    "running_play": true
  }
},
...
]
```

6. **Query and Reasoning Engine (Small LLM):** Upon receiving a user query, feed the structured data and query into VideoDB search interface. The engine processes these inputs and returns relevant, accurate play-by-play results.

## Pricing: VideoDB vs. Gemini @ 1 fps

| 60-min NFL Game | Frames Analysed | VideoDB (Balanced tier) | Gemini 1.5 Pro\* |
| - | - | - | - |
| **1 fps, 1080p** | 3600 | \*\*$2.00** index + ≈$0.35 tokens | $1.1 – $7.4 |
| **5 fps** | 18000 | **\$10.00** index | $5.6 – $37.0 |
| **30 fps** | 108000 | **\$12.00** index | $33 – $220 |

*Prices use Google’s published rate card: $0.10 /M input tokens, $0.40 /M output; HD frames tokenize into 1 024–4 128 tokens each.*

As frame-rate or resolution rises, VideoDB’s flat visual-index pricing stays predictable while pure-Gemini costs explode.

## Why Choose VideoDB

* **Event-aligned indexing** – cut by play, scene, or any custom timeline—not crude 1s slices
* **Hybrid reasoning pipelines** – blend stats, embeddings, and VLMs to slice hallucinations to \~11%
* **Serverless scale** – ingest petabytes or a single clip; zero idle cost
* **Developer-first API** – Python, JS, REST; one call for streams, one for scene queries
* **Transparent pricing** – pay once for storage + index; pick Entry / Advanced / SOTA LLM pricing per query

## Explore the Docs

Learn more about the techniques and capabilities used in this case study:

> **[Visual Search Pipelines](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> Build sophisticated multi-modal search workflows

> **[Scene Indexing](/pages/understand/indexing-pipelines/create-an-index)**
>
> Segment and index video at semantic boundaries

> **[Custom Annotations](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> Add domain-specific metadata to video content

> **[Prompt Engineering](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Optimize prompts for better VLM results

Questions or feedback? Email us at [engg@videodb.io](mailto:engg@videodb.io)—we love hearing what you’re building!


# Character Extraction
Source: https://docs.videodb.io/examples-and-tutorials/video-rag/character-clips

Extract clips featuring specific people or characters

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Create%20Clip%20Faces.ipynb)

## Overview

We all have our favorite characters in TV shows that we love. What if you could create the clips of the scenes when they appear? You can use it in your content creation, analysis workflow or simply observe how they talk, dress or act in an episode.

**VideoDB** is the database for your AI applications and enables it with ease. No need for fancy editing software or waiting around – you get to view your video right away. ⚡️

![Face recognition and character extraction example using AWS Rekognition](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/face-clip.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=9d15081e1d0bd0efff619fad6739b322)

## Workflow

Here's a [15-minute video](https://www.youtube.com/watch?v=NNAgJ5p4CIY) from HBO's Silicon Valley show. We've already identified instances when `Gilfoyle`, `Jian Yang`, `Erlich`, `Jared`, `Dinesh` and `Richard` make appearances with the help of [AWS Rekognition API](https://docs.aws.amazon.com/rekognition/).

We've pinpointed the timestamps of the characters' appearances in the video and mapped out where they appear in `persons_data`.

Next, we'll upload the video to VideoDB and use these timestamps to clip the video. It's as easy as querying a database⚡️

## Setup

### Installing VideoDB in your environment

`VideoDB` is available as a [python package](https://pypi.org/project/videodb) and [npm package](https://www.npmjs.com/package/videodb)

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

###

### Setting Up a connection

To connect to **VideoDB**, simply create a `Connection` object.

This can be done by either providing your VideoDB API key directly to the constructor or by setting the `VIDEO_DB_API_KEY` environment variable with your API key.

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). ( Free for first 50 uploads, No credit card required)

```python Python theme={null}
from videodb import connect
conn = connect(api_key="YOUR_API_KEY")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
```

## Timeline Data

Here's the json we compiled with the timestamps of appearances of each character. The **persons\_data** contains **timeline** for each character, representing the timestamps of shots when they were present in the video.

```json theme={null}
persons_data = {
    "gilfoyle": {
        "timeline": [[0, 4], [160, 185], [330, 347], [370, 378], [382, 391], [391, 400]]
    },
    "jianyang": {
        "timeline": [[232, 271], [271, 283], [284, 308], [312, 343], [398, 407]]
    },
    "erlich": {
        "timeline": [[0, 8], [12, 30], [31, 41], [44, 52], [56, 97], [97, 124], [147, 165], [185, 309], [316, 336], [336, 345], [348, 398], [398, 408]]
    },
    "jarad": {
        "timeline": [[0, 15], [148, 165], [182, 190], [343, 355], [358, 381], [384, 393]]
    },
    "dinesh": {
        "timeline": [[0, 4], [160, 189], [343, 354], [374, 383], [392, 402]]
    },
    "richard": {
        "timeline": [[12, 41], [127, 137], [137, 154], [159, 167], [360, 378], [381, 398], [399, 407]]
    }
}
```

## Creating instant clips using VideoDB

The idea behind `VideoDB` is straightforward: it functions as a database specifically for videos. Similar to uploading tables or JSON data to a standard database, you can upload your videos to VideoDB.

You can also retrieve your videos through queries, much like accessing regular data from a database.

You can pass timeline of each person in `video.generate_stream()` and get the streaming link almost instantly.

```python Python theme={null}
v_url = "https://www.youtube.com/watch?v=NNAgJ5p4CIY"
coll = conn.get_collection()
video = coll.upload(url=v_url)

# store stream of each person
for person in persons_data:
  person_data = persons_data[person]
  stream_link = video.generate_stream(timeline=person_data["timeline"])
  person_data["clip"] = stream_link
```

```javascript Node.js theme={null}
const vUrl = "https://www.youtube.com/watch?v=NNAgJ5p4CIY";
const coll = await conn.getCollection();
const video = await coll.uploadURL({ url: vUrl });

// store stream of each person
for (const person in personsData) {
  const personData = personsData[person];
  const streamLink = await video.generateStream({ timeline: personData.timeline });
  personData.clip = streamLink;
}
```

## Results

We effortlessly uploaded our video to VideoDB and generated clips for each character in just 30 seconds.

Now, it's time to check out our results.

Let's take a look at a clip featuring `Erlich Bachman` (feel free to choose your favorite character by changing the name field below)

```python Python theme={null}
from videodb import play_stream

name = "erlich"
play_stream(persons_data[name]["clip"])
```

```javascript Node.js theme={null}
const name = "erlich";
console.log(personsData[name].clip);
```

[Character Clips Result](https://console.videodb.io/player?url=https://d27qzqw9ehjjni.cloudfront.net/v3/published/manifests/339a6a0b-3a79-4328-94f2-a8f5467d3204.m3u8)

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Create%20Clip%20Faces.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Keyword Search](/examples-and-tutorials/video-rag/keyword-search)**
>
> Create custom compilations by searching for spoken words

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine visual and spoken content for comprehensive queries


# Overview
Source: https://docs.videodb.io/examples-and-tutorials/video-rag/index

Find and understand video content using natural language queries and multimodal search

## Video Search and Understanding

**Turn video archives into searchable, understandable knowledge.** Find exactly what you're looking for in hours of video content using natural language queries. Extract insights from spoken words, visual content, and on-screen information.

### When to Use This

* You have meeting recordings and need to find "what did the client say about the budget?"
* You want to extract all clips where a specific person appears
* You're building video Q\&A for your content library
* You need to search across both what's spoken AND what's shown
* You want to extract slides, diagrams, or on-screen content from recordings

### What You'll Build

> **[Keyword Compilations](/examples-and-tutorials/video-rag/keyword-search)**
>
> Search for specific words/phrases and auto-generate highlight reels

> **[Character Extraction](/examples-and-tutorials/video-rag/character-clips)**
>
> Find all scenes where a specific person appears using face recognition

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine spoken word + visual search for comprehensive queries

> **[NFL Game Analysis](/examples-and-tutorials/video-rag/case-study-nfl)**
>
> Search and analyze sports footage with multimodal understanding

> **[Conference Slide Scraper](/examples-and-tutorials/video-rag/use-case-conference-slides)**
>
> Extract slides and on-screen content from conference recordings

## How It Works

```
Upload Video → Index (Spoken + Visual) → Search → Get Timestamped Results → Play/Export
```

Every search result includes:

* **Timestamps** - Exact start/end times
* **Playable URLs** - Stream the matching segment instantly
* **Relevance scores** - Confidence in the match
* **Text evidence** - What was said/shown

## Related Documentation

> **[Natural Language Query](/pages/understand/search-and-retrieval/natural-language-query)**
>
> How semantic search works under the hood

> **[Collection Search](/pages/understand/search-and-retrieval/collection-search)**
>
> Search across your entire video library

> **[Indexes and Search](/pages/core-concepts/indexes-and-search)**
>
> Core concepts behind video indexing

> **[Timestamps & Clips](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> Working with search results


# Keyword Search
Source: https://docs.videodb.io/examples-and-tutorials/video-rag/keyword-search

Create custom video compilations by searching for keywords and phrases

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Keyword_Search_1.ipynb)

From an hour long video, want to create a fun compilation of every moment Mark Zuckerberg says 'metaverse'?

[Keyword Search Result - Metaverse Compilation](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/3bc3a55b-29aa-46d5-9368-cd965af66518.m3u8)

## Overview

In this tutorial, let's explore the powerful functionality of Keyword Search in VideoDB. This feature enables users to efficiently locate any keyword or phrase within their video assets, streamlining the process of content discovery.

![Fun keyword search example with Mark Zuckerberg metaverse compilation](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/examples/fun-with-keyword-search.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=4f85bc908b7929bfb78b8f043a6aa5a1)

## Setup

### Installing packages

```python Python theme={null}
!pip install videodb
```

```javascript Node.js theme={null}
npm install videodb
```

### API Keys

Before proceeding, ensure access to [VideoDB](https://videodb.io)

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs). ( Free for first 50 uploads, No credit card required)

## Steps

### Step 1: Connect to VideoDB

Begin by establishing a connection to VideoDB using your API key

```python Python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect({ apiKey: process.env.VIDEO_DB_API_KEY });
const coll = await conn.getCollection();
```

### Step 2: Upload Video

Upload the video to your VideoDB collection. You can upload the video asset from your local device or from a YouTube URL to upload the video from its source. This works as the base video for all the Keyword Search queries.

```python Python theme={null}
video = coll.upload(url="https://www.youtube.com/watch?v=Uvufun6xer8")
video.play()
```

```javascript Node.js theme={null}
const video = await coll.uploadURL({
  url: "https://www.youtube.com/watch?v=Uvufun6xer8"
});
console.log(video.playerUrl);
```

You can upload from your local file system too by passing `file_path` in `upload()`.

For this tutorial, we'll run a Keyword Search on the following video:

[Keyword Search Tutorial Video](https://www.youtube.com/embed/Uvufun6xer8)

### Step 3: Index Spoken Words

Index the spoken words in your video to enable accurate keyword search.

```python Python theme={null}
import re
understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")
word_records = []
seen_words = set()
for scene in transcript_analyzer.get_output()["scenes"]:
    for word in scene.get("data", {}).get("words", []):
        text = re.sub(r"^\W+|\W+$", "", word.get("text", "")).casefold()
        start, end = float(word["start"]), float(word["end"])
        key = (text, round(start, 3), round(end, 3))
        if text and key not in seen_words:
            seen_words.add(key)
            word_records.append({"start": start, "end": end, "text": text})
word_records.sort(key=lambda record: (record["start"], record["end"]))
word_index = video.index(
    name="transcript_keyword_words",
    source=word_records,
    use_for=["query"],
    fields={"fts": ["text"], "filter": ["text"]},
)
word_index.wait_until_complete()
```

```javascript Node.js theme={null}
const understanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await understanding.waitUntilComplete();

const transcriptAnalyzer = await understanding.getAnalyzer("transcript");
const transcriptOutput = await transcriptAnalyzer.getOutput();
const wordRecords = transcriptOutput.scenes
  .flatMap((scene) => scene.data.words || [])
  .map((word) => ({
    start: word.start,
    end: word.end,
    text: word.text.replace(/^\W+|\W+$/g, "").toLowerCase(),
  }))
  .filter((word) => word.text);
wordRecords.sort((left, right) => left.start - right.start || left.end - right.end);

const wordIndex = await video.index(wordRecords, {
  name: "transcript_keyword_words",
  useFor: ["query"],
  fields: { fts: ["text"], filter: ["text"] },
});
await wordIndex.waitUntilComplete();
```

### Step 4: Search for any keyword

Use `video.query(index_id=..., filter=...)` to retrieve the timed transcript moments that contain a word or phrase. The word-record index keeps both `fts` and `filter` support on its `text` field.

```python Python theme={null}
keyword = "metaverse".casefold()
results = video.query(
    index_id=word_index.index_id,
    filter=[{"field": "text", "op": "==", "value": keyword}],
    limit=50,
    sort=[("start", "asc")],
)
for shot in results.get_shots()[:10]:
    print(f"{shot.start:.2f}s – {shot.end:.2f}s: {shot.text}")
```

```javascript Node.js theme={null}
const keyword = "metaverse";
const results = await video.query({
  indexId: wordIndex.indexId,
  filter: [{ field: "text", op: "==", value: keyword }],
  limit: 50,
  sort: [["start", "asc"]],
});

for (const shot of results.getShots().slice(0, 10)) {
  console.log(`${shot.start.toFixed(2)}s – ${shot.end.toFixed(2)}s: ${shot.text}`);
}
```

### Step 5: Preview and Share

Preview your video with a compilation of all the clips matching your search query. You can access the stream link alongside the preview to share the Keyword Search result with others.

```python Python theme={null}
stream_url = results.compile()
print("Stream URL:", stream_url)
results.play()
```

```javascript Node.js theme={null}
await results.compile();
console.log(results.playerUrl);
```

[Keyword Search Result - Metaverse Compilation](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/3bc3a55b-29aa-46d5-9368-cd965af66518.m3u8)

### Bonus: Refining Keyword Search results by adding padding

Some keyword search results/compilations may appear slightly choppy, or the cuts may feel abrupt. We can solve this issue by using VideoDB’s padding controls. Here’s how it works:

The resulting shots can be made smoother by including a little more context from before and after the matching timestamps. That's exactly what padding controls enable:

Using the Editor SDK's `Track` and `Clip` pattern, we can create a timeline with padding:

1. Create a timeline and track using `Timeline()` and `Track()`
2. Create a `VideoAsset` with `id` and `start` parameters (where `start` is adjusted by subtracting padding)
3. Wrap each asset in a `Clip` with the appropriate duration (adding padding on both ends)
4. Add clips to the track sequentially using `track.add_clip(start_time, clip)`

```python Python theme={null}
from videodb import play_stream
from videodb.editor import Timeline, Track, Clip, VideoAsset

timeline = Timeline(conn)

# Add padding for smoother cuts
padding = 0.4

# Create main track
track = Track()
seeker = 0

# Compile Video from search results
for shot in results.get_shots():
    start_with_padding = max(0, shot.start - padding)
    duration = (shot.end + padding) - start_with_padding

    asset = VideoAsset(id=shot.video_id, start=start_with_padding)
    clip = Clip(asset=asset, duration=duration)
    track.add_clip(seeker, clip)

    seeker += duration

timeline.add_track(track)

stream_url = timeline.generate_stream()
play_stream(stream_url)
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset } from "videodb";

const timeline = new EditorTimeline(conn);
const padding = 0.4;
const track = new Track();
let seeker = 0;

for (const shot of results.getShots()) {
  const startWithPadding = Math.max(0, shot.start - padding);
  const duration = shot.end + padding - startWithPadding;
  const asset = new EditorVideoAsset({
    id: shot.videoId,
    start: startWithPadding,
  });
  const clip = new Clip({ asset, duration });
  track.addClip(seeker, clip);
  seeker += duration;
}

timeline.addTrack(track);

const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

### Here's the result for the same video, but improved using padding control.

[YouTube video player](https://www.youtube.com/embed/MQVTtET7LM8)

### Conclusion

Keyword Search in VideoDB empowers users to extract valuable insights from their video assets with ease.

## More Examples

Check out these fun experiments with Keyword search 👇

1. So basically it's "basically"

[YouTube video player](https://www.youtube.com/embed/4gjl-WCpbXg)

2. The untold story of "generative" AI

[YouTube video player](https://www.youtube.com/embed/FW-OgmJLEWM)

## Next Steps

> **[Explore Full Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/Keyword_Search_1.ipynb)**
>
> Open the complete implementation in Google Colab with all code examples.

## Related Tutorials

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine spoken word and visual search for comprehensive queries

> **[Character Extraction](/examples-and-tutorials/video-rag/character-clips)**
>
> Extract clips featuring specific characters or people instantly


# Multimodal Search
Source: https://docs.videodb.io/examples-and-tutorials/video-rag/multimodal-search

Combine spoken word and visual search for comprehensive queries

## Search Spoken and Visual Evidence Together

Create one timed transcript artifact and one visual scene artifact, then keep their indexes and playback results distinct before combining matching intervals.

Install VideoDB, then connect to your default collection and upload the video used in the following steps.

```bash theme={null}
pip install videodb
```

```python theme={null}
from videodb import connect, play_stream

conn = connect()
coll = conn.get_collection()
video = coll.upload(url="https://www.youtube.com/watch?v=libKVRa01L8")
```

```python theme={null}
understanding = video.understand(
    segmentation={"type": "time", "seconds": 2},
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"strategy": "uniform", "frame_count": 2},
            "config": {
                "prompt": "Describe the scene in detail",
                "schema": {"description": "string"},
            },
        },
    ],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")
scene_analyzer = understanding.get_analyzer("scene")
transcript_index = video.index(
    name="transcript",
    source=transcript_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["text"]},
)
transcript_index.wait_until_complete()
scene_index = video.index(
    name="scene",
    source=scene_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["description"]},
)
scene_index.wait_until_complete()
```

```python theme={null}
transformation_prompt = """
Split the following multimodal video query into one query for spoken content and one query for visual content.
Return JSON with exactly these fields:
{{"spoken_query": "string", "visual_query": "string"}}

Query: {query}
"""

def divide_query(query):
    response = coll.generate_text(
        prompt=transformation_prompt.format(query=query),
        model_name="pro",
        response_type="json",
    )
    divided_query = response["output"]
    return divided_query["spoken_query"], divided_query["visual_query"]

query = "Show me where the narrator discusses the formation of the solar system and visualize the milky way galaxy"
spoken_query, visual_query = divide_query(query)

spoken_results = video.semantic_search(
    query=spoken_query,
    index_names=[transcript_index.name],
    top_k=10,
)
spoken_results.play()
scene_results = video.semantic_search(
    query=visual_query,
    index_names=[scene_index.name],
    top_k=10,
    score_threshold=0.1,
)
scene_results.play()
```

Combine the timed shots with either a union or an intersection. Use the intersection when both the spoken and visual evidence must be present in the same interval.

```python theme={null}
def process_shots(left, right, operation):
    def merge_intervals(intervals):
        if not intervals:
            return []
        intervals.sort(key=lambda interval: interval[0])
        merged = [intervals[0]]
        for interval in intervals[1:]:
            if interval[0] <= merged[-1][1]:
                merged[-1][1] = max(merged[-1][1], interval[1])
            else:
                merged.append(interval)
        return merged

    def intersection(intervals1, intervals2):
        i = j = 0
        result = []
        while i < len(intervals1) and j < len(intervals2):
            low = max(intervals1[i][0], intervals2[j][0])
            high = min(intervals1[i][1], intervals2[j][1])
            if low < high:
                result.append([low, high])
            if intervals1[i][1] < intervals2[j][1]:
                i += 1
            else:
                j += 1
        return result

    if operation.lower() == "intersection":
        return intersection(merge_intervals(left), merge_intervals(right))
    if operation.lower() == "union":
        return merge_intervals(left + right)
    raise ValueError("Choose 'intersection' or 'union'.")


def combine_results(spoken_results, scene_results, operation):
    spoken_timestamps = [[shot.start, shot.end] for shot in spoken_results.get_shots()]
    scene_timestamps = [[shot.start, shot.end] for shot in scene_results.get_shots()]
    return process_shots(spoken_timestamps, scene_timestamps, operation)


results = combine_results(spoken_results, scene_results, "intersection")
stream_link = video.generate_stream(results)
play_stream(stream_link)
```

Videos are inherently multimodal — they present both visual and audio content simultaneously, creating a unified experience. Our brains naturally use these modalities to store and retrieve information. With advancements in retrieval technology, we now have the opportunity to develop an assistant or agent that can mimic our cognitive processes for storing and retrieving information externally.

VideoDB allows you to index both spoken and visual content, creating a modular architecture optimized for multimodal search queries. This can significantly benefit your users by enabling them to:

* Watch streams or footage instantly.
* Extract information or content for their workflows.

Multimodal search and reasoning enable more human-like behaviors when retrieving information from videos. This approach offers various types of searches and solves a wide range of use cases. Let’s explore a few examples:

### Watch the footage instantly: Multimodal Search in Action:

**Show me the footage of the suspects being caught on camera stealing at the mall and the news anchor discussing their identities.**

This query is a classic example of a multimodal search as it seeks both visual content (the footage of the theft) and spoken content (the news anchor's discussion). The search engine needs to process video data for visual evidence and audio data for the spoken segment, making it a multimodal search.

These kinds of queries are common in many critical scenarios, for example:

* **Law Enforcement**: Helps in quickly retrieving crucial evidence from vast amounts of surveillance and news footage.
* **Media and Journalism**: Facilitates the process of locating specific segments within hours of news broadcasts, aiding in efficient reporting and fact-checking.
* **Public Safety**: Enhances the ability of authorities to disseminate important information to the public by quickly identifying and sharing relevant content.

### Extracting Content from the Screen: Enhanced User Experience

"What was on the screen when 'quantum entanglement' was spoken?"

Another powerful application of multimodal search and information retrieval lies in the ability to extract and share content displayed on screens. This feature is particularly useful for taking notes or sharing information with others, especially in dynamic and multimedia-rich environments. Some examples:

* **Educational Settings:** A student is watching an online lecture and wants to capture the slide that was displayed when the professor mentioned "quantum entanglement."
* **Business Meetings:** During a virtual meeting, a project manager wants to save the presentation slide that was shown when the team discussed "budget allocations."
* **Content Creation:** A content creator is reviewing a webinar and wants to capture the visual content displayed when the speaker talked about "social media strategies."

In this section, you'll find tutorials, notebooks, and blogs designed to help you unlock the potential of multimodal video retrieval for your video library. These resources will empower your Retrieval-Augmented Generation (RAG) pipeline, enhance AI-driven video content creation, and optimize the search for multimodal information.

## Use Cases

### Law Enforcement

Retrieve crucial evidence from surveillance and news footage for investigations and case building.

### Media & Journalism

Locate specific segments within news broadcasts for efficient reporting and fact-checking.

### Public Safety

Disseminate important information by quickly identifying and sharing relevant safety content.

### Education

Capture and share lecture slides and educational content discussed in recorded classes.

### Business Meetings

Save presentation slides and visual content from meetings for future reference and sharing.

### Content Creation

Extract and repurpose visual content and key moments from webinars and presentations.

## Related Tutorials

> **[Keyword Search](/examples-and-tutorials/video-rag/keyword-search)**
>
> Search for specific words and auto-generate clips

> **[Character Extraction](/examples-and-tutorials/video-rag/character-clips)**
>
> Find scenes where specific people appear

## Learn More

> **[Multimodal Search Quickstart](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Step-by-step implementation guide for multimodal video search

> **[Conference Slide Scraper](/examples-and-tutorials/video-rag/use-case-conference-slides)**
>
> Detailed walkthrough of extracting screen content with multimodal search


# Use Case: Conference Slide Extraction
Source: https://docs.videodb.io/examples-and-tutorials/video-rag/use-case-conference-slides

Extract and search slide content from conference videos by combining spoken content with visual understanding for instant information retrieval.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/examples/conference_slide_scraper.ipynb)

When watching a talk or presentation, it's common to take notes or share interesting points with others. Often, the content on the slides captures our attention. At VideoDB, we follow many top engineering processes and regularly take notes from talks and conferences, which we share internally on our Slack channel.

However, when trying to recall specific parts of these talks, only a few keywords might come to mind. To address this, we built an internal tool that stores all these talks in VideoDB, allowing us to find and share the content on the screen in text form using a search query.

Let's explore the problem:

What was on the screen when the speaker discussed the "hard and fast rule" in the following video?

[YouTube video player](https://www.youtube.com/embed/IEe-5VOv0Js)

> [!NOTE]
>
> This notebook is a step towards creating a Slack bot that posts valuable engineering practices from top tech talks daily. Stay Tuned!

## Introduction

In this tutorial, we'll explore an advanced yet accessible technique for retrieving visual information from video content based on what the speaker was discussing. Specifically, we'll focus on finding information on slides in a video recording of a speech.

As video content continues to grow in volume and importance, being able to quickly find specific information within videos becomes crucial. Imagine being able to locate a particular statistic mentioned in an hour-long presentation without watching the entire video. That's the power of multimodal video search!

This approach combines structured transcript queries with visual understanding to create a robust, multimodal search pipeline. Don't worry if these terms sound complex - we'll break everything down step by step!

## Setup

### Installing packages

```
!pip install videodb
```

### API Keys

Before proceeding, ensure access to [VideoDB](https://videodb.io). If not, sign up for API access on the respective platforms.

Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) (Free for first 50 uploads, No credit card required).

### Step 1: Connect to VideoDB

Gear up by establishing a connection to VideoDB.

```python theme={null}
import videodb

# Set your API key
api_key = "your_api_key"

# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```

### Step 2: Upload the Video

Next, let's upload our sample video:

```python theme={null}
# Upload a video by URL
video = coll.upload(url="https://www.youtube.com/watch?v=IEe-5VOv0Js")
```

### Step 3: Understand Spoken Content and Slides

Build the multimodal workflow in two clear stages:

1. Create a queryable transcript from spoken content.
2. Compare slide configurations and prompts before analyzing the full video.

### Understand Spoken Content

```python theme={null}
import re

transcript_understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
transcript_understanding.wait_until_complete()
transcript_analyzer = transcript_understanding.get_analyzer("transcript")
transcript_output = transcript_analyzer.get_output()
sentence_records = []
sentence_words = []
for scene in transcript_output["scenes"]:
    for word in scene["data"]["words"]:
        sentence_words.append(word)
        if re.search(r"""[.!?]+["'”’)\]]*$""", word["text"]):
            sentence_records.append(
                {
                    "start": sentence_words[0]["start"],
                    "end": sentence_words[-1]["end"],
                    "text": " ".join(item["text"] for item in sentence_words),
                }
            )
            sentence_words = []
if sentence_words:
    sentence_records.append(
        {
            "start": sentence_words[0]["start"],
            "end": sentence_words[-1]["end"],
            "text": " ".join(item["text"] for item in sentence_words),
        }
    )
transcript_index = video.index(
    name="conference_transcript_sentences",
    source=sentence_records,
    use_for=["query"],
    fields={"filter": ["text"]},
)
transcript_index.wait_until_complete()

print(f"✅ Transcript index is ready with {len(sentence_records)} timed sentences!")
```

This function transcribes the speech in the video and indexes it, making it searchable.

### Find the Right Configuration for Slide Understanding

Build the slide workflow in two stages:

1. Compare slide configurations and prompts on extracted scenes.
2. Analyze the full video with the selected configuration.

To learn more about video understanding, explore [Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts) and [Analyzer Outputs](/pages/understand/indexing-pipelines/analyzer-outputs).

1. **Compare Slide Segmentation Configurations**

Start with the default scene extraction so there is a visual baseline for comparison.

```python theme={null}
from IPython.display import Image, display
import requests


# Helper function to preview extracted scenes
def display_scenes(scenes, images=True):
    for scene in scenes:
        print(f"{scene.id} : {scene.start}-{scene.end}")
        if images:
            for frame in scene.frames:
                image = Image(requests.get(frame.url, stream=True).content)
                display(image)
        print("----")


scene_collection_default = video.extract_scenes()
display_scenes(scene_collection_default.scenes)
```

Conference videos can contain subtle slide changes that the default configuration misses. Lower the shot threshold and compare the extracted scenes again.

```python theme={null}
from videodb import SceneExtractionType

scene_collection = video.extract_scenes(
    extraction_type=SceneExtractionType.shot_based,
    extraction_config={"threshold": 10},
)
display_scenes(scene_collection.scenes)
```

#### Test the Slide Prompt on Sample Scenes

Test the prompt on a few scenes before analyzing the entire video. This keeps the experiment focused while you check that the analyzer describes slide content and returns `None` when a scene does not show a slide.

```python theme={null}
slide_prompt = (
    "Give the content written on the slides, output None if it isn't the slides."
)

for scene in scene_collection.scenes[20:23]:
    description = scene.describe(slide_prompt)
    print(f"{scene.id} : {scene.start}-{scene.end}")
    print(description)
    print("-----")
```

The lower threshold and sample prompt results provide the configuration for the full slide-analysis run.

### Analyze the Video With the Finalized Configuration

The final slide run uses the selected configuration:

1. Segment the video into shots with threshold 10.
2. Sample one representative frame of each shot.
3. Return a structured slide description for every timed segment.

```python theme={null}
def display_scene_index(scene_index):
    for scene in scene_index:
        print(f"{scene['start']} - {scene['end']}")
        print(scene["description"])
        print("----")


slide_prompt = (
    "Give the content written on the slides, output None if it isn't the slides."
)
slide_segmentation = {"type": "shot", "threshold": 10}
slide_sampling = {"strategy": "uniform", "frame_count": 1}
slide_schema = {"description": "string"}

slide_understanding = video.understand(
    segmentation=slide_segmentation,
    analyzers=[
        {
            "type": "vlm",
            "name": "slides",
            "sampling": slide_sampling,
            "config": {"prompt": slide_prompt, "schema": slide_schema},
        }
    ],
)
slide_understanding.wait_until_complete()

slides_analyzer = slide_understanding.get_analyzer("slides")
slides_output = slides_analyzer.get_output()

scene_index = [
    {
        "start": float(segment["start"]),
        "end": float(segment["end"]),
        "description": segment["data"]["description"],
    }
    for segment in slides_output["scenes"]
]
scene_index.sort(key=lambda scene: (scene["start"], scene["end"]))
display_scene_index(scene_index)
```

### Step 4: Search Pipeline Implementation

The heart of this approach is the search pipeline, which combines a structured transcript query with slide-understanding artifact records.

This pipeline does the following:

1. Runs a structured transcript query with a text filter
2. Extracts time ranges from the returned transcript records
3. Filters slide-understanding artifact records by their overlap with those ranges
4. Returns the matching slide descriptions and their time ranges

```python theme={null}
def simple_filter_scenes(time_ranges, scene_dicts):
    def is_in_range(scene, range_start, range_end):
        scene_start = scene["start"]
        scene_end = scene["end"]
        return scene_start < range_end and range_start < scene_end

    filtered_scenes = []
    for start, end in time_ranges:
        filtered_scenes.extend(
            scene
            for scene in scene_dicts
            if is_in_range(scene, start, end)
        )

    # Remove duplicates while preserving order
    seen = set()
    return [
        scene
        for scene in filtered_scenes
        if not (
            (scene["start"], scene["end"]) in seen
            or seen.add((scene["start"], scene["end"]))
        )
    ]
```

```python theme={null}
def search_pipeline(query, video):
    search_result = video.query(
        index_id=transcript_index.index_id,
        filter=[{"field": "text", "op": "contains", "value": query}],
        limit=100,
        sort=[("start", "asc")],
    )
    time_ranges = [(shot.start, shot.end) for shot in search_result.get_shots()]
    final_result = simple_filter_scenes(time_ranges, scene_index)
    result_text = "\n\n".join(item["description"] for item in final_result
                               if item.get("description", "").lower().strip() != "none")
    result_timeline = [(item.get("start"), item.get("end")) for item in final_result]
    return result_text, result_timeline
```

### Step 5: Viewing the Search Results

Finally, let's use our search pipeline:

```python theme={null}
from videodb import play_stream

query = "hard and fast rule"

result_text, result_timeline = search_pipeline(query, video)

# Truncate result_text before printing if it's too long
max_chars = 500
if len(result_text) > max_chars:
    print(result_text[:max_chars] + "...\n<TRUNCATED>")
else:
    print(result_text)

# Get the video's actual length from the video object
video_length = video.length

# Clip the end times in result_timeline to not exceed video_length
clipped_result_timeline = [
    (start, min(end, video_length)) for start, end in result_timeline
]

stream_link = video.generate_stream(clipped_result_timeline)
play_stream(stream_link)
```

It returns slide-understanding artifact records whose time ranges overlap the matching structured transcript results.

Here's the result for this particular search query "hard and fast rule":

### The content written on the slide is

```
IT'S ALL IN THE DETAILS

- Prefer American English for naming
- Avoid payment-industry jargon
- Timestamp fields should use <verbed_at>
- Amount properties should also provide a currency
- API resources with IDs are top-level
- New API resources should be retrieved and listed one way
- API resource mutations should be reflected in API responses
- Use nested structures for future extensibility
- Prefer enums to booleans for new properties
- Use a type field for polymorphic objects
- Use verbs for properties with side effects
- Use top-level namespaces for product APIs
- Evaluate new features in the Dashboard before building an API
- Use simple, unambiguous language
- Always paginate unbounded lists
- Iterate on designs with beta users with the feature behind a gate
```

[Hard and Fast Rule Result](https://console.videodb.io/player?url=https://dseetlpshk2tb.cloudfront.net/v3/published/manifests/7c35b4af-4390-4269-914e-c32cf54b142b.m3u8)

Here are some other query outputs using the same search pipeline:

### Search for "stripe api review"

### API REVIEW CHECKLIST

```
API Review: [Insert Title Here]
Ziec: Gavel jar, link for API review join creation

Gavel block
To ping PM when Q/A and other stakeholders have

Summary
(Please include a short description of the change you would like to make...)
```

[Stripe API Review Result](https://console.videodb.io/player?url=https://dseetlpshk2tb.cloudfront.net/v3/published/manifests/80cce472-2883-487a-b107-64d94d9524a3.m3u8)

### Search for "Friction Log"

### Internal Terminal Dogfooding Instructions

```
https://go/terminal-dogfooding-instructions

Stripe! Thanks a ton for your help in dogfooding ahead of our Terminal GA launch!

We’re very close to launching Terminal in public beta and then GA.

And we could use your help! There are a ton of different use cases of these integrations to test and polish, and we want to stress test what we've made and ship these paths and accompanying docs and dashboard flows in a developer-friendly as possible in the time we have before the Terminal GA launch.

If you’re arriving at this doc after having signed up to dogfood, continue to Steps below.
If you haven’t signed up yet, please signup here, and we’ll get back to you when you have a slot your test.

Steps
1. If you’re dogfooding remotely and haven’t received instructions on ordering hardware or attending demos, please email terminal-dogfooding@stripe.com or ping in #terminal-dogfooding

2. If you're dogfooding the iOS or Android SDK, you'll need to set up your environment as described in the links...
```

[Friction Log Result](https://console.videodb.io/player?url=https://dseetlpshk2tb.cloudfront.net/v3/published/manifests/1feab696-7991-4589-8659-fc51a38f2dd4.m3u8)

## Conclusion

This document has outlined a sophisticated approach to multimodal video search, combining structured transcript queries with visual understanding. The pipeline finds specific visual content, such as slide information, based on what was discussed.

This technique has broad applications beyond just searching for slides in speeches. It could be adapted for various use cases where visual information needs to be retrieved based on audio content, such as:

* Finding product demonstrations in long-form video content
* Identifying key moments in educational videos
* Searching for specific visual elements in recorded meetings or presentations

As video content continues to grow in importance and volume, tools and techniques like these will become increasingly valuable for efficient information retrieval and analysis.

## Further Resources

To learn more about understanding artifacts and analyzer outputs, explore the following guides:

> **[Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Create reusable timed artifacts from video understanding runs.

> **[Analyzer Outputs](/pages/understand/indexing-pipelines/analyzer-outputs)**
>
> Inspect the timestamped records produced by each analyzer.

## Get Support

If you have any questions or feedback, feel free to reach out to us:

> **[Discord Community](https://discord.gg/py9P639jGz)**
>
> Connect with the VideoDB community for support and discussions

> **[GitHub](https://github.com/video-db)**
>
> Explore our open source projects and contribute

> **[VideoDB](https://videodb.io)**
>
> Visit our main website for more information


# Welcome to VideoDB
Source: https://docs.videodb.io/index

The perception, memory, and action for AI agents

Your agents can read text and static images. But the real world is live, continuous, and always changing. To operate with real context, your agent needs real-time access to video calls, camera feeds, screen recordings, and live internet streams.

VideoDB is the perception layer that lets agents see, hear, remember, and act on continuous media. Most AI development focuses on text and static images, but video remains a significant hurdle because of its density and lack of structure. VideoDB turns raw pixel data into structured context that agents can query, reason about, and act upon in real time.

For agents to move beyond text boxes and interact with the physical or digital world via screens and cameras, they need a way to parse continuous visual and auditory data. VideoDB provides this through a specialized database that indexes video at the scene level - making it possible for an agent to "recall" specific events or "see" real-time occurrences without excessive compute costs.

> **[Quickstart](/pages/getting-started/quickstart)**
>
> Give your agent perception in 5 minutes

> **[Core Concepts](/pages/core-concepts/overview)**
>
> Understand the platform architecture

### How It Works

The platform operates through three stages: **See**, **Understand**, and **Act**.

| Stage | What Happens |
| :- | :- |
| **See** | Capture SDK or live stream integration takes in media from files, desktops, or cameras |
| **Understand** | Build specialized indexes for transcripts, visual scenes, or custom prompts |
| **Act** | Query, search, edit, and export - agents can generate summaries or clips based on findings |

Rather than merely storing video files, the platform indexes frames and audio to support semantic retrieval. This allows an agent to ask for a specific moment in a continuous stream without downloading or processing the entire file.

The architecture sits above transport protocols and below the reasoning engine. This separation means you can use VideoDB with any Large Language Model or Large Video Model. By consolidating transcription, frame extraction, vector indexing, and video playback into a single platform, VideoDB addresses the high total cost of ownership typically associated with video AI.

### Skills: Native Agent Experiences

Since VideoDB handles server-side video processing, indexing, and retrieval, developers can use [skills](/pages/getting-started/agent-skills) to create agent workflows that feel native to their environment. Skills give agents like Claude Code and Codex structured perception primitives - capture, search, edit, stream - without writing infrastructure code.

```bash theme={null}
npx skills add video-db/skills
```

***

## What You Can Build

> **Desktop Agents**
>
> Stream screen, mic, and camera. Get real-time context about what the user is doing and saying.
>
> [Call.md →](/examples-and-tutorials/ai-copilots/call-md)

> **Video Search**
>
> Search across hours of meetings, lectures, or archives. Get timestamped moments with playable evidence.
>
> [Multimodal Search →](/examples-and-tutorials/video-rag/multimodal-search)

> **Real-time Monitoring**
>
> Connect RTSP cameras and drones. Detect events as they happen. Trigger alerts and automations.
>
> [Intrusion Detection →](/examples-and-tutorials/live-intelligence/intrusion-detection)

> **Media Automation**
>
> Compose videos with code. Generate voice, music, and images. Export to any format.
>
> [Faceless Video Creator →](/examples-and-tutorials/content-factory/faceless-video-creator)

> **Agent Skills**
>
> Add real-time perception to coding assistants and autonomous agents. Screen capture, audio indexing, and searchable context.
>
> [Agent Skills →](/pages/getting-started/agent-skills)

> **MCP Server**
>
> Connect Claude or any MCP client to your VideoDB account. Upload, index, search, edit, and stream from a conversation.
>
> [MCP Server →](/pages/mcp/index)

> **[Browse All Examples](/examples-and-tutorials)**
>
> Explore examples across AI Copilots, Video Search, Live Intelligence, Content Factory, and more

***

## Example: Real-time Alerting

```python Python theme={null}
import videodb

conn = videodb.connect()

# See: Get an active stream (from desktop capture or RTSP)
coll = conn.get_collection()
rtstream = coll.get_rtstream("rts-abc123")

# Understand: create a persistent visual description output for the live stream
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {
                "prompt": "Describe visible user activity and any visible deadlines or due dates."
            },
        }
    ],
    store=True,
)
scene_output = understanding.outputs.get("scene")
scene_index = rtstream.index(
    source=scene_output,
    name="desktop_activity",
    use_for=["semantic"],
)

# Act: Create an event and attach an alert
event_id = conn.create_event(
    event_prompt="Detect a visible deadline or due date",
    label="deadline_visible",
)
alert = scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/webhooks/deadline-mentioned"
)

# Alert evidence is event- and server-payload-dependent; inspect received data
# before handling the relevant time range or clip.
```

```javascript Node.js theme={null}
import { connect } from "videodb";

const conn = connect();
const coll = await conn.getCollection();
const rtstream = await coll.getRTStream("rts-abc123");

const understanding = await rtstream.understand({
    segmentation: { type: "time", window: "5s" },
    analyzers: [
        {
            type: "vlm",
            name: "scene",
            sampling: { frame_count: 2 },
            config: {
                prompt: "Describe visible user activity and any visible deadlines or due dates.",
            },
        },
    ],
    store: true,
});

const sceneIndex = await rtstream.index({
    source: { understanding_id: understanding.id, output: "scene" },
    name: "desktop_activity",
    useFor: ["semantic"],
});

const eventId = await conn.createEvent(
    "Detect a visible deadline or due date",
    "deadline_visible",
);
const alert = await sceneIndex.createAlert(
    eventId,
    "https://example.com",
);
```

The Node.js descriptor explicitly identifies the continuous `scene` output.

***

## Install the SDK

```bash Python theme={null}
pip install videodb
```

```bash Node.js theme={null}
npm install videodb
```

> **[Python SDK](/pages/getting-started/python)**
>
> GitHub, PyPI, and setup guide

> **[Node.js SDK](/pages/getting-started/node)**
>
> npm, TypeScript, and setup guide

***

## Philosophy

Why perception is the next frontier for AI agents.

> **[Why AI Agents Are Blind Today](https://www.videodb.io/blogs/why-agents-are-blind)**
>
> The gap between human perception and agent perception

> **[Perception Is the Missing Layer](https://www.videodb.io/blogs/perception-is-the-missing-layer)**
>
> The stack that gives agents eyes and ears

> **[MP4 Is the Wrong Primitive](https://www.videodb.io/blogs/mp4-is-wrong-primitive)**
>
> Why video files don't work for AI

> **[What Episodic Memory Means for Agents](https://www.videodb.io/blogs/episodic-memory-for-agents)**
>
> Remember experiences, not just facts

***


# Generative Media Overview
Source: https://docs.videodb.io/pages/act/generative-media/index

Generate media assets programmatically: images, music, sound effects, voices, and video clips. Translate and dub existing content into new languages.

## Capabilities

| Category | What You Can Create |
| :- | :- |
| **Images** | Text-to-image with aspect ratio control |
| **Audio** | Music, sound effects, text-to-speech |
| **Video** | Short AI-generated clips (5-8 seconds) |
| **Localization** | Dubbing, transcript translation |

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Generate an image
image = coll.generate_image(
    prompt="Futuristic city skyline at sunset",
    aspect_ratio="16:9"
)
print(image.generate_url())

# Generate a voiceover
voice = coll.generate_voice(
    text="Welcome to our product demo",
    voice_name="Sarah"
)

# Dub a video to Spanish
dubbed = coll.dub_video(
    video_id="m-xxx",
    language_code="es"
)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();

// Generate an image
const image = await coll.generateImage(
    "Futuristic city skyline at sunset",
    "16:9"
);
console.log(await image.generateUrl());

// Generate a voiceover
const voice = await coll.generateVoice(
    "Welcome to our product demo",
    "Sarah"
);

// Dub a video to Spanish
const dubbed = await coll.dubVideo("m-xxx", "es");
```

***

## Cheat Sheet

```python Python theme={null}
# Image
coll.generate_image(prompt, aspect_ratio='1:1', callback_url=None)

# Music
coll.generate_music(prompt, duration=5, callback_url=None)

# Sound Effect
coll.generate_sound_effect(prompt, duration=2, callback_url=None)

# Voice
coll.generate_voice(text, voice_name='Default', config={}, callback_url=None)

# Video
coll.generate_video(prompt, duration=5, callback_url=None)

# Dub
coll.dub_video(video_id, language_code, callback_url=None)

# Translate
video.translate_transcript(language, additional_notes='', callback_url=None)
```

```javascript Node.js theme={null}
// Image
await coll.generateImage(prompt, '1:1', null);

// Music
await coll.generateMusic(prompt, 5, null);

// Sound Effect
await coll.generateSoundEffect(prompt, 2, null);

// Voice
await coll.generateVoice(text, 'Default', {}, null);

// Video
await coll.generateVideo(prompt, 5, null);

// Dub
await coll.dubVideo(videoId, languageCode, null);

// Translate
await video.translateTranscript(language, '', null);
```

***

## Async Processing

All generative calls support asynchronous processing:

1. When `callback_url` is not provided: SDK returns an asset object once the process is done (synchronous processing)
2. When `callback_url` is provided: SDK returns immediately and sends a webhook when complete (asynchronous processing)
3. Call `.generate_url()` to get the finished file URL

### Callback Payload

```json theme={null}
{
  "success": true,
  "data": {
    "id": "img-z-019c4cb6-207c-7d23-aab1-e8ef7a5fcde1",
    "collection_id": "c-3b0a66f2-fac5-4816-b29e-e606986c0490",
    "name": "Futuristic City Sunset Skyline",
    "extension": "png",
    "size": "1607220"
  }
}
```

***

## Guide Index

> **[Images, Audio & Video](/pages/act/generative-media)**
>
> Generate images, music, SFX, voices, and video

> **[Translation & Dubbing](/pages/act/generative-media/translation-and-dubbing)**
>
> Localize content into 20+ languages

***

## What You Can Build

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Add narration to silent footage with AI-generated voices

> **[Video Dubbing](/examples-and-tutorials/content-factory/dubbing)**
>
> Localize videos into multiple languages

> **[Voice Cloning](/examples-and-tutorials/content-factory/voice-cloning)**
>
> Replace voices with cloned audio preserving tone and style

> **[Trailer Narration](/examples-and-tutorials/content-factory/trailer-narration)**
>
> Create dramatic trailers with compelling voiceovers


# Translation and Dubbing
Source: https://docs.videodb.io/pages/act/generative-media/translation-and-dubbing

Localize videos with automated dubbing and transcript translation

Dub videos into new languages or translate transcripts for subtitles. Support for 20+ languages with automated voice matching.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
video = coll.get_video("m-xxx")

# Dub video to Spanish
dubbed = coll.dub_video(
    video_id=video.id,
    language_code="es"
)
dubbed.play()

# Translate transcript to French
video.generate_transcript()
french_transcript = video.translate_transcript(language="fr")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();
const video = await coll.getVideo("m-xxx");

// Dub video to Spanish
const dubbed = await coll.dubVideo(video.id, "es");
await dubbed.play();

// Translate transcript to French
await video.generateTranscript();
const frenchTranscript = await video.translateTranscript("fr");
```

***

## Video Dubbing

Replace the audio track with AI-generated voices in another language.

```python Python theme={null}
dubbed = coll.dub_video(
    video_id=video.id,
    language_code="hi",  # Hindi
    callback_url="https://your-backend.com/webhooks/dubbing"
)

# Returns a new video object
dubbed.play()
```

```javascript Node.js theme={null}
const dubbed = await coll.dubVideo(
    video.id,
    "hi",  // Hindi
    "https://your-backend.com/webhooks/dubbing"
);

// Returns a new video object
await dubbed.play();
```

### Parameters

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `video_id` | str | required | Source video ID |
| `language_code` | str | required | Target language (ISO 639-1) |
| `callback_url` | str | None | Webhook URL when complete |

### Supported Languages

| Language | Code | Language | Code |
| :- | :- | :- | :- |
| Spanish | `es` | Hindi | `hi` |
| French | `fr` | Japanese | `ja` |
| German | `de` | Korean | `ko` |
| Italian | `it` | Chinese | `zh` |
| Portuguese | `pt` | Arabic | `ar` |
| Dutch | `nl` | Russian | `ru` |
| Polish | `pl` | Turkish | `tr` |

***

## Transcript Translation

Translate the spoken content for subtitles or text-based use cases.

### Step 1: Generate a Transcript

```python Python theme={null}
# Required before translation
video.generate_transcript()
```

```javascript Node.js theme={null}
// Required before translation
await video.generateTranscript();
```

### Step 2: Translate

```python Python theme={null}
# Translate to French
french_text = video.translate_transcript(
    language="fr",
    additional_notes="Use formal tone"
)

print(french_text)
```

```javascript Node.js theme={null}
// Translate to French
const frenchText = await video.translateTranscript(
    "fr",
    "Use formal tone"
);

console.log(frenchText);
```

### Parameters

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `language` | str | required | Target language (ISO 639-1) |
| `additional_notes` | str | `""` | Style guidance for translation |
| `callback_url` | str | None | Webhook URL |

### Style Guidance Examples

```python theme={null}
# Formal business content
additional_notes = "Use formal business language appropriate for corporate presentations"

# Casual content
additional_notes = "Use casual, conversational tone"

# Technical content
additional_notes = "Preserve technical terms in English where no equivalent exists"

# Marketing content
additional_notes = "Adapt cultural references for the target audience"
```

***

## Use Cases

### Global Product Launch

```python theme={null}
# Upload marketing video
video = coll.upload(url="https://example.com/product-launch.mp4")

# Create versions for key markets
languages = ["es", "fr", "de", "ja", "zh"]

for lang in languages:
    dubbed = coll.dub_video(
        video_id=video.id,
        language_code=lang,
        callback_url=f"https://backend.com/webhooks/dubbing/{lang}"
    )
```

### Educational Content

```python theme={null}
# Upload lecture video
lecture = coll.upload(url="https://example.com/lecture.mp4")
lecture.generate_transcript()

# Generate subtitles in multiple languages
for lang in ["es", "fr", "pt", "zh"]:
    translated = lecture.translate_transcript(
        language=lang,
        additional_notes="Educational content, maintain academic tone"
    )
    # Use with CaptionAsset for subtitles
```

### Customer Support

```python theme={null}
# Dub support video for regional markets
support_video = coll.get_video("m-support-tutorial")

# Create localized versions
regional_versions = {
    "LATAM": "es",
    "Brazil": "pt",
    "France": "fr",
    "Germany": "de"
}

for region, lang in regional_versions.items():
    coll.dub_video(
        video_id=support_video.id,
        language_code=lang,
        callback_url=f"https://backend.com/webhooks/{region}"
    )
```

***

## Best Practices

### Pre-Dubbing Checklist

| Check | Why |
| :- | :- |
| Clear audio | Reduces transcription errors |
| Single speaker | Better voice matching |
| Minimal background music | Cleaner dubbing result |
| Proper pacing | Allows for language expansion |

### Language Expansion

Different languages have different word counts for the same content:

| Original (English) | Expansion |
| :- | :- |
| German | +15-20% |
| French | +15-20% |
| Spanish | +10-15% |
| Japanese | -10-15% |
| Chinese | -20-30% |

Plan for this when timing is critical.

### Quality Review

After dubbing:

1. Check lip sync on close-ups
2. Verify tone matches content
3. Review technical term pronunciation
4. Test with native speakers

***

## Async Processing

Dubbing is a long-running operation. Use callbacks:

```python Python theme={null}
# Start dubbing (returns immediately when callback_url is provided)
dubbed = coll.dub_video(
    video_id=video.id,
    language_code="ja",
    callback_url="https://your-backend.com/webhooks/dubbing"
)

print(f"Processing video ID: {dubbed.id}")

# When callback_url is not provided, SDK returns video object once process is done
dubbed = coll.dub_video(
    video_id=video.id,
    language_code="ja"
)
# This will wait and return the completed video object
```

```javascript Node.js theme={null}
// Start dubbing (returns immediately when callback_url is provided)
const dubbed = await coll.dubVideo(
    video.id,
    "ja",
    "https://your-backend.com/webhooks/dubbing"
);

console.log(`Processing video ID: ${dubbed.id}`);

// When callback_url is not provided, SDK returns video object once process is done
const dubbedSync = await coll.dubVideo(
    video.id,
    "ja"
);
// This will wait and return the completed video object
```

### Webhook Response

When the dubbing process completes, your webhook receives:

```json theme={null}
{
  "success": true,
  "data": {
    "id": "m-dubbed-xxx",
    "collection_id": "c-xxx",
    "name": "dubbed_video",
    "extension": "mp4",
    "size": "12345678"
  }
}
```

***

## What You Can Build

> **[Video Dubbing](/examples-and-tutorials/content-factory/dubbing)**
>
> Localize videos into multiple languages automatically

> **[AI Voiceovers](/examples-and-tutorials/content-factory/voiceovers)**
>
> Generate narration in different languages

***

## Next Steps

> **[Safety & Approvals](/pages/act/generative-media)**
>
> Review workflows before publishing

> **[Subtitles Guide](/pages/act/subtitles/subtitles-guide)**
>
> Add styled subtitles to videos


# Alerts and Callbacks
Source: https://docs.videodb.io/pages/act/live-action/alerts-and-callbacks

Wire events to delivery channels for real-time notifications

Alerts connect events to delivery channels. Use a callback URL for server-to-server notifications, and optionally attach a WebSocket for real-time dashboards.

## Quick Example

```python Python theme={null}
# Create alert with webhook delivery
alert_id = scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts"
)

# With WebSocket delivery; a callback URL is still required
ws = conn.connect_websocket()
await ws.connect()

CALLBACK_URL = "https://example.com"  # Replace with a reachable endpoint you control.
alert_id = scene_index.create_alert(
    event_id=event_id,
    callback_url=CALLBACK_URL,
    ws_connection_id=ws.connection_id
)
```

```javascript Node.js theme={null}
// Create alert with webhook delivery
const webhookAlertId = await sceneIndex.createAlert(
    eventId,
    "https://your-backend.com/alerts"
);

// With WebSocket delivery; a callback URL is still required
const ws = await conn.connectWebsocket();
await ws.connect();

const callbackUrl = "https://example.com"; // Replace with a reachable endpoint you control.
const websocketAlertId = await sceneIndex.createAlert(
    eventId,
    callbackUrl,
    ws.connectionId
);
```

***

## Delivery Methods

| Method | Latency | Use Case |
| :- | :- | :- |
| WebSocket | Real-time | Dashboards, live UI updates |
| Webhook | Under 1s | Server-to-server, automation |

You can use both simultaneously for redundancy.

Alert creation requires a callback URL even when a WebSocket connection is attached. Replace `https://example.com` in the examples with a reachable endpoint you control.

***

## Webhook Delivery

### Create Webhook Alert

```python Python theme={null}
alert_id = scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/webhooks/alerts"
)
```

```javascript Node.js theme={null}
const alertId = await sceneIndex.createAlert(
    eventId,
    "https://your-backend.com/webhooks/alerts"
);
```

### Illustrative Callback Payload

Callback payload fields can vary by pipeline and backend. Inspect and validate the payload your endpoint receives before relying on individual fields:

```json theme={null}
{
  "event_id": "event-3fd4174feceb6162",
  "label": "traffic_violation",
  "confidence": 0.95,
  "explanation": "A red sedan ran through the intersection while the light was red",
  "timestamp": "2024-01-15T10:30:45Z",
  "start_time": 1234.5,
  "end_time": 1238.0,
  "stream_url": "https://stream.videodb.io/v3/...",
  "player_url": "https://console.videodb.io/player?url=..."
}
```

### Possible Payload Fields

| Field | Type | Description |
| :- | :- | :- |
| `event_id` | string | ID of the triggered event |
| `label` | string | Human-readable event label |
| `confidence` | float | Detection confidence (0-1) |
| `explanation` | string | AI-generated description of what was detected |
| `timestamp` | string | ISO 8601 timestamp |
| `start_time` | float | Video timestamp where event starts (seconds) |
| `end_time` | float | Video timestamp where event ends (seconds) |
| `stream_url` | string | HLS stream URL for the clip |
| `player_url` | string | Web player URL |

***

## WebSocket Delivery

### Connect and Listen

```python Python theme={null}
ws = conn.connect_websocket()
await ws.connect()

# Pass the connection ID; a callback URL is still required
CALLBACK_URL = "https://example.com"  # Replace with a reachable endpoint you control.
alert_id = scene_index.create_alert(
    event_id=event_id,
    callback_url=CALLBACK_URL,
    ws_connection_id=ws.connection_id
)

# WebSocket messages are arbitrary decoded payloads; inspect before accessing fields.
async for event in ws.receive():
    print(event)
```

```javascript Node.js theme={null}
const ws = await conn.connectWebsocket();
await ws.connect();

// Pass the connection ID; a callback URL is still required
const callbackUrl = "https://example.com"; // Replace with a reachable endpoint you control.
const alertId = await sceneIndex.createAlert(
    eventId,
    callbackUrl,
    ws.connectionId
);

// WebSocket messages are arbitrary decoded payloads; inspect before accessing fields.
for await (const event of ws.receive()) {
    console.log(event);
}
```

### WebSocket Channels

The current V2 alert channel is produced by the V2 scene index:

| Channel | Source | Content |
| :- | :- | :- |
| `alert` | `scene_index.create_alert()` | Alert notifications |

The following channel names are available through RTStream helper APIs:

| Channel | Source | Content |
| :- | :- | :- |
| `transcript` | `start_transcript()` | Speech-to-text |
| `scene_index` | `index_visuals()` | Visual analysis |
| `audio_index` | `index_audio()` | Audio analysis |

***

## Managing Alerts

### List Alerts

```python Python theme={null}
alerts = scene_index.list_alerts()
for alert in alerts:
    print(f"{alert['alert_id']}: {alert['event_id']} - {alert['status']}")
```

```javascript Node.js theme={null}
const alerts = await sceneIndex.listAlerts();
for (const alert of alerts) {
    console.log(`${alert.alertId}: ${alert.eventId} - ${alert.status}`);
}
```

### Enable/Disable Alerts

```python Python theme={null}
# Temporarily disable
scene_index.disable_alert(alert_id)

# Re-enable
scene_index.enable_alert(alert_id)
```

```javascript Node.js theme={null}
// Temporarily disable
await sceneIndex.disableAlert(alertId);

// Re-enable
await sceneIndex.enableAlert(alertId);
```

## Dual Delivery Pattern

Use both channels for critical alerts:

```python Python theme={null}
# WebSocket for real-time UI
ws = conn.connect_websocket()
await ws.connect()

# Create alert with both delivery methods
alert_id = scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts",  # webhook
    ws_connection_id=ws.connection_id  # websocket
)
```

```javascript Node.js theme={null}
// WebSocket for real-time UI
const ws = await conn.connectWebsocket();
await ws.connect();

// Create alert with both delivery methods
const alertId = await sceneIndex.createAlert(
    eventId,
    "https://your-backend.com/alerts",  // webhook
    ws.connectionId  // websocket
);
```

**Benefits:**

* WebSocket delivers instantly to connected clients
* Webhook provides reliable server-side processing
* If WebSocket disconnects, webhook still works

***

## Latency Profile

| Event Type | Typical Latency |
| :- | :- |
| Alert trigger | Under 1s |
| Transcript event | 1-2s |
| Visual index event | 2-5s |
| Audio index event | 2-5s |

***

## Delivery Guarantees

| Method | Guarantee | Notes |
| :- | :- | :- |
| WebSocket | At-most-once | May miss events if disconnected |
| Webhook | At-least-once | May receive duplicates; use idempotency |

***

## Next Steps

> **[Webhooks and Reliability](/pages/act/live-action/webhooks-and-reliability)**
>
> Handle webhooks at scale with idempotency

> **[Event Detection Patterns](/pages/act/live-action/event-detection-patterns)**
>
> Create effective detection rules


# Event Detection Patterns
Source: https://docs.videodb.io/pages/act/live-action/event-detection-patterns

Events are reusable detection rules that define what to look for in video. Create once, attach to any index, and receive alerts when conditions match.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()

# Create a reusable event
event_id = conn.create_event(
    event_prompt="Detect when a person enters a restricted area",
    label="intrusion_detected"
)

# Attach to a scene index
scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts"
)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();

// Create a reusable event
const eventId = await conn.createEvent(
    "Detect when a person enters a restricted area",
    "intrusion_detected"
);

// Attach to a scene index
await sceneIndex.createAlert(
    eventId,
    "https://your-backend.com/alerts"
);
```

***

## Events vs Alerts

| Concept | What It Is | Scope |
| :- | :- | :- |
| **Event** | Detection rule (prompt + label) | Account-level, reusable |
| **Alert** | Wiring between event and index | Index-specific |

Think of events as templates. Create them once, then wire them to multiple indexes via alerts.

***

## Creating Events

### Basic Event

```python Python theme={null}
event_id = conn.create_event(
    event_prompt="Detect when someone falls down",
    label="fall_detected"
)
```

```javascript Node.js theme={null}
const eventId = await conn.createEvent(
    "Detect when someone falls down",
    "fall_detected"
);
```

### Event Prompt Best Practices

The `event_prompt` is what the AI uses to evaluate each indexed scene. Be specific:

```python theme={null}
# Too vague - will trigger on many scenes
event_prompt = "Detect anything unusual"

# Better - specific condition
event_prompt = "Detect when a vehicle runs a red light"

# Best - specific with context
event_prompt = "Detect when a vehicle enters the intersection while the traffic light is red"
```

***

## Detection Patterns by Use Case

### Security & Safety

```python Python theme={null}
# Intrusion detection
conn.create_event(
    event_prompt="Detect when a person enters the warehouse after hours",
    label="after_hours_entry"
)

# Fall detection
conn.create_event(
    event_prompt="Detect when a person falls or collapses",
    label="fall_detected"
)

# Unauthorized access
conn.create_event(
    event_prompt="Detect when someone accesses the server room without a badge",
    label="unauthorized_access"
)
```

```javascript Node.js theme={null}
// Intrusion detection
await conn.createEvent(
    "Detect when a person enters the warehouse after hours",
    "after_hours_entry"
);

// Fall detection
await conn.createEvent(
    "Detect when a person falls or collapses",
    "fall_detected"
);

// Unauthorized access
await conn.createEvent(
    "Detect when someone accesses the server room without a badge",
    "unauthorized_access"
);
```

### Retail & Operations

```python Python theme={null}
# Queue detection
conn.create_event(
    event_prompt="Detect when more than 5 people are waiting in line",
    label="queue_long"
)

# Spill detection
conn.create_event(
    event_prompt="Detect liquid spills on the floor",
    label="spill_detected"
)

# Shelf monitoring
conn.create_event(
    event_prompt="Detect when a shelf appears empty or low on products",
    label="shelf_empty"
)
```

```javascript Node.js theme={null}
// Queue detection
await conn.createEvent(
    "Detect when more than 5 people are waiting in line",
    "queue_long"
);

// Spill detection
await conn.createEvent(
    "Detect liquid spills on the floor",
    "spill_detected"
);

// Shelf monitoring
await conn.createEvent(
    "Detect when a shelf appears empty or low on products",
    "shelf_empty"
);
```

### Traffic & Transportation

```python Python theme={null}
# Traffic violation
conn.create_event(
    event_prompt="Detect when a vehicle runs a red light or stop sign",
    label="traffic_violation"
)

# Wrong-way driving
conn.create_event(
    event_prompt="Detect a vehicle driving in the wrong direction",
    label="wrong_way"
)

# Congestion
conn.create_event(
    event_prompt="Detect when traffic has stopped or is moving very slowly",
    label="congestion_detected"
)
```

```javascript Node.js theme={null}
// Traffic violation
await conn.createEvent(
    "Detect when a vehicle runs a red light or stop sign",
    "traffic_violation"
);

// Wrong-way driving
await conn.createEvent(
    "Detect a vehicle driving in the wrong direction",
    "wrong_way"
);

// Congestion
await conn.createEvent(
    "Detect when traffic has stopped or is moving very slowly",
    "congestion_detected"
);
```

***

## Managing Events

### List Events

```python Python theme={null}
events = conn.list_events()
for event in events:
    print(f"{event['event_id']}: {event['label']}")
```

```javascript Node.js theme={null}
const events = await conn.listEvents();
for (const event of events) {
    console.log(`${event.eventId}: ${event.label}`);
}
```

## Prompt Engineering Tips

### Be Specific About Conditions

```python theme={null}
# Weak: ambiguous threshold
"Detect crowding"

# Strong: clear threshold
"Detect when more than 10 people are visible in the frame"
```

### Include Context

```python theme={null}
# Weak: missing context
"Detect a person running"

# Strong: includes context
"Detect a person running in the parking lot (not jogging normally)"
```

### Describe What "Detected" Means

```python theme={null}
# Weak: unclear criteria
"Detect suspicious activity"

# Strong: observable criteria
"Detect when someone looks into car windows repeatedly or tries door handles"
```

***

## Event-Index Pairing

Match your event to the right index configuration:

| Detection Type | Recommended Index Config |
| :- | :- |
| Static objects | 1 frame per scene |
| Motion/activity | 3-5 frames per scene |
| Quick events | Short intervals (2-5s) |
| Sustained conditions | Longer intervals (10-30s) |

***

## What You Can Build

> **[Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Real-time alerts when unauthorized access is detected

> **[Traffic Violations](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Detect red light and stop sign violations automatically

> **[Beep Profanity](/examples-and-tutorials/safety-compliance/beep-profanity)**
>
> Audio event detection to censor inappropriate language

> **[Copyright Detection](/examples-and-tutorials/safety-compliance/copyright-detection)**
>
> Detect copyrighted content in video streams

***

## Next Steps

> **[Alerts and Callbacks](/pages/act/live-action/alerts-and-callbacks)**
>
> Wire events to delivery channels

> **[Webhooks and Reliability](/pages/act/live-action/webhooks-and-reliability)**
>
> Handle alerts at scale


# Webhooks and Reliability
Source: https://docs.videodb.io/pages/act/live-action/webhooks-and-reliability

Handle webhooks at scale with idempotency, retries, and queues

Production systems need reliable webhook handling. This guide covers idempotency, retry logic, and queue patterns.

## Quick Example

```python Python theme={null}
from flask import Flask, request, jsonify
import hashlib

app = Flask(__name__)
processed_events = set()  # Use Redis in production

@app.route("/webhooks/alerts", methods=["POST"])
def handle_alert():
    payload = request.json

    # Generate idempotency key
    idempotency_key = hashlib.sha256(
        f"{payload['event_id']}:{payload['timestamp']}".encode()
    ).hexdigest()

    # Skip if already processed
    if idempotency_key in processed_events:
        return jsonify({"status": "duplicate"}), 200

    # Process the alert
    process_alert(payload)

    # Mark as processed
    processed_events.add(idempotency_key)

    return jsonify({"status": "ok"}), 200
```

```javascript Node.js theme={null}
import express from 'express';
import crypto from 'crypto';

const app = express();
app.use(express.json());

const processedEvents = new Set();  // Use Redis in production

app.post("/webhooks/alerts", (req, res) => {
    const payload = req.body;

    // Generate idempotency key
    const idempotencyKey = crypto
        .createHash('sha256')
        .update(`${payload.event_id}:${payload.timestamp}`)
        .digest('hex');

    // Skip if already processed
    if (processedEvents.has(idempotencyKey)) {
        return res.json({ status: "duplicate" });
    }

    // Process the alert
    processAlert(payload);

    // Mark as processed
    processedEvents.add(idempotencyKey);

    res.json({ status: "ok" });
});
```

***

## Idempotency

Webhooks may be delivered multiple times. Always implement idempotency.

### Generate Idempotency Keys

Use a combination of unique fields:

```python theme={null}
# Option 1: Event ID + timestamp
key = f"{payload['event_id']}:{payload['timestamp']}"

# Option 2: Hash of entire payload
key = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
```

### Store Processed Keys

```python Python theme={null}
import redis

r = redis.Redis()

def is_duplicate(key: str) -> bool:
    """Check if we've already processed this event"""
    # Set with 24h expiry
    return not r.set(f"webhook:{key}", "1", nx=True, ex=86400)

@app.route("/webhooks/alerts", methods=["POST"])
def handle_alert():
    payload = request.json
    key = generate_key(payload)

    if is_duplicate(key):
        return jsonify({"status": "duplicate"}), 200

    process_alert(payload)
    return jsonify({"status": "ok"}), 200
```

```javascript Node.js theme={null}
import Redis from 'ioredis';

const redis = new Redis();

async function isDuplicate(key) {
    // SET with NX (only if not exists) and 24h expiry
    const result = await redis.set(`webhook:${key}`, '1', 'NX', 'EX', 86400);
    return result === null;
}

app.post("/webhooks/alerts", async (req, res) => {
    const payload = req.body;
    const key = generateKey(payload);

    if (await isDuplicate(key)) {
        return res.json({ status: "duplicate" });
    }

    await processAlert(payload);
    res.json({ status: "ok" });
});
```

***

## Respond Quickly

Return 200 immediately, then process asynchronously:

```python Python theme={null}
from celery import Celery
from flask import Flask, request, jsonify

app = Flask(__name__)
celery = Celery('tasks', broker='redis://localhost:6379/0')

@celery.task
def process_alert_async(payload):
    """Heavy processing happens here"""
    # Notify team
    send_slack_notification(payload)
    # Store in database
    save_to_db(payload)
    # Trigger downstream actions
    trigger_automation(payload)

@app.route("/webhooks/alerts", methods=["POST"])
def handle_alert():
    payload = request.json

    # Acknowledge immediately
    process_alert_async.delay(payload)

    return jsonify({"status": "queued"}), 200
```

```javascript Node.js theme={null}
import Bull from 'bull';

const alertQueue = new Bull('alerts', 'redis://localhost:6379');

// Worker processes jobs asynchronously
alertQueue.process(async (job) => {
    const payload = job.data;
    // Heavy processing
    await sendSlackNotification(payload);
    await saveToDb(payload);
    await triggerAutomation(payload);
});

app.post("/webhooks/alerts", async (req, res) => {
    const payload = req.body;

    // Acknowledge immediately, process async
    await alertQueue.add(payload);

    res.json({ status: "queued" });
});
```

***

## Queue Patterns

### Basic Queue Architecture

```
VideoDB → Webhook Endpoint → Queue → Worker → Actions
              ↓
         Return 200 fast
```

### Priority Queues

Handle critical alerts first:

```python Python theme={null}
from celery import Celery

celery = Celery('tasks', broker='redis://localhost:6379/0')

@app.route("/webhooks/alerts", methods=["POST"])
def handle_alert():
    payload = request.json
    label = payload.get("label", "")

    # High priority for safety alerts
    if label in ["fall_detected", "intrusion_detected", "fire_detected"]:
        process_alert_async.apply_async(args=[payload], priority=0)
    else:
        process_alert_async.apply_async(args=[payload], priority=5)

    return jsonify({"status": "queued"}), 200
```

```javascript Node.js theme={null}
import Bull from 'bull';

const highPriorityQueue = new Bull('high-priority-alerts');
const normalQueue = new Bull('normal-alerts');

app.post("/webhooks/alerts", async (req, res) => {
    const payload = req.body;
    const label = payload.label || "";

    // High priority for safety alerts
    const safetyLabels = ["fall_detected", "intrusion_detected", "fire_detected"];

    if (safetyLabels.includes(label)) {
        await highPriorityQueue.add(payload);
    } else {
        await normalQueue.add(payload);
    }

    res.json({ status: "queued" });
});
```

***

## Error Handling

### Retry Failed Processing

```python Python theme={null}
from celery import Celery
from celery.exceptions import MaxRetriesExceededError

celery = Celery('tasks', broker='redis://localhost:6379/0')

@celery.task(bind=True, max_retries=3, default_retry_delay=60)
def process_alert_async(self, payload):
    try:
        # Process the alert
        send_notification(payload)
        save_to_db(payload)
    except Exception as e:
        # Retry with exponential backoff
        raise self.retry(exc=e, countdown=60 * (2 ** self.request.retries))
```

```javascript Node.js theme={null}
const alertQueue = new Bull('alerts', {
    defaultJobOptions: {
        attempts: 3,
        backoff: {
            type: 'exponential',
            delay: 60000  // 1 minute base
        }
    }
});

alertQueue.process(async (job) => {
    // Automatic retries on failure
    await sendNotification(job.data);
    await saveToDb(job.data);
});

// Handle failed jobs
alertQueue.on('failed', (job, err) => {
    console.error(`Job ${job.id} failed after ${job.attemptsMade} attempts`);
    // Send to dead letter queue or alert ops team
});
```

### Dead Letter Queue

Handle permanently failed jobs:

```python Python theme={null}
@celery.task(bind=True, max_retries=3)
def process_alert_async(self, payload):
    try:
        process(payload)
    except MaxRetriesExceededError:
        # Move to dead letter queue
        save_to_dead_letter(payload)
        alert_ops_team(payload)
    except Exception as e:
        raise self.retry(exc=e)
```

```javascript Node.js theme={null}
const deadLetterQueue = new Bull('dead-letter');

alertQueue.on('failed', async (job, err) => {
    if (job.attemptsMade >= job.opts.attempts) {
        // Move to dead letter queue
        await deadLetterQueue.add({
            originalPayload: job.data,
            error: err.message,
            failedAt: new Date()
        });
    }
});
```

***

## Monitoring

### Track Webhook Health

```python theme={null}
from prometheus_client import Counter, Histogram

webhook_received = Counter(
    'webhook_received_total',
    'Total webhooks received',
    ['event_label']
)

webhook_latency = Histogram(
    'webhook_processing_seconds',
    'Time to process webhook'
)

@app.route("/webhooks/alerts", methods=["POST"])
def handle_alert():
    payload = request.json

    webhook_received.labels(event_label=payload.get("label")).inc()

    with webhook_latency.time():
        process_alert(payload)

    return jsonify({"status": "ok"}), 200
```

### Alert on Issues

```python theme={null}
# Alert if processing takes too long
if processing_time > 5.0:
    send_ops_alert("Webhook processing slow")

# Alert if queue is backing up
queue_size = celery.control.inspect().active()
if queue_size > 1000:
    send_ops_alert("Alert queue backing up")
```

***

## Checklist

Before going to production:

* Idempotency keys stored in Redis/database
* Webhook returns 200 within 500ms
* Async processing with queue (Celery, Bull, etc.)
* Retry logic with exponential backoff
* Dead letter queue for failed jobs
* Monitoring and alerting
* Key expiry to prevent memory growth

***

## Next Steps

> **[Event Detection Patterns](/pages/act/live-action/event-detection-patterns)**
>
> Create effective detection rules

> **[Alerts and Callbacks](/pages/act/live-action/alerts-and-callbacks)**
>
> Wire events to delivery channels


# Publishing Patterns
Source: https://docs.videodb.io/pages/act/output-and-delivery/publishing-patterns

Embed players, share content, and deliver video to end users

Deliver video content to end users through embedded players, shareable links, and social media integrations.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
video = coll.get_video("m-xxx")

# Get streaming URL
stream_url = video.generate_stream()

# Get thumbnail
thumbnail_url = video.generate_thumbnail()

# Get download link
download_url = video.download(name="sample video")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();
const video = await coll.getVideo("m-xxx");

// Get streaming URL
const streamUrl = await video.generateStream();

// Get thumbnail URL
const thumbnailUrl = await video.generateThumbnail();

// Get download link
const downloadUrl = await video.download("sample video");
```

***

## Embed Player

### Basic HTML Embed

```html theme={null}
<video
  id="player"
  controls
  width="100%"
  poster="{thumbnail_url}"
>
  <source
    src="{stream_url}"
    type="application/x-mpegURL"
  >
</video>

<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
<script>
  const video = document.getElementById('player');
  const src = video.querySelector('source').src;

  if (Hls.isSupported()) {
    const hls = new Hls();
    hls.loadSource(src);
    hls.attachMedia(video);
  }
</script>
```

### React Component

```jsx theme={null}
import Hls from 'hls.js';
import { useRef, useEffect } from 'react';

function VideoPlayer({ streamUrl, thumbnailUrl, title }) {
  const videoRef = useRef(null);

  useEffect(() => {
    const video = videoRef.current;
    if (!video) return;

    if (Hls.isSupported()) {
      const hls = new Hls();
      hls.loadSource(streamUrl);
      hls.attachMedia(video);

      return () => hls.destroy();
    } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
      // Safari native HLS
      video.src = streamUrl;
    }
  }, [streamUrl]);

  return (
    <video
      ref={videoRef}
      controls
      poster={thumbnailUrl}
      title={title}
      style={{ width: '100%' }}
    />
  );
}
```

### Vue Component

```vue theme={null}
<template>
  <video
    ref="videoPlayer"
    controls
    :poster="thumbnailUrl"
    style="width: 100%"
  />
</template>

<script setup>
import Hls from 'hls.js';
import { ref, onMounted, onUnmounted } from 'vue';

const props = defineProps(['streamUrl', 'thumbnailUrl']);
const videoPlayer = ref(null);
let hls = null;

onMounted(() => {
  if (Hls.isSupported()) {
    hls = new Hls();
    hls.loadSource(props.streamUrl);
    hls.attachMedia(videoPlayer.value);
  }
});

onUnmounted(() => {
  if (hls) hls.destroy();
});
</script>
```

***

## Embed Code Generation

### `get_embed_code()`

Generate an HTML iframe embed string directly from SDK objects. Available on `Video`, `Shot`, `SearchResult`, `Timeline`, `RTStream`, `RTStreamShot`, `RTStreamExportResult`, and `Editor` `Timeline`.

```python theme={null}
embed_html = obj.get_embed_code(
    width="100%",             # iframe width
    height=405,               # iframe height in pixels
    title="VideoDB Player",   # iframe title attribute
    allow_fullscreen=True,    # allow fullscreen
    auto_generate=True        # auto-call generate_stream() if player_url missing
)
```

Returns an HTML `<iframe>` string. Raises `ValueError` if `player_url` is not available and cannot be auto-generated.

### Video Embed

```python theme={null}
video = coll.get_video("m-xxx")

# Auto-generates stream URL if needed
embed_html = video.get_embed_code()
print(embed_html)
# <iframe src="https://console.videodb.io/player?url=..." width="100%" height="405" ...></iframe>

# Custom dimensions
embed_html = video.get_embed_code(width="640px", height=360, title="My Video")
```

### Search Result Embed

```python theme={null}
results = video.search("product demo")

# Embed the compiled search results
embed_html = results.get_embed_code(height=480)
```

### RTStream Note

`RTStream` does **not** support `auto_generate`. You must call `generate_stream(start, end)` explicitly before calling `get_embed_code()`:

```python theme={null}
rt_stream.generate_stream(start=0, end=120)
embed_html = rt_stream.get_embed_code(auto_generate=False)
```

### `build_iframe_embed_code()` Utility

A standalone helper when you already have a player URL:

```python theme={null}
from videodb import build_iframe_embed_code

embed_html = build_iframe_embed_code(player_url, width="100%", height=405)
```

***

## Thumbnail Generation

### Get Video Thumbnail

```python Python theme={null}
# Default thumbnail (first frame)
thumbnail_url = video.generate_thumbnail()

# Thumbnail at specific time
thumbnail = video.generate_thumbnail(time=30.5)
thumbnail_url = thumbnail.generate_url()
```

```javascript Node.js theme={null}
// Default thumbnail (first frame)
const defaultThumbnailUrl = await video.generateThumbnail();

// Thumbnail at specific time
const timeThumbnail = await video.generateThumbnail(30.5);
const timeThumbnailUrl = typeof timeThumbnail === "string"
  ? timeThumbnail
  : await timeThumbnail.generateUrl();
```

### Multiple Thumbnails for Preview

```python Python theme={null}
def generate_preview_thumbnails(video, count=5):
    """Generate evenly spaced thumbnails across video"""
    duration = video.length
    interval = duration / (count + 1)

    thumbnails = []
    for i in range(1, count + 1):
        time = interval * i
        thumbnail = video.generate_thumbnail(time=time)
        url = thumbnail.generate_url()
        thumbnails.append({
            "time": time,
            "url": url
        })

    return thumbnails

previews = generate_preview_thumbnails(video, count=5)
```

```javascript Node.js theme={null}
async function generatePreviewThumbnails(video, count = 5) {
    const duration = video.length;
    const interval = duration / (count + 1);

    const thumbnails = [];
    for (let i = 1; i <= count; i++) {
        const time = interval * i;
        const thumbnail = await video.generateThumbnail(time);
        const url = typeof thumbnail === "string"
            ? thumbnail
            : await thumbnail.generateUrl();
        thumbnails.push({ time, url });
    }

    return thumbnails;
}

const previews = await generatePreviewThumbnails(video, 5);
```

***

## Share Links

### VideoDB Console Player

Generate shareable links using the console player:

```python Python theme={null}
import urllib.parse

stream_url = video.generate_stream()
encoded_url = urllib.parse.quote(stream_url, safe='')

share_link = f"https://console.videodb.io/player?url={encoded_url}"
```

```javascript Node.js theme={null}
const streamUrl = await video.generateStream();
const encodedUrl = encodeURIComponent(streamUrl);

const shareLink = `https://console.videodb.io/player?url=${encodedUrl}`;
```

### Share Specific Clips

```python Python theme={null}
# Share a specific segment
clip_url = video.generate_stream([(120, 180)])
share_clip = f"https://console.videodb.io/player?url={urllib.parse.quote(clip_url)}"
```

```javascript Node.js theme={null}
// Share a specific segment
const clipUrl = await video.generateStream([[120, 180]]);
const shareClip = `https://console.videodb.io/player?url=${encodeURIComponent(clipUrl)}`;
```

***

## Social Media Metadata

### Open Graph Tags

```html theme={null}
<!-- Video metadata for social sharing -->
<meta property="og:type" content="video.other" />
<meta property="og:title" content="{video_title}" />
<meta property="og:description" content="{video_description}" />
<meta property="og:image" content="{thumbnail_url}" />
<meta property="og:video" content="{stream_url}" />
<meta property="og:video:type" content="application/x-mpegURL" />
<meta property="og:video:width" content="1920" />
<meta property="og:video:height" content="1080" />
```

### Twitter Card

```html theme={null}
<meta name="twitter:card" content="player" />
<meta name="twitter:title" content="{video_title}" />
<meta name="twitter:description" content="{video_description}" />
<meta name="twitter:image" content="{thumbnail_url}" />
<meta name="twitter:player" content="{embed_url}" />
<meta name="twitter:player:width" content="1280" />
<meta name="twitter:player:height" content="720" />
```

### Generate Metadata Object

```python Python theme={null}
def generate_social_metadata(video, title, description):
    """Generate metadata for social sharing"""
    stream_url = video.generate_stream()
    thumbnail_url = video.generate_thumbnail()

    return {
        "title": title,
        "description": description,
        "thumbnail": thumbnail_url,
        "video_url": stream_url,
        "og": {
            "type": "video.other",
            "title": title,
            "description": description,
            "image": thumbnail_url,
            "video": stream_url
        },
        "twitter": {
            "card": "player",
            "title": title,
            "description": description,
            "image": thumbnail_url
        }
    }
```

```javascript Node.js theme={null}
async function generateSocialMetadata(video, title, description) {
    const streamUrl = await video.generateStream();
    const thumbnailUrl = await video.generateThumbnail();

    return {
        title,
        description,
        thumbnail: thumbnailUrl,
        videoUrl: streamUrl,
        og: {
            type: "video.other",
            title,
            description,
            image: thumbnailUrl,
            video: streamUrl
        },
        twitter: {
            card: "player",
            title,
            description,
            image: thumbnailUrl
        }
    };
}
```

***

## CDN Delivery

### Stream URL Structure

VideoDB serves content through a global CDN:

```
https://stream.videodb.io/v3/published/manifests/{manifest-id}.m3u8
```

| Endpoint | Purpose |
| :- | :- |
| `stream.videodb.io` | HLS streaming |
| `cdn.videodb.io` | Direct file downloads |

### URL Caching Strategy

```python Python theme={null}
from functools import lru_cache

class VideoURLCache:
    def __init__(self):
        self.cache = {}

    def get_stream_url(self, video, timestamps=None):
        key = f"{video.id}:{timestamps}"

        if key in self.cache:
            return self.cache[key]

        # Generate new URL
        if timestamps:
            url = video.generate_stream(timestamps)
        else:
            url = video.generate_stream()

        self.cache[key] = url
        return url

# Usage
url_cache = VideoURLCache()
stream_url = url_cache.get_stream_url(video)
```

```javascript Node.js theme={null}
class VideoURLCache {
    constructor() {
        this.cache = new Map();
    }

    async getStreamUrl(video, timestamps = null) {
        const key = `${video.id}:${JSON.stringify(timestamps)}`;

        if (this.cache.has(key)) {
            return this.cache.get(key);
        }

        // Generate new URL
        const url = timestamps
            ? await video.generateStream(timestamps)
            : await video.generateStream();

        this.cache.set(key, url);
        return url;
    }
}

// Usage
const urlCache = new VideoURLCache();
const streamUrl = await urlCache.getStreamUrl(video);
```

***

## Player Features

### Autoplay with Mute

```html theme={null}
<video
  id="player"
  autoplay
  muted
  playsinline
  poster="{thumbnail_url}"
>
  <source src="{stream_url}" type="application/x-mpegURL">
</video>
```

### Loop Playback

```html theme={null}
<video id="player" loop controls>
  <source src="{stream_url}" type="application/x-mpegURL">
</video>
```

### Custom Controls

```jsx theme={null}
function CustomPlayer({ streamUrl }) {
  const videoRef = useRef(null);
  const [playing, setPlaying] = useState(false);
  const [progress, setProgress] = useState(0);

  const togglePlay = () => {
    if (playing) {
      videoRef.current.pause();
    } else {
      videoRef.current.play();
    }
    setPlaying(!playing);
  };

  return (
    <div className="player-container">
      <video
        ref={videoRef}
        onTimeUpdate={(e) => {
          const pct = (e.target.currentTime / e.target.duration) * 100;
          setProgress(pct);
        }}
      />
      <div className="controls">
        <button onClick={togglePlay}>
          {playing ? 'Pause' : 'Play'}
        </button>
        <div className="progress-bar" style={{ width: `${progress}%` }} />
      </div>
    </div>
  );
}
```

***

## Responsive Embed

### Aspect Ratio Container

```css theme={null}
.video-container {
  position: relative;
  width: 100%;
  padding-bottom: 56.25%; /* 16:9 aspect ratio */
}

.video-container video {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
}
```

```html theme={null}
<div class="video-container">
  <video controls>
    <source src="{stream_url}" type="application/x-mpegURL">
  </video>
</div>
```

### Multiple Aspect Ratios

```css theme={null}
.video-16-9 { padding-bottom: 56.25%; }
.video-4-3 { padding-bottom: 75%; }
.video-1-1 { padding-bottom: 100%; }
.video-9-16 { padding-bottom: 177.78%; }
```

***

## Best Practices

| Practice | Reason |
| :- | :- |
| Cache stream URLs | Avoid regenerating on every request |
| Use thumbnails | Improve perceived load time |
| Preload metadata | `preload="metadata"` for faster starts |
| Lazy load off-screen | Defer loading until visible |
| Handle errors | Show fallback on stream failure |

### Error Handling

```jsx theme={null}
function VideoPlayer({ streamUrl, fallbackUrl }) {
  const [error, setError] = useState(false);

  if (error && fallbackUrl) {
    return <img src={fallbackUrl} alt="Video unavailable" />;
  }

  return (
    <video
      controls
      onError={() => setError(true)}
    >
      <source src={streamUrl} type="application/x-mpegURL" />
    </video>
  );
}
```

***

## Next Steps

> **[Streams and Exports](/pages/act/output-and-delivery/streams-and-exports)**
>
> Generate clips and export video

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Compose video programmatically


# Streams and Exports
Source: https://docs.videodb.io/pages/act/output-and-delivery/streams-and-exports

Generate playable streams, downloadable clips, and export videos

Generate playable HLS streams from search results, timestamps, or timeline compositions. Export clips for download or embed streams in your application.

The semantic-search examples assume the video already has a ready semantic index.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
video = coll.get_video("m-xxx")

# Stream from search results
results = video.search("product demo")
stream_url = results.compile()

# Stream from timestamps
stream_url = video.generate_stream([(10, 30), (45, 60)])

# Get download URL
download_url = video.download(name="sample video")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();
const video = await coll.getVideo("m-xxx");

// Stream from search results
const results = await video.semanticSearch("product demo", {
    topK: 10,
});
const streamUrl = await results.compile();

// Stream from timestamps
const streamUrl2 = await video.generateStream([[10, 30], [45, 60]]);

// Get download URL
const downloadUrl = await video.download("sample video");
```

***

## Stream Generation

### From Search Results

Search returns matching segments. Generate a stream to play them as a single video.

```python Python theme={null}
# Search for content
results = video.search("key moments")

# Generate playable stream
stream_url = results.compile()
# https://stream.videodb.io/v3/published/manifests/{id}.m3u8

# Play in browser
results.play()
```

```javascript Node.js theme={null}
// Search for content
const results = await video.semanticSearch("key moments", {
    topK: 10,
});

// Generate playable stream
const streamUrl = await results.compile();
// https://stream.videodb.io/v3/published/manifests/{id}.m3u8

// Play in browser
await results.play();
```

### From Timestamps

Create streams from specific time ranges.

```python Python theme={null}
# Define time segments (start, end) in seconds
timestamps = [
    (0, 15),      # Intro
    (120, 180),   # Main content
    (300, 330)    # Conclusion
]

stream_url = video.generate_stream(timestamps)
```

```javascript Node.js theme={null}
// Define time segments [start, end] in seconds
const timestamps = [
    [0, 15],      // Intro
    [120, 180],   // Main content
    [300, 330]    // Conclusion
];

const streamUrl = await video.generateStream(timestamps);
```

### From Timeline

Export composed timelines as streams.

```python Python theme={null}
from videodb.editor import Clip, Timeline, Track, VideoAsset

# Build timeline
timeline = Timeline(conn)
track = Track()
track.add_clip(0, Clip(asset=VideoAsset(id=video.id, start=10), duration=20))
track.add_clip(20, Clip(asset=VideoAsset(id=video.id, start=60), duration=30))
timeline.add_track(track)

# Generate stream
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Timeline, VideoAsset } from 'videodb';

// Build timeline
const timeline = new Timeline(conn);
timeline.addInline(new VideoAsset(video.id, { start: 10, end: 30 }));
timeline.addInline(new VideoAsset(video.id, { start: 60, end: 90 }));

// Generate stream
const streamUrl = await timeline.generateStream();
```

***

## Stream URL Format

All generated streams use HLS format:

```
https://stream.videodb.io/v3/published/manifests/{manifest-id}.m3u8
```

| Property | Value |
| :- | :- |
| Format | HLS (HTTP Live Streaming) |
| Container | .m3u8 manifest with .ts segments |
| Compatibility | All modern browsers, native apps |

***

## Download and Export

### Get Video URL

```python Python theme={null}
# Full video download URL
download_url = video.download(name="sample video")
# https://cdn.videodb.io/v3/{collection}/{video}.mp4
```

```javascript Node.js theme={null}
// Full video download URL
const downloadUrl = await video.download("sample video");
// https://cdn.videodb.io/v3/{collection}/{video}.mp4
```

### Get Audio URL

```python Python theme={null}
# Audio-only download
audio_url = audio.generate_url()
```

```javascript Node.js theme={null}
// Audio-only download
const audioUrl = await audio.generateUrl();
```

### Get Image URL

```python Python theme={null}
# Generated image URL
image_url = image.generate_url()
```

```javascript Node.js theme={null}
// Generated image URL
const imageUrl = await image.generateUrl();
```

***

## Embedding Streams

### HTML5 Video

```html theme={null}
<video controls>
  <source
    src="https://stream.videodb.io/v3/published/manifests/{id}.m3u8"
    type="application/x-mpegURL"
  >
</video>
```

### With HLS.js

```html theme={null}
<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
<video id="video" controls></video>

<script>
const video = document.getElementById('video');
const streamUrl = 'https://stream.videodb.io/v3/...';

if (Hls.isSupported()) {
  const hls = new Hls();
  hls.loadSource(streamUrl);
  hls.attachMedia(video);
}
</script>
```

### React Component

```jsx theme={null}
import Hls from 'hls.js';
import { useRef, useEffect } from 'react';

function VideoPlayer({ streamUrl }) {
  const videoRef = useRef(null);

  useEffect(() => {
    if (Hls.isSupported()) {
      const hls = new Hls();
      hls.loadSource(streamUrl);
      hls.attachMedia(videoRef.current);
      return () => hls.destroy();
    }
  }, [streamUrl]);

  return <video ref={videoRef} controls />;
}
```

### VideoDB Console Player

Use the built-in player:

```
https://console.videodb.io/player?url={encoded_stream_url}
```

***

## Clip Export Patterns

### Export Search Results as Clips

```python Python theme={null}
results = video.search("highlight moments")

# Export each match as separate stream
clips = []
for shot in results.get_shots():
    clip_url = shot.generate_stream()
    clips.append({
        "start": shot.start,
        "end": shot.end,
        "url": clip_url,
        "text": shot.text
    })
```

```javascript Node.js theme={null}
const results = await video.search("highlight moments");

// Export each match as separate stream
const clips = [];
for (const shot of results.shots) {
    const clipUrl = await shot.generateStream();
    clips.push({
        start: shot.start,
        end: shot.end,
        url: clipUrl,
        text: shot.text
    });
}
```

### Batch Export

```python Python theme={null}
def export_clips(video, timestamps, prefix="clip"):
    """Export multiple clips from a video"""
    exports = []

    for i, (start, end) in enumerate(timestamps):
        stream_url = video.generate_stream([(start, end)])
        exports.append({
            "name": f"{prefix}_{i+1}",
            "start": start,
            "end": end,
            "duration": end - start,
            "url": stream_url
        })

    return exports

# Export intro, middle, outro
clips = export_clips(video, [
    (0, 30),
    (120, 180),
    (300, 330)
])
```

```javascript Node.js theme={null}
async function exportClips(video, timestamps, prefix = "clip") {
    const exports = [];

    for (let i = 0; i < timestamps.length; i++) {
        const [start, end] = timestamps[i];
        const streamUrl = await video.generateStream([[start, end]]);
        exports.push({
            name: `${prefix}_${i + 1}`,
            start,
            end,
            duration: end - start,
            url: streamUrl
        });
    }

    return exports;
}

// Export intro, middle, outro
const clips = await exportClips(video, [
    [0, 30],
    [120, 180],
    [300, 330]
]);
```

***

## Combine Multiple Videos

### Merge Clips from Different Videos

```python Python theme={null}
from videodb.editor import Clip, Timeline, Track, VideoAsset

video1 = coll.get_video("m-first")
video2 = coll.get_video("m-second")
video3 = coll.get_video("m-third")

# Create combined timeline
timeline = Timeline(conn)
track = Track()
track.add_clip(0, Clip(asset=VideoAsset(id=video1.id, start=0), duration=30))
track.add_clip(30, Clip(asset=VideoAsset(id=video2.id, start=10), duration=30))
track.add_clip(60, Clip(asset=VideoAsset(id=video3.id, start=0), duration=20))
timeline.add_track(track)

# Generate merged stream
merged_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Timeline, VideoAsset } from 'videodb';

const video1 = await coll.getVideo("m-first");
const video2 = await coll.getVideo("m-second");
const video3 = await coll.getVideo("m-third");

// Create combined timeline
const timeline = new Timeline(conn);
timeline.addInline(new VideoAsset(video1.id, { start: 0, end: 30 }));
timeline.addInline(new VideoAsset(video2.id, { start: 10, end: 40 }));
timeline.addInline(new VideoAsset(video3.id, { start: 0, end: 20 }));

// Generate merged stream
const mergedUrl = await timeline.generateStream();
```

### Collection Highlights

```python Python theme={null}
from videodb.editor import Timeline, VideoAsset

# Search across collection
results = coll.search("best moments")

# Build highlight reel from top results
timeline = Timeline(conn)
for shot in results.get_shots()[:10]:  # Top 10
    video_asset = VideoAsset(id=shot.video_id, start=shot.start)
    clip = Clip(asset=video_asset, duration=shot.end - shot.start)
    track = Track()
    track.add_clip(0, clip)
    timeline.add_track(track)

highlight_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset } from 'videodb';

// Search across collection
const results = await coll.search("best moments");

// Build highlight reel from top results
const timeline = new EditorTimeline(conn);
for (const shot of results.shots.slice(0, 10)) {  // Top 10
    const videoAsset = new EditorVideoAsset({ id: shot.videoId, start: shot.start });
    const clip = new Clip({ asset: videoAsset, duration: shot.end - shot.start });
    const track = new Track();
    track.addClip(0, clip);
    timeline.addTrack(track);
}

const highlightUrl = await timeline.generateStream();
```

***

## URL Lifecycle

| Stage | Duration | Notes |
| :- | :- | :- |
| Generation | Instant | Stream URL returned immediately |
| Regeneration | Anytime | Call generate\_stream again for new URL |

### Reuse Generated URLs

```python Python theme={null}
def get_stream(video, timestamps, cache={}):
    """Get or reuse a stream URL"""
    cache_key = f"{video.id}:{timestamps}"

    # Check cache
    if cache_key in cache:
        return cache[cache_key]

    # Generate URL
    url = video.generate_stream(timestamps)
    cache[cache_key] = url
    return url
```

```javascript Node.js theme={null}
const cache = new Map();

async function getStream(video, timestamps) {
    const cacheKey = `${video.id}:${JSON.stringify(timestamps)}`;

    // Check cache
    if (cache.has(cacheKey)) {
        return cache.get(cacheKey);
    }

    // Generate URL
    const url = await video.generateStream(timestamps);
    cache.set(cacheKey, url);
    return url;
}
```

***

## Reframe and Aspect Ratio Conversion

### `reframe()`

Convert a video to a different aspect ratio. Useful for repurposing landscape content to vertical formats for social media.

```python Python theme={null}
# Reframe to vertical (9:16) for social media
vertical_video = video.reframe(target="vertical", mode="smart")

# Custom aspect ratio (square)
square_video = video.reframe(target={"width": 1, "height": 1})

# Reframe a specific time range
clip = video.reframe(start=10.0, end=60.0, target="vertical")

# Async with callback
video.reframe(
    start=10.0,
    end=60.0,
    target="vertical",
    callback_url="https://example.com/webhook"
)
```

| Parameter | Type | Description |
| :- | :- | :- |
| `target` | `str` or `dict` | `"vertical"`, `"square"`, `"landscape"`, or `{"width": int, "height": int}` |
| `mode` | `str` | `"simple"` or `"smart"` (default). Smart mode uses object-aware tracking. |
| `start` | `float` | Optional start time in seconds |
| `end` | `float` | Optional end time in seconds |
| `callback_url` | `str` | Optional URL for async completion callback |

### `smart_vertical_reframe()`

Convenience method that reframes to vertical (9:16) with smart object tracking in a single call:

```python theme={null}
vertical = video.smart_vertical_reframe()
```

***

## Next Steps

> **[Publishing Patterns](/pages/act/output-and-delivery/publishing-patterns)**
>
> Embed players and share content

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Build complex compositions


# Aspect Ratio Control
Source: https://docs.videodb.io/pages/act/programmable-editing/aspect-ratio-control

Control how assets scale when dimensions don't match the timeline. The fit parameter handles the mismatch (crop, letterbox, stretch), while position and offset control placement.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/fit_position_aspect_ratios.ipynb)

## Fit Modes

The fit parameter determines how an asset scales to match the timeline resolution.

### `Fit.crop` (default)

* Scales asset to fill viewport while maintaining aspect ratio
* Edges are cropped if aspect ratios differ

This is the default because it creates clean, full-frame compositions without black bars. The tradeoff is you might lose some content at the edges, but everything stays proportional and the frame stays filled.

<video>
  <source type="application/x-mpegURL" />
</video>

### `Fit.contain`

* Scales asset to fit entirely within viewport while maintaining aspect ratio
* May add letterboxing (black bars) if aspect ratios differ

Use this when you need to guarantee that all of your content is visible, even if it means having black bars on the sides or top and bottom. Nothing gets cropped, but you sacrifice the full-frame look.

<video>
  <source type="application/x-mpegURL" />
</video>

### `Fit.cover`

* Stretches asset to fill viewport, ignoring aspect ratio
* May cause distortion

This mode stretches the content to fill the frame regardless of proportions. You’ll rarely need this unless you specifically want the stretched effect, or you’re working with content that already matches your timeline’s aspect ratio exactly.

<video>
  <source type="application/x-mpegURL" />
</video>

### `Fit.none`

* Preserves original pixel dimensions
* No scaling applied

This keeps your asset at its exact original size, which is useful for things like logos, overlays, or picture-in-picture effects where you want pixel-perfect control. The content appears at its native resolution regardless of timeline size.

<video>
  <source type="application/x-mpegURL" />
</video>

### Code Example

```python Python theme={null}
from videodb.editor import Fit

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    fit=Fit.crop  # or Fit.contain, Fit.cover, Fit.none
)
```

```javascript Node.js theme={null}
import { Fit, Clip, EditorVideoAsset } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    fit: Fit.crop  // or Fit.contain, Fit.cover, Fit.none
});
```

Note: You can also use fit=None (Python None) which is equivalent to `Fit.none`.

## Position

The position parameter places the asset in one of 9 preset zones on the viewport. Think of your screen divided into a 3x3 grid - you can anchor your content to any of these zones, from corners to edges to dead center.

```
Position.top_left      Position.top        Position.top_right
Position.left          Position.center     Position.right
Position.bottom_left   Position.bottom     Position.bottom_right
```

**Example with `Position.left`**

<video>
  <source type="application/x-mpegURL" />
</video>

### Code Example

```python Python theme={null}
from videodb.editor import Position

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    position=Position.top_left,
    fit=None
)
```

```javascript Node.js theme={null}
import { Position, Clip, EditorVideoAsset } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    position: Position.topLeft,
    fit: null
});
```

## Offset

The offset parameter provides fine-grained position control using relative coordinates. Sometimes the 9 preset zones aren’t quite right - maybe you want something centered but nudged slightly to the left, or positioned with a specific margin. That’s where offset comes in, letting you tweak positions with precision.

**Example `Offset(x=-0.25)`**

<video>
  <source type="application/x-mpegURL" />
</video>

### Code Example

```python Python theme={null}
from videodb.editor import Offset

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    position=Position.center,
    offset=Offset(x=0.15, y=0.1)  # x: horizontal, y: vertical
)
```

```javascript Node.js theme={null}
import { Offset, Clip, EditorVideoAsset, Position } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    position: Position.center,
    offset: new Offset({ x: 0.15, y: 0.1 })  // x: horizontal, y: vertical
});
```

Coordinate System:

* `x` : Horizontal shift (`-1.0` to `1.0`)
  * Positive values move right
  * Negative values move left
* `y` : Vertical shift (`-1.0` to `1.0`)
  * Positive values move down
  * Negative values move up
* Values are relative to viewport dimensions

Example: `offset=Offset(x=0.1, y=0)` on a 1080px wide viewport moves the clip 108px to the right.

## Parameter Reference

| Parameter | Type | Description |
| :- | :- | :- |
| fit | Fit | Scaling behavior: `Fit.crop`, `Fit.contain`, `Fit.cover`, `Fit.none` |
| position | Position | Anchor point: `Position.top_left`, `Position.center`, etc. (9 options) |
| offset | Offset | Fine-tuning position with `x`/`y` coordinates |

```python Python theme={null}
from videodb.editor import Clip, VideoAsset, Fit, Position, Offset

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    fit=Fit.crop,
    position=Position.right,
    offset=Offset(x=-0.1, y=0)
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Fit, Position, Offset } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    fit: Fit.crop,
    position: Position.right,
    offset: new Offset({ x: -0.1, y: 0 })
});
```

## What You Can Build

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Create vertical 9:16 videos optimized for TikTok and Reels

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Position logos and overlays precisely using fit and offset

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Control aspect ratios for multi-platform content

***

## Next Steps

> **[Fit, Position & Aspect Ratios](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/fit_position_aspect_ratios.ipynb)**
>
> Hands-on experimentation with all fit modes, 9-zone positioning, and aspect ratio handling.


# Audio Asset Layering
Source: https://docs.videodb.io/pages/act/programmable-editing/audio-asset-layering

Layer music and voiceover audio over video and build practical mixes with AudioAsset.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/audio_asset_layering.ipynb)

`AudioAsset` adds an audio file from your VideoDB collection to an Editor composition. Use it with `VideoAsset.volume` when the visual source and its original audio need separate treatment.

The focused snippets below assume `conn`, `video`, and the named audio assets already exist. See [Timeline Architecture](/pages/act/programmable-editing/timeline-architecture) for complete setup and [Trimming Vs Timing](/pages/act/programmable-editing/trimming-vs-timing) for the `AudioAsset` fields.

## Add background music

Place a video clip and a music clip at the same timeline position to combine their audio.

```python Python theme={null}
from videodb.editor import AudioAsset, Clip, Timeline, Track, VideoAsset

timeline = Timeline(conn)
timeline.background = "#2B2B2B"

video_clip = Clip(
    asset=VideoAsset(id=video.id, volume=1.0),
    duration=10,
)
music_clip = Clip(
    asset=AudioAsset(id=music.id, volume=0.1),
    duration=10,
)

track = Track()
track.add_clip(0, video_clip)
track.add_clip(0, music_clip)
timeline.add_track(track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorAudioAsset, EditorTimeline, EditorVideoAsset, Track } from 'videodb';

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";

const videoClip = new Clip({
  asset: new EditorVideoAsset({ id: video.id, volume: 1.0 }),
  duration: 10,
});
const musicClip = new Clip({
  asset: new EditorAudioAsset({ id: music.id, volume: 0.1 }),
  duration: 10,
});

const track = new Track();
track.addClip(0, videoClip);
track.addClip(0, musicClip);
timeline.addTrack(track);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Layer voiceover and music

Mute the source video when the mix should consist only of narration and music. The levels below are examples from the notebook, not universal requirements.

```python Python theme={null}
from videodb.editor import AudioAsset, Clip, Timeline, Track, VideoAsset

timeline = Timeline(conn)
timeline.background = "#2B2B2B"

video_clip = Clip(
    asset=VideoAsset(id=video.id, volume=0),
    duration=10,
)
voiceover_clip = Clip(
    asset=AudioAsset(id=voiceover.id, volume=1.0),
    duration=10,
)
music_clip = Clip(
    asset=AudioAsset(id=music.id, volume=0.1),
    duration=10,
)

track = Track()
track.add_clip(0, video_clip)
track.add_clip(0, voiceover_clip)
track.add_clip(0, music_clip)
timeline.add_track(track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorAudioAsset, EditorTimeline, EditorVideoAsset, Track } from 'videodb';

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";

const videoClip = new Clip({
  asset: new EditorVideoAsset({ id: video.id, volume: 0 }),
  duration: 10,
});
const voiceoverClip = new Clip({
  asset: new EditorAudioAsset({ id: voiceover.id, volume: 1.0 }),
  duration: 10,
});
const musicClip = new Clip({
  asset: new EditorAudioAsset({ id: music.id, volume: 0.1 }),
  duration: 10,
});

const track = new Track();
track.addClip(0, videoClip);
track.addClip(0, voiceoverClip);
track.addClip(0, musicClip);
timeline.addTrack(track);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

The full [Audio Asset Layering notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/audio_asset_layering.ipynb) explores additional mixes and source combinations.


# Caption Asset
Source: https://docs.videodb.io/pages/act/programmable-editing/caption-asset

CaptionAsset synchronizes text to audio timestamps, creating subtitles that move with spoken words. Unlike TextAsset which displays static text overlays, CaptionAsset is built for speech-driven content where timing matters.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/caption_asset_auto_subtitles.ipynb)

## `CaptionAsset` vs `TextAsset`

| Feature | TextAsset | CaptionAsset |
| :- | :- | :- |
| Timeline Sync | No | Yes (word-level timestamps) |
| Data Source | Manual text input | Auto-generated from speech |
| Animation | Static only | reveal, karaoke, supersize, box\_highlight |
| Format | Font/Border/Shadow objects | ASS (Advanced SubStation Alpha) |

`CaptionAsset` uses ASS format for subtitle rendering, which enables time-synchronized animations and professional subtitle styling.

## Timed Caption Assets

Create current caption assets from a base64-encoded SRT payload with the timings you want to render. The [Track Layering guide](/pages/act/programmable-editing/track-layering) shows how to build timed SRT from a spoken-words artifact.

```python theme={null}
import base64
from videodb.editor import CaptionAsset, Clip, Track

srt = """1
00:00:00,000 --> 00:00:02,000
Hello from a timed caption asset.
"""
srt_src = base64.b64encode(srt.encode()).decode()

caption_clip = Clip(asset=CaptionAsset(src=srt_src), duration=2)
track = Track()
track.add_clip(0, caption_clip)
```

### Auto-Caption Compatibility

> [!NOTE]
>
> `CaptionAsset(src="auto")` and its `index_spoken_words()` prerequisite below are a current compatibility path. They do not make an Understand artifact alone a source for `src="auto"`.

### Compatibility Usage

```python Python theme={null}
from videodb.editor import CaptionAsset, Clip, Track

video.index_spoken_words()

caption_clip = Clip(
    asset=CaptionAsset(src="auto"),
    duration=float(video.length)
)

track = Track()
track.add_clip(0, caption_clip)
```

```javascript Node.js theme={null}
import { CaptionAsset, Clip, Track } from 'videodb';

await video.indexSpokenWords();

const captionClip = new Clip({
    asset: new CaptionAsset({ src: "auto" }),
    duration: parseFloat(video.length)
});

const track = new Track();
track.addClip(0, captionClip);
```

Example:

<video>
  <source type="application/x-mpegURL" />
</video>

The caption clip duration should match or exceed the video duration to ensure all words display.

> [!NOTE]
>
> The animation examples below continue the `src="auto"` compatibility path. Use the timed SRT source above for current caption assets.

## Animation Types

`CaptionAsset` supports six animation modes that make your subtitles more dynamic:

| Animation | Effect |
| :- | :- |
| reveal | Words appear one-by-one as they're spoken |
| karaoke | Active word changes color (primary → secondary) while speaking |
| supersize | Active word scales up in size for emphasis |
| box\_highlight | Active word is highlighted with a background box |
| impact | Only the active word appears on the screen |
| color\_highlight | Active word highlights with a distinct color for emphasis |

Code Example

```python Python theme={null}
from videodb.editor import CaptionAnimation

caption_asset = CaptionAsset(
    src="auto",
    animation=CaptionAnimation.karaoke,
    primary_color="&H00FFFFFF",    # White
    secondary_color="&H0000FFFF"   # Yellow highlight
)
```

```javascript Node.js theme={null}
import { CaptionAsset, CaptionAnimation } from 'videodb';

const captionAsset = new CaptionAsset({
    src: "auto",
    animation: CaptionAnimation.karaoke,
    primaryColor: "&H00FFFFFF",    // White
    secondaryColor: "&H0000FFFF"   // Yellow highlight
});
```

Example with `CaptionAnimation.karaoke`

<video>
  <source type="application/x-mpegURL" />
</video>

## ASS Color Format

ASS (Advanced SubStation Alpha) is a professional subtitle format that’s been used in video production for years.

It uses BGR (Blue-Green-Red) byte order with an alpha channel - which is backwards from the RGB format you might be used to from web colors.

This quirk exists for historical reasons in subtitle rendering systems.

### Format Structure

`&HAABBGGRR` or `&H00BBGGRR`

* `AA` = Alpha (`00` = opaque, `FF` = transparent)
* `BB` = Blue channel
* `GG` = Green channel
* `RR` = Red channel

### HTML to ASS Conversion

To convert HTML colors to ASS format:

1. HTML `#RRGGBB` → Extract RGB bytes
2. Reverse to BGR order
3. Add prefix `&H00` (opaque) or `&HAA` (with transparency)

Example: HTML `#FF6600` (orange)

* RGB: Red=`FF`, Green=`66`, Blue=`00`
* BGR: `00-66-FF`
* ASS: `&H000066FF`

### Common Colors

| HTML | ASS | Color |
| :- | :- | :- |
| `#FFFFFF` | `&H00FFFFFF` | White |
| `#000000` | `&H00000000` | Black |
| `#FF0000` | `&H000000FF` | Red |
| `#FFFF00` | `&H0000FFFF` | Yellow |
| `#00FF00` | `&H0000FF00` | Green |

## Styling Parameters

`CaptionAsset` styling is organized into three parameter groups: `FontStyling`, `Positioning`, and `BorderAndShadow`.

### `FontStyling`

Controls how your subtitle text looks - the font face, size, and whether it's bold or italic. Think of this as the basic typography settings for making your captions readable and on-brand.

```python Python theme={null}
from videodb.editor import FontStyling

FontStyling(
    size=36,          # Font size in points
    bold=True,        # Bold weight
    italic=False,     # Italic style
    name="Arial"      # Font family
)
```

```javascript Node.js theme={null}
import { FontStyling } from 'videodb';

new FontStyling({
    size: 36,          // Font size in points
    bold: true,        // Bold weight
    italic: false,     // Italic style
    name: "Arial"      // Font family
});
```

| Parameter | Type | Description |
| :- | :- | :- |
| size | int | Font size in points (not pixels) |
| bold | bool | Bold weight (True) or normal (False) |
| italic | bool | Italic style |
| name | str | Font family name (must be available on server) |

![Chart showing all supported fonts for text and caption assets](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/reference/supported-fonts.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=a97596482d63a7c6b3ad294bba6f2363)

### `Positioning`

Controls where on the screen your captions appear and how much spacing you want from the edges. You can place captions at the bottom like traditional subtitles, or anywhere else on screen with precise margin control.

```python Python theme={null}
Positioning(
    alignment=CaptionAlignment.bottom_center,
    margin_v=100,     # Vertical margin in pixels
    margin_l=20,      # Left margin in pixels
    margin_r=20       # Right margin in pixels
)
```

```javascript Node.js theme={null}
import { Positioning, CaptionAlignment } from 'videodb';

new Positioning({
    alignment: CaptionAlignment.bottomCenter,
    marginV: 100,     // Vertical margin in pixels
    marginL: 20,      // Left margin in pixels
    marginR: 20       // Right margin in pixels
});
```

| Parameter | Type | Description |
| :- | :- | :- |
| alignment | CaptionAlignment | Where on screen the captions appear (see alignment options below) |
| margin\_v | int | Vertical margin in pixels from top or bottom edge |
| margin\_l | int | Left margin in pixels from left edge |
| margin\_r | int | Right margin in pixels from right edge |

```python Python theme={null}
# Corners
CaptionAlignment.top_left
CaptionAlignment.top_right
CaptionAlignment.bottom_left
CaptionAlignment.bottom_right

# Edges
CaptionAlignment.top
CaptionAlignment.top_center
CaptionAlignment.bottom
CaptionAlignment.left
CaptionAlignment.right

# Center positions
CaptionAlignment.middle_center
CaptionAlignment.bottom_center
```

```javascript Node.js theme={null}
// Corners
CaptionAlignment.topLeft
CaptionAlignment.topRight
CaptionAlignment.bottomLeft
CaptionAlignment.bottomRight

// Edges
CaptionAlignment.top
CaptionAlignment.topCenter
CaptionAlignment.bottom
CaptionAlignment.left
CaptionAlignment.right

// Center positions
CaptionAlignment.middleCenter
CaptionAlignment.bottomCenter
```

Example:

```python Python theme={null}
position=Positioning(
            alignment=CaptionAlignment.bottom_center,
            margin_v=50  # 50px from bottom
),
font=FontStyling(
    size=48,
    bold=True,
    name = "Clear Sans",
)
```

```javascript Node.js theme={null}
const captionOptions = {
  position: new Positioning({
    alignment: CaptionAlignment.bottomCenter,
    marginV: 50,  // 50px from bottom
  }),
  font: new FontStyling({
    size: 48,
    bold: true,
    name: "Clear Sans",
  }),
};
```

<video>
  <source type="application/x-mpegURL" />
</video>

### `BorderAndShadow`

Controls outlines and shadows that make your text readable over any background.

These parameters are crucial because subtitles need to be legible whether they're over bright skies, dark scenes, or complex imagery - borders and shadows ensure the text always stands out.

```python Python theme={null}
from videodb.editor import BorderAndShadow, CaptionBorderStyle

BorderAndShadow(
    style=CaptionBorderStyle.outline_and_shadow,
    outline=3.0,                # Outline width in pixels
    shadow=2.0,                 # Shadow depth in pixels
    outline_color="&H00000000" # Black outline (ASS format)
)
```

```javascript Node.js theme={null}
import { BorderAndShadow, CaptionBorderStyle } from 'videodb';

new BorderAndShadow({
    style: CaptionBorderStyle.outlineAndShadow,
    outline: 3.0,                // Outline width in pixels
    shadow: 2.0,                 // Shadow depth in pixels
    outlineColor: "&H00000000"  // Black outline (ASS format)
});
```

| Parameter | Type | Description |
| :- | :- | :- |
| style | CaptionBorderStyle | How the border/background is rendered |
| outline | float | Outline width in pixels around each letter |
| shadow | float | Shadow depth in pixels for drop shadow effect |
| outline\_color | str | Outline color in ASS format |

`CaptionBorderStyle` Options:

* `CaptionBorderStyle.outline_and_shadow` - Outline + drop shadow
* `CaptionBorderStyle.opaque_box` - Solid background box

Example:

```python Python theme={null}
border=BorderAndShadow(
            style=CaptionBorderStyle.outline_and_shadow,
            outline=5,
            outline_color="&H00000000",  # Black outline
            shadow=3
        )
```

```javascript Node.js theme={null}
const captionBorder = new BorderAndShadow({
  style: CaptionBorderStyle.outlineAndShadow,
  outline: 5,
  outlineColor: "&H00000000",  // Black outline
  shadow: 3,
});
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Complete Example

From the notebook, here's a complete `CaptionAsset` with all styling parameters:

```python Python theme={null}
from videodb.editor import CaptionAsset, CaptionAnimation, Positioning, CaptionAlignment, FontStyling, BorderAndShadow

caption_asset = CaptionAsset(
    src="auto",
    animation=CaptionAnimation.supersize,
    primary_color="&H00FFFFFF",      # White text
    secondary_color="&H0000FFFF",    # Yellow highlight for animations
    position=Positioning(
        alignment=CaptionAlignment.middle_center,
        margin_v=100
    ),
    font=FontStyling(
        size=44,
        bold=True
    )
)

caption_clip = Clip(
  asset=caption_asset,
  duration=100
)
```

```javascript Node.js theme={null}
import { CaptionAsset, CaptionAnimation, Positioning, CaptionAlignment, FontStyling, Clip } from 'videodb';

const captionAsset = new CaptionAsset({
    src: "auto",
    animation: CaptionAnimation.supersize,
    primaryColor: "&H00FFFFFF",      // White text
    secondaryColor: "&H0000FFFF",    // Yellow highlight for animations
    position: new Positioning({
        alignment: CaptionAlignment.middleCenter,
        marginV: 100
    }),
    font: new FontStyling({
        size: 44,
        bold: true
    })
});

const captionClip = new Clip({
  asset: captionAsset,
  duration: 100
});
```

## Parameter Reference

| Parameter | Type | Description |
| :- | :- | :- |
| src | str | Set to `"auto"` for auto-generation from speech or add manually in `.srt` or `.ass` format converted to `base64` |
| animation | CaptionAnimation | Animation mode: `reveal`, `karaoke`, `supersize`, `box_highlight`, `impact`, `color_highlight` |
| primary\_color | str | Main text color (ASS format: `&H00BBGGRR`) |
| secondary\_color | str | Highlight color for animations (ASS format) |
| position | Positioning | Spatial placement and margins |
| font | FontStyling | Typography settings |
| border\_and\_shadow | BorderAndShadow | Outline and shadow styling |

## Next Steps

> **[Caption Asset & Auto Subtitles](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/caption_asset_auto_subtitles.ipynb)**
>
> Hands-on experimentation with styling options, animations, and color formatting.


# Clip Parameters
Source: https://docs.videodb.io/pages/act/programmable-editing/clip-parameters

The Clip object wraps an Asset and controls how it appears on screen. Complete reference for duration, geometry, visual effects, filters, transitions, and layering.

The Clip object wraps an Asset and controls how it appears on screen. Think of the Asset as your raw content (the video file, image, or text), and the Clip as all the presentation decisions - where it appears, how big it is, what color effects it has, how it fades in and out. This guide documents all available Clip parameters and their interactions.

## Editor Architecture

Asset → Clip → Track → Timeline

* Asset: Raw content (what to show)
* Clip: Presentation control (how to show it)
* Track: Layering container (for stacking multiple clips)
* Timeline: Final composition (the output video)

This separation lets you reuse the same asset in multiple clips with different visual treatments - same video, but one clip shows it full-screen while another shows it as a small picture-in-picture overlay.

## Clip Parameters

### Core Parameters

| Parameter | Type | Description |
| :- | :- | :- |
| asset | Asset | The content to display (`VideoAsset`, `ImageAsset`, `AudioAsset`, `TextAsset`, `CaptionAsset`) |
| duration | float | Clip length in seconds |

### Geometry Parameters

| Parameter | Type | Description |
| :- | :- | :- |
| fit | Fit | Scaling behavior: `Fit.crop`, `Fit.contain`, `Fit.cover`, `Fit.none` |
| position | Position | Anchor point (9 zones: `top_left`, `center`, `bottom_right`, etc.) |
| offset | Offset | Fine-tune position with `x`/`y` coordinates |
| scale | float | Size multiplier (`0.0` to `10.0`, default: `1.0`) |

### Visual Effect Parameters

| Parameter | Type | Description |
| :- | :- | :- |
| filter | Filter | Color treatment (`greyscale`, `blur`, `contrast`, etc.) |
| opacity | float | Transparency (`0.0` = invisible, `1.0` = opaque) |
| transition | Transition | Fade `in_`/`out` effects |

## Scale Parameter

The scale parameter is a size multiplier applied after fit mode. First, the fit mode handles the aspect ratio and scales your content to match the timeline, then scale multiplies that result. This is useful for creating picture-in-picture effects (scale=0.3 for a tiny corner video) or zoom effects (scale=1.5 to enlarge).

```python Python theme={null}
from videodb.editor import Clip, VideoAsset, Position

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    scale=0.5,  # 50% of original size
    position=Position.top_left
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Position } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    scale: 0.5,  // 50% of original size
    position: Position.topLeft
});
```

Range: `0.0` to `10.0` (default: `1.0`)

Example with `scale=0.5` and `Position.top_left`

<video>
  <source type="application/x-mpegURL" />
</video>

## Opacity Parameter

The opacity parameter controls transparency, letting you create semi-transparent overlays, subtle watermarks, or fade effects. At `1.0` your clip is fully solid, at `0.5` it's half-transparent, and at `0.0` it's completely invisible.

```python Python theme={null}
from videodb.editor import Clip, VideoAsset

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    opacity=0.5  # 50% transparent
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    opacity: 0.5  // 50% transparent
});
```

Range: `0.0` (invisible) to `1.0` (opaque)

Example with `opacity=0.3`

<video>
  <source type="application/x-mpegURL" />
</video>

## Filter Parameter

The filter parameter applies color and visual treatments to your clip. These are global effects that change the entire clip's appearance - you can make it black and white, blur it for backgrounds, adjust contrast, or create stylistic looks. Each clip can have one filter applied.

```python Python theme={null}
Filter.greyscale   # Remove color (black and white)
Filter.blur        # Blur the video
Filter.contrast    # Increase contrast
Filter.boost       # Boost contrast and saturation
Filter.muted       # Reduce saturation and contrast
Filter.darken      # Darken the scene
Filter.lighten     # Lighten the scene
Filter.negative    # Invert colors
```

```javascript Node.js theme={null}
Filter.greyscale   // Remove color (black and white)
Filter.blur        // Blur the video
Filter.contrast    // Increase contrast
Filter.boost       // Boost contrast and saturation
Filter.muted       // Reduce saturation and contrast
Filter.darken      // Darken the scene
Filter.lighten     // Lighten the scene
Filter.negative    // Invert colors
```

```python Python theme={null}
from videodb.editor import Clip, VideoAsset, Filter

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    filter=Filter.greyscale
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Filter } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    filter: Filter.greyscale
});
```

Example with `Filter.greyscale`

<video>
  <source type="application/x-mpegURL" />
</video>

## Transition Parameter

The transition parameter controls fade in/out effects, making your clips appear and disappear smoothly instead of cutting abruptly. The fade happens over the first and last N seconds of your clip - so a 2-second fade means the first 2 seconds gradually appear, and the last 2 seconds gradually disappear.

```python Python theme={null}
from videodb.editor import Clip, VideoAsset, Transition

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    transition=Transition(
        in_="fade",   # Fade in effect (note the underscore)
        out="fade",   # Fade out effect
        duration=2    # Transition duration in seconds
    )
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Transition } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    transition: new Transition({
        in: "fade",   // Fade in effect
        out: "fade",  // Fade out effect
        duration: 2   // Transition duration in seconds
    })
});
```

Example:

<video>
  <source type="application/x-mpegURL" />
</video>

Parameters:

* `in_`: Transition type for entry (use `in_` with underscore because `in` is a Python keyword)
* `out`: Transition type for exit
* `duration`: Length of transition in seconds

## Complete Example

Here's a clip using multiple parameters:

```python Python theme={null}
from videodb.editor import Clip, VideoAsset, Position, Filter, Transition

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    position=Position.bottom_left,
    scale=0.7,
    opacity=0.3,
    filter=Filter.greyscale,
    transition=Transition(in_="fade", out="fade", duration=3),
    fit=None
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Position, Filter, Transition, Fit } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    position: Position.bottomLeft,
    scale: 0.7,
    opacity: 0.3,
    filter: Filter.greyscale,
    transition: new Transition({ in: "fade", out: "fade", duration: 3 }),
    fit: Fit.none
});
```

## Parameter Reference

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| asset | Asset | Required | Content to display |
| duration | float | Required | Clip length in seconds |
| fit | Fit | Fit.crop | Scaling mode |
| position | Position | None | Anchor point (9 zones) |
| offset | Offset | None | Fine position adjustment |
| scale | float | 1.0 | Size multiplier (`0.0`-`10.0`) |
| opacity | float | 1.0 | Transparency (`0.0`-`1.0`) |
| filter | Filter | None | Color treatment |
| transition | Transition | None | Fade effects |

## What You Can Build

> **[Intro & Outro Automation](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Use clip positioning and transitions for seamless brand intros

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Overlay logos and watermarks with opacity and positioning

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Combine clips with text overlays and visual effects

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Layer multiple clips with filters and transitions

***

## Next Steps

> **[Clip Control Layer](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/clip_control_layer.ipynb)**
>
> Hands-on experimentation with all Clip parameters, effects, and transitions.


# Filters and Transitions
Source: https://docs.videodb.io/pages/act/programmable-editing/filters-and-transitions

Give individual clips a visual treatment and an independent entrance or exit transition in VideoDB Editor.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/filters_and_transitions.ipynb)

Filters change a clip's look. Transitions control how that clip enters or exits. Both belong on `Clip`, so the same source asset can appear more than once with different treatments. The focused snippets below assume `conn` and the source videos already exist. For the core parameters and other clip controls, see [Clip Parameters](/pages/act/programmable-editing/clip-parameters); for complete composition setup, see [Timeline Architecture](/pages/act/programmable-editing/timeline-architecture).

## Filters

Each clip can carry one visual treatment. The comparison below uses one source segment for every released filter and adds synchronized labels on a second track.

### Eight-filter showcase

The notebook places eight five-second clips on one track, each referencing the same source segment with one of the released filter values.

```python Python theme={null}
from videodb.editor import (
    Background, Clip, Filter, Font, TextAlignment, TextAsset,
    Timeline, Track, VideoAsset
)

timeline = Timeline(conn)
timeline.background = "#2B2B2B"

filters = [
    Filter.greyscale, Filter.blur, Filter.contrast, Filter.darken,
    Filter.lighten, Filter.boost, Filter.muted, Filter.negative,
]

track = Track()
for start, filter_value in enumerate(filters):
    track.add_clip(
        start * 5,
        Clip(
            asset=VideoAsset(id=video1.id, start=10),
            duration=5,
            filter=filter_value,
        ),
    )
timeline.add_track(track)

filter_names = [
    "Greyscale", "Blur", "Contrast", "Darken",
    "Lighten", "Boost", "Muted", "Negative",
]

text_track = Track()
for start, filter_name in enumerate(filter_names):
    text_asset = TextAsset(
        text=filter_name,
        font=Font(family="Clear Sans", size=36, color="#FFFFFF"),
        background=Background(
            color="#000000",
            opacity=0.6,
            height=60,
            width=200,
            text_alignment=TextAlignment.center,
        ),
    )
    text_track.add_clip(start * 5, Clip(asset=text_asset, duration=5))
timeline.add_track(text_track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import {
  Background, Clip, EditorTextAsset, EditorTimeline, EditorVideoAsset,
  Filter, Font, TextAlignment, Track
} from "videodb";

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";

const filters = [
  Filter.greyscale, Filter.blur, Filter.contrast, Filter.darken,
  Filter.lighten, Filter.boost, Filter.muted, Filter.negative,
];

const track = new Track();
filters.forEach((filter, index) => {
  track.addClip(index * 5, new Clip({
    asset: new EditorVideoAsset({ id: video1.id, start: 10 }),
    duration: 5,
    filter,
  }));
});
timeline.addTrack(track);

const filterNames = [
  "Greyscale", "Blur", "Contrast", "Darken",
  "Lighten", "Boost", "Muted", "Negative",
];

const textTrack = new Track();
filterNames.forEach((filterName, index) => {
  const textAsset = new EditorTextAsset({
    text: filterName,
    font: new Font({ family: "Clear Sans", size: 36, color: "#FFFFFF" }),
    background: new Background({
      color: "#000000",
      opacity: 0.6,
      height: 60,
      width: 200,
      textAlignment: TextAlignment.center,
    }),
  });
  textTrack.addClip(index * 5, new Clip({ asset: textAsset, duration: 5 }));
});
timeline.addTrack(textTrack);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Transitions

Beyond the basic fade covered in [Clip Parameters](/pages/act/programmable-editing/clip-parameters), the notebook demonstrates motion-based transition names and mixed timings. Python uses `in_` because `in` is a keyword; Node uses `in` inside the configuration object.

### Reveal

`reveal` is the demonstrated motion-based alternative to a fade.

```python Python theme={null}
from videodb.editor import Clip, Timeline, Track, Transition, VideoAsset

timeline = Timeline(conn)
timeline.background = "#2B2B2B"

clip = Clip(
    asset=VideoAsset(id=video1.id),
    duration=10,
    transition=Transition(in_="reveal", out="reveal", duration=2),
)

track = Track()
track.add_clip(2, clip)
timeline.add_track(track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorTimeline, EditorVideoAsset, Track, Transition } from "videodb";

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";

const clip = new Clip({
  asset: new EditorVideoAsset({ id: video1.id }),
  duration: 10,
  transition: new Transition({ in: "reveal", out: "reveal", duration: 2 }),
});

const track = new Track();
track.addClip(2, clip);
timeline.addTrack(track);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

### Wipe and shuffle combinations

The notebook also demonstrates `wipe_left`, `wipe_right`, `shuffle_top_right`, `shuffle_bottom_right`, `shuffle_bottom_left`, and `shuffle_top_left`. These are demonstrated transition names, not a claim that they exhaust the server-side strings accepted by `Transition`.

```python Python theme={null}
from videodb.editor import Clip, Timeline, Track, Transition, VideoAsset

timeline = Timeline(conn)
timeline.background = "#2B2B2B"

transition_names = [
    "wipe_left", "wipe_right", "shuffle_top_right",
    "shuffle_bottom_right", "shuffle_bottom_left", "shuffle_top_left",
]

transition_track = Track()
for start, name in enumerate(transition_names):
    transition_track.add_clip(
        start * 5,
        Clip(
            asset=VideoAsset(id=video1.id, start=10),
            duration=5,
            transition=Transition(in_=name, out=name, duration=1),
        ),
    )
timeline.add_track(transition_track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorTimeline, EditorVideoAsset, Track, Transition } from "videodb";

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";

const transitionNames = [
  "wipe_left", "wipe_right", "shuffle_top_right",
  "shuffle_bottom_right", "shuffle_bottom_left", "shuffle_top_left",
];

const transitionTrack = new Track();
transitionNames.forEach((name, index) => {
  transitionTrack.addClip(index * 5, new Clip({
    asset: new EditorVideoAsset({ id: video1.id, start: 10 }),
    duration: 5,
    transition: new Transition({ in: name, out: name, duration: 1 }),
  }));
});
timeline.addTrack(transitionTrack);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Combine filters and transitions

The properties are independent. This output uses three sequential clips with distinct filters and fade durations.

```python Python theme={null}
from videodb.editor import Clip, Filter, Timeline, Track, Transition, VideoAsset

timeline = Timeline(conn)
timeline.background = "#2B2B2B"

showcase_track = Track()

clips = [
    Clip(
        asset=VideoAsset(id=video1.id, start=5),
        duration=7,
        filter=Filter.greyscale,
        transition=Transition(in_="fade", out="fade", duration=2),
    ),
    Clip(
        asset=VideoAsset(id=video2.id, start=10),
        duration=7,
        filter=Filter.boost,
        transition=Transition(in_="fade", out="fade", duration=1),
    ),
    Clip(
        asset=VideoAsset(id=video1.id, start=30),
        duration=7,
        filter=Filter.negative,
        transition=Transition(in_="fade", out="fade", duration=2),
    ),
]

for start, clip in zip([0, 7, 14], clips):
    showcase_track.add_clip(start, clip)
timeline.add_track(showcase_track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorTimeline, EditorVideoAsset, Filter, Track, Transition } from "videodb";

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";

const showcaseTrack = new Track();

const clips = [
  new Clip({
    asset: new EditorVideoAsset({ id: video1.id, start: 5 }),
    duration: 7,
    filter: Filter.greyscale,
    transition: new Transition({ in: "fade", out: "fade", duration: 2 }),
  }),
  new Clip({
    asset: new EditorVideoAsset({ id: video2.id, start: 10 }),
    duration: 7,
    filter: Filter.boost,
    transition: new Transition({ in: "fade", out: "fade", duration: 1 }),
  }),
  new Clip({
    asset: new EditorVideoAsset({ id: video1.id, start: 30 }),
    duration: 7,
    filter: Filter.negative,
    transition: new Transition({ in: "fade", out: "fade", duration: 2 }),
  }),
];

[0, 7, 14].forEach((start, index) => {
  showcaseTrack.addClip(start, clips[index]);
});
timeline.addTrack(showcaseTrack);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

For sequential placement and source trimming, continue with [Trimming Vs Timing](/pages/act/programmable-editing/trimming-vs-timing). For mixing effects with layout controls, return to [Clip Parameters](/pages/act/programmable-editing/clip-parameters).


# Image Asset
Source: https://docs.videodb.io/pages/act/programmable-editing/image-asset

Add still images to an Editor timeline for branding, slideshows, overlays, and layered compositions

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/image_asset_usage.ipynb)

`ImageAsset` places an uploaded image on an Editor timeline. The snippets below assume `conn`, a base `video`, and the named images already exist in your collection.

## ImageAsset fields

| Field | Python | Node.js | Description |
| :- | :- | :- | :- |
| `id` | `ImageAsset(id=image.id)` | `new EditorImageAsset({ id: image.id })` | Required image identifier |
| `crop` | `crop=Crop(left=0.3)` | `crop: new Crop({ left: 0.3 })` | Optional edge crop |

```python Python theme={null}
from videodb.editor import Crop, ImageAsset

image_asset = ImageAsset(
    id=landscape_image.id,
    crop=Crop(left=0.3),
)
```

```javascript Node.js theme={null}
import { Crop, EditorImageAsset } from 'videodb';

const imageAsset = new EditorImageAsset({
  id: landscapeImage.id,
  crop: new Crop({ left: 0.3 }),
});
```

`Fit`, `Position`, `Offset`, and clip-level controls such as duration, scale, opacity, filters, and transitions belong to the clip. See [Aspect Ratio Control](/pages/act/programmable-editing/aspect-ratio-control) and [Clip Parameters](/pages/act/programmable-editing/clip-parameters) for those controls.

## Add a watermark

Place the base video on one track, then add a later image track for the logo. Later visual tracks render above earlier ones.

```python Python theme={null}
from videodb.editor import Clip, Fit, ImageAsset, Offset, Position, Timeline, Track, VideoAsset

timeline = Timeline(conn)

video_track = Track()
video_track.add_clip(0, Clip(asset=VideoAsset(id=video.id), duration=10))

logo_track = Track()
logo_track.add_clip(
    0,
    Clip(
        asset=ImageAsset(id=logo_image.id),
        duration=10,
        fit=Fit.none,
        position=Position.top_right,
        scale=0.04,
        offset=Offset(x=-0.05, y=0.05),
        opacity=0.8,
    ),
)

timeline.add_track(video_track)
timeline.add_track(logo_track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import {
  Clip, EditorImageAsset, EditorTimeline, EditorVideoAsset,
  Fit, Offset, Position, Track
} from 'videodb';

const timeline = new EditorTimeline(conn);

const videoTrack = new Track();
videoTrack.addClip(0, new Clip({
  asset: new EditorVideoAsset({ id: video.id }),
  duration: 10,
}));

const logoTrack = new Track();
logoTrack.addClip(0, new Clip({
  asset: new EditorImageAsset({ id: logoImage.id }),
  duration: 10,
  fit: Fit.none,
  position: Position.topRight,
  scale: 0.04,
  offset: new Offset({ x: -0.05, y: 0.05 }),
  opacity: 0.8,
}));

timeline.addTrack(videoTrack);
timeline.addTrack(logoTrack);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Build a slideshow

Put still-image clips back to back on one track. A short fade transition softens each handoff.

```python Python theme={null}
from videodb.editor import Clip, ImageAsset, Timeline, Track, Transition

timeline = Timeline(conn)
slides = Track()

for start, image in [(0, landscape_image), (5, portrait_image), (10, square_image)]:
    slides.add_clip(
        start,
        Clip(
            asset=ImageAsset(id=image.id),
            duration=5,
            transition=Transition(in_="fade", out="fade", duration=1),
        ),
    )

timeline.add_track(slides)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorImageAsset, EditorTimeline, Track, Transition } from 'videodb';

const timeline = new EditorTimeline(conn);
const slides = new Track();

[
  [0, landscapeImage],
  [5, portraitImage],
  [10, squareImage],
].forEach(([start, image]) => {
  slides.addClip(start, new Clip({
    asset: new EditorImageAsset({ id: image.id }),
    duration: 5,
    transition: new Transition({ in: 'fade', out: 'fade', duration: 1 }),
  }));
});

timeline.addTrack(slides);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Create a staged zoom

Use consecutive clips of the same image at increasing scales. This is a staged zoom with crossfades, not a continuously animated Ken Burns effect.

```python Python theme={null}
from videodb.editor import Clip, Fit, ImageAsset, Position, Timeline, Track, Transition

timeline = Timeline(conn)
zoom = Track()

zoom_steps = [
    (0, 1.0, Transition(out="fade", duration=0.5)),
    (2.5, 1.3, Transition(in_="fade", out="fade", duration=0.5)),
    (5, 1.6, Transition(in_="fade", duration=0.5)),
]

for start, scale, transition in zoom_steps:
    zoom.add_clip(
        start,
        Clip(
            asset=ImageAsset(id=landscape_image.id),
            duration=3,
            scale=scale,
            fit=Fit.crop,
            position=Position.center,
            transition=transition,
        ),
    )

timeline.add_track(zoom)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import {
  Clip, EditorImageAsset, EditorTimeline,
  Fit, Position, Track, Transition
} from 'videodb';

const timeline = new EditorTimeline(conn);
const zoom = new Track();

const zoomSteps = [
  [0, 1.0, new Transition({ out: 'fade', duration: 0.5 })],
  [2.5, 1.3, new Transition({ in: 'fade', out: 'fade', duration: 0.5 })],
  [5, 1.6, new Transition({ in: 'fade', duration: 0.5 })],
];

zoomSteps.forEach(([start, scale, transition]) => {
  zoom.addClip(start, new Clip({
    asset: new EditorImageAsset({ id: landscapeImage.id }),
    duration: 3,
    scale,
    fit: Fit.crop,
    position: Position.center,
    transition,
  }));
});

timeline.addTrack(zoom);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Layer multiple images

Build an overlapping three-layer composition with a base video, a full-frame image treatment, and a logo. Add the tracks in bottom-to-top order to control which visual appears above the others.

```python Python theme={null}
from videodb.editor import (
    Clip, Filter, Fit, ImageAsset, Offset, Position, Timeline, Track, VideoAsset
)

timeline = Timeline(conn)

video_track = Track()
video_track.add_clip(0, Clip(asset=VideoAsset(id=video.id), duration=10))

overlay_track = Track()
overlay_track.add_clip(
    0,
    Clip(
        asset=ImageAsset(id=square_image.id),
        duration=10,
        opacity=0.3,
        filter=Filter.boost,
    ),
)

logo_track = Track()
logo_track.add_clip(
    0,
    Clip(
        asset=ImageAsset(id=logo_image.id),
        duration=10,
        position=Position.bottom_right,
        fit=Fit.none,
        scale=0.04,
        offset=Offset(x=-0.05, y=-0.05),
        opacity=0.9,
    ),
)

timeline.add_track(video_track)
timeline.add_track(overlay_track)
timeline.add_track(logo_track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import {
  Clip, EditorImageAsset, EditorTimeline, EditorVideoAsset,
  Filter, Fit, Offset, Position, Track
} from 'videodb';

const timeline = new EditorTimeline(conn);

const videoTrack = new Track();
videoTrack.addClip(0, new Clip({
  asset: new EditorVideoAsset({ id: video.id }),
  duration: 10,
}));

const overlayTrack = new Track();
overlayTrack.addClip(0, new Clip({
  asset: new EditorImageAsset({ id: squareImage.id }),
  duration: 10,
  opacity: 0.3,
  filter: Filter.boost,
}));

const logoTrack = new Track();
logoTrack.addClip(0, new Clip({
  asset: new EditorImageAsset({ id: logoImage.id }),
  duration: 10,
  position: Position.bottomRight,
  fit: Fit.none,
  scale: 0.04,
  offset: new Offset({ x: -0.05, y: -0.05 }),
  opacity: 0.9,
}));

timeline.addTrack(videoTrack);
timeline.addTrack(overlayTrack);
timeline.addTrack(logoTrack);
const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

## Next steps

> **[Image Asset Usage Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/image_asset_usage.ipynb)**
>
> Explore image cropping, overlays, transitions, and layered compositions.

> **[Track Layering](/pages/act/programmable-editing/track-layering)**
>
> Control time and visual z-order across tracks.


# Text Asset
Source: https://docs.videodb.io/pages/act/programmable-editing/text-asset

Overlay text elements on videos with customizable fonts, colors, and positioning

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/text_asset_titles_overlays.ipynb)

`TextAsset` creates static text for titles, labels, lower-thirds, and other overlays. It defines the text and its styling; a `Clip` controls how long it appears, while `Track` and `Timeline` place it in the composition.

For speech-synchronized subtitles, use [Caption Asset](/pages/act/programmable-editing/caption-asset) instead.

## Create a text overlay

Start with the smallest useful composition: centered white text on a black canvas.

```python Python theme={null}
from videodb.editor import Clip, Font, TextAsset, Timeline, Track

timeline = Timeline(conn)
timeline.background = "#000000"
timeline.resolution = "1280x720"

text_asset = TextAsset(
    text="Hello World",
    font=Font(family="Clear Sans", size=48, color="#FFFFFF"),
)

track = Track()
track.add_clip(0, Clip(asset=text_asset, duration=5))
timeline.add_track(track)

stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { Clip, EditorTextAsset, EditorTimeline, Font, Track } from 'videodb';

const timeline = new EditorTimeline(conn);
timeline.background = "#000000";
timeline.resolution = "1280x720";

const textAsset = new EditorTextAsset({
    text: "Hello World",
    font: new Font({ family: "Clear Sans", size: 48, color: "#FFFFFF" }),
});

const track = new Track();
track.addClip(0, new Clip({ asset: textAsset, duration: 5 }));
timeline.addTrack(track);

const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

***

## TextAsset Parameters

| Parameter | Type | Description |
| :- | :- | :- |
| `text` | str / string | The text to display |
| `font` | Font | Font styling (family, size, color, opacity) |
| `background` | Background | Background box styling |
| `alignment` | Alignment | Position on screen |
| `border` | Border | Border styling |
| `shadow` | Shadow | Shadow styling |
| `tabsize` | int / number | Tab size for text formatting |
| `line_spacing` / `lineSpacing` | float / number | Spacing between lines in Python / Node.js |
| `width` | int / number | Width of text box in pixels |
| `height` | int / number | Height of text box in pixels |

***

## Font

Controls text appearance.

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `family` | str / string | `"Clear Sans"` | Font family name |
| `size` | int / number | `48` | Font size in pixels |
| `color` | str / string | `"#FFFFFF"` | Text color (hex) |
| `opacity` | float / number | `1.0` | Text opacity (0.0 to 1.0) |

```python Python theme={null}
from videodb.editor import Font

font = Font(
    family="Inter",
    size=56,
    color="#FFFFFF"
)
```

```javascript Node.js theme={null}
const font = new Font({
    family: "Inter",
    size: 56,
    color: "#FFFFFF"
});
```

**Supported fonts:** Sans, Inter, Roboto, Open Sans, Lato, Montserrat, Oswald, Raleway, Poppins, Ubuntu, Clear Sans, Arial, Times New Roman, Courier New, Georgia, Verdana

Node.js also accepts `weight` in `new Font({ ... })`. Python's `Font` constructor does not take that option.

***

## Background

Controls the background box behind text.

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `color` | str / string | `"#000000"` | Background color (hex) |
| `opacity` | float / number | `1.0` | Background opacity (0.0 to 1.0) |
| `width` | float / number | `0` | Box width in pixels |
| `height` | float / number | `0` | Box height in pixels |
| `border_width` / `borderWidth` | float / number | `0` | Border thickness in Python / Node.js |
| `text_alignment` / `textAlignment` | TextAlignment | `center` | Text alignment within box in Python / Node.js |

```python Python theme={null}
from videodb.editor import Background, TextAlignment

background = Background(
    color="#000000",
    width=600,
    height=120,
    border_width=2.0,
    text_alignment=TextAlignment.center
)
```

```javascript Node.js theme={null}
const background = new Background({
    color: "#000000",
    width: 600,
    height: 120,
    borderWidth: 2.0,
    textAlignment: TextAlignment.center
});
```

***

## Border and Shadow

Use `Border(color, width)` and `Shadow(color, x, y)` in Python, or `new Border({ color, width })` and `new Shadow({ color, x, y })` in Node.js. Both are optional TextAsset fields. `width`, `x`, and `y` default to `0`; the default color is `"#000000"`.

***

## Multiline Text

Use `line_spacing` in Python and `lineSpacing` in Node.js to control the space between lines.

```python Python theme={null}
title = TextAsset(
    text="Line One \n Line Two \n Line Three",
    font=Font(family="Clear Sans", size=48, color="#FFFFFF"),
    line_spacing=0.8,
)
```

```javascript Node.js theme={null}
const title = new EditorTextAsset({
    text: "Line One \n Line Two \n Line Three",
    font: new Font({ family: "Clear Sans", size: 48, color: "#FFFFFF" }),
    lineSpacing: 0.8,
});
```

<video>
  <source type="application/x-mpegURL" />
</video>

***

## Alignment

Controls where the text appears on screen.

| Parameter | Type | Description |
| :- | :- | :- |
| `horizontal` | HorizontalAlignment | `left`, `center`, `right` |
| `vertical` | VerticalAlignment | `top`, `center`, `bottom` |

```python Python theme={null}
from videodb.editor import (
    Alignment, Border, Clip, Font, HorizontalAlignment,
    TextAsset, Timeline, Track, VerticalAlignment
)

timeline = Timeline(conn)
timeline.background = "#2B2B2B"
timeline.resolution = "1280x720"

positions = [
    (HorizontalAlignment.left, VerticalAlignment.top, "Top-Left"),
    (HorizontalAlignment.center, VerticalAlignment.top, "Top-Center"),
    (HorizontalAlignment.right, VerticalAlignment.top, "Top-Right"),
    (HorizontalAlignment.left, VerticalAlignment.center, "Mid-Left"),
    (HorizontalAlignment.center, VerticalAlignment.center, "Center"),
    (HorizontalAlignment.right, VerticalAlignment.center, "Mid-Right"),
    (HorizontalAlignment.left, VerticalAlignment.bottom, "Bot-Left"),
    (HorizontalAlignment.center, VerticalAlignment.bottom, "Bot-Center"),
    (HorizontalAlignment.right, VerticalAlignment.bottom, "Bot-Right"),
]

track = Track()
for horizontal, vertical, label in positions:
    text_asset = TextAsset(
        text=label,
        font=Font(family="Clear Sans", size=60, color="#FFD700"),
        border=Border(color="#000000", width=2.0),
        alignment=Alignment(horizontal=horizontal, vertical=vertical),
    )
    track.add_clip(0, Clip(asset=text_asset, duration=5))
timeline.add_track(track)
```

```javascript Node.js theme={null}
import {
    Alignment, Border, Clip, EditorTextAsset, EditorTimeline, Font,
    HorizontalAlignment, Track, VerticalAlignment
} from 'videodb';

const timeline = new EditorTimeline(conn);
timeline.background = "#2B2B2B";
timeline.resolution = "1280x720";

const positions = [
    [HorizontalAlignment.left, VerticalAlignment.top, "Top-Left"],
    [HorizontalAlignment.center, VerticalAlignment.top, "Top-Center"],
    [HorizontalAlignment.right, VerticalAlignment.top, "Top-Right"],
    [HorizontalAlignment.left, VerticalAlignment.center, "Mid-Left"],
    [HorizontalAlignment.center, VerticalAlignment.center, "Center"],
    [HorizontalAlignment.right, VerticalAlignment.center, "Mid-Right"],
    [HorizontalAlignment.left, VerticalAlignment.bottom, "Bot-Left"],
    [HorizontalAlignment.center, VerticalAlignment.bottom, "Bot-Center"],
    [HorizontalAlignment.right, VerticalAlignment.bottom, "Bot-Right"],
];

const track = new Track();
positions.forEach(([horizontal, vertical, label]) => {
    const textAsset = new EditorTextAsset({
        text: label,
        font: new Font({ family: "Clear Sans", size: 60, color: "#FFD700" }),
        border: new Border({ color: "#000000", width: 2.0 }),
        alignment: new Alignment({ horizontal, vertical }),
    });
    track.addClip(0, new Clip({ asset: textAsset, duration: 5 }));
});
timeline.addTrack(track);
```

<video>
  <source type="application/x-mpegURL" />
</video>

***

## Branded Lower-Third

```python Python theme={null}
from videodb.editor import (
    Timeline, Track, Clip, VideoAsset, TextAsset,
    Font, Background, Border, Shadow, Alignment,
    HorizontalAlignment, VerticalAlignment, Offset, Transition
)

video_clip = Clip(
    asset=VideoAsset(id=video.id, start=0, volume=0),
    duration=5,
)

lower_third = TextAsset(
    text="Dr. Sarah Chen: Tech Officer",
    font=Font(family="Clear Sans", size=36, color="#FFFFFF", opacity=1.0),
    background=Background(
        width=600,
        height=120,
        color="#1E88E5",
        opacity=0.95,
    ),
    border=Border(color="#FFC107", width=4.0),
    shadow=Shadow(color="#000000", x=3, y=3),
    alignment=Alignment(
        horizontal=HorizontalAlignment.left,
        vertical=VerticalAlignment.bottom,
    ),
)

lower_third_clip = Clip(
    asset=lower_third,
    duration=6,
    offset=Offset(x=0.03, y=-0.05),
    transition=Transition(in_="fade", out="fade", duration=2),
)

timeline = Timeline(conn)
track = Track()
track.add_clip(0, video_clip)
track.add_clip(0, lower_third_clip)

timeline.add_track(track)

stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import {
    EditorTimeline, Track, Clip, EditorVideoAsset, EditorTextAsset,
    Font, Background, Border, Shadow, Alignment,
    HorizontalAlignment, VerticalAlignment, Offset, Transition
} from 'videodb';

const videoClip = new Clip({
    asset: new EditorVideoAsset({ id: video.id, start: 0, volume: 0 }),
    duration: 5,
});

const lowerThird = new EditorTextAsset({
    text: "Dr. Sarah Chen: Tech Officer",
    font: new Font({ family: "Clear Sans", size: 36, color: "#FFFFFF", opacity: 1.0 }),
    background: new Background({
        width: 600,
        height: 120,
        color: "#1E88E5",
        opacity: 0.95,
    }),
    border: new Border({ color: "#FFC107", width: 4.0 }),
    shadow: new Shadow({ color: "#000000", x: 3, y: 3 }),
    alignment: new Alignment({
        horizontal: HorizontalAlignment.left,
        vertical: VerticalAlignment.bottom,
    }),
});

const lowerThirdClip = new Clip({
    asset: lowerThird,
    duration: 6,
    offset: new Offset({ x: 0.03, y: -0.05 }),
    transition: new Transition({ in: "fade", out: "fade", duration: 2 }),
});

const timeline = new EditorTimeline(conn);
const track = new Track();
track.addClip(0, videoClip);
track.addClip(0, lowerThirdClip);
timeline.addTrack(track);

const streamUrl = await timeline.generateStream();
```

<video>
  <source type="application/x-mpegURL" />
</video>

***

## Next Steps

> **[Text Asset Titles and Overlays Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/text_asset_titles_overlays.ipynb)**
>
> Build titles, lower-thirds, multiline overlays, and text styling examples.

> **[CaptionAsset](/pages/act/programmable-editing/caption-asset)**
>
> Auto-generated subtitles synced to speech

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Complete guide to the 4-layer editing system


# Timeline Architecture
Source: https://docs.videodb.io/pages/act/programmable-editing/timeline-architecture

Imagine building videos like coding - declarative, composable, and infinitely reusable. A comprehensive guide to VideoDB's code-first video editing architecture.

VideoDB Editor lets you create videos programmatically using code instead of clicking timelines. You define what you want (assets, effects, timing), and the engine handles the rendering.

This guide is your complete conceptual introduction. By the end, you’ll understand how to compose anything from simple clips to complex multi-layer productions - all through code.

## Why Code-First Video Editing?

Traditional video editors are built for one-off productions. But what if you need to:

* Generate 100 personalized videos from a template
* Build a TikTok content pipeline that runs daily
* Create video variations for A/B testing
* Automate highlight reels from live streams

**Code changes everything:**

* **Reusability** – One video asset, infinite variations
* **Scalability** – Loop over data to generate hundreds of videos
* **Version control** – Git-track your compositions
* **Automation** – Integrate with AI, databases, APIs

## The 4-Layer Architecture

VideoDB Editor uses a hierarchy where each layer has one job. Understanding this structure is the key to mastering composition:

**Asset → Clip → Track → Timeline**

![Timeline architecture overview showing the 4-layer hierarchy: Asset, Clip, Track, and Timeline](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/timeline-architecture.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=f5bb2dfed99b8ec5aaffc483fb40d8eb)

Let's walk through each layer using the simplest possible example: **one video asset playing for 10 seconds**. This is the "Hello World" of Editor - understanding this foundation lets you build anything.

### Installing `VideoDB` in your environment

`VideoDB` is available as a [Python package](https://pypi.org/project/videodb) and [Node.js package](https://www.npmjs.com/package/videodb).

```bash Python theme={null}
pip install videodb
```

```bash Node.js theme={null}
npm install videodb
```

## Layer 1: Assets – Your Raw Materials

**Assets are your content library.** They reference media that exists in your VideoDB collection but don’t define how or when it plays.

### VideoAsset

Your main video content. Each `VideoAsset` points to a video file via its ID.

**Key parameters:**

* `id` (required) – The VideoDB media ID
* `start` (optional) – Trim point in seconds (e.g., `start=10` skips first 10s of source)
* `volume` (optional) – Audio level: `0.0` (muted) to `2.0` (200%), default `1.0`
* `crop` (optional) – Crop the sides of an asset by a relative amount. The size of the crop is specified using a scale between `0` and `1`. A left crop of `0.5` will crop half of the asset from the left, a top crop of `0.25` will crop the top by a quarter of the asset.

Real example:

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

video_asset = VideoAsset(
  # Create a VideoAsset pointing to a video file in your collection
    id=video.id,
    start=0,
    volume=1
)
# Ready to use in a Clip
```

```javascript Node.js theme={null}
import { EditorVideoAsset } from 'videodb';

const videoAsset = new EditorVideoAsset({
  // Create a VideoAsset pointing to a video file in your collection
  id: video.id,
  start: 0,
  volume: 1
});
// Ready to use in a Clip
```

This says: "Use the video from your VideoDB collection, start from the beginning (`start=0`), and keep original volume (`volume=1`)."

**Important distinction:** `VideoAsset.start` trims the *source file*. Where it appears on the timeline is controlled later at the Track layer. This "double start" concept is critical - we'll explore it more in Layer 3 (Tracks).

### AudioAsset

Background music, voiceovers, or sound effects. Works exactly like `VideoAsset`.

**Key parameters:**

* `id` (required) – The VideoDB audio file ID
* `start` (optional) – Same trim behavior as VideoAsset
* `volume` (optional) – `0.0`-`2.0` range (0.2 = 20% volume)

### ImageAsset

Logos, watermarks, title cards, or static backgrounds.

**Key parameters:**

* `id` (required) – The VideoDB image ID

Images are static by nature - duration, position, and size are controlled at the [Clip layer](/pages/act/programmable-editing/clip-parameters).

### TextAsset

Custom text overlays with full typography control.

**Key parameters:**

* `text` (required) – The string to display
* `font` (optional) – Font object with family, size, color, opacity
* `border`, `shadow`, `background` (optional) – Styling objects
* `alignment` (optional) – Position on screen
* `tabsize` (optional) – Tab size for text formatting
* `line_spacing` (optional) – Spacing between lines
* `width` (optional) – Width of text box in pixels
* `height` (optional) – Height of text box in pixels

**Color format:** ASS-style `&HAABBGGRR` in hex (e.g., `&H00FFFFFF` = white)

![TextAsset example showing custom text overlays with typography control](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/text-asset.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=cbe4ddad200610a0393af4ee7e4ae4ef)

### CaptionAsset

Auto-generated subtitles synced to speech. **This is where VideoDB gets magical.**

**Important:** `CaptionAsset` is a separate asset type from `TextAsset`. While `TextAsset` is for custom text overlays you write yourself, `CaptionAsset` automatically generates subtitles from video speech.

**Key parameters:**

* `src` (required) – Set to `"auto"` to generate captions from video speech
* `animation` (optional) – How words appear: `reveal`, `karaoke`, `supersize`, `box_highlight`, `impact`, `color_highlight`
* `primary_color`, `secondary_color` (optional) – ASS-style colors
* `font`, positioning, border, shadow styling (optional)

For current timed captions, build an SRT-backed `CaptionAsset` with the [Track Layering guide](/pages/act/programmable-editing/track-layering).

![CaptionAsset example showing auto-generated subtitles synced to speech with animation](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/caption-asset.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=0cd064beaabfca4c9eb101049a866c27)

### Supported Fonts for Text and Caption Assets

![Chart showing all supported fonts for text and caption assets](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/reference/supported-fonts.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=a97596482d63a7c6b3ad294bba6f2363)

**Supported Indic fonts:**

* Noto Sans Kannada
* Noto Sans Devanagari
* Noto Sans Gujarati
* Noto Sans Gurmukhi

**Recap:** Assets answer "What content exists?" They don't yet define timing, size, position, or effects. That's the Clip layer's job.

## Layer 2: Clips – The Presentation Engine

**Clips wrap Assets and define** ***how*** **and** ***how long*** **they appear.** This is your effects layer.

![Clip layer showing how assets are wrapped with effects, positioning, and duration](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/clip.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=6b93fe5bfbfd91ffdc034d0805404fc8)

Every Clip must have an `asset` and a `duration`. Everything else is optional.

### Duration – How Long It Plays

`duration` is a float in seconds. It defines how long the clip plays on the timeline.

Real example:

```python Python theme={null}
from videodb.editor import Clip
clip = Clip(
  asset=video_asset,
  duration=10
)
```

```javascript Node.js theme={null}
import { Clip } from 'videodb';
const clip = new Clip({
  asset: videoAsset,
  duration: 10
});
```

"Play this VideoAsset for 10 seconds."

**Key insight:** Duration is *independent* of the source file's length. If your source is 2 minutes but you set `duration=10`, only 10 seconds play (starting from `VideoAsset.start`).

### Fit – How It Scales to Canvas

When your asset’s aspect ratio doesn’t match the timeline’s, `fit` controls scaling behavior.

**Four modes:**

* `Fit.crop` (most common) – Fills the canvas completely, cropping edges if needed
  * Use when: Filling the frame is priority, cropping is acceptable
  * Example: 16:9 video on a 9:16 (vertical) timeline
* `Fit.contain` – Fits the entire asset inside the canvas, adding bars if needed
  * Use when: Showing all content is priority, bars are acceptable
  * Example: Preserving widescreen footage in a square format
* `Fit.cover` – Stretches to fill canvas (distortion possible)
  * Use when: Artistic effect or abstract content
* `Fit.none` – Uses native pixel dimensions (no scaling)
  * Use when: Precise pixel control needed (e.g., 1:1 pixel mapping)

Real example:

```python Python theme={null}
clip = Clip(
  asset=video_asset,
  duration=10,
  fit=Fit.crop
)
```

```javascript Node.js theme={null}
import { Fit } from 'videodb';
const clip = new Clip({
  asset: videoAsset,
  duration: 10,
  fit: Fit.crop
});
```

"Fill the canvas completely, crop edges if aspect ratios don't match."

### Position – Where It Appears

Position uses a 9-zone grid system:

```
top_left      top        top_right
center_left   center     center_right
bottom_left   bottom     bottom_right
```

![Position 9-zone grid system showing placement options for clips](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/position.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=a1008a7a0455f359e7860d9333e6a60b)

Real example:

```python Python theme={null}
logo_clip = Clip(
  asset=logo,
  duration=30,
  position=Position.top_right
)
```

```javascript Node.js theme={null}
import { Position } from 'videodb';
const logoClip = new Clip({
  asset: logo,
  duration: 30,
  position: Position.topRight
});
```

"Place the logo in the top-right corner."

### Offset – For *fine-tuned* positioning

```python Python theme={null}
from videodb.editor import Offset

clip = Clip(
    asset=logo,
    duration=30,
    position=Position.center,
    offset=Offset(x=0.3, y=-0.2)
)
```

```javascript Node.js theme={null}
import { Offset } from 'videodb';

const clip = new Clip({
  asset: logo,
  duration: 30,
  position: Position.center,
  offset: new Offset({ x: 0.3, y: -0.2 })
});
```

This shifts the logo 30% right, 20% up from center.

![Offset fine-tuning positioning with x and y adjustments](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/offset.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=6a5a4fdd5612905f3ab2babb928a87c1)

### Scale – Size Adjustment

`scale` is a multiplier applied *after* fit. Default is `1.0`.

Real example:

```python Python theme={null}
pip_clip = Clip(
  asset=overlay_video,
  duration=15,
  scale=0.3
)
```

```javascript Node.js theme={null}
const pipClip = new Clip({
  asset: overlayVideo,
  duration: 15,
  scale: 0.3
});
```

"Shrink this video to 30% of its fitted size" (perfect for picture-in-picture).

### Opacity – Transparency

`opacity` ranges from `0.0` (invisible) to `1.0` (opaque).

Real example:

```python Python theme={null}
watermark_clip = Clip(
  asset=logo,
  duration=30,
  opacity=0.6
)
```

```javascript Node.js theme={null}
const watermarkClip = new Clip({
  asset: logo,
  duration: 30,
  opacity: 0.6
});
```

"Make the logo 60% opaque (semi-transparent)."

### Filter – Visual Effects

Apply color/blur effects:

```python Python theme={null}
from videodb.editor import Filter

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    filter=Filter.greyscale
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Filter } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    filter: Filter.greyscale
});
```

Available filters: `greyscale`, `blur`, `boost` (saturation), `contrast`, `darken`, `lighten`, `muted`, `negative`.

| Filter | Effect |
| :- | :- |
| `Filter.greyscale` | Removes all color, creating a black-and-white look |
| `Filter.blur` | Blurs the scene for artistic or privacy effects |
| `Filter.contrast` | Increases contrast, making darks darker and lights lighter |
| `Filter.darken` | Darkens the entire scene |
| `Filter.lighten` | Lightens the entire scene |
| `Filter.boost` | Boosts both contrast and saturation for vibrant colors |
| `Filter.muted` | Reduces saturation and contrast for a subdued look |
| `Filter.negative` | Inverts colors for a surreal, negative effect |

### Transition

Apply a transition effect at the start (`in_`) and/or end (`out`) of a clip:

```python Python theme={null}
from videodb.editor import Transition

clip = Clip(
    asset=VideoAsset(id=video.id),
    duration=10,
    transition=Transition(
      in_="fade",
      out="fade",
      duration=2
    )
)
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Transition } from 'videodb';

const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id }),
    duration: 10,
    transition: new Transition({
      in: "fade",
      out: "fade",
      duration: 2
    })
});
```

**Available transition types:**

| Value | Description |
| :- | :- |
| `fade` | Fade to/from black |
| `reveal` | Slide reveal from edge |
| `shuffle` | Shuffle from center |
| `shuffle_top_right` | Shuffle from top-right corner |
| `shuffle_bottom_right` | Shuffle from bottom-right corner |
| `shuffle_bottom_left` | Shuffle from bottom-left corner |
| `shuffle_top_left` | Shuffle from top-left corner |

**Recap:** A Clip wraps an Asset and defines *how long* it plays (`duration`) and *how* it appears (fit, position, scale, opacity, filter, transition). Now let’s see how to place clips on the timeline.

## Layer 3: Tracks – Sequencing and Layering

**Tracks are timeline lanes.** They control *when* clips play and *how* they stack.

![Track layer showing timeline lanes for sequencing and layering clips](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/track.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=c81305835b4a430f51ef351c004c49aa)

### The Track Object

A Track is a container you add clips to:

```python Python theme={null}
from videodb.editor import Track

track = Track()
track.add_clip(0, clip)  # Add clip at 0 seconds
```

```javascript Node.js theme={null}
import { Track } from 'videodb';

const track = new Track();
track.addClip(0, clip);  // Add clip at 0 seconds
```

`track.add_clip(start, clip)` has two parameters:

* `start` (float, seconds) – When the clip begins on the timeline
* `clip` (Clip object) – The clip to add

### Sequential Playback (Same Track)

Clips on the *same track* play **one after another**:

```python Python theme={null}
track = Track()
track.add_clip(0, clip1)    # 0s-5s
track.add_clip(5, clip2)    # 5s-10s
track.add_clip(10, clip3)   # 10s-15s
```

```javascript Node.js theme={null}
const track = new Track();
track.addClip(0, clip1);    // 0s-5s
track.addClip(5, clip2);    // 5s-10s
track.addClip(10, clip3);   // 10s-15s
```

This creates a montage - three clips in sequence.

### Simultaneous Playback (Different Tracks)

Clips on *different tracks* at the same timestamp play **simultaneously**:

![Simultaneous playback showing clips on different tracks playing at the same time](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/simultaneous-playback.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=a047d7c10ad426070c4b4e784b958af4)

```python Python theme={null}
track1 = Track()
track1.add_clip(0, clip1)  # First layer

track2 = Track()
track2.add_clip(0, clip2)  # Second layer (plays at same time)
```

```javascript Node.js theme={null}
const track1 = new Track();
track1.addClip(0, clip1);  // First layer

const track2 = new Track();
track2.addClip(0, clip2);  // Second layer (plays at same time)
```

Both start at 0 seconds, so they play together. This is how you create layered compositions.

### Z-Order (Layering)

**Later tracks render** ***on top*** **of earlier tracks.**

![Z-order layering showing tracks rendered on top of each other](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/z-order.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=e6264aa8b7fcb477d9a0b229f766bded)

```python Python theme={null}
timeline.add_track(track1)  # Bottom layer
timeline.add_track(track2)  # Renders above track1
timeline.add_track(track3)  # Renders above track2
```

```javascript Node.js theme={null}
timeline.addTrack(track1);  // Bottom layer
timeline.addTrack(track2);  // Renders above track1
timeline.addTrack(track3);  // Renders above track2
```

This is how you create overlays: put background content on track1, overlays on track2.

### The “Double Start” Concept

There are **two separate “start” parameters**:

* `Asset.start` – Trims the *source file*
* `track.add_clip(start=...)` – Places clip on the *timeline*

Real example:

```python Python theme={null}
# Source video is 2 minutes long

video_asset = VideoAsset(
  id=video.id,
  start=30
)  # Skip first 30s of source

clip = Clip(
  asset=video_asset,
  duration=40
)  # Use 40s (from 0:30 to 1:10 of source)

track = Track()
track.add_clip(5, clip)  # Place it at 5-second mark on timeline
```

```javascript Node.js theme={null}
// Source video is 2 minutes long

const videoAsset = new EditorVideoAsset({
  id: video.id,
  start: 30
});  // Skip first 30s of source

const clip = new Clip({
  asset: videoAsset,
  duration: 40
});  // Use 40s (from 0:30 to 1:10 of source)

const track = new Track();
track.addClip(5, clip);  // Place it at 5-second mark on timeline
```

Result: The timeline plays seconds 0:30-01:10 of the source video, but it appears at the 5-second mark of the final output.

**Why this matters:** You can extract any segment from source media and place it anywhere on the timeline, independently.

*For multi-track layering examples (video + music + captions + overlays), see the Advanced Clip Control guide and creative tutorials.*

## Layer 4: Timeline – The Final Canvas

**Timeline is your export settings.** It defines resolution, background color, and combines all tracks.

![Timeline layer showing the final canvas combining all tracks and export settings](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/timeline.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=3313e9c34e78639a3b18593e4a0f707e)

```python Python theme={null}
from videodb.editor import Timeline

timeline = Timeline(conn)  # conn is your VideoDB connection
timeline.background = "#808080"  # Grey background (hex color)
timeline.resolution = "600x1060"  # Custom resolution
```

```javascript Node.js theme={null}
import { EditorTimeline } from 'videodb';

const timeline = new EditorTimeline(conn);  // conn is your VideoDB connection
timeline.background = "#808080";  // Grey background (hex color)
timeline.resolution = "600x1060";  // Custom resolution
```

### Resolution

Format: `"WIDTHxHEIGHT"`

Common presets:

* `"1280x720"` – 16:9 horizontal (YouTube, landscape)
* `"608x1080"` – 9:16 vertical (TikTok, Shorts, Reels)
* `"1080x1080"` – 1:1 square (Instagram feed)
* `"600x1060"` – Custom dimensions

### Background

The color shown behind/around clips when they don't fill the canvas (e.g., when using `Fit.contain`). Format: hex color string.

![Background color shown behind clips when they don't fill the entire canvas](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/background.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=35bd70e5ab293db223f6e245c03d1be4)

### Adding Tracks

```python Python theme={null}
timeline.add_track(track)
```

```javascript Node.js theme={null}
timeline.addTrack(track);
```

For multiple tracks, order matters - this sets the z-order (layering). Later tracks render on top.

### Rendering

```python Python theme={null}
stream_url = timeline.generate_stream()
print(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await timeline.generateStream();
console.log(streamUrl);
```

This sends your composition to VideoDB's rendering engine and returns a playable stream URL.

### Complete example:

![Complete timeline example showing all layers working together](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/editor-sdk/complete-timeline.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=3eb639d70ca7f64735372859bb737465)

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

# Create timeline
timeline = Timeline(conn)
timeline.background = "#FFA629"
timeline.resolution = "600x1060"

# Create asset
video_asset = VideoAsset(id=video.id, start=0, volume=1)

# Wrap in clip
clip = Clip(asset=video_asset, duration=10)

# Add to track
track = Track()
track.add_clip(0, clip)

# Add track to timeline
timeline.add_track(track)

# Render
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset } from 'videodb';

// Create timeline
const timeline = new EditorTimeline(conn);
timeline.background = "#FFA629";
timeline.resolution = "600x1060";

// Create asset
const videoAsset = new EditorVideoAsset({ id: video.id, start: 0, volume: 1 });

// Wrap in clip
const clip = new Clip({ asset: videoAsset, duration: 10 });

// Add to track
const track = new Track();
track.addClip(0, clip);

// Add track to timeline
timeline.addTrack(track);

// Render
const streamUrl = await timeline.generateStream();
```

You've just composed your first video programmatically: one video asset playing for 10 seconds. This simple pattern scales to any complexity - just add more assets, clips, and tracks.

### Concept Guides (Detailed Explanations)

These guides expand on specific concepts with design principles, edge cases, and best practices:

> **[Fit and Position](/pages/act/programmable-editing/aspect-ratio-control)**
>
> Deep dive into aspect ratios, 9-zone positioning, offset mechanics, and framing patterns

> **[Trimming vs Timing](/pages/act/programmable-editing/trimming-vs-timing)**
>
> Complete explanation of the "double start" concept with formulas and multi-clip workflows

> **[Caption Asset](/pages/act/programmable-editing/caption-asset)**
>
> Animation styles, ASS color format, positioning, accessibility, and styling best practices

> **[Clip Parameters](/pages/act/programmable-editing/clip-parameters)**
>
> Filters, transitions, opacity patterns, and complex multi-layer compositions

## What You Can Build

See real-world applications of timeline composition and programmatic video creation:

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Transform music into viral vertical clips with AI backgrounds and synced lyrics

> **[Video Statistics Recaps](/examples-and-tutorials/content-factory/year-in-frames)**
>
> Turn analytics into cinematic recaps with dynamic animations

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Build complete faceless videos with AI scripts and voiceovers

> **[Chess Match Montages](/examples-and-tutorials/programmatic-editing/chess-montage)**
>
> Automatically extract highlights with AI move detection

## Get Started

> **[Timeline Basics](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/timeline_basic.ipynb)**
>
> See all concepts in action with hands-on examples and interactive code.


# Track Layering
Source: https://docs.videodb.io/pages/act/programmable-editing/track-layering

Control sequencing, simultaneous playback, and z-order across video, audio, image, text, and caption clips

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/tracks_layering_model.ipynb)

Tracks control two dimensions of an Editor composition:

* **Time:** A clip's start value determines when it plays.
* **Layering:** The order in which visual tracks are added determines what appears on top.

Each track can contain multiple clips. Use one track for a simple sequence or multiple tracks to organize video, audio, graphics, text, and captions.

## Place Clips in Time

The first argument to `add_clip()` or `addClip()` is the clip's start time in the final timeline.

```python Python theme={null}
from videodb.editor import Track

track = Track()
track.add_clip(0, intro_clip)    # Starts at 0s
track.add_clip(4, main_clip)     # Starts at 4s
track.add_clip(24, outro_clip)   # Starts at 24s
```

```javascript Node.js theme={null}
import { Track } from 'videodb';

const track = new Track();
track.addClip(0, introClip);     // Starts at 0s
track.addClip(4, mainClip);      // Starts at 4s
track.addClip(24, outroClip);    // Starts at 24s
```

Clip start times are explicit. They can create sequential playback, gaps, or overlaps.

## Sequence, Gap, or Overlap

Assume each clip is five seconds long:

| Placement | Result |
| :- | :- |
| Starts at `0`, `5`, and `10` | Back-to-back sequence |
| Starts at `0`, `7`, and `14` | Two-second gaps |
| Starts at `0`, `3`, and `6` | Two-second overlaps |

Audio clips that overlap are mixed. Overlapping visual clips are composed according to track and clip layering.

## Visual Z-Order

Tracks added later render above earlier visual tracks.

```python Python theme={null}
timeline.add_track(video_track)  # Base layer
timeline.add_track(image_track)  # Above video
timeline.add_track(text_track)   # Above image
timeline.add_track(caption_track)  # Top layer
```

```javascript Node.js theme={null}
timeline.addTrack(videoTrack);    // Base layer
timeline.addTrack(imageTrack);    // Above video
timeline.addTrack(textTrack);     // Above image
timeline.addTrack(captionTrack);  // Top layer
```

> [!WARNING]
>
> If a full-frame image is added after a text track, it can cover the text. Add background content first and overlays afterward.

## Build a Multi-Layer Composition

The following pattern combines a muted base video, background music, a logo, and a title.

```python Python theme={null}
from videodb.editor import (
    Timeline, Track, Clip,
    VideoAsset, AudioAsset, ImageAsset, TextAsset,
    Font, Background, Alignment,
    HorizontalAlignment, VerticalAlignment,
    Fit, Position, Offset
)

timeline = Timeline(conn)
timeline.background = "#000000"
timeline.resolution = "1280x720"

# Layer 1: base video
video_track = Track()
video_track.add_clip(
    0,
    Clip(
        asset=VideoAsset(id=video.id, volume=0),
        duration=15,
    ),
)

# Layer 2: background music
audio_track = Track()
audio_track.add_clip(
    0,
    Clip(
        asset=AudioAsset(id=music.id, volume=0.2),
        duration=15,
    ),
)

# Layer 3: logo
image_track = Track()
image_track.add_clip(
    0,
    Clip(
        asset=ImageAsset(id=logo.id),
        duration=15,
        fit=Fit.none,
        position=Position.top_right,
        scale=0.12,
        offset=Offset(x=-0.04, y=0.04),
    ),
)

# Layer 4: title
title = TextAsset(
    text="Product Update",
    font=Font(size=52, color="#FFFFFF"),
    background=Background(color="#111827", opacity=0.8),
    alignment=Alignment(
        horizontal=HorizontalAlignment.center,
        vertical=VerticalAlignment.top,
    ),
)
text_track = Track()
text_track.add_clip(1, Clip(asset=title, duration=5))

timeline.add_track(video_track)
timeline.add_track(audio_track)
timeline.add_track(image_track)
timeline.add_track(text_track)

stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import {
  EditorTimeline, Track, Clip,
  EditorVideoAsset, EditorAudioAsset, EditorImageAsset, EditorTextAsset,
  Font, Background, Alignment,
  HorizontalAlignment, VerticalAlignment,
  Fit, Position, Offset
} from 'videodb';

const timeline = new EditorTimeline(conn);
timeline.background = '#000000';
timeline.resolution = '1280x720';

const videoTrack = new Track();
videoTrack.addClip(
  0,
  new Clip({
    asset: new EditorVideoAsset({ id: video.id, volume: 0 }),
    duration: 15
  })
);

const audioTrack = new Track();
audioTrack.addClip(
  0,
  new Clip({
    asset: new EditorAudioAsset({ id: music.id, volume: 0.2 }),
    duration: 15
  })
);

const imageTrack = new Track();
const logoClip = new Clip({
  asset: new EditorImageAsset({ id: logo.id }),
  duration: 15,
  position: Position.topRight,
  scale: 0.12,
  offset: new Offset({ x: -0.04, y: 0.04 })
});
logoClip.fit = Fit.none;
imageTrack.addClip(0, logoClip);

const title = new EditorTextAsset({
  text: 'Product Update',
  font: new Font({ size: 52, color: '#FFFFFF' }),
  background: new Background({ color: '#111827', opacity: 0.8 }),
  alignment: new Alignment({
    horizontal: HorizontalAlignment.center,
    vertical: VerticalAlignment.top
  })
});
const textTrack = new Track();
textTrack.addClip(1, new Clip({ asset: title, duration: 5 }));

timeline.addTrack(videoTrack);
timeline.addTrack(audioTrack);
timeline.addTrack(imageTrack);
timeline.addTrack(textTrack);

const streamUrl = await timeline.generateStream();
```

## Add Timed Captions as the Top Layer

Create a timed spoken-word artifact from the source video, then use those word timings to build an explicit subtitle source for the caption clip.

```python Python theme={null}
from videodb.editor import (
    Track,
    Clip,
    CaptionAsset,
    CaptionAnimation,
    Positioning,
    CaptionAlignment,
    FontStyling,
)

# Create a timed spoken-word artifact for the caption track
caption_understanding = video.understand(
    analyzers=[
        {
            "type": "spoken_words",
            "name": "caption_words",
        }
    ]
)
caption_understanding.wait_until_complete()

caption_analyzer = caption_understanding.get_analyzer("caption_words")
caption_output = caption_analyzer.get_output()

caption_words = [
    {
        "text": word["text"],
        "start": word["start"],
        "end": word["end"],
    }
    for scene in caption_output["scenes"]
    for word in scene["data"]["words"]
]

caption_words.sort(key=lambda word: (word["start"], word["end"]))

print(f"✓ Timed caption words ready: {len(caption_words)}")

import base64

source_start = 0
source_end = 15

def format_srt_time(seconds):
    total_ms = max(0, int(round(float(seconds) * 1000)))
    hours, remainder = divmod(total_ms, 3_600_000)
    minutes, remainder = divmod(remainder, 60_000)
    secs, milliseconds = divmod(remainder, 1000)
    return f"{hours:02}:{minutes:02}:{secs:02},{milliseconds:03}"

def build_caption_cues(words, interval_start, interval_end, max_words=8):
    selected_words = []
    for word in words:
        if word["end"] <= interval_start or word["start"] >= interval_end:
            continue

        text = " ".join(word["text"].split())
        start = max(word["start"], interval_start)
        end = min(word["end"], interval_end)
        if text and end > start:
            selected_words.append({"text": text, "start": start, "end": end})

    cues = []
    group = []
    for word in selected_words:
        group.append(word)
        closes_sentence = word["text"].endswith((".", "!", "?"))
        if len(group) >= max_words or closes_sentence:
            cues.append(
                {
                    "start": group[0]["start"] - interval_start,
                    "end": group[-1]["end"] - interval_start,
                    "text": " ".join(item["text"] for item in group),
                }
            )
            group = []

    if group:
        cues.append(
            {
                "start": group[0]["start"] - interval_start,
                "end": group[-1]["end"] - interval_start,
                "text": " ".join(item["text"] for item in group),
            }
        )

    return [cue for cue in cues if cue["end"] > cue["start"]]

caption_cues = build_caption_cues(caption_words, source_start, source_end)
newline = chr(10)
caption_srt = newline.join(
    f"{number}{newline}"
    f"{format_srt_time(cue['start'])} --> {format_srt_time(cue['end'])}{newline}"
    f"{cue['text']}{newline}"
    for number, cue in enumerate(caption_cues, start=1)
)
caption_srt_base64 = base64.b64encode(
    caption_srt.encode("utf-8")
).decode("utf-8")

# Create the caption clip from the explicit timed subtitle source
caption_asset = CaptionAsset(
    src=caption_srt_base64,
    animation=CaptionAnimation.supersize,
    primary_color="&H00FFFFFF",  # White text (ASS format)
    secondary_color="&H0000D7FF",  # Orange for emphasized text
    position=Positioning(
        alignment=CaptionAlignment.bottom_center
    ),
    font=FontStyling(
        bold=True,
        size=32
    )
)

caption_clip = Clip(
    asset=caption_asset,
    duration=15
)

# Add to a new track
caption_track = Track()
caption_track.add_clip(0, caption_clip)

# Add this track to the timeline (topmost layer)
timeline.add_track(caption_track)

print("✓ Layer 5 added: Timed captions (bottom-center)")
```

## The Two Start Values

Source trimming and timeline placement are independent:

```python Python theme={null}
from videodb.editor import Track, Clip, VideoAsset

track = Track()
video_clip = Clip(
    asset=VideoAsset(
        id=video.id,
        start=10,  # Skip 10s in the source
    ),
    duration=8,
)

track.add_clip(3, video_clip)  # Show it at 3s in the output
```

```javascript Node.js theme={null}
import { Track, Clip, EditorVideoAsset } from 'videodb';

const track = new Track();
const videoClip = new Clip({
  asset: new EditorVideoAsset({
    id: video.id,
    start: 10  // Skip 10s in the source
  }),
  duration: 8
});

track.addClip(3, videoClip);   // Show it at 3s in the output
```

The output remains blank or shows lower layers for the first three seconds. At three seconds, the clip appears using media from the ten-second mark of the source.

## Track Organization Patterns

* Keep a base video sequence on the first visual track.
* Put persistent branding on a later image or text track.
* Use dedicated tracks for captions and timed callouts.
* Separate audio by role when the composition has several music, voice, or effect layers.
* Add tracks to the timeline in deliberate bottom-to-top order.

## Next Steps

> **[Track Layering Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/tracks_layering_model.ipynb)**
>
> Build a complete video, audio, image, text, and caption composition and compare z-order changes.

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Understand how assets, clips, tracks, and timelines fit together.


# Trimming Vs Timing
Source: https://docs.videodb.io/pages/act/programmable-editing/trimming-vs-timing

VideoDB Editor uses two separate start parameters that control different aspects of video composition. Understanding these parameters is key to controlling what plays and when.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/trimming_vs_timing.ipynb)

VideoDB Editor uses two separate "start" parameters that control different aspects of video composition. This trips up almost everyone at first - you set start=10 and expect the video to play 10 seconds into your timeline, but it actually just skips the first 10 seconds of the source video. Understanding these two parameters is key to controlling both what plays and when it plays.

## The Two Start Parameters

### `Asset.start` - Source Trimming

`VideoAsset(start=…)` skips the beginning of the source file.

```python Python theme={null}
asset = VideoAsset(
    id=video.id,
    start=10  # Skip first 10 seconds of source video
)
```

```javascript Node.js theme={null}
const asset = new VideoAsset(video.id, {
    start: 10  // Skip first 10 seconds of source video
});
```

<video>
  <source type="application/x-mpegURL" />
</video>

This controls which part of your source material gets used. Think of it like fast-forwarding to a specific timestamp before hitting play - you’re extracting a segment from your original video.

### `track.add_clip(start=…)` - Timeline Positioning

Positions when the clip appears in the final timeline.

```python Python theme={null}
track.add_clip(
    start=5,  # Place clip at 5-second mark in final video
    clip=clip
)
```

```javascript Node.js theme={null}
track.addClip(
    5,    // Place clip at 5-second mark in final video
    clip
);
```

<video>
  <source type="application/x-mpegURL" />
</video>

This controls when your clip plays in the composed output - like dragging a clip to a specific position on an editing timeline. It’s completely independent of what part of the source is playing.

## Key Concept: Independent Control

These parameters are completely independent, operating in two different coordinate systems.

`Asset.start` works in "source video time" (which part of the original file), while `track.add_clip(start=…)` works in "output timeline time" (when it appears in the final video).

You can extract any segment from your source and place it anywhere on your timeline.

```python Python theme={null}
# Extract seconds 10-20 from source, place at 5-second mark in timeline
clip = Clip(
    asset=VideoAsset(id=video.id, start=10),  # TRIMMING: skip first 10s
    duration=10  # Play for 10 seconds
)

track = Track()
track.add_clip(start=5, clip=clip)  # TIMING: appears at 5s in final
```

```javascript Node.js theme={null}
import { Clip, EditorVideoAsset, Track } from 'videodb';

// Extract seconds 10-20 from source, place at 5-second mark in timeline
const clip = new Clip({
    asset: new EditorVideoAsset({ id: video.id, start: 10 }),  // TRIMMING: skip first 10s
    duration: 10  // Play for 10 seconds
});

const track = new Track();
track.addClip(5, clip);  // TIMING: appears at 5s in final
```

<video>
  <source type="application/x-mpegURL" />
</video>

Timeline structure:

* 0-5 seconds: Blank (background color)
* 5-15 seconds: Video plays (showing seconds 10-20 from source)

## Asset Parameters

### `VideoAsset`

| Parameter | Type | Description |
| :- | :- | :- |
| id | str | Video ID from VideoDB collection |
| start | int | Timestamp (seconds) where playback begins in source file. Default: `0` |
| volume | float | Audio volume multiplier. `1.0` = original, `0.0` = mute, `2.0` = double |
| crop | Crop | Crop edges of source video |

### `AudioAsset`

| Parameter | Type | Description |
| :- | :- | :- |
| id | str | Audio ID from VideoDB collection |
| start | int | Timestamp (seconds) where playback begins in source file |
| volume | float | Audio volume multiplier (`0.0` - `2.0`) |

## Common Patterns

### Sequential Clips (No Gaps)

```python Python theme={null}
# Clips play one after another
track.add_clip(0, clip1)   # 0-10s
track.add_clip(10, clip2)  # 10-20s
track.add_clip(20, clip3)  # 20-30s
```

```javascript Node.js theme={null}
// Clips play one after another
track.addClip(0, clip1);   // 0-10s
track.addClip(10, clip2);  // 10-20s
track.addClip(20, clip3);  // 20-30s
```

### Clips with Gaps

```python Python theme={null}
# Intentional pauses between clips
track.add_clip(0, clip1)    # 0-10s
track.add_clip(15, clip2)   # 15-25s (5s gap)
track.add_clip(35, clip3)   # 35-45s (10s gap)
```

```javascript Node.js theme={null}
// Intentional pauses between clips
track.addClip(0, clip1);    // 0-10s
track.addClip(15, clip2);   // 15-25s (5s gap)
track.addClip(35, clip3);   // 35-45s (10s gap)
```

### Overlapping Clips (Different Tracks)

```python Python theme={null}
main_track = Track()
main_track.add_clip(0, main_clip)

# Overlay appears at specific time
overlay_track = Track()
overlay_track.add_clip(20, overlay_clip)  # Appears at 20s

timeline.add_track(main_track)
timeline.add_track(overlay_track)
```

```javascript Node.js theme={null}
const mainTrack = new Track();
mainTrack.addClip(0, mainClip);

// Overlay appears at specific time
const overlayTrack = new Track();
overlayTrack.addClip(20, overlayClip);  // Appears at 20s

timeline.addTrack(mainTrack);
timeline.addTrack(overlayTrack);
```

## Complete Example

```python Python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

timeline = Timeline(conn)

# Extract 15 seconds starting at 30s from source
clip1 = Clip(
    asset=VideoAsset(
        id=video.id,
        start=30  # Skip first 30s of source
    ),
    duration=15  # Play for 15 seconds (shows 30s-45s of source)
)

track = Track()
track.add_clip(start=10, clip=clip1)  # Place at 10s in final timeline

timeline.add_track(track)
stream_url = timeline.generate_stream()
```

```javascript Node.js theme={null}
import { EditorTimeline, Track, Clip, EditorVideoAsset } from 'videodb';

const timeline = new EditorTimeline(conn);

// Extract 15 seconds starting at 30s from source
const clip1 = new Clip({
    asset: new EditorVideoAsset({
        id: video.id,
        start: 30  // Skip first 30s of source
    }),
    duration: 15  // Play for 15 seconds (shows 30s-45s of source)
});

const track = new Track();
track.addClip(10, clip1);  // Place at 10s in final timeline

timeline.addTrack(track);
const streamUrl = await timeline.generateStream();
```

Result: Final video has blank 0-10s, then shows seconds 30-45 from source at 10-25s position.

## Next Steps

> **[Trimming vs Timing](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/editor/feature/trimming_vs_timing.ipynb)**
>
> Hands-on practice with trimming and timing patterns, multi-clip workflows, and timing precision.


# Integrations Overview
Source: https://docs.videodb.io/pages/automate/integrations-overview

Connect your videos to any app to generate instant summaries, clips, subtitles, thumbnails, alerts and more - zero code required!

## Automation Platforms

VideoDB integrates with leading automation platforms, allowing you to build powerful no-code video workflows without writing any code.

### n8n

**n8n** is an open-source, self-hosted workflow automation platform that gives you complete control over your video processing pipelines.

* **Self-hosted or cloud** – Deploy on your own infrastructure or use n8n Cloud
* **Visual workflow builder** – Create complex workflows with an intuitive drag-and-drop interface
* **Extensive app ecosystem** – Connect to 400+ apps and services
* **Enterprise-ready** – Advanced features like error handling, scheduling, and monitoring

Perfect for teams that want full control over their automation infrastructure and need to process videos at scale.

> **[Explore n8n Workflows](/pages/automate/n8n/index)**
>
> View all n8n workflow templates and get started

### Zapier

**Zapier** is a cloud-based automation platform that connects thousands of apps without code.

* **No infrastructure needed** – Everything runs in the cloud, nothing to set up or maintain
* **Millions of users** – Trusted by teams worldwide for no-code automation
* **App marketplace** – Connect to 6000+ apps including Google Drive, Slack, Notion, and more
* **Instant setup** – Get workflows running in minutes with pre-built templates

Ideal for teams that want to quickly automate video workflows across their favorite tools without managing any infrastructure.

> **[Explore Zapier Workflows](/pages/automate/zapier/index)**
>
> View all Zapier workflow templates and get started

***

## Which Platform Should I Choose?

| Feature | n8n | Zapier |
| :- | :- | :- |
| **Hosting** | Self-hosted or cloud | Cloud-only |
| **Setup Complexity** | Simple (cloud) or moderate (self-hosted) | Very simple (instant) |
| **Cost Structure** | Pay for infrastructure | Per-task pricing |
| **Control** | Full customization | Limited customization |
| **Best For** | Teams needing full control, complex workflows | Quick automation, pre-built integrations |

Both platforms offer powerful video automation capabilities. Choose **n8n** if you need full control and self-hosting, or choose **Zapier** if you want the fastest way to get started with minimal setup.


# n8n Workflows
Source: https://docs.videodb.io/pages/automate/n8n/index

n8n is an open-source workflow automation platform that integrates with VideoDB. Build visual workflows to process videos, extract insights, and distribute content.

## Available Workflows

> **[Meeting Intelligence](/pages/automate/n8n/n8n-workflow-meeting-intelligence)**
>
> Automate meeting recording, AI analysis, and multi-platform distribution

> **[Video Dubbing](/pages/automate/n8n/n8n-workflow-dubbing)**
>
> Dub videos into different languages and save to Google Drive

> **[Subtitle Generation](/pages/automate/n8n/n8n-workflow-subtitle-generation)**
>
> Generate and embed professional subtitles in videos

> **[YouTube to Notion](/pages/automate/n8n/n8n-workflow-youtube-notion)**
>
> Monitor YouTube channels and create summaries in Notion

> **[HubSpot CRM Sync](/pages/automate/n8n/n8n-workflow-hubspot-crm)**
>
> Extract sales call insights and sync to HubSpot CRM

> **[Interview Evaluation](/pages/automate/n8n/n8n-workflow-interview-evaluation)**
>
> Generate structured candidate evaluations from interviews

> **[Meeting Summaries](/pages/automate/n8n/n8n-workflow-meeting-summaries)**
>
> Create meeting summaries with action items and decisions

## Getting Started

To use VideoDB with n8n, you need:

1. A running n8n instance (cloud or self-hosted)
2. A VideoDB API key from the [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)

Each workflow guide includes prerequisites, step-by-step instructions, and video tutorials.


# Video Dubbing Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-dubbing

Automatically dub videos into different languages and save to Google Drive using n8n and VideoDB.

This workflow takes video URLs, uploads them to VideoDB, dubs them into different languages, and saves the dubbed videos to Google Drive. It enables automatic video localization for global content distribution.

The workflow is useful for content creators who need to create multilingual versions of their videos for international audiences without manual dubbing processes.

[Video Dubbing Workflow](https://www.youtube.com/embed/atuBLPrW53I)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **Google Drive Account**: With API access for file uploads
4. **Video URLs**: Access to video content that needs dubbing

## Workflow Steps

### Step 1: Video Input and Upload

1. Form trigger collects YouTube video URL for dubbing
2. Upload video content to VideoDB for processing
3. Wait for upload completion before proceeding

### Step 2: Video Processing

1. Monitor upload status via HTTP request polling
2. Verify video upload is complete
3. Prepare video for dubbing operations

### Step 3: Video Dubbing

1. Use VideoDB dubbing operation to create dubbed version
2. Process video with target language dubbing
3. Monitor dubbing completion status

### Step 4: Google Drive Storage

1. Download dubbed video from VideoDB
2. Upload dubbed video to Google Drive folder
3. Store video with appropriate naming for language identification

## Conclusion

This workflow automates video dubbing and storage, enabling easy creation of multilingual video content that can be distributed through Google Drive or integrated into other workflows.


# HubSpot CRM Sync Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-hubspot-crm

Capture sales conversations and sync deal information directly to HubSpot CRM using n8n and VideoDB.

This workflow captures sales conversations and syncs deal information directly to HubSpot CRM. It eliminates manual data entry by extracting key deal details from sales calls and updating CRM records automatically.

The workflow ensures consistent CRM data entry and prevents missed sales opportunities through automated deal tracking.

[HubSpot CRM Sync Workflow](https://www.youtube.com/embed/FB60KHfvQRk)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **HubSpot CRM**: With API access for deal and contact management
4. **Meeting Recording**: Capability for sales call recording
5. **HubSpot Pipeline**: Configured deal pipeline with appropriate stages

## Workflow Steps

### Step 1: Sales Call Recording

1. Record sales conversations automatically
2. Ensure complete call capture before processing
3. Verify audio quality for accurate transcript generation

### Step 2: Video Processing

1. Index spoken words from sales call audio
2. Generate detailed transcript of the conversation
3. Prepare content for deal analysis

### Step 3: Deal Intelligence Extraction

1. Extract company/prospect name for deal identification
2. Assess current deal stage based on conversation content
3. Identify deal value and monetary discussions
4. Extract next steps and follow-up requirements
5. Categorize deals into appropriate HubSpot pipeline stages

### Step 4: HubSpot CRM Integration

1. Create new deal records or update existing ones
2. Populate CRM fields with:
   * Deal name and estimated value
   * Current pipeline stage
   * Meeting summary and discussion points
   * Next steps and follow-up tasks
3. Log meeting as activity in deal timeline

## Conclusion

This workflow maintains accurate, up-to-date CRM data by automatically capturing and structuring sales conversation details in HubSpot, supporting consistent deal tracking and sales process management.


# Interview Evaluation Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-interview-evaluation

Convert interview recordings into structured candidate evaluations with AI-powered assessments using n8n and VideoDB.

This workflow converts interview recordings into structured candidate evaluations with assessments and recommendations. It standardizes the interview evaluation process and distributes results to both Coda and Slack for hiring team collaboration.

The workflow ensures consistent evaluation criteria across all interviews, making candidate comparison more objective.

[Interview Evaluation Workflow](https://www.youtube.com/embed/siATcrJGhOw)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **Meeting Platform**: Platform with bot recording capability for interviews
4. **Coda Workspace**: For candidate tracking and evaluation storage
5. **Slack Workspace**: For hiring team notifications

## Workflow Steps

### Step 1: Interview Recording

1. Form trigger collects meeting URL for the interview session
2. VideoDB joins the interview and records the session
3. Wait for interview completion before processing

### Step 2: Video Processing

1. Index spoken words from the interview recording
2. Generate transcript with speaker identification (interviewer vs candidate)
3. Prepare audio content for AI analysis

### Step 3: AI Evaluation

1. Extract structured candidate information:
   * Candidate name and background details
   * Role they're applying for
   * Personality and communication assessment
2. Generate hiring recommendation (Strongly Recommended/Recommended/Consider with Reservations/Not Recommended)
3. Evaluate key strengths, areas for development, and cultural fit (High/Medium/Low)

### Step 4: Results Distribution

1. Add candidate evaluation to Coda hiring database with structured fields
2. Send formatted evaluation summary to Slack hiring team channels
3. Ensure consistent format for easy candidate comparison

## Conclusion

This workflow provides hiring teams with standardized candidate evaluations distributed across both documentation (Coda) and communication (Slack) platforms for comprehensive hiring decision support.


# Meeting Intelligence Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-meeting-intelligence

Automate the complete meeting lifecycle from recording to analysis and distribution with n8n and VideoDB.

This workflow automates the complete meeting lifecycle from recording to analysis and distribution. It processes different meeting types (Sync, Planning, Interview, Sales) and distributes insights to Slack, Coda, and HubSpot based on the meeting context.

The workflow handles different meeting types with intelligent branching instead of requiring separate workflows for each type.

[Meeting Intelligence Workflow](https://www.youtube.com/embed/jkujIeEv96c)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **Meeting Platform**: Platform with bot recording support (Zoom, Teams, Google Meet)
4. **Slack Workspace**: With bot permissions for notifications
5. **Coda Workspace**: With API token for meeting documentation
6. **HubSpot Account**: With API key for sales meeting integration

## Workflow Steps

### Step 1: Meeting Recording

1. Form trigger collects meeting URL and meeting type (Sync/Planning/Interview/Sales)
2. VideoDB joins the meeting and records the session
3. Wait for meeting completion before processing

### Step 2: Video Processing

1. Index spoken words from the recorded meeting
2. Generate transcript with speaker identification and mapping
3. Process video content for analysis readiness

### Step 3: AI Analysis with Branching

1. Switch workflow based on meeting type selection
2. Apply custom AI prompts for each meeting type:

   * **Planning**: Extract strategic outcomes, decisions, and action items
   * **Interview**: Generate candidate evaluation with recommendation
   * **Sales**: Identify deal details, stage, amount, and next steps
   * **Sync**: Create summaries with follow-ups and task assignments

### Step 4: Multi-Platform Distribution

1. Send formatted updates to Slack channels
2. Update Coda databases with meeting insights
3. For sales meetings: Create or update HubSpot deal records
4. Route data to appropriate platforms based on meeting type

## Conclusion

This workflow consolidates multiple meeting types into one automated system, ensuring consistent documentation and appropriate distribution across your business tools.


# Meeting Summaries Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-meeting-summaries

Convert recorded meetings into structured summaries with action items and key decisions using n8n and VideoDB.

This workflow converts recorded meetings into structured summaries with action items and key decisions. It serves as a universal meeting processor for general business meetings and distributes summaries to both Coda and Slack.

The workflow ensures consistent meeting documentation and team notification for any type of business meeting.

[Meeting Summaries Workflow](https://www.youtube.com/embed/kP7fHIEFEvg)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **Meeting Recording**: Access to meeting recording (any platform with recording capability)
4. **Coda Workspace**: For meeting documentation storage
5. **Slack Workspace**: For team notifications

## Workflow Steps

### Step 1: Meeting Recording

1. Process recorded meeting files or live meeting recordings
2. Initiate VideoDB processing for the meeting content
3. Wait for recording completion before analysis

### Step 2: Video Processing

1. Index spoken words and generate comprehensive transcript
2. Create timestamped transcript for reference
3. Prepare content for AI summarization

### Step 3: AI Summarization

1. Extract meeting title and purpose
2. Identify key discussion points and decisions made
3. Extract action items with ownership assignments and due dates
4. Structure information into consistent format

### Step 4: Documentation and Distribution

1. Create new entry in Coda project database with structured fields:
   * Date and meeting metadata
   * Executive summary
   * Action points with assignments
   * Decision log
2. Send formatted meeting summary to Slack with action item notifications

## Conclusion

This workflow creates a complete meeting documentation system that stores structured summaries in Coda while keeping teams informed through Slack notifications.


# Subtitle Generation Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-subtitle-generation

Generate and embed professional subtitles in videos, then save to Google Drive using n8n and VideoDB.

This workflow takes video URLs, uploads them to VideoDB, indexes spoken content, adds subtitle overlays directly to the video, and saves the subtitled videos to Google Drive. It creates professional subtitle overlays that are permanently embedded in the video.

The workflow is useful for creating accessible video content with professional subtitle styling that works across all platforms, unlike platform-specific auto-generated subtitles.

[Subtitle Generation Workflow](https://www.youtube.com/embed/lUP0XmNOXtA)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **Google Drive Account**: With API access for file uploads
4. **Video URLs**: Access to video content that needs subtitles

## Workflow Steps

### Step 1: Video Input and Upload

1. Form trigger collects YouTube video URL for subtitle processing
2. Upload video content to VideoDB for processing
3. Wait for upload completion before proceeding

### Step 2: Spoken Content Indexing

1. Monitor upload status via HTTP request polling
2. Index spoken words from the video content
3. Generate timestamped transcript for subtitle creation

### Step 3: Subtitle Generation and Overlay

1. Use VideoDB subtitle operation to add subtitle overlays
2. Create professional subtitle styling embedded in video
3. Process video with precisely timed subtitle overlays

### Step 4: Google Drive Storage

1. Download subtitled video from VideoDB
2. Upload subtitled video to Google Drive folder
3. Store video with subtitles permanently embedded

## Conclusion

This workflow creates professional subtitled videos with embedded subtitle overlays, making content accessible across all platforms while maintaining consistent subtitle styling and timing.


# YouTube to Notion Workflow
Source: https://docs.videodb.io/pages/automate/n8n/n8n-workflow-youtube-notion

Monitor YouTube channels for new content and automatically create summaries in Notion using n8n and VideoDB.

This workflow monitors YouTube channels for new content and automatically creates summaries in Notion. It builds a searchable knowledge base of video content without requiring manual video watching.

The workflow helps teams track multiple content sources and extract insights from educational or industry content automatically.

[YouTube to Notion Workflow](https://www.youtube.com/embed/3xPvFNmplK4)

## Prerequisites

1. **n8n Instance**: Running n8n automation platform (cloud or self-hosted)
2. **VideoDB API Key**: Obtain from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. **Notion Workspace**: With page/database access for content storage
4. **YouTube Channels**: Specific channels to monitor via RSS feed
5. **RSS Configuration**: n8n RSS trigger setup for channel monitoring

## Workflow Steps

### Step 1: Content Discovery

1. RSS feed trigger monitors specified YouTube channels for new uploads
2. Capture video metadata (title, description, author, publication date, URL)
3. Activate immediately when new video is published

### Step 2: Video Upload and Processing

1. Upload video content to VideoDB for processing
2. Monitor upload and indexing completion status
3. Index spoken words for transcript extraction

### Step 3: Content Analysis

1. Generate full transcript of video content with timestamps
2. Apply AI summarization to extract:
   * Key topics and themes discussed
   * Important quotes and insights
   * Technical concepts explained
   * Action items or recommendations mentioned

### Step 4: Notion Integration

1. Create new Notion page with structured video summary
2. Include comprehensive documentation:
   * Video title and source channel information
   * Publication date and metadata
   * Detailed summary with key insights
   * Direct link to original video
3. Add entry to Notion database with appropriate tags for searching

## Conclusion

This workflow automates video content monitoring and creates a searchable knowledge repository in Notion, enabling teams to stay informed about relevant content without manual video consumption.


# Zapier Workflows
Source: https://docs.videodb.io/pages/automate/zapier/index

Zapier is a no-code automation platform that connects VideoDB with thousands of apps. Build automated workflows to process videos, generate content, and sync data.

## Available Workflows

> **[YouTube Summaries to Notion](/pages/automate/zapier/zapier-workflow-youtube-summaries)**
>
> Summarize new YouTube videos and save to Notion

> **[Video Dubbing to Drive](/pages/automate/zapier/zapier-workflow-dubbing-drive)**
>
> Dub videos into Spanish and store in Google Drive

> **[GenAI Video to YouTube](/pages/automate/zapier/zapier-workflow-notion-to-youtube)**
>
> Generate AI videos from Notion ideas and publish to YouTube

> **[Profanity Detection](/pages/automate/zapier/zapier-workflow-profanity-detection)**
>
> Detect profanity in videos and send Slack alerts

> **[Subtitle Generation](/pages/automate/zapier/zapier-workflow-subtitle-generation)**
>
> Generate subtitles and save videos to Google Drive

> **[Video Highlights](/pages/automate/zapier/zapier-workflow-video-highlights)**
>
> Extract highlights and summaries to Notion

> **[Video Q\&A Support](/pages/automate/zapier/zapier-workflow-video-qa)**
>
> Answer Zendesk tickets using video content

## Getting Started

To use VideoDB with Zapier, you need:

1. A Zapier account
2. A VideoDB API key from the [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)
3. Connected accounts for the apps you want to integrate (Google Drive, Notion, Slack, etc.)

Each workflow guide includes prerequisites, step-by-step instructions, and video tutorials.


# Video Dubbing to Google Drive
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-dubbing-drive

Automate video dubbing into Spanish and store in Google Drive using Zapier and VideoDB.

This workflow automates the process of dubbing videos into Spanish and storing them in a dedicated Google Drive folder using Zapier and VideoDB. The goal is to enhance content accessibility, optimize workflow efficiency, and conserve resources, allowing for seamless management of multilingual video content.

## Goals

1. Automate the process of dubbing videos into Spanish
2. Store the dubbed videos in a specified Google Drive folder
3. Optimize workflow efficiency by minimizing unnecessary triggers and conserving credits

[Video Dubbing to Google Drive](https://www.youtube.com/embed/ixObggmYS0w)

## Prerequisites

* A Zapier account to create and manage workflows
* Access to VideoDB with an active API key
* A configured Google Drive account with a folder called "Dubbed Videos" for storage
* The ability to create forms via interfaces.zapier.com for video submissions
* A basic understanding of how Zapier workflows (zaps) function

## Workflow 1: Upload and Indexing Process

### Step 1: Form Creation

Use interfaces.zapier.com to create a video submission form. The form should capture the video name and URL, which can be a downloadable link or a YouTube link.

### Step 2: Trigger Setup

1. Create a new Zap in Zapier
2. Set the trigger to be a form submission from Zapier Interfaces
3. Choose the interface and specific form created for video submissions

### Step 3: Video Upload and Indexing

1. Add an action step using VideoDB to upload and index the video
2. Connect to VideoDB using your API key
3. Pass the video name and URL from the form submission
4. Use "dubvideo" as the target label to ensure specific workflow triggering
5. Test the step to verify that a job ID is returned, indicating the process is in progress

### Step 4: Publish Workflow 1

Once verified, publish this initial workflow to make it active.

## Workflow 2: Dubbing and Storage Process

### Step 1: Trigger Setup

1. Create a new Zap in Zapier
2. Set the trigger to "New Video Uploaded and Indexed" from VideoDB
3. Use the target label "dubvideo" to ensure this workflow is only triggered for designated tasks

### Step 2: Dubbing Process

1. Add an action step for autodubbing the video using VideoDB
2. Set the language code to "ES" for Spanish
3. Test the step to receive a new job ID, indicating the dubbing process has started

### Step 3: Delay for Processing

Insert a delay action using Zapier's flow control to wait for a specified time (e.g., 5 minutes) to allow the dubbing process to complete.

### Step 4: Check Dubbing Status

After the delay, check the job status using VideoDB's action to confirm the dubbing has finished.

### Step 5: Google Drive Storage

1. Add an action step to upload the dubbed video to Google Drive
2. Select the appropriate account and folder ("Dubbed Videos")
3. Use the downloadable URL from the completed video job for the file upload
4. Use the video name with ".mp4" as the extension for the file name

### Step 6: Test and Publish Workflow 2

Test the entire workflow to ensure a dubbed video is correctly uploaded to Google Drive. Once complete, publish this workflow to activate it.

## Workflow Templates

> **[Part 1: Upload and Indexing Process](https://zapier.com/shared/f66991c88b7ed1d0e79dd207acf5c7bd8604d0e2)**
>
> Upload videos and index them in VideoDB

> **[Part 2: Dubbing and Storage Process](https://zapier.com/shared/166e664d5238cdfe610c38b2bf88b759b07dcf09)**
>
> Dub videos into different languages and save to Google Drive


# GenAI Video to YouTube
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-notion-to-youtube

Automate AI-generated video creation from Notion ideas and publish directly to YouTube using Zapier and VideoDB.

This workflow explains how to automate the creation and publishing of AI-generated (GenAI) videos to YouTube using VideoDB. The goal is to enable a fully automated YouTube channel where video ideas added in Notion automatically result in generated videos uploaded to YouTube.

## Goals

1. Automate the generation of AI-based videos from Notion database entries
2. Upload generated videos directly to a connected YouTube channel
3. Use target labels to route tasks efficiently and avoid duplicate triggers
4. Enable creators to scale video publishing with minimal manual effort

[GenAI Video to YouTube](https://www.youtube.com/embed/ihgeH6slPqk)

## Prerequisites

* A Zapier account to create and manage workflows
* Access to VideoDB with an active API key
* A Notion account with a database for storing video ideas and descriptions
* A YouTube account connected to Zapier for video uploads
* Basic familiarity with Zapier workflows and triggers

## Workflow 1: Generate AI Video

### Step 1: Notion Database Setup

Create a new database in Notion called *GenAI Video Ideas*. Add fields such as *Name* (video title) and *Description* (prompt for video content).

### Step 2: Trigger Setup in Zapier

1. Create a new Zap
2. Set the trigger to *New Database Item* in Notion
3. Select the *GenAI Video Ideas* database

### Step 3: Generate Video with VideoDB

1. Add an action using VideoDB - *Generate Video*
2. Authenticate with API key from the VideoDB dashboard
3. Pass in the Notion fields (*Name* and *Description*) as the prompt
4. Configure settings (e.g., 30 seconds duration, vertical aspect ratio, with subtitles)
5. Assign a **target label** (e.g., `YT upload`) to ensure the correct next workflow is triggered
6. Test and verify that a Job ID (token) is returned

### Step 4: Publish Workflow 1

Once tested, activate this workflow to automatically start video generation.

## Workflow 2: Upload Generated Video to YouTube

### Step 1: Trigger Setup

1. Create another Zap
2. Set the trigger to *AI Generated Video Created* in VideoDB
3. Use the same target label (`YT upload`) so only matching jobs trigger this workflow

### Step 2: YouTube Upload Configuration

1. Add an action using YouTube - *Upload Video*
2. Connect your YouTube account to Zapier
3. Map video fields:
   * Title: from Notion *Name*
   * Description: e.g., *"GenAI Video created by VideoDB"*
   * Video File: use the *Download URL* from VideoDB output
4. Optional settings: leave thumbnail empty, mark video as *Public*, and set "not made for kids"

### Step 3: Test and Publish Workflow 2

Run a test to verify upload works. Publish the workflow to activate automatic uploads.

## Workflow Templates

> **[Part 1: Generate AI Video](https://zapier.com/shared/1c89ba5342e1e14bacffab645281d492230a9e9e)**
>
> Generate AI videos from Notion ideas and prompts

> **[Part 2: Upload Generated Video to YouTube](https://zapier.com/shared/4d2d476f15d277f8bd694a1f3c31b0d611765e13)**
>
> Publish generated videos directly to YouTube


# Profanity Detection to Slack
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-profanity-detection

Detect profanity in uploaded videos and send alerts to Slack using Zapier and VideoDB.

This automation is split into two workflows: the first handles video upload and indexing, and the second checks for profanity and sends the results via Slack DM. This design ensures longer-running indexing tasks complete properly while still delivering actionable insights.

## Goals

1. Automate detection of profanity or vulgar language in uploaded videos
2. Split the workflow into two parts to handle longer processing times without Zapier timeouts
3. Send detected profanity results and timestamps directly to Slack for review
4. Use target labels to ensure only the intended second workflow is triggered

[Profanity Detection to Slack](https://www.youtube.com/embed/jYoT4AcGvgA)

## Prerequisites

* A Zapier account to manage and run workflows
* Access to VideoDB with an active API key
* A Slack workspace with permissions to send direct messages or channel messages
* A Zapier Interface form to collect video name and URL submissions
* Basic understanding of asynchronous job handling in Zapier

## Workflow 1: Upload and Index Video

### Step 1: Form Creation (Zapier Interfaces)

Build a form with fields for *Video Name* and *Video URL*. Example: Enter a video title and a YouTube link.

### Step 2: Trigger Setup

1. Set trigger to *Form Submission Created* (Zapier Interfaces)
2. Select the created form and page as the source

### Step 3: Upload and Index with VideoDB

1. Add an action - VideoDB - *Upload and Index Video*
2. Authenticate using your VideoDB API key
3. Map fields: Video Name and URL from form submission
4. Set **prompt**: *Check for any profanity or vulgar content*
5. Assign a **target label** (e.g., `Profanity Check`) to connect with the second workflow
6. Test - confirm Job ID is returned (asynchronous task)

### Step 4: Publish Workflow 1

Activate the workflow so all new form submissions trigger video uploads automatically.

## Workflow 2: Profanity Detection and Slack Notification

### Step 1: Trigger Setup

1. Trigger: *New Video Uploaded Job Completed* (VideoDB)
2. Filter by target label `Profanity Check`

### Step 2: Find Video Moment (Profanity Check)

1. Action: VideoDB - *Find Video Moment*
2. Use dynamic video ID from the trigger
3. Prompt: *Detect profanity or vulgar content*
4. Content type: *Multimodal* (spoken + visual)
5. This action returns a new Job ID for processing

### Step 3: Delay for Processing

Insert a delay (e.g., 3 minutes) to allow VideoDB to finish analyzing.

### Step 4: Check Job Status

1. Action: VideoDB - *Check Job Status*
2. Input the Job ID from the "Find Video Moment" step
3. Retrieve results: detected profanity, timestamps, and URLs

### Step 5: Slack Notification

1. Action: Slack - *Send Channel or DM Message*
2. Configure message format:
   * Video Name
   * Player/Download URL
   * Profanity results (text + timestamps)
3. Customize bot name (e.g., *Profanity Checker*)
4. Test - confirm profanity results are delivered in Slack

### Step 6: Publish Workflow 2

Activate the workflow to automatically run profanity checks and push results to Slack.

## Workflow Templates

> **[Part 1: Upload and Index Video](https://zapier.com/shared/5fce07a54798db1e85d2f3775d19aa04ca08675e)**
>
> Upload videos and index them in VideoDB

> **[Part 2: Profanity Detection and Slack Notification](https://zapier.com/shared/98663dbde48b7c4218177f415ff1ac9d0397e035)**
>
> Detect profanity and send real-time alerts to Slack


# Subtitle Generation
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-subtitle-generation

Automate subtitle generation for videos and save to Google Drive using Zapier and VideoDB.

This workflow automates the process of generating subtitles for uploaded videos and storing the subtitled videos in a designated Google Drive folder. It improves accessibility for audiences who need captions.

## Goals

1. Automate the process of generating subtitles for uploaded videos
2. Store the subtitled videos in a designated Google Drive folder
3. Improve accessibility for audiences who need captions

[Subtitle Generation](https://www.youtube.com/embed/Uv866TBP5i4)

## Prerequisites

* A Zapier account to create and manage workflows
* Access to VideoDB with an active API key
* A configured Google Drive account with a folder (e.g., "Videos with Subtitles") for storing processed videos
* A Zapier Interface form to collect video name and video URL submissions
* Basic understanding of Zapier workflows (Zaps) and triggers

## Workflow 1: Form Submission and Upload

### Step 1: Form Creation (Zapier Interfaces)

1. Create a blank form titled *Subtitle Video*
2. Add two fields:
   * Video Name (short text, required)
   * Video URL (valid URL field, required)
3. Share the form link for submissions

### Step 2: Trigger Setup

1. In Zapier, set the trigger to *Form Submission Created* from Interfaces
2. Select the subtitle video form and page

### Step 3: Upload and Index with VideoDB

1. Add an action to upload and index the video via VideoDB
2. Authenticate using your VideoDB API key
3. Map video name and URL from the form data
4. Set a target label (e.g., `subtitle video`) to connect with the second workflow
5. Test to confirm a Job ID is returned, indicating the task has started

### Step 4: Publish Workflow 1

Activate the workflow so form submissions trigger video uploads automatically.

## Workflow 2: Subtitle Generation and Storage

### Step 1: Trigger Setup

1. Create a new Zap with trigger *New Video Uploaded Job* from VideoDB
2. Filter by target label `subtitle video` to ensure only relevant jobs trigger this workflow

### Step 2: Subtitle Generation

1. Add an action to *Generate Subtitles* using VideoDB
2. Select dynamic video ID from the trigger event
3. Configure subtitle style (e.g., *supersize* animation to enlarge the spoken word)
4. Test to confirm the output returns a stream URL, download URL, and new video ID

### Step 3: Google Drive Storage

1. Add an action to *Upload File* to Google Drive
2. Authenticate with your Google account and select the designated folder (e.g., *Videos with Subtitles*)
3. Use the downloadable URL from VideoDB as the input file
4. Set the file name dynamically from the form submission with ".mp4" extension
5. Test to confirm the subtitled video is stored in the correct folder

### Step 4: Publish Workflow 2

Activate this workflow to automatically process and store subtitled videos.

## Workflow Templates

> **[Part 1: Form Submission and Upload](https://zapier.com/shared/55c30af3c6b74590de9bb94835785a085e2806fb)**
>
> Collect videos via form and upload to VideoDB

> **[Part 2: Subtitle Generation and Storage](https://zapier.com/shared/9e99d44362db51149cb2508eb939ed243e73b254)**
>
> Generate subtitles automatically and save to Google Drive


# Video Highlights to Notion
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-video-highlights

Automate video highlight clip extraction and summary generation, stored in Notion using Zapier and VideoDB.

This Zap automates the generation of video highlight clips and summaries from YouTube videos using VideoDB, with the results automatically stored in Notion. The goal is to simplify knowledge capture, optimize workflow efficiency, and create a system that generates insights with minimal manual effort.

## Goals

1. Automate the extraction of highlight clips from YouTube videos
2. Generate concise, insightful summaries for each video
3. Store both highlights and summaries in a designated Notion page
4. Optimize efficiency by leveraging asynchronous processing and targeted workflow triggers

[Video Highlights to Notion](https://www.youtube.com/embed/Dmgu-V2YcUs)

## Prerequisites

* A Zapier account to create and manage workflows
* Access to VideoDB with an active API key
* A configured YouTube playlist (e.g., "Informative Videos and Lectures")
* A Notion account with a parent page prepared for storing video highlights and summaries
* Basic understanding of how Zapier workflows (Zaps) function

## Workflow 1: Upload and Indexing Process

### Step 1: Trigger Setup

1. Create a new Zap in Zapier
2. Set the trigger to *New Video in Playlist* (YouTube)
3. Select the playlist (e.g., "Informative Videos and Lectures")

### Step 2: Video Upload and Indexing

1. Add an action step using VideoDB to upload and index the video
2. Connect to VideoDB using the API key
3. Pass the video title and YouTube URL
4. Set a **target label** (e.g., `YT highlights`) to direct the workflow
5. Confirm that a **Job ID** is returned as a token for asynchronous processing

### Step 3: Publish Workflow 1

Once verified, publish this workflow to make it active.

## Workflow 2: Highlight Clips and Summaries

### Step 1: Trigger Setup

1. Create a new Zap in Zapier
2. Set the trigger to *Upload Job Completed* from VideoDB
3. Filter by the target label (`YT highlights`)

### Step 2: Highlight Clip Generation

1. Add an action to generate highlight clips using VideoDB
2. Use prompts such as *"Extract interesting and insightful information"*
3. Select dynamic video ID from the trigger
4. Configure clip aspect ratio and subtitles (optional)
5. Note: this action returns a new **asynchronous Job ID**

### Step 3: Summary Generation

1. Add another action to summarize the video with VideoDB
2. Prompt: *"Create Notion markdown notes capturing key insights"*
3. Use the same video ID dynamically
4. Another asynchronous Job ID will be returned

### Step 4: Delay for Processing

Insert a Zapier delay (e.g., 5-6 minutes) to allow highlight and summary jobs to complete.

### Step 5: Job Status Checks

1. Add steps to check the status of both highlight and summary jobs
2. Verify the results are available (player link for highlights, markdown text for summary)

### Step 6: Notion Storage

1. Add an action to create a new Notion page under the parent "Playlist Highlights" page
2. Use the video title as the page title
3. Insert the summary text first
4. Add a "Highlights" section, including the **VideoDB Player URL** for generated highlight clips

### Step 7: Test and Publish Workflow 2

Test the full process end-to-end. Once successful, publish this workflow.

## Workflow Templates

> **[Part 1: Upload and Indexing Process](https://zapier.com/shared/3baccadd46bf12d5be5bc0a9ed95527b9adb13d4)**
>
> Upload videos and index them in VideoDB

> **[Part 2: Highlight Clips and Summaries](https://zapier.com/shared/33cb81a9ef621614bb3047138e259fd9192fa7dd)**
>
> Extract highlights and create summaries in Notion


# Video Q&A Support
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-video-qa

Automatically answer Zendesk support tickets using relevant video content via Zapier and VideoDB.

This workflow is designed to automatically send Zendesk ticket notifications to Slack while also finding relevant information from a specific video to provide quick answers to the queries raised in the tickets.

## Goals

1. Automate the notification of new Zendesk tickets
2. Retrieve relevant video clips that provide answers to the ticket queries
3. Send both the ticket information and the relevant video clips to a Slack channel for quick resolution

[Video Q\&A Support](https://www.youtube.com/embed/PQSZr8aS61I)

## Prerequisites

* A Zendesk account with API access
* A VideoDB account to find and manage video content
* A Slack account to receive notifications
* A Zapier account to create and manage the integration

## Detailed Steps

### Step 1: Setup in Zapier

1. Navigate to Zapier and create a new Zap
2. Select Zendesk as the trigger app with the "New Ticket" event
3. Configure Zendesk account settings using the domain, admin email, and API token from Zendesk

### Step 2: Configure Zendesk

Access the Zendesk dashboard and retrieve the API token required to connect with Zapier.

### Step 3: VideoDB Integration

1. Use the "Find Video Moment" action in Zapier to connect VideoDB
2. Sign in to VideoDB using Google or GitHub and obtain the API key
3. Choose the relevant video from which answers will be retrieved based on the ticket subject and description

### Step 4: Control Flow with Delay

Add a "Delay by Zapier" action to wait for a set time (e.g., 3 minutes) for the VideoDB processing to complete before proceeding.

### Step 5: Check Job Status

Add an action to check the status of the VideoDB job using the job ID obtained from the previous step.

### Step 6: Send Notifications to Slack

1. Select Slack as the action app and configure it to send a direct message or channel message
2. Customize the message with ticket subject, description, VideoDB results, and a link to the video

### Step 7: Test and Finalize

Test the workflow to ensure that the message is correctly sent to Slack with all necessary details.

## Workflow Template

> **[Video Q\&A Support Workflow](https://zapier.com/shared/00fc160d0e3749a6de4002bd69dff4e31b299625)**
>
> Answer support tickets using video content


# YouTube Summaries to Notion
Source: https://docs.videodb.io/pages/automate/zapier/zapier-workflow-youtube-summaries

Automatically summarize new YouTube videos and save to Notion using Zapier and VideoDB.

This Zapier workflow automates the process of summarizing newly uploaded YouTube videos and saving those summaries into Notion using VideoDB. The goal is to create an end-to-end pipeline that detects channel uploads, sends videos to VideoDB for indexing, requests an automated summary, and stores the Markdown-formatted summary as a new Notion page.

## Goals

1. Automatically detect new YouTube uploads on a specific channel
2. Upload and index each new video in VideoDB and tag it with a target label (e.g., `YT summary`)
3. Generate a Markdown-formatted summary of the indexed video using VideoDB
4. Create a new Notion page containing the summary and relevant metadata for easy storage and reference

[YouTube Summaries to Notion](https://www.youtube.com/embed/1C6vZr2eVMA)

## Prerequisites

* A Zapier account to create and manage the two workflows (zaps)
* Access to VideoDB with an active API key
* A Notion account and a parent Notion page (e.g., **YouTube video summaries**) where new pages will be created
* The YouTube channel ID (Note: YouTube typically displays channel usernames/handles like `@channelname` instead of the actual channel ID. Use a converter tool such as [StreamWeasels YouTube Channel ID Converter](https://www.streamweasels.com/tools/youtube-channel-id-and-user-id-convertor/) to convert the username to a channel ID)
* Basic familiarity with Zapier actions (triggers, async jobs, delay/flow control) and dynamic field mapping

## Workflow 1: YouTube Upload to VideoDB Upload and Index

### Step 1: Trigger Setup (YouTube)

1. Create a new Zap with the trigger **New Video in Channel** (YouTube)
2. Provide the correct YouTube **channel ID**. Note that YouTube displays channel usernames/handles (like `@channelname`) instead of the actual channel ID. Use a converter tool such as [StreamWeasels YouTube Channel ID Converter](https://www.streamweasels.com/tools/youtube-channel-id-and-user-id-convertor/) to get the channel ID from the username.
3. Test the trigger to ensure Zapier can fetch a recent video from the channel

### Step 2: Upload and Index with VideoDB

1. Add a Zapier action: **VideoDB - Upload an Index job**
2. Authenticate using your VideoDB API key (retrieve from the VideoDB console / Access Control)
3. Map fields: video title, video URL (YouTube link), and a descriptive prompt (e.g., "informative video - summarize accordingly")
4. **Set a target label** (e.g., `YT summary`) so VideoDB knows which downstream workflow(s) to trigger when indexing completes
5. Test the step and confirm VideoDB returns a **Job ID** (indexing is asynchronous)

### Step 3: Publish Workflow 1

Activate (publish) the zap so that every new upload on the specified channel triggers this upload/index job automatically.

## Workflow 2: VideoDB Done to Summarize to Notion Page

### Step 1: Trigger Setup (VideoDB)

1. Create a second Zap with trigger **New video uploaded & indexed** from VideoDB
2. Filter by the **same target label** (`YT summary`) to ensure only relevant indexing jobs trigger this workflow
3. Test to obtain a sample payload (includes `video_id`, stream/download URLs, length, etc.)

### Step 2: Request Video Summarization

1. Add an action: **VideoDB - Summarize Video**
2. Use the **dynamic** `video_id` from the VideoDB trigger (do not use a static video id)
3. Provide a summarization prompt tailored for Notion, for example: "Summarize the technical content of this video. Provide the final notes in Markdown format suitable for Notion."
4. Run the step - VideoDB returns a **summary job id** because summarization is asynchronous

### Step 3: Wait / Poll for Completion

1. Insert a Zapier **Delay** (Flow Control) - e.g., **Delay for 3 minutes** - to allow time for the async summarization job
2. After the delay, add **VideoDB - Check Job Status** and pass the summary job id (dynamic)
3. Test the check step; once complete it will return the summarized text (already formatted per the prompt)

### Step 4: Create Notion Page

1. Add an action: **Notion - Create Page** under the parent page **YouTube video summaries**
2. Authenticate and grant Zapier access to that Notion page
3. Configure the page:
   * **Title:** map dynamically from the original YouTube upload (video name from the first trigger)
   * **Body/content:** use the Markdown summary returned by the VideoDB Check Job Status step (Notion supports Markdown import)
   * Optionally map metadata (video URL, video length, VideoDB job id) into properties or content for traceability
4. Test creating the page to confirm formatting and placement are correct

### Step 5: Publish Workflow 2

Activate (publish) the zap so VideoDB completion events automatically trigger summarization and Notion page creation.

## Workflow Templates

> **[Part 1: YouTube Upload to VideoDB Upload and Index](https://zapier.com/shared/c9fdbeb01205775e439de528c530793aea952711)**
>
> Detect YouTube uploads and automatically index them in VideoDB

> **[Part 2: VideoDB Done to Summarize to Notion Page](https://zapier.com/shared/ec4e5046352427d462615426269eb19c5a32521d)**
>
> Generate AI summaries and save as Notion pages


# Create Custom Agent
Source: https://docs.videodb.io/pages/build-with-agents/director/create-custom-agent

Learn how to design, develop, and integrate custom agents into the Director framework with best practices for architecture, communication, and error handling.

This playbook will guide you through the process of creating your own agents within the Director framework. **You'll learn:**

* How to plan and structure an effective agent
* Best practices for development and integration
* Techniques for handling user communication and errors
* Ways to leverage Director's powerful video processing capabilities

## Understanding the Architecture

Before diving into agent creation, let's understand how Director works as a system. Director follows a modular architecture that enables seamless interaction between users and AI-powered video processing capabilities.

### System Overview

Director's architecture is designed around three core principles:

1. **Modularity**: Each component has a specific responsibility and can be developed or modified independently
2. **Scalability**: The system can handle multiple requests and complex video operations efficiently
3. **Extensibility**: New agents and tools can be easily added to expand functionality

Looking at the system architecture diagram below, you can see how these principles come together:

![Director system architecture showing modularity, scalability, and extensibility principles](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/director/system-architecture.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=4548cae36fc128ab09cabaa2a61ccfd8)

### Understanding the Director Framework

Director consists of several key components working together:

#### Reasoning Engine

The brain of the system that interprets natural language commands, coordinates multiple agents, maintains conversation context, and manages workflows and decision-making.

![Reasoning engine showing how it interprets commands, coordinates agents, and manages workflows](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/director/reasoning-engine.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=6fc42f16d267a3d3ec37070312280be4)

#### Agents

Specialized workers that handle specific tasks. Agents collaborate to complete complex video operations.

> **[SearchAgent](https://github.com/video-db/Director/blob/main/backend/director/agents/search.py)**
>
> Finds specific content within videos using semantic search

> **[ThumbnailAgent](https://github.com/video-db/Director/blob/main/backend/director/agents/thumbnail.py)**
>
> Generates preview images and thumbnails

> **[UploadAgent](https://github.com/video-db/Director/blob/main/backend/director/agents/upload.py)**
>
> Handles media uploads to VideoDB

> **[View All Agents](https://github.com/video-db/Director/tree/main/backend/director/agents)**
>
> Explore the complete collection of built-in agents in the repository

#### Tools

Reusable functions that agents can leverage to perform their tasks:

> **[VideoDB Tool](https://github.com/video-db/Director/blob/main/backend/director/tools/videodb_tool.py)**
>
> Core video database operations and queries

> **[AI Model Integrations](https://github.com/video-db/Director/tree/main/backend/director/llm)**
>
> OpenAI, Anthropic, and other LLM providers

> **[External Connections](https://github.com/video-db/Director/tree/main/backend/director/tools)**
>
> Slack, Composio, ElevenLabs, and more

#### Session Management

Handles state and context across interactions, ensuring consistent conversation flow and data persistence.

### The Reasoning Engine in Detail

The Reasoning Engine is the orchestrator of all agent activities. It performs four critical functions:

> **Processes User Input**
>
> Understands natural language requests, maintains conversation history and context, and determines required actions and sequence

> **Orchestrates Agents**
>
> Selects appropriate agents from the pool, coordinates multiple agents for complex tasks, and manages dependencies between operations

> **Handles Communication**
>
> Provides real-time progress updates, manages error scenarios, and returns formatted responses to the user

> **Maintains State**
>
> Tracks ongoing operations, manages session data, and ensures context persistence across interactions

### Bringing It All Together

The architecture enables powerful workflows like:

* A user requests a `video summary` through the chat interface
* The Flask server processes this request and routes it to the Reasoning Engine
* The Reasoning Engine coordinates multiple agents to analyze and process the video
* Real-time updates flow back through WebSocket connections
* The final result is presented in the video player

This architectural foundation is what makes Director so powerful. When you create a new agent, it becomes part of this ecosystem, leveraging all these capabilities to perform its tasks efficiently.

With the fundamentals covered, **let's start building!**

## Planning Phase

The success of an agent heavily depends on thorough planning and requirements gathering. Before writing any code:

1. **Question Everything**

* Compile a comprehensive list of questions across all aspects
* Include edge cases and potential future requirements
* Consider integration points with other agents/systems

1. **Categorize Requirements**

```
MUST-HAVE (v1)
- Core functionality requirements
- Essential error handling
- Basic user feedback

SHOULD-HAVE (v2)
- Enhanced features
- Performance optimizations
- Additional provider support

NICE-TO-HAVE (v3)
- Advanced customization
- Extra integration points
- Optional enhancements
```

1. **Define Constraints**

```python theme={null}
TECHNICAL_LIMITS = {
    "max_input_length": "Clear limits",
    "rate_limits": "API constraints",
    "storage_requirements": "Resource needs",
    "performance_expectations": "Response times"
}
```

Investing time in this planning phase:

* Prevents scope creep during development
* Ensures clear alignment with team expectations
* Makes code structure more maintainable
* Helps predict potential issues before they arise
* Creates clear testing boundaries
* Provides systematic upgrade paths for future versions

> Remember: **It's easier to adjust plans than refactor code. Take time to ask questions and challenge assumptions before implementation begins.**

## Pre-Development Checklist

### 1. Purpose

Define a clear, single-responsibility purpose. Study these examples from the codebase:

> **[SearchAgent](https://github.com/video-db/Director/blob/main/backend/director/agents/search.py)**
>
> Finds and retrieves specific content within videos

> **[ThumbnailAgent](https://github.com/video-db/Director/blob/main/backend/director/agents/thumbnail.py)**
>
> Generates preview images from video frames

> **[UploadAgent](https://github.com/video-db/Director/blob/main/backend/director/agents/upload.py)**
>
> Handles media uploads with format validation and processing

### 2. Background Check

Review the [director/agents/](https://github.com/video-db/Director/tree/main/backend/director/agents) directory for similar functionality

* Consider extending existing agents by:
  * Optimizing core functionality or improving performance
  * Adding new capabilities
  * Enhancing integration points (adding support for better tools, platforms or models)

### 3. Agent architecture

> > **I/O Contract**

The **I/O Contract** defines how your agent interacts with the system, including the expected inputs, outputs, and how they are structured. It ensures consistency and clarity in communication between the agent, the system, and the end user. This contract is critical for integrating your agent with the infrastructure and enabling seamless interaction with other components.

**Input Contract**

The input contract specifies the parameters your agent expects to receive. These parameters can be simple (e.g., a string or number) or complex (e.g., a nested JSON object). The input contract is defined in two parts:

1. **Function Signature**: The `run` method of your agent defines the expected parameters.
2. **JSON Schema**: For complex inputs, a JSON schema is used to describe the structure and constraints of the input data.

**Simple Input Example**

For agents with straightforward inputs, you can define the parameters directly in the `run` method. Here’s an example for a Slack agent:

```python theme={null}
def run(self, message: str, channel: str = None) -> AgentResponse:
    """Send a message to a Slack channel.
    
    Args:
        message (str): The content to send.
        channel (str, optional): The target Slack channel. Defaults to None.
    """
    # Agent logic here
    return AgentResponse(status=AgentStatus.SUCCESS, message="Message sent successfully")
```

**Complex Input Example**

For agents requiring structured inputs (e.g., video generation), you define a JSON schema. This schema is used to validate the input and provide clear documentation for API consumers. Here’s an example for a video generation agent:

```python theme={null}
VIDEO_GENERATION_PARAMETERS = {
    "type": "object",
    "properties": {
        "prompt": {
            "type": "string",
            "description": "Text prompt for video generation",
        },
        "config": {
            "type": "object",
            "properties": {
                "duration": {
                    "type": "integer",
                    "description": "Duration of the video in seconds",
                },
                "style": {
                    "type": "string",
                    "description": "Visual style for the video (e.g., cinematic, photorealistic)",
                },
            },
            "required": ["duration"],  # Mandatory fields
        },
    },
    "required": ["prompt"],  # Mandatory fields
}
```

In this example:

* The `prompt` field is a required string.
* The `config` field is an optional object with nested properties (`duration` and `style`).
* The `required` keyword ensures mandatory fields are validated.

**Output Contract**

The output contract defines how your agent communicates results, errors, and progress updates. This includes:

1. **AgentResponse**: A standardized response format for success or failure.
2. **Progress Updates**: Real-time updates using the `output_message` object.
3. **Frontend Content Handling**: Structured content for rendering in the frontend (e.g., text, video, images).

**AgentResponse**

The `AgentResponse` object is used to return the result of the agent's execution. It includes:

* `status`: Indicates success (`AgentStatus.SUCCESS`) or failure (`AgentStatus.ERROR`).
* `message`: A human-readable message describing the result.
* `data`: Additional data returned by the agent (e.g., generated content).

Example:

```python theme={null}
return AgentResponse(
    status=AgentStatus.SUCCESS,
    message="Video generated successfully",
    data={"video_url": "https://example.com/video.mp4"},
)
```

**Progress Updates**

Use the `output_message` object to provide real-time updates during the agent's execution. For example:

```python theme={null}
self.output_message.actions.append("Generating video...")
self.output_message.push_update()
```

**Frontend Content Handling**

Agents can return different types of content (e.g., text, video, images) using the `output_message.content` list. Each content type is represented by a specific model (e.g., `TextContent`, `VideoContent`). Example:

```python theme={null}
video_content = VideoContent(
    agent_name=self.agent_name,
    status=MsgStatus.progress,
    status_message="Generating video...",
    video=VideoData(stream_url="https://example.com/video.mp4"),
)
self.output_message.content.append(video_content)
self.output_message.publish()
```

> > **Workflow Definition**

Plan and fix the steps to go from input → output. Keep the following factors in mind:

* Input validation & preprocessing
* Resource initialization
* Core processing steps
* Progress updates
* Result formatting & cleanup

> > **Agent & Tool Composition**

Look for composition opportunities with existing agents or tools

* Example 1: ComparisonAgent leveraging VideoGenerationAgent
* Example 2: AudioGenerationAgent using ElevenLabs tool

### 4. Session Management

* The `session` parameter is crucial for maintaining context across multiple interactions, preventing the agent from handling requests without retaining state.
* Assigning a unique name and description aids in debugging and log analysis, making it easier to track agent behavior.
* Flexible parameter handling: Your agent can either get its settings on-the-fly from user input (dynamic parameters), or use pre-defined parameters that you specify upfront. Simple agents often work best with dynamic parameters, while complex agents usually need pre-defined configurations.

```python theme={null}
class NewAgent(BaseAgent):
    def __init__(self, session: Session, **kwargs):
        self.agent_name = "unique_name"
        self.description = "Clear, specific description"
        self.parameters = self.get_parameters()
        super().__init__(session=session, **kwargs)
```

> Note: Parameter handling `self.parameters` changes based on the complexity of the parameters required for the agent:

```python theme={null}
#in case of simple docstring
        self.parameters = self.get_parameters() 

#in case of dictionary of complex parameters:
        self.parameters = AGENT_PARAMETERS
```

**Key Points of User Communication:**

> > **Director's Log (Showing Steps)**

Think of this like a progress bar or status updates you see when installing software. It tells users what's happening behind the scenes.

```python theme={null}
# Shows up as a list of actions in the chat
self.output_message.actions.append("Starting video processing...")
self.output_message.actions.append("Loading AI model...")
self.output_message.actions.append("Creating final video...")
```

> > **Progress Updates (Showing Content Responses)**

This handles the actual content (videos, images, text) and its current state. Think of this like when you upload a file to Google Drive - you see both the file and its upload status.

```python theme={null}
# Create a video player with loading state
content = VideoContent(
    status=MsgStatus.progress,  # Can be progress/success/error
    status_message="Your video is being generated...",  # User-friendly message
)

# Add it to the chat
self.output_message.content.append(content)

# Show it to the user
self.output_message.push_update()
```

This shows up in the chat as:

* A video player
* A loading indicator
* A message saying "Your video is being generated..."

> > **Final Cut (Returning Results)**

When the agent finishes, it returns three things:

* `status:` Did it work? (success/error)
* `message:` What happened? (user-friendly explanation)
* `data:` The actual results (video URLs, text, etc.)

```python theme={null}
self.output_message.status = MsgStatus.success  # Can also be error
self.output_message.status_message = "Your video is ready!"
self.output_message.data = {"video_url": generated_video_url}  # Or relevant result data

# Send the final message to the user
self.output_message.publish()
```

### 5. Error Handling

Clear error responses help the reasoning engine interpret failures, guide users effectively, and determine the next appropriate actions in the workflow.

```python theme={null}
try:
    # Main logic
    self.output_message.actions.append("Current action...")
    self.output_message.push_update()
except Exception as e:
    logger.exception(f"Error in {self.agent_name}")
    content.status = MsgStatus.error
    content.status_message = "User-friendly error message"
    self.output_message.publish()
    return AgentResponse(status=AgentStatus.ERROR, message=str(e))
```

## Best Practices

1. **Clear Agent & Parameter Descriptions**

* Agent definitions help Reasoning Engine select the appropriate agent
* Parameter definitions guide RE in providing correct instructions

```python theme={null}
self.description = "Generates video with AI models, supporting multiple providers and configurations"
```

1. **Use explicit status updates**

```python theme={null}
self.output_message.actions.append("Specific action...")
self.output_message.push_update()
```

1. **Resource Management**

```python theme={null}
try:
    # Resource operations
finally:
    # Cleanup
```

1. **Validate inputs early:**

```python theme={null}
if not required_param:
    raise ValueError("Clear error message explaining what's missing and how to fix")
```

1. **Use appropriate content \[response] types:**

```python theme={null}
`VideoContent`, `TextContent`, `ImageContent`, `SearchResultsContent`
```

## Further Resources and Next Steps

Ready to build your first agent? Start with our sample agents in the `director/agents/` directory or check out these resources:

> **[Director Documentation](https://docs.director.videodb.io/index.html)**
>
> Comprehensive guides and API references for the Director framework

> **[Sample Agents Repository](https://github.com/video-db/Director/tree/main/backend/director/agents)**
>
> Real-world examples and implementations of custom agents

> **[OpenAI Function Schema Guide](https://platform.openai.com/docs/guides/function-calling)**
>
> Learn about parameter schema design and function calling

> **[Discord Community](https://discord.com/invite/py9P639jGz)**
>
> Connect with other agent developers and get support

Need help? Reach out to our community or open an issue on [GitHub](https://github.com/video-db).


# Director Framework
Source: https://docs.videodb.io/pages/build-with-agents/director/index

Director is your agentic framework for video tasks - think of it as ChatGPT for videos.

## What is Director?

Director is your agentic framework for video tasks - think of it as ChatGPT for videos.

It's open source and allows you to build intelligent video agents that can handle complex operations like searching through content, editing videos, creating compilations, and generating new content, all through natural language commands.

For example, a simple natural language command like: `Upload this video and send the highlights to my Slack`, sets everything in motion - Director's reasoning will orchestrate the different agents intelligently to complete the task for you.

### Quick Access

> **[Try Hosted Version](https://chat.videodb.io/)**
>
> Explore Director in the cloud - no setup required

> **[GitHub Repository](https://github.com/video-db/Director)**
>
> View the open source code and contribute

Director framework is built on **VideoDB's 'video-as-data' infrastructure**, enabling your agents to:

* Process and analyze videos at scale
* Search for specific moments or content
* Create clips and compilations instantly
* Add overlays and modifications in real-time
* Integrate with various AI tools and APIs

*You can use it to create quick demos, POCs, showing the power of AI to a non-technical audience, and integration with your own systems.*

Built with flexibility in mind, Director is perfect for developers, creators, and teams looking to harness AI to simplify media workflows and unlock new possibilities.

## How do I try it?

You can quickly try out the hosted version to explore the interface at [https://chat.videodb.io/](https://chat.videodb.io/). It functions similarly to ChatGPT but is specifically designed for videos. Here, multiple agents collaborate to complete tasks. This platform also serves as a playground for exploring your video content uploaded on VideoDB.

The Director's backend and frontend are open source and available for you to modify and customize under the MIT license. Our goal is to help you get started quickly and provide insights into designing systems with Large Language Models (LLMs) at their core, as well as planning and building agentic processes. The frontend is polished and well-designed to integrate into your existing project or app, while the backend is scalable and easy to deploy.

### Open Source Components

> **[Main Project Repo](https://github.com/video-db/Director)**
>
> Core Director framework and backend

> **[Chat Interface](https://github.com/video-db/videodb-chat)**
>
> Web-based chat UI for Director

> **[Video Player](https://github.com/video-db/videodb-player)**
>
> Embeddable video player component

## Features of Director

1. **Chat-Based Video Interaction**

* Interact with your videos seamlessly via a chat-based UI.
* Perform tasks like summarization, chapter creation, and more through simple prompts.

1. **AI-Driven Video Agents**

* Agents like `upload`, `summary`, `search`, `web search`, `dubbing`, `branding`, `dynamic editing`, and `clipping` are available to streamline workflows.

1. **Backend Reasoning Engine**

* An intelligent engine loops over agents based on input and context provided by an LLM, ensuring efficient task management.

1. **Video Player Integration**

* Built-in video player with a **collection view** to organize and navigate through your video assets.

1. **Customization and Extensibility**

* Add new agents and integrate your specific workflows effortlessly.

1. **Open Source and Developer-Friendly**

* Fully open-sourced to encourage collaboration and innovation.

## Demos

View the complete [Director demo playlist](https://www.youtube.com/playlist?list=PLhxAMFLSSK039xl1UgcZmoFLnb-qNRYQw) for more examples.

## Learn More About Director

> **[Setup Director](/pages/build-with-agents/director/setup-director)**
>
> Get started with Director installation and configuration

> **[Create Custom Agent](/pages/build-with-agents/director/create-custom-agent)**
>
> Build your own agents within the Director framework

> **[MCP Server](/pages/mcp/index)**
>
> Integrate MCP servers with your Director agents


# Setup Director
Source: https://docs.videodb.io/pages/build-with-agents/director/setup-director

Install and configure the Director framework locally to build custom video agents and workflows for your applications.

* **Clone the repository:**

Begin by cloning the **Director** repository and navigating to the project directory:

```bash theme={null}
git clone https://github.com/video-db/Director.git
cd Director
```

* **Run the setup script:**

Execute the setup script to install all necessary dependencies:

```bash theme={null}
./setup.sh
```

* **Run the Application**

You can start both the backend and frontend servers with:

```bash theme={null}
make run
```

If you want to run them separately:

* **Backend only:** make run-be (Runs at [http://127.0.0.1:8000](http://127.0.0.1:8000))
* **Frontend only:** make run-fe (Runs at [http://127.0.0.1:8080](http://127.0.0.1:8080))

You are now ready to begin creating an agent in VideoDB Director!


# About Us
Source: https://docs.videodb.io/pages/community/about-us

Building the perception layer AI deserves

## Building the Perception Layer AI Deserves

We are a team of engineers who have spent years working at the intersection of AI, video systems, and cloud infrastructure. We came together because we saw the same problem: video is stuck in the past.

Early experiments with LLMs made one thing clear to us: Agents are coming, but the infrastructure isn't. The tools we have today were built for humans watching screens, not for software that needs to perceive the world.

For AI to be truly useful in real-world environments, it needs more than just text processing. It needs a real-time perception layer—the ability to see, understand, and act on visual data instantly.

So we're building something new. Video infrastructure that treats video as a real-time data stream, not a static file on a server. Searchable. Programmable. Intelligent.

This is the foundation AI needs to work with vision and audio the way it works with text.

**This is VideoDB.**

***

## Who We Are

![VideoDB Team](https://mintcdn.com/videodb/Szr-nrLcIsMc77bo/images/team.jpg?fit=max\&auto=format\&n=Szr-nrLcIsMc77bo\&q=85\&s=1ac9e1896ff512b3071b16acffb21db7)

We are a remote tribe. Headquartered in San Francisco, building from India. You'll find us where nature, good vibes, music, and creativity thrive—often trading city smog for the quiet peaks of Dharamshala.

We've been together for the past five years, solving complex challenges in video infrastructure for AI. We believe that if AI can see what we see, hear what we hear, and listen to us in real time, it can amplify human expression.

***

## What We Built

Over the years, we've built a unique media format that's optimized for AI processing. It powers:

* A universal ingestion engine
* A stack that lets you plug in models to understand your frames and audio without touching raw video files or streams
* The ability to build indexes and retrieve episodic memory
* A streaming engine that your AI can use
* A video editing stack that AI can control to edit video information in real time

Together, this lets you build agents that can see, understand, and act.

## Join Us

> **[Internship Program](/pages/community/internship-program)**
>
> Work on cutting-edge AI and video infrastructure

> **[Connect With Us](/pages/community/connect)**
>
> Discord, GitHub, Twitter, and more


# Agentic Systems
Source: https://docs.videodb.io/pages/community/agentic-systems

Collaborate with us on agentic AI and physical AI research - multi-agent systems, embodied agents, and real-world AI applications

We're building the perception layer for AI agents - giving machines eyes and ears. If you're working on agentic systems or physical AI, we'd love to collaborate.

***

## Areas of Interest

### Agentic AI Systems

> **Multi-Agent Collaboration**
>
> Frameworks for agents to work together - sharing context, delegating tasks, and coordinating actions across video, audio, and text modalities.

> **Agent Memory & Perception**
>
> How agents remember, recall, and reason about continuous media. Building long-term memory systems that work with video streams and recordings.

> **Autonomous Video Understanding**
>
> Agents that watch, understand, and act on video content - from surveillance feeds to screen recordings to live streams.

> **Tool Use & Action**
>
> Enabling agents to take meaningful actions based on what they see and hear - editing, annotating, searching, and generating video content.

***

### Physical AI & Embodied Agents

> **Robotics & Perception**
>
> Vision systems for robots and autonomous machines. Processing real-world video streams for navigation, manipulation, and interaction.

> **Embodied Learning**
>
> Training agents that learn from video demonstrations. Sim-to-real transfer and video-based imitation learning.

> **Real-World Deployment**
>
> Taking agentic systems from lab to production. Handling edge cases, failures, and real-world complexity.

***

### Video Understanding for Agents

> **Scene & Activity Detection**
>
> Identifying scenes, activities, and events in video streams. Creating coherent segments for agent reasoning.

> **Temporal Reasoning**
>
> Understanding causality, sequences, and time in video. What happened, what's happening, what might happen next.

> **Multimodal RAG**
>
> Retrieval systems that work across video, audio, and text. Finding relevant moments and context for agent decision-making.

***

### Code & Model Development

> **Open Source Models**
>
> Training and fine-tuning video models. Managing terabytes of training data, annotations, and evaluation pipelines.

> **Benchmarking**
>
> Evaluating vision and video models. Pushing beyond current benchmarks to test real-world agent capabilities.

> **Code Generation from Video**
>
> Systems that watch tutorials, demos, or documentation and generate working code.

***

## Let's Collaborate

If you're working on agentic AI or physical AI and want to collaborate, reach out. We can provide:

* **Infrastructure support** for managing video data at scale
* **API access** for video understanding and manipulation
* **Technical collaboration** with our team on shared research

> **[Get in Touch](mailto:hello@videodb.io)**
>
> Email [hello@videodb.io](mailto:hello@videodb.io) to discuss collaboration

> **[Join Our Discord](https://discord.gg/py9P639jGz)**
>
> Chat with our team and community


# Love from the community
Source: https://docs.videodb.io/pages/community/community-love

Real posts from the devs, founders, and engineers<br/>shipping on VideoDB.

[Join Our Discord ](https://discord.gg/py9P639jGz)


# Connect
Source: https://docs.videodb.io/pages/community/connect

Join our community - Discord, GitHub, Twitter, YouTube, and LinkedIn

## Join the Community

> **[Discord](https://discord.gg/py9P639jGz)**
>
> Chat with developers, get help, share what you're building

> **[GitHub](https://github.com/video-db)**
>
> Star our repos, contribute, report issues

> **[Twitter / X](https://x.com/videodb_io)**
>
> Follow for updates, demos, and announcements

> **[YouTube](https://www.youtube.com/@video_db)**
>
> Tutorials, demos, and deep dives

> **[LinkedIn](https://www.linkedin.com/company/videodb)**
>
> Company updates and team news

***

## Workshops & Meetups

We host workshops and builder sessions across the world. Come build with us, learn from the community, and meet fellow developers working on video AI and agentic systems.

> **[VideoDB Events on Luma](https://lu.ma/videodb)**
>
> Check for upcoming workshops in your city and join our builder sessions

Look for a workshop in your town — we'd love to meet you in person. Our builder sessions are hands-on, collaborative, and a great way to dive deep into video infrastructure and AI agents.

***

## Meet the Founder

> **[Ashutosh Trivedi](https://www.linkedin.com/in/ashutoshtrivedi/)**
>
> Connect with Ashutosh on LinkedIn - always happy to chat about video AI, agents, and building developer tools

![Ashutosh Trivedi](https://mintcdn.com/videodb/2LcKqKr9brao1QUY/images/ashutosh-trivedi.jpg?fit=max\&auto=format\&n=2LcKqKr9brao1QUY\&q=85\&s=a6da82706cd148d6ae45448ba6fd570c)

Ashu has spent decades at the crossroads of machine learning and distributed computing.

He believes beauty emerges when complex systems are made simple to use.

VideoDB has been a journey to find a tribe that loves innovation, grow together, and build a foundation for technology and research where we create useful things that truly matter.

If you catch him offline, talk to him about philosophy, music, maths, and physics.

> **[LinkedIn](https://www.linkedin.com/in/ashutoshtrivedi/)**
>
> Connect on LinkedIn

> **[Twitter/X](https://twitter.com/ashu_trv)**
>
> Follow on Twitter

> **[GitHub](https://github.com/codeAshu)**
>
> View on GitHub

## Get in Touch

> **[General Inquiries](mailto:hello@videodb.io)**
>
> [hello@videodb.io](mailto:hello@videodb.io)

> **[Careers & Internships](mailto:careers@videodb.io)**
>
> [careers@videodb.io](mailto:careers@videodb.io)

***

## Website

> **[videodb.io](https://videodb.io)**
>
> Visit our main website


# Internship Program
Source: https://docs.videodb.io/pages/community/internship-program

Join VideoDB and work on the perception layer for AI agents

## What We're Building

VideoDB is the perception, memory, and action layer for AI agents operating on video and audio. We turn continuous media—files, live streams, and desktop capture—into structured context, searchable moments with playable evidence, and event triggers that drive automations.

**The core loop: See → Understand → Act**

* **See**: Ingest video and audio from files, RTSP streams, or desktop capture
* **Understand**: Build indexes that convert raw media into timestamped, searchable memory
* **Act**: Trigger alerts, automate workflows, and edit video programmatically

We're building for developers who need to give their agents eyes and ears—whether that's media archives, real-time monitoring, desktop agents, or automation workflows.

***

## What You'll Work On

> **Platform Engineering**
>
> * Backend systems in Python, Node.js, or Go
> * APIs, serverless architecture, and scalable video infrastructure
> * Real-time pipelines for live streams and desktop capture

> **AI & Vision Systems**
>
> * Build indexes that turn video into structured understanding
> * Work on scene segmentation, frame sampling, and multimodal search
> * Develop agent memory and retrieval systems

> **Open Source**
>
> * Contribute to [Director](https://github.com/video-db/Director)—our open source AI video agent
> * Build with [Capture SDK](https://github.com/video-db/videodb-capture-sdk) for screen and audio capture
> * Create workflow templates for n8n, Zapier, and MCP-based tools

> **Developer Experience**
>
> * Build intuitive interfaces and developer tools
> * Create tutorials, demos, and documentation
> * Grow our community of builders

***

## Details

* **Location**: Remote-first
* **Duration**: 3-6 months, flexible start dates
* **Compensation**: Competitive stipend + mentorship

***

## Apply

Send your application to **[careers@videodb.io](mailto:careers@videodb.io)** with subject: **Internship Application**.

Include your GitHub or relevant projects if you have them.


# LlamaIndex Retriever
Source: https://docs.videodb.io/pages/community/open-source/llamaindex-retriever

VideoDB's integration with LlamaIndex enables RAG applications to produce multimodal outputs including video streams alongside text results.

> **[LlamaIndex Documentation](https://docs.llamaindex.ai/en/stable/examples/retrievers/videodb_retriever.html)**
>
> View the official LlamaIndex VideoDB Retriever documentation

Today, most of the RAG applications are limited to generating text responses. However, VideoDB's integration with the popular RAG framework, LlamaIndex, introduces a groundbreaking dimension to RAG, upon video repositories.

This integration enables RAG to produce multimodal outputs, including video streams alongside text results, enhancing the verifiability of these results. It also opens up numerous practical applications where viewing the stream is essential, such as in security camera footage, OTT content, and news and media.

![Multimodal RAG with LlamaIndex architecture showing video and text integration](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/open-source/llama-index.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=abf0959a9d6ff2c025d92d3dbccbff58)

To quickly experiment with the VideoDB Retriever within the LlamaIndex framework, check out the [Simple Video Search Notebook](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/integrations/llama-index/simple_video_rag.ipynb).


# PromptClip
Source: https://docs.videodb.io/pages/community/open-source/promptclip

AI-powered video editing with natural language prompts - find perfect moments in videos instantly using spoken and visual content analysis.

> **[Try PromptClip](https://github.com/video-db/PromptClip)**
>
> Open source repo for AI-driven video editing

PromptClip is your new superpower for video editing with prompts. While editing, we all wanted to find that perfect moment in a video with ease. Say goodbye to manually skimming and seeking through video, and say hello to instant, AI-driven video consumption and creation.

LLMs are great with text, but they don't help you consume or create video clips. We've built [PromptClip](https://github.com/video-db/PromptClip), an open source repo to get you started on AI-driven video editing. For example, you can just say:

"*Find the moment where Mr. Bean is attempting to cheat by peeking at the answer sheet of the man beside him*"

[Video player](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/f711075f-4573-4435-a190-7cb1817ad8b8.m3u8)

### Process

**1. Describe what you want**

Use natural language to describe what you want.

**2. Run prompts**

Run prompts on the visual scenes and spoken content with the choice of your LLM.

**3. Get video stream**

Instantly get video compilation stream from VideoDB.

![PromptClip architecture showing the process of describing prompts, running analysis, and getting video streams](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/open-source/prompt-clip.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=9ec22084ead55ce030f1fdc3730c7ea4)

## Download PromptClip

**1. Get your API key**

Get your API key from the [VideoDB console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) (Free for first 50 uploads, no credit card required)

**2. Set up your environment**

* Add your `VIDEO_DB_API_KEY` to the `.env` file
* Include your preferred LLM API key (`OPENAI_API_KEY` or `ANTHROPIC_KEY`) in `.env`

**3. Install dependencies**

Run `pip install -r requirements.txt`

**4. Explore notebooks**

Launch the Jupyter notebooks to start experimenting:

* Create clips using only spoken content: [PromptClip Spoken Notebook](https://github.com/video-db/PromptClip/blob/scene-support/PromptClip_spoken.ipynb)
* Create clips by analyzing visual content: [PromptClip Visual Notebook](https://github.com/video-db/PromptClip/blob/scene-support/PromptClip_visual.ipynb)
* Using combined audio-visual analysis: [PromptClip Multimodal Notebook](https://github.com/video-db/PromptClip/blob/scene-support/PromptClip_multimodal.ipynb)

## PromptClip in Action

### 1. Spoken Content Analysis

**Prompt: Find every moment where a deal was offered**

[Video player](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/f46f8de8-725a-410f-a83b-3b5e73a6b29a.m3u8)

**Source:** [Shark Tank](https://www.youtube.com/watch?v=HpUR7-Oe1ss)

**Prompt: Show me moments in the video where the host discusses or reveals the pricing of the gadgets.**

[Video player](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/5c4f4752-eca4-4f76-9d68-1a855d75de88.m3u8)

**Source:** [Useful Gadgets](https://www.youtube.com/watch?v=bGmXrMW9ucU)

### 2. Visual/Scene Analysis

**Prompt: Find the moment where Mr. Bean is attempting to cheat by peeking at the answer sheet of the man beside him**

[Video player](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/f711075f-4573-4435-a190-7cb1817ad8b8.m3u8)

### 3. Using both Visual and Spoken

**Prompt: Find scenes explaining cricket rules using infographics**

[Video player](https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/e6698737-6fc6-4bd2-b2e9-7833c03f99d7.m3u8)

### Why PromptClip Will Transform Your Work

> **Efficiency**
>
> Reduce hours of manual video searching to mere seconds.

> **Creativity**
>
> Generate fresh perspectives and ideas from your existing content.

> **Precision**
>
> Extract exactly what you need, not just approximate matches.

> **Flexibility**
>
> Works with various video types, topics, and [languages](/pages/understand/indexing-pipelines/understanding-artifacts).

PromptClip is more than a tool; it's a new way of content interaction — The natural language way! Whether you're a content creator, researcher, or developer, PromptClip empowers you to unlock the full potential of your video content.

## Watch PromptClip in Action

Here you can check more PromptClip examples:

**Prompt: "Find details about every sponsor"**

[YouTube video player](https://www.youtube.com/embed/tYsU-fcIWL8)

**Prompt: "Find sentences where anxiety is discussed"**

[YouTube video player](https://www.youtube.com/embed/0s4fxzqaYws)

**Prompt: "How to control sugar cravings"**

[YouTube video player](https://www.youtube.com/embed/4uhIMnUnvJc)

## The Magic Behind PromptClip: VideoDB's Indexing Methods

PromptClip leverages VideoDB's advanced indexing techniques and Large Language Models (LLMs) to analyze and manipulate video content with unmatched power.

> **Spoken Word Indexing**
>
> * **Process**: VideoDB transcribes the audio content and creates a searchable index of spoken words.
> * **Implementation**: Uses advanced speech recognition algorithms to generate accurate transcripts.

> **Visual/Scene Indexing**
>
> * **Process**: Analyzes video frames to identify objects, actions, and scene compositions.
> * **Implementation**: Employs sophisticated scene extraction algorithms and vision models for comprehensive visual understanding.
> * **Customization**: Allows for different extraction algorithms and custom prompts for scene description. Find detailed documentation in our [Scene Indexing Guide](/pages/understand/indexing-pipelines/create-an-index).

## Join the VideoDB Community

Your ideas drive PromptClip's evolution. Get involved:

> **[GitHub](https://github.com/video-db/PromptClip)**
>
> Report issues, suggest features, or contribute code via pull requests

> **[Discord](https://discord.gg/py9P639jGz)**
>
> Share your innovative use cases and connect with the community

> [!NOTE]
>
> For more information and support, refer to the [VideoDB documentation](https://docs.videodb.io/)


# Data Model
Source: https://docs.videodb.io/pages/core-concepts/data-model

Core objects in VideoDB: Collections, Videos, RTStreams, Indexes, and more. Understanding these objects helps you navigate the SDK and API.

## When to Use This Reference

Use this page when you need to:

* Understand what each SDK class represents
* Know which object to use for your use case
* Map between the SDK, API, and documentation

## Quick Example

```python theme={null}
import videodb
from videodb.editor import Timeline, Track, Clip, VideoAsset

conn = videodb.connect()                    # Connection
coll = conn.get_collection()                # Collection
video = coll.upload(url="...")              # Video (file-based)
rtstream = coll.connect_rtstream(...)       # RTStream (live)
understanding = video.understand(...)       # Understanding
analyzer = understanding.get_analyzer(...)  # Analyzer / artifact handle
index = video.index(name="scene", source=analyzer)  # Index
results = video.semantic_search(...)        # SearchResult

# Editor (separate module)
timeline = Timeline(conn)                   # Timeline
```

## Core Objects

### Connection

The entry point to VideoDB. Holds your API key and provides access to collections, events, and WebSocket connections.

```python theme={null}
conn = videodb.connect()  # Uses VIDEO_DB_API_KEY env var
conn = videodb.connect(api_key="your-key")
```

**Key methods:**

* `get_collection()` - Get default or specific collection
* `create_event()` - Create reusable detection rules
* `connect_websocket()` - Establish real-time connection

***

### Collection

A container for organizing media assets. Every video and stream belongs to a collection.

```python theme={null}
coll = conn.get_collection()
coll = conn.get_collection(collection_id="c-xxx")
```

**Key methods:**

* `upload(url=...)` - Upload video from URL
* `upload(file_path=...)` - Upload local file
* `connect_rtstream(...)` - Connect live stream
* `search(...)` - Search across all assets in collection
* `get_videos()` - List video assets
* `list_rtstreams()` - List live streams

***

### Video

A file-based media asset. Represents uploaded videos that can be indexed, searched, and edited.

```python theme={null}
video = coll.upload(url="https://example.com/video.mp4")
video = coll.get_video(video_id="m-xxx")
```

**Key attributes:**

* `id` - Unique identifier (m-xxx)
* `name` - Display name
* `length` - Duration in seconds
* `stream_url` - Playback URL

**Key methods:**

* `understand(...)` - Create finite analyzer artifacts
* `index(name=..., source=...)` - Index an analyzer artifact or temporal records
* `semantic_search(...)`, `query(...)`, `aggregate(...)` - Retrieve indexed video context
* `generate_stream(...)` - Get playback URL for time range
* `play()` - Open in browser

***

### RTStream

A real-time media stream. Represents live video from RTSP feeds, cameras, or desktop capture.

```python theme={null}
rtstream = coll.connect_rtstream(
    name="Lobby Camera",
    url="rtsp://user:pass@host:554/stream"
)
```

**Key attributes:**

* `id` - Unique identifier (rts-xxx)
* `name` - Display name
* `status` - connected, stopped, etc.

**Key methods:**

* `understand(...)` - Start a continuous VLM understanding job
* `index(name=..., source=...)` - Index its VLM output descriptor
* `search(query, index_id=...)` - Search indexed stream windows
* `generate_stream(start, end)` - Get playback URL

***

### CaptureSession

Desktop capture for AI agents. Stream screen, mic, and system audio from any desktop with real-time AI events.

```python theme={null}
# Backend creates session (holds API key)
cap = conn.create_capture_session(
    end_user_id="user_abc",
    callback_url="https://your-backend.com/webhooks",
    metadata={"app": "my-ai-copilot"}
)

# Generate short-lived token for desktop client
token = conn.generate_client_token(expires_in=600)
```

**Key attributes:**

* `id` - Unique identifier (cap-xxx)
* `status` - created, starting, active, stopped, exported, or failed
* `end_user_id` - Your user identifier

**Key concepts:**

* **Two-component architecture** - Backend holds API key, desktop client uses tokens
* **Creates RTStreams** - Each active channel (mic, screen, audio) creates an RTStream
* **Real-time events** - Transcripts, visual indexes, and alerts delivered via webhook/WebSocket

**Capture channels:**

* Microphone - User's voice
* Screen/Display - Visual content
* System Audio - Audio from apps
* Camera - Webcam feed

***

### Understanding, Analyzer, and Index

For a finite video, `Understanding` runs one or more analyzers; each named `Analyzer` exposes a reusable artifact. An `Index` turns that artifact into a retrieval-ready structure.

```python theme={null}
# On a finite video
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {"prompt": "Describe the scene and identify key moments"},
        }
    ]
)
understanding.wait_until_complete()
scene = understanding.get_analyzer("scene")
index = video.index(name="scene", source=scene)
index.wait_until_complete()

# On an RTStream, VLM understanding exposes a continuous output descriptor.
live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Monitor for safety violations"},
        }
    ],
    store=True,
)
live_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)
```

**Key concepts:**

* Analyzers are **prompt-driven** - you define what to extract
* Indexes are **additive** - create multiple indexes on the same media
* Indexes are **non-destructive** - add or remove without affecting source
* Finite indexes can use visual or spoken analyzer artifacts; initial RTStream indexes use VLM descriptors

**Key methods:**

* `get_analyzer(name)` - Retrieve a named finite analyzer handle
* `get_output()` - Inspect an analyzer artifact

**RTStreamIndex-only methods:**

* `create_alert(event_id, ...)` - Attach a live alert trigger
* `start()` / `stop()` - Control continuous indexing

***

### SearchResult

The result of a search query. Contains shots (moments) with timestamps and playable evidence.

```python theme={null}
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {"prompt": "Describe the scene and identify key moments"},
        }
    ]
)
understanding.wait_until_complete()
scene = understanding.get_analyzer("scene")
index = video.index(name="scene", source=scene)
index.wait_until_complete()

results = video.semantic_search(
    query="important announcement",
    index_ids=[index.index_id],
    top_k=5,
)

for shot in results.get_shots():
    print(f"{shot.start}s - {shot.end}s")
    shot.play()  # Verify the result
```

**Key attributes:**

* `shots` - List of matching moments
* `collection_id` - Source collection

***

### Shot

A single moment from search results. Contains timestamps, text, and playable stream URL.

**Key attributes:**

* `start` - Start timestamp (seconds or Unix time)
* `end` - End timestamp
* `text` - Content/description at this moment
* `search_score` - Relevance score (0-1)
* `stream_url` - Direct playback URL

**Key methods:**

* `generate_stream()` - Get playback URL
* `play()` - Open in browser

***

### Event

A reusable detection rule defined in plain English. Events describe what to detect; alerts define where and how to deliver.

```python theme={null}
event_id = conn.create_event(
    event_prompt="Detect when someone enters restricted area",
    label="intrusion_detected"
)
```

***

### Alert

Wires an event to an index and defines delivery. When the event condition is detected, the alert fires.

```python theme={null}
websocket = conn.connect_websocket(coll.id)
await websocket.connect()

alert_id = live_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.example/webhooks/alerts",
    ws_connection_id=websocket.connection_id  # Optional for real-time
)
```

**Delivery options:**

* `callback_url` - Webhook for automation
* `ws_connection_id` - WebSocket for real-time

***

### Editor Module (Programmable Editing)

The action layer for programmable video editing lives in `videodb.editor`. It uses a 4-layer architecture:

**Asset → Clip → Track → Timeline**

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

# 1. Asset - your raw content
video_asset = VideoAsset(id=video.id, start=0)

# 2. Clip - how it appears (duration, position, effects)
clip = Clip(asset=video_asset, duration=10)

# 3. Track - when it plays (timeline lane)
track = Track()
track.add_clip(0, clip)  # Start at 0 seconds

# 4. Timeline - the final canvas
timeline = Timeline(conn)
timeline.add_track(track)
stream_url = timeline.generate_stream()
```

**Asset types:**

* `VideoAsset` - Video content
* `AudioAsset` - Music, voiceover, sound effects
* `ImageAsset` - Logos, watermarks, backgrounds
* `TextAsset` - Custom text overlays
* `CaptionAsset` - Auto-generated subtitles

**Key concepts:**

* Clips on the **same track** play sequentially
* Clips on **different tracks** play simultaneously (layered)
* Later tracks render **on top** of earlier tracks

***

## Object Hierarchy

```
videodb
├── Connection (videodb.connect())
│   ├── Collection
│   │   ├── Video (file-based media)
│   │   │   ├── Understanding
│   │   │   │   └── Analyzer / Artifact
│   │   │   │       └── Index
│   │   │   └── semantic_search() → SearchResult → Shot[]
│   │   │   └── search() → SearchResponse → Shot[]
│   │   ├── RTStream (live media)
│   │   │   ├── Understanding
│   │   │   │   └── VLM output descriptor
│   │   │   │       └── Index
│   │   │   │           └── Alert
│   │   │   └── search() → RTStreamSearchResult
│   │   └── search() → SearchResponse (collection-wide)
│   ├── CaptureSession (desktop capture)
│   │   ├── creates RTStreams (mic, screen, system audio)
│   │   └── generate_client_token() → short-lived token
│   ├── Event (reusable detection rules)
│   └── WebSocket (real-time events)
│
videodb.editor (programmable editing)
├── Timeline (final canvas, resolution, rendering)
├── Track (timeline lanes, sequencing)
├── Clip (duration, position, effects)
└── Assets
    ├── VideoAsset
    ├── AudioAsset
    ├── ImageAsset
    ├── TextAsset
    └── CaptionAsset
```

## Next Steps

> **[Indexes Deep Dive](/pages/core-concepts/indexes-and-search)**
>
> Learn how to create and use indexes effectively

> **[Search & Retrieval](/pages/core-concepts/indexes-and-search)**
>
> Master search patterns and result handling


# Events & Real-time
Source: https://docs.videodb.io/pages/core-concepts/events-and-realtime

Events and alerts turn understanding into action. VideoDB processes live media with sub-second latency, delivering events via WebSocket or webhook.

## Quick Example

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
rtstream = coll.get_rtstream("rts-xxx")
ws = conn.connect_websocket()
await ws.connect()

# Continuously create and store scene output
scene_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe activity and detect unusual behavior"},
        }
    ],
    store=True,
)

# Index the stored scene output before attaching alerts
scene_index = rtstream.index(
    name="security_monitor",
    source=scene_understanding.outputs["scene"],
    use_for=["semantic"],
)

# Create detection rule (reusable)
event_id = conn.create_event(
    event_prompt="Detect intruder",
    label="security_alert"
)

# Wire to the V2 scene index; callback delivery remains required with WebSocket
scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts",  # Webhook
    ws_connection_id=ws.connection_id                 # Real-time
)

# Inspect WebSocket messages before accessing fields; their decoded payload shape is pipeline-specific.
async for event in ws.receive():
    print(event)
```

***

## Events vs Alerts

| Concept | Purpose | Example |
| :- | :- | :- |
| **Event** | What to detect (reusable rule) | "Detect person without helmet" |
| **Alert** | Where to deliver (wiring) | Webhook URL, WebSocket ID |

```python theme={null}
# Reuse the same event with the V2 scene index
safety_event = conn.create_event(
    event_prompt="Detect safety violation",
    label="safety"
)

# Alerts wire events to specific indexes
scene_index.create_alert(
    event_id=safety_event,
    callback_url="https://your-backend.com/alerts",
)
```

***

## Delivery Methods

| Method | Latency | Use Case |
| :- | :- | :- |
| **WebSocket** | Real-time | Dashboards, interactive apps |
| **Webhook** | Near real-time | Automation, integrations |

### WebSocket

```python theme={null}
ws = conn.connect_websocket()
await ws.connect()

async for event in ws.receive():
    print(event)  # Inspect the decoded payload before accessing fields.
```

### Webhook

```python theme={null}
scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts"
)
```

Illustrative callback payload; inspect and validate the delivery your endpoint receives:

```json theme={null}
{
  "channel": "alert",
  "rtstream_id": "rts-xxx",
  "data": {
    "event_id": "event-xxx",
    "label": "intrusion",
    "triggered": true,
    "confidence": 0.92,
    "start": 1710000012340,
    "end": 1710000018900
  }
}
```

***

## Latency Profile

| Operation | Typical Latency |
| :- | :- |
| Alert trigger | \< 1 second |
| Transcript event | 1-2 seconds |
| Visual index event | 2-5 seconds |
| Search query | \< 500ms |

***

## Delivery Guarantees

| Method | Guarantee | Handle |
| :- | :- | :- |
| WebSocket | At-most-once | Client reconnects |
| Webhook | At-least-once | Idempotency checks |

```python theme={null}
# Webhook idempotency
@app.post("/webhooks")
async def handle(request):
    event = await request.json()
    if already_processed(event["event_id"]):
        return {"status": "ok"}
    process(event)
```

***

## Event Channels

The current V2 alert channel is produced by the V2 scene index above:

| Channel | Source | Content |
| :- | :- | :- |
| `alert` | `scene_index.create_alert()` | Alert notifications |

The following channel names are available through RTStream helper APIs:

| Channel | Source | Content |
| :- | :- | :- |
| `transcript` | `start_transcript()` | Speech-to-text |
| `scene_index` | `index_visuals()` | Visual analysis |
| `audio_index` | `index_audio()` | Audio analysis |

***

## Best Practices

1. **Make events reusable** - Define once, use across streams
2. **Be specific in prompts** - "Detect person falling" beats "detect problems"
3. **Use both methods** - WebSocket for UI, webhooks for automation
4. **Handle idempotency** - Webhooks may deliver duplicates

## What You Can Build

> **[Baby Crib Monitoring](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> Real-time infant monitoring with AI-powered alerts

> **[Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Sub-second alerts when unauthorized access is detected

> **[Traffic Violations](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Real-time detection of red light and stop sign violations

***

## Next Steps

> **[RTStream Reference](/pages/ingest/live-streams/rtsp-ingest)**
>
> Complete real-time API

> **[Webhooks Guide](/pages/act/live-action/alerts-and-callbacks)**
>
> Setting up callbacks


# Indexes & Search
Source: https://docs.videodb.io/pages/core-concepts/indexes-and-search

Understand VideoDB's new Understand → Index → Retrieve workflow.

VideoDB separates video retrieval into three steps:

```text theme={null}
Understand → Index → Retrieve
```

## The model

| Step | What it does | SDK surface |
| - | - | - |
| **Understand** | Runs analyzers over video and creates reusable artifacts such as transcripts, objects, OCR, brand signals, activity, location, or VLM scene outputs. | `video.understand(...)` |
| **Index** | Chooses which artifact fields should become searchable, filterable, aggregatable, or sortable. | `video.index(...)` |
| **Retrieve** | Searches, filters, aggregates, and returns timestamped moments as playable evidence with selected fields from indexed artifacts. | `search()`, `semantic_search()`, `query()`, `aggregate()` |

Understanding finds what is in the video. Indexing organizes those findings for retrieval. Search, semantic search, query, and aggregation use those indexes to answer user questions.

> [!NOTE]
>
> Sandbox Compute is an independent runtime option for supported models. Use it during the Understand step when you want open-weight or specialized models; the Index step still uses the same `video.index(...)` interface.

> [!NOTE]
>
> RTStreams follow the same model with continuous jobs: `rtstream.understand(...)` keeps producing VLM artifacts from live windows, and `rtstream.index(...)` keeps materializing them for search and alerts. Initial RTStream support is VLM-only with time-based segmentation. See [Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index).

***

## Quick example

```python theme={null}
import videodb

conn = videodb.connect()
video = conn.get_collection().get_video("m-xxx")

# 1. Understand the video with defaults
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {"type": "vlm", "name": "scene"},
    ],
)

understanding.wait_until_complete()

transcript = understanding.get_analyzer("transcript")
scene = understanding.get_analyzer("scene")

# 2. Index selected artifacts and wait for semantic retrieval
transcript_index = video.index(name="transcript", source=transcript)
scene_index = video.index(name="scene", source=scene)
transcript_index.wait_until_complete()
scene_index.wait_until_complete()

# 3. Retrieve playable evidence
results = video.semantic_search(
    query="person talking about a product while holding a phone",
    index_ids=[transcript_index.index_id, scene_index.index_id],
    top_k=10,
)

for shot in results.get_shots():
    print(shot.start, shot.end)
    stream_url = shot.generate_stream()
```

***

## What is an artifact?

An artifact is the output of one analyzer. Examples:

| Analyzer | Artifact | Example contents |
| - | - | - |
| `spoken_words` | `transcript` | timed spoken text |
| `object_detection` | `objects` | labels, counts, boxes |
| `vlm` | `scene` | scene descriptions and structured visual reasoning |
| `ocr` | `ocr` | visible text |
| `brand_detection` | `brands` | detected brands/logos/signals |

Artifacts are scene-aware: each output is tied to timestamps such as `start` and `end`.

***

## Retrieval modes

| API | Use when you want to... | Example |
| - | - | - |
| `search()` | use high-level natural-language search | “find scenes where someone talks about Nike while holding a phone” |
| `semantic_search()` | directly search semantic/vector fields | “happy reunion outside a shop” against the `scene` index |
| `query()` | apply explicit structured filters | `object_labels contains "phone"` |
| `aggregate()` | count, group, or facet indexed fields | count scenes by brand |

Moment-returning retrieval APIs include playable evidence by default. A retrieval request can also combine multiple indexes. For example:

> Find scenes where a person talks about Nike.

VideoDB can combine:

* a scene index for visual/person context
* a transcript index for “talks about Nike”
* a brand index for `Nike`
* playable evidence, transcript text, frames, and timestamps requested with `return_fields`

***

## Next steps

> **[Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Create reusable video artifacts with analyzers.

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Turn artifacts into searchable, queryable indexes.

> **[Sandbox Compute](/pages/core-concepts/sandbox-compute)**
>
> Run supported open-weight and specialized models.


# Core Concepts Overview
Source: https://docs.videodb.io/pages/core-concepts/overview

VideoDB is the perception, memory, and action layer for AI agents operating on video and audio. Every workflow follows the same loop, whether you're processing files, live streams, or desktop capture.

## The Platform Loop

```
See (Ingest) → Process → Understand (Indexes) → Remember → Retrieve (Search) → Act
```

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

# SEE: Ingest from any source
video = coll.upload(url="https://example.com/video.mp4")

# UNDERSTAND: Create a reusable scene artifact and index it
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {"prompt": "Extract key moments"},
        }
    ]
)
understanding.wait_until_complete()

scene = understanding.get_analyzer("scene")
scene_index = video.index(name="key_moments", source=scene)
scene_index.wait_until_complete()

# RETRIEVE: Search with natural language
results = video.semantic_search(
    query="important announcement",
    index_ids=[scene_index.index_id],
    top_k=5,
)

# ACT: Generate outputs, trigger actions
for shot in results.shots:
    print(f"{shot.start}s - {shot.end}s")
    shot.play()  # Playable evidence
```

***

## See (Ingest)

Get video and audio from anywhere into VideoDB.

| Source | Method |
| :- | :- |
| File URL | `coll.upload(url="https://...")` |
| Local file | `coll.upload(file_path="./video.mp4")` |
| RTSP stream | `coll.connect_rtstream(url="rtsp://...", name="Camera")` |
| Desktop capture | Capture SDK (screen, mic, camera) |

```python theme={null}
# File-based
video = coll.upload(url="https://example.com/meeting.mp4")

# Live stream
rtstream = coll.connect_rtstream(
    name="Security Camera",
    url="rtsp://user:pass@host:554/stream"
)
```

***

## Process

Understanding configuration converts raw media into reusable, processable artifacts before indexing.

* **Scene segmentation** - Time-based or shot-based
* **Frame sampling** - Control which frames to analyze
* **Audio chunking** - Word, sentence, or time-based segments

```python theme={null}
# Time-based finite-video analysis: one frame every 10 seconds
understanding = video.understand(
    segmentation={"type": "time", "seconds": 10},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe the scene."},
        }
    ],
)
understanding.wait_until_complete()

# RTStreams continuously produce VLM output descriptors.
live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe this stream window."},
        }
    ],
    store=True,
)
live_scene_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)
```

This is where cost control happens - sampling policies trade compute for recall.

***

## Understand (Indexes)

Indexes are programmable interpretation layers. You define what to extract with prompts.

* **Prompt-driven** - Natural language instructions
* **Model-orchestrated** - LLMs and VLMs do the work
* **Additive** - Multiple indexes on same media
* **Multimodal** - Visual and spoken

```python theme={null}
# Create visual and spoken artifacts in one finite understanding run.
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "visual",
            "config": {"prompt": "Identify key moments and describe activities"},
        },
        {"type": "spoken_words", "name": "transcript"},
    ]
)
understanding.wait_until_complete()

visual = understanding.get_analyzer("visual")
transcript = understanding.get_analyzer("transcript")
visual_index = video.index(name="visual", source=visual)
transcript_index = video.index(name="transcript", source=transcript)
visual_index.wait_until_complete()
transcript_index.wait_until_complete()
```

***

## Remember

Indexes are stored as episodic memory. This is automatic by default.

**What gets stored:**

* Transcripts and embeddings
* Scene descriptions and tags
* Structured metadata
* Retrieval structures

For a live stream, set `store=True` when you plan to index its continuous VLM output:

```python theme={null}
live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe this stream window."},
        }
    ],
    store=True,
)
live_scene_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)
```

***

## Retrieve (Search)

Search across indexed content with natural language. Results include playable evidence.

```python theme={null}
# Single video
results = video.semantic_search(
    query="product demo",
    index_ids=[visual_index.index_id],
    top_k=10,
)

# Single stream
results = rtstream.search("intrusion", index_id=live_scene_index.id)

# Collection-wide
results = coll.search("quarterly results", top_k=10)
```

**Results include:**

* **Timestamps** - Exact start/end times
* **Text** - What was detected
* **Score** - Relevance ranking
* **Stream URL** - Playable link

```python theme={null}
for shot in results.shots:
    print(f"{shot.start}s - {shot.end}s")
    shot.play()  # Verify the result
```

***

## Act

Go from understanding to automation and outputs.

### Event Detection

Continuously understand and index the live VLM output before adding an alert:

```python theme={null}
live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe activity in this stream window."},
        }
    ],
    store=True,
)
live_scene_index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)

event_id = conn.create_event(
    event_prompt="Detect intruder",
    label="security_alert"
)

live_scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/alerts"
)
```

### Programmable Editing

Compose outputs using the 4-layer editor architecture:

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

video_asset = VideoAsset(id=video.id, start=10)
clip = Clip(asset=video_asset, duration=20)

track = Track()
track.add_clip(0, clip)

timeline = Timeline(conn)
timeline.add_track(track)
output = timeline.generate_stream()
```

***

## Architecture Patterns

The loop applies to different use cases:

| Use Case | See | Understand | Act |
| :- | :- | :- | :- |
| Video Search | Upload files | Index with domain prompts | Search + retrieve |
| Monitoring | Connect RTSP | Real-time indexing | Alerts + webhooks |
| Desktop Agent | Capture SDK | Index screen/mic | Context for LLM |
| Media Automation | Upload + transcode | Index for editing | Timeline + export |

***

## Next Steps

> **[Data Model](/pages/core-concepts/data-model)**
>
> Collections, Videos, RTStreams, and other core objects

> **[Indexes](/pages/core-concepts/indexes-and-search)**
>
> Turn media into searchable knowledge

> **[Search & Retrieval](/pages/core-concepts/indexes-and-search)**
>
> How search returns playable evidence

> **[Events & Alerts](/pages/core-concepts/events-and-realtime)**
>
> Real-time detection and automation


# Programmable Editing
Source: https://docs.videodb.io/pages/core-concepts/programmable-editing

VideoDB Editor lets you create videos programmatically using code instead of clicking timelines. Build videos using the 4-layer architecture: Asset → Clip → Track → Timeline.

## When to Use Programmable Editing

Use the editor when you need to:

* Generate personalized videos at scale
* Build automated content pipelines
* Create video variations for A/B testing
* Compose clips from search results
* Add overlays, captions, and effects programmatically

## Quick Example

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

# 1. Asset - your raw content
video_asset = VideoAsset(id=video.id, start=0)

# 2. Clip - how it appears
clip = Clip(asset=video_asset, duration=10)

# 3. Track - when it plays
track = Track()
track.add_clip(0, clip)

# 4. Timeline - the final canvas
timeline = Timeline(conn)
timeline.add_track(track)
stream_url = timeline.generate_stream()
```

## Next Steps

> **[Timeline Architecture](/pages/act/programmable-editing/timeline-architecture)**
>
> Complete guide to the 4-layer architecture

> **[Clip Parameters](/pages/act/programmable-editing/clip-parameters)**
>
> Filters, transitions, and effects

***

## The 4-Layer Architecture

Think of video composition like building with LEGO:

```
Asset → Clip → Track → Timeline
```

| Layer | Purpose | Example |
| :- | :- | :- |
| **Asset** | Raw content (what) | `VideoAsset(id=video.id)` |
| **Clip** | Presentation (how) | `Clip(asset=..., duration=10)` |
| **Track** | Sequencing (when) | `track.add_clip(0, clip)` |
| **Timeline** | Canvas (output) | `timeline.generate_stream()` |

***

## Layer 1: Assets

Assets reference your content but don't define timing or effects.

```python theme={null}
from videodb.editor import VideoAsset, AudioAsset, ImageAsset, TextAsset

# Video content
video_asset = VideoAsset(id=video.id, start=0, volume=1.0)

# Audio (music, voiceover)
audio_asset = AudioAsset(id=audio.id, start=0, volume=0.8)

# Images (logos, watermarks)
image_asset = ImageAsset(id=image.id)

# Custom text
text_asset = TextAsset(text="Hello World")
```

**Key parameters:**

* `id` - The VideoDB media ID
* `start` - Trim point in source (skip first N seconds)
* `volume` - Audio level (0.0 to 2.0)

***

## Layer 2: Clips

Clips wrap assets and define **how** and **how long** they appear.

```python theme={null}
from videodb.editor import Clip, Fit, Position

clip = Clip(
    asset=video_asset,
    duration=10,           # How long it plays
    fit=Fit.crop,          # How it scales to canvas
    position=Position.center,  # Where it appears
    scale=1.0,             # Size multiplier
    opacity=1.0            # Transparency (0-1)
)
```

**Fit modes:**

* `Fit.crop` - Fill canvas, crop edges if needed
* `Fit.contain` - Fit inside, add bars if needed
* `Fit.cover` - Stretch to fill (may distort)

**Position options:**

* 9-zone grid: `top_left`, `top`, `top_right`, `center_left`, `center`, `center_right`, `bottom_left`, `bottom`, `bottom_right`

***

## Layer 3: Tracks

Tracks are timeline lanes that control **when** clips play.

```python theme={null}
from videodb.editor import Track

track = Track()
track.add_clip(0, clip1)    # Starts at 0s
track.add_clip(10, clip2)   # Starts at 10s
```

### Sequencing Rules

**Same track** = sequential playback:

```python theme={null}
track.add_clip(0, clip1)   # 0s-5s
track.add_clip(5, clip2)   # 5s-10s
```

**Different tracks** = simultaneous playback (layered):

```python theme={null}
track1.add_clip(0, video_clip)   # Background
track2.add_clip(0, logo_clip)    # Overlay (on top)
```

### Z-Order

Later tracks render **on top** of earlier tracks:

```python theme={null}
timeline.add_track(background_track)  # Bottom layer
timeline.add_track(overlay_track)     # Renders above
timeline.add_track(caption_track)     # Top layer
```

***

## Layer 4: Timeline

Timeline is your final canvas and export settings.

```python theme={null}
from videodb.editor import Timeline

timeline = Timeline(conn)
timeline.resolution = "1280x720"    # WIDTHxHEIGHT
timeline.background = "#000000"     # Hex color

timeline.add_track(track)
stream_url = timeline.generate_stream()
```

**Common resolutions:**

* `"1280x720"` - 16:9 landscape (YouTube)
* `"608x1080"` - 9:16 vertical (TikTok, Reels)
* `"1080x1080"` - 1:1 square (Instagram)

***

## Common Patterns

### Highlight Reel from Search Results

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

# Search for moments
results = video.search("key announcement")

# Build highlight reel
timeline = Timeline(conn)
track = Track()
current_time = 0

for shot in results.shots[:5]:
    asset = VideoAsset(id=video.id, start=shot.start)
    clip = Clip(asset=asset, duration=shot.end - shot.start)
    track.add_clip(current_time, clip)
    current_time += shot.end - shot.start

timeline.add_track(track)
highlights = timeline.generate_stream()
```

### Video with Logo Overlay

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset, ImageAsset, Position

# Main video
video_asset = VideoAsset(id=video.id)
video_clip = Clip(asset=video_asset, duration=30)
video_track = Track()
video_track.add_clip(0, video_clip)

# Logo overlay
logo_asset = ImageAsset(id=logo.id)
logo_clip = Clip(
    asset=logo_asset,
    duration=30,
    position=Position.top_right,
    scale=0.2,
    opacity=0.8
)
logo_track = Track()
logo_track.add_clip(0, logo_clip)

# Combine (video first, logo on top)
timeline = Timeline(conn)
timeline.add_track(video_track)
timeline.add_track(logo_track)
output = timeline.generate_stream()
```

### Timed Captioned Video

```python theme={null}
import base64
from videodb.editor import CaptionAnimation, Timeline, Track, Clip, VideoAsset, CaptionAsset

# Build this base64 SRT from your timed caption workflow.
srt = """1
00:00:00,000 --> 00:00:02,000
Captioned with explicit timing.
"""
srt_src = base64.b64encode(srt.encode()).decode()

# Video track
video_asset = VideoAsset(id=video.id)
video_clip = Clip(asset=video_asset, duration=60)
video_track = Track()
video_track.add_clip(0, video_clip)

# Caption track
caption_asset = CaptionAsset(src=srt_src, animation=CaptionAnimation.karaoke)
caption_clip = Clip(asset=caption_asset, duration=60)
caption_track = Track()
caption_track.add_clip(0, caption_clip)

# Combine
timeline = Timeline(conn)
timeline.add_track(video_track)
timeline.add_track(caption_track)
output = timeline.generate_stream()
```

See [Track Layering](/pages/act/programmable-editing/track-layering) for the timed SRT construction flow.

***

## What You Can Build

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Vertical videos with synced lyrics and AI backgrounds

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Complete AI-generated videos with scripts and voiceovers

> **[Intro & Outro Automation](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Add branded intros and outros to all videos automatically

> **[Brand Elements](/examples-and-tutorials/programmatic-editing/brand-elements)**
>
> Overlay logos, watermarks, and lower thirds programmatically

***

## Best Practices

1. **Think in layers** - Background video first, overlays on top

2. **Use Asset.start for trimming** - Skip to the interesting part of source media

3. **Match clip durations** - Overlays should match or exceed video duration

4. **Test with generate\_stream()** - Preview before expensive exports

5. **Reuse assets** - One VideoAsset can be used in multiple clips


# Sandbox Compute
Source: https://docs.videodb.io/pages/core-concepts/sandbox-compute

Run supported open-weight and specialized models in VideoDB workflows by starting a sandbox and passing sandbox_id.

Sandbox Compute is VideoDB's managed runtime for supported open-weight and specialized models.

Create a sandbox, wait until it is active, then pass `sandbox_id` to supported APIs. VideoDB routes that job to your sandbox runtime instead of the default hosted path.

Sandbox is not only an indexing feature. It can power object detection analyzers, indexing pipelines, transcription, and generation workflows such as text-to-speech and text generation.

For hosted/default models, omit `sandbox_id`. For Sandbox Compute/open-weight models, pass `sandbox_id` explicitly.

> **[Try the Sandbox quickstart notebook](https://github.com/video-db/videodb-cookbook/blob/preview/guides/sandbox/quickstart.ipynb)**
>
> Create a sandbox and run supported models in a hands-on example.

***

## When to use Sandbox Compute

Use Sandbox Compute when you want:

* **Open-weight model access** for object detection, speech, or text workflows.
* **Specialized models** such as Qwen3-4B, Whisper, OmniVoice, or RT-DETR.
* **Predictable routing** by pinning a job to a specific sandbox ID.
* **Runtime-based pricing** for compatible workloads.

***

## Create a sandbox

Choose a tier based on the largest model you plan to run. Creating a sandbox returns immediately while compute is provisioning.

```python theme={null}
import videodb
from videodb import SandboxTier

conn = videodb.connect()
coll = conn.get_collection()

sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    models=["Qwen/Qwen3-4B"],
)

sandbox.wait_for_ready(timeout=300, interval=5)
print(sandbox.id)
```

You can prepare the sandbox by model category instead of exact model names:

```python theme={null}
sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    model_categories=["text_generation", "object_detection", "text_to_speech"],
)

sandbox.wait_for_ready()
```

Supported category names include:

| Category | Use for |
| - | - |
| `object_detection` | object detection models such as RT-DETR |
| `speech_to_text` | speech recognition models such as Whisper |
| `text_to_speech` | speech generation models such as OmniVoice |
| `text_generation` | text generation models such as Qwen3-4B |

> [!NOTE]
>
> Submit sandbox-backed jobs only after the sandbox is active.

***

## Pick a model

Sandbox supports text generation, speech-to-text, text-to-speech, and object detection models. All currently listed models run on the `small` tier.

> **[Sandbox Models](/pages/core-concepts/sandbox-models)**
>
> See the current model catalog and minimum tier for each model.

***

## Route jobs to the sandbox

Pass `sandbox_id=sandbox.id` to supported APIs.

If `sandbox_id` is omitted, VideoDB uses the hosted/default path. For production workflows with Sandbox models, pass the ID explicitly so routing is predictable.

***

## Use Sandbox with Understanding and Indexing

Sandbox affects the [**understanding** step](/pages/understand/indexing-pipelines/understanding-artifacts): it controls where the model runs. The **indexing** step is unchanged: index the resulting artifact with `video.index(...)`.

For sandbox-backed object detection, choose the detection model in the analyzer config. Configure object filters and aggregations when you create the index.

```python theme={null}
sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    models=["rtdetr-v2-r50vd"],
)
sandbox.wait_for_ready()

video = coll.get_video("m-xxx")

understanding = video.understand(
    analyzers=[
        {
            "type": "object_detection",
            "name": "objects",
            "config": {
                "model": "rtdetr-v2-r50vd",
                "sandbox_id": sandbox.id,
            },
        }
    ],
)

understanding.wait_until_complete()
objects = understanding.get_analyzer("objects")

video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["object_labels", "object_counts"],
        "aggregate": ["object_labels"],
    },
)
```

For complete indexing options, see [Create an Index](/pages/understand/indexing-pipelines/create-an-index).

***

## Use Sandbox for generation workflows

Sandbox can also route supported generation jobs. These APIs use `model_name` plus `sandbox_id`.

```python theme={null}
sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    models=["k2-fsa/OmniVoice", "Qwen/Qwen3-4B"],
)
sandbox.wait_for_ready()
```

### OmniVoice text-to-speech

```python theme={null}
job = coll.generate_voice(
    text="Welcome to VideoDB Sandbox Compute.",
    model_name="k2-fsa/OmniVoice",
    sandbox_id=sandbox.id,
)

audio = job.wait(timeout=900, interval=5)
print(audio.id)
```

### Voice design

Use `config.instructions` to guide the voice style.

```python theme={null}
job = coll.generate_voice(
    text="Breaking update: your video workflows now have dedicated inference compute.",
    model_name="k2-fsa/OmniVoice",
    sandbox_id=sandbox.id,
    config={"instructions": "A deep, authoritative news anchor voice"},
)

audio = job.wait(timeout=900, interval=5)
```

### Reusable voice clone

Create a voice clone once from a reference audio asset, then reuse it by passing `voice_clone_id`.

```python theme={null}
ref_audio = coll.upload(
    url="https://www.youtube.com/shorts/8GrguhmR6oQ",
    media_type="audio",
)

voice_clone = coll.create_voice_clone(
    ref_audio_id=ref_audio.id,
    name="Product Narrator",
    description="Reusable narration voice",
    ref_text="Sample reference text for the audio clip",
    language="en",
)

job = coll.generate_voice(
    text="This narration uses a reusable voice clone.",
    model_name="k2-fsa/OmniVoice",
    sandbox_id=sandbox.id,
    voice_clone_id=voice_clone.id,
)

audio = job.wait(timeout=900, interval=5)
```

### Text generation

```python theme={null}
response = coll.generate_text(
    prompt="Summarize the key visual events from this scene description list.",
    model_name="Qwen/Qwen3-4B",
    sandbox_id=sandbox.id,
    max_tokens=300,
    temperature=0.2,
)

print(response)
```

***

## Validation and permissions

When you pass `sandbox_id`, VideoDB validates that:

1. the sandbox exists and belongs to your workspace/account,
2. the sandbox is active,
3. the sandbox tier supports the requested model,
4. the sandbox was created with a matching `models` or `model_categories` value, when model selection was provided.

If validation fails, create or start a compatible sandbox and retry the job with that sandbox ID.

***

## Manage sandboxes

```python theme={null}
# Refresh one sandbox
sandbox.refresh()
print(sandbox.status, sandbox.is_active)

# List sandboxes
for sb in conn.list_sandboxes():
    print(sb.id, sb.name, sb.tier, sb.status)

# Get a sandbox by ID
sb = conn.get_sandbox(sandbox.id)
print(sb)
```

***

## Stop the sandbox

Sandbox billing is based on runtime. Stop the sandbox when your workload is complete.

```python theme={null}
sandbox.stop()
sandbox.wait_for_stop(timeout=120, interval=5)
print(f"Sandbox {sandbox.id} final status: {sandbox.status}")
```

***

## Pricing and limits

Sandbox billing is runtime-based. Billing is recorded when the sandbox stops and is calculated from `started_at` to `stopped_at`, rounded to 2 decimal hours.

| Sandbox tier | Price |
| - | -: |
| `small` | `$1/hour` |
| `medium` | `$3.50/hour` |

| Sandbox tier | Concurrent sandbox limit | Max runtime |
| - | -: | -: |
| `small` | 5 | 24 hours |
| `medium` | 3 | 24 hours |

***

## Best practices

* Create one sandbox per workflow and reuse it across compatible jobs.
* Use the `small` tier; all currently listed models run on it.
* Use `models=[...]` when you know the exact models your workflow needs.
* Use `model_categories=[...]` when your workflow may use several models in the same category.
* Wait until the sandbox is active before submitting jobs.
* Pass `sandbox_id=sandbox.id` explicitly for production indexing and generation jobs.
* Use `job.wait(timeout=900, interval=5)` for long-running generation jobs.
* Log `sandbox.id` with each job so runs can be debugged or retried.
* Stop the sandbox when work is complete.

> **[Sandbox Models](/pages/core-concepts/sandbox-models)**
>
> See supported models and minimum tiers.

> **[Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Configure analyzers and model selection.

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Index artifacts produced by hosted or sandbox models.


# Sandbox Models
Source: https://docs.videodb.io/pages/core-concepts/sandbox-models

Available models for VideoDB Sandbox Compute and their minimum tiers.

Sandbox models run on VideoDB Sandbox Compute. Pick a model, create a sandbox on a compatible tier, wait until it is active, and pass the sandbox ID to a supported workflow.

Sandbox models can be used across VideoDB workflows, including object detection analyzers, indexing pipelines, transcription, text-to-speech, and text generation.

> [!NOTE]
>
> Use model names exactly as listed unless your VideoDB team contact gives you an alias.

***

## Model catalog

| Model | Use case | Minimum tier | Good for |
| - | - | - | - |
| `Qwen/Qwen3-4B` | text generation | `small` | lightweight text generation |
| `openai/whisper-large-v3-turbo` | speech-to-text | `small` | transcription and speech recognition |
| `k2-fsa/OmniVoice` | text-to-speech | `small` | TTS, voice design, and voice cloning |
| `rtdetr-v2-r50vd` | object detection | `small` | object detection workflows |

***

## Choose a tier

All currently listed models run on the `small` tier.

```python theme={null}
from videodb import SandboxTier

small_sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    model_categories=[
        "text_generation",
        "speech_to_text",
        "text_to_speech",
        "object_detection",
    ],
)
```

Reuse the sandbox across compatible jobs in the same workflow.

***

## Where to pass the model name

| Workflow | Model field | Sandbox field |
| - | - | - |
| Object detection analyzer | analyzer `config.model` | analyzer `config.sandbox_id` |
| Generation APIs | `model_name` | `sandbox_id` |

For hosted/default models, omit `sandbox_id`. For models listed on this page, pass a compatible sandbox ID.

***

## Use a model in Understanding and Indexing

Use the sandbox model during [understanding](/pages/understand/indexing-pipelines/understanding-artifacts), then index the produced artifact with the standard `video.index(...)` interface.

For object detection, configure the model and sandbox in the analyzer, then choose filter and aggregation fields at indexing time:

```python theme={null}
sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    models=["rtdetr-v2-r50vd"],
)
sandbox.wait_for_ready()

understanding = video.understand(
    analyzers=[
        {
            "type": "object_detection",
            "name": "objects",
            "config": {
                "model": "rtdetr-v2-r50vd",
                "sandbox_id": sandbox.id,
            },
        }
    ],
)

understanding.wait_until_complete()
objects = understanding.get_analyzer("objects")

video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["object_labels", "object_counts"],
        "aggregate": ["object_labels"],
    },
)
```

***

## Use a model in generation APIs

Generation APIs use `model_name` and `sandbox_id`.

```python theme={null}
sandbox = conn.create_sandbox(
    tier=SandboxTier.small,
    models=["k2-fsa/OmniVoice", "Qwen/Qwen3-4B"],
)
sandbox.wait_for_ready()

job = coll.generate_voice(
    text="Welcome to VideoDB Sandbox Compute.",
    model_name="k2-fsa/OmniVoice",
    sandbox_id=sandbox.id,
)

audio = job.wait(timeout=900, interval=5)
```

```python theme={null}
response = coll.generate_text(
    prompt="Summarize the key visual events from this scene description list.",
    model_name="Qwen/Qwen3-4B",
    sandbox_id=sandbox.id,
    max_tokens=300,
    temperature=0.2,
)

print(response)
```

***

## Pricing and limits

Sandbox pricing and concurrency limits are based on the sandbox tier, not the individual model. See [Sandbox Compute](/pages/core-concepts/sandbox-compute#pricing-and-limits) for current pricing and limits.

> **[Sandbox Compute](/pages/core-concepts/sandbox-compute)**
>
> Create, use, and stop sandbox compute.

> **[Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Configure analyzers and model selection.

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Index artifacts produced by sandbox models.


# Security & Privacy
Source: https://docs.videodb.io/pages/core-concepts/security-privacy

VideoDB handles sensitive media data. Understand storage modes, data retention, and privacy patterns to build compliant systems.

## When to Reference This

Use this page when you're:

* Designing systems with privacy requirements
* Choosing storage and retention policies
* Implementing consent workflows
* Securing API access patterns

## Quick Example

```python theme={null}
import videodb

# Secure connection
conn = videodb.connect()  # Uses VIDEO_DB_API_KEY env var

# Desktop capture with client tokens (never expose API key)
cap = conn.create_capture_session(
    end_user_id="user_abc",
    callback_url="https://your-backend.com/webhooks"
)
token = conn.generate_client_token(expires_in=600)  # Short-lived
```

## Next Steps

> **[Capture Session](/pages/core-concepts/data-model#capturesession)**
>
> Secure desktop capture patterns

> **[API Reference](/api-reference/authentication/index)**
>
> API key management

***

## Storage Modes

### Persistent Storage (Default)

Media and derived data are stored for retrieval and search:

```python theme={null}
video = coll.upload(url="https://...")  # Stored
understanding = video.understand(
    analyzers=[{"type": "vlm", "name": "scene"}]
)
understanding.wait_until_complete()
scene = understanding.get_analyzer("scene")
index = video.index(name="scene", source=scene)  # Retrieval-ready
index.wait_until_complete()
```

**Use for:**

* Video archives and libraries
* Searchable knowledge bases
* Long-term agent memory

### Data Minimization

Choose capture channels deliberately and retain only the media your workflow needs. The released RTStream V2 indexing flow requires `store=True` when a continuous VLM output will be indexed.

```python theme={null}
# Indexing a continuous live output requires stored records.
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Monitor for activity"},
        }
    ],
    store=True,
)
index = rtstream.index(
    name="live_scene",
    source=understanding.outputs["scene"],
    use_for=["semantic"],
)
```

**Use for:**

* Live monitoring dashboards with deliberate channel selection
* Privacy-sensitive contexts that minimize captured inputs
* Workflows with documented retention controls

***

## Retention Patterns

### Define Your Retention Policy

* Document what is captured, why it is retained, and who can access it
* Use the capture SDK's per-channel `store` control to minimize collection
* Confirm product- and account-specific retention controls before making guarantees to users

### Manual Deletion

```python theme={null}
# Delete video and all associated data
video.delete()

# Delete specific index
index.delete()

# Delete collection
coll.delete()
```

### Recommended Patterns

| Use Case | Pattern |
| :- | :- |
| Temporary analysis | Select only necessary capture channels and define retention |
| GDPR compliance | Implement deletion on user request |
| Meeting recordings | Define retention policy, batch delete |
| Security footage | Time-based archival and deletion |

***

## API Key Security

### Key Management

```python theme={null}
# Environment variable (recommended)
conn = videodb.connect()  # Uses VIDEO_DB_API_KEY

# Explicit key (server-side only)
conn = videodb.connect(api_key="your-key")
```

**Best practices:**

* Never embed keys in client applications
* Use environment variables
* Rotate keys periodically
* Use separate keys for dev/prod

Manage API keys through the VideoDB platform controls. The released Python and Node SDKs do not expose API-key creation, listing, or deletion methods; see the [Authentication API reference](/api-reference/authentication/index) for the supported platform workflow.

***

## Client Token Pattern

For desktop and mobile clients, use short-lived tokens instead of API keys:

```python theme={null}
# Backend (holds API key)
token = conn.generate_client_token(expires_in=600)  # 10 minutes

# Client (uses token, never sees API key)
client = CaptureClient(client_token=token)
```

**Why this matters:**

* Tokens expire automatically
* Tokens have limited scope
* Compromised tokens have limited blast radius
* API key never leaves your backend

***

## Consent Patterns

### User Consent for Capture

Before capturing screen, mic, or camera:

```python theme={null}
# Request explicit permission
await client.request_permission("microphone")
await client.request_permission("screen_capture")

# User must grant permission in OS dialog
channels = await client.list_channels()
```

### Data Subject Access

Map consent and capture records to your own user identity, then implement authenticated access and deletion-request workflows in your application. Python can list capture sessions by collection and status, but it has no `end_user_id` filter and `CaptureSession.delete()` is not available. Keep the user-to-resource mapping and deletion orchestration in your application until a supported lifecycle API is available.

***

## Network Security

### HTTPS

All API communication uses HTTPS. No configuration needed.

### Webhook Security

Verify webhook payloads to prevent spoofing:

```python theme={null}
@app.post("/webhooks/videodb")
async def handle_webhook(request):
    # Verify signature (implementation depends on your setup)
    signature = request.headers.get("X-VideoDB-Signature")
    if not verify_signature(request.body, signature):
        return {"error": "Invalid signature"}, 401

    event = await request.json()
    process_event(event)
    return {"status": "ok"}
```

***

## Compliance Considerations

### GDPR

* Implement data access endpoints
* Implement deletion endpoints
* Document data processing purposes
* Minimize stored channels when storage is not needed

### HIPAA

* Minimize captured and stored data for sensitive content
* Implement strict access controls
* Audit all data access
* Consider on-premise deployment for PHI

### SOC 2

* VideoDB maintains SOC 2 compliance
* Implement access logging
* Use separate keys per environment
* Regular key rotation

***

## Best Practices Summary

1. **Never expose API keys** - Use client tokens for untrusted clients

2. **Minimize collection** - Only persist data when needed

3. **Implement deletion** - Honor data deletion requests

4. **Short token lifetimes** - 10-15 minutes for desktop capture

5. **Audit access** - Log who accesses what data

6. **Encrypt at rest** - VideoDB encrypts stored data

7. **Get consent** - Always get user permission before capture


# Supported Languages
Source: https://docs.videodb.io/pages/core-concepts/supported-languages

Language codes supported by VideoDB speech-to-text and spoken-word indexing.

Use these language codes with speech-to-text workflows, including:

* `video.understand(analyzers=[{"type": "spoken_words", "config": {"language": "..."}}])`
* `video.generate_transcript(language_code="...")`
* `audio.generate_transcript(language_code="...")`

If you omit the language code, VideoDB uses the default behavior for the selected transcription engine. Pass a code when you want to provide an explicit language hint.

> [!NOTE]
>
> Use lower-case codes. Regional English variants use underscores, such as `en_us`, `en_uk`, and `en_au`.

## Supported speech-to-text languages

| Language | Code | Language | Code |
| :- | :- | :- | :- |
| Afrikaans | `af` | Albanian | `sq` |
| Amharic | `am` | Arabic | `ar` |
| Armenian | `hy` | Assamese | `as` |
| Azerbaijani | `az` | Bashkir | `ba` |
| Basque | `eu` | Belarusian | `be` |
| Bengali | `bn` | Bosnian | `bs` |
| Breton | `br` | Bulgarian | `bg` |
| Burmese | `my` | Catalan | `ca` |
| Chinese | `zh` | Croatian | `hr` |
| Czech | `cs` | Danish | `da` |
| Dutch | `nl` | English | `en` |
| English (Australia) | `en_au` | English (United Kingdom) | `en_uk` |
| English (United States) | `en_us` | Estonian | `et` |
| Faroese | `fo` | Finnish | `fi` |
| French | `fr` | Galician | `gl` |
| Georgian | `ka` | German | `de` |
| Greek | `el` | Gujarati | `gu` |
| Haitian Creole | `ht` | Hausa | `ha` |
| Hawaiian | `haw` | Hebrew | `he` |
| Hindi | `hi` | Hungarian | `hu` |
| Icelandic | `is` | Indonesian | `id` |
| Italian | `it` | Japanese | `ja` |
| Javanese | `jw` | Kannada | `kn` |
| Kazakh | `kk` | Khmer | `km` |
| Korean | `ko` | Lao | `lo` |
| Latin | `la` | Latvian | `lv` |
| Lingala | `ln` | Lithuanian | `lt` |
| Luxembourgish | `lb` | Macedonian | `mk` |
| Malagasy | `mg` | Malay | `ms` |
| Malayalam | `ml` | Maltese | `mt` |
| Maori | `mi` | Marathi | `mr` |
| Manipuri | `mni` | Mongolian | `mn` |
| Nepali | `ne` | Norwegian | `no` |
| Norwegian Nynorsk | `nn` | Occitan | `oc` |
| Odia | `or` | Pashto | `ps` |
| Persian | `fa` | Polish | `pl` |
| Portuguese | `pt` | Punjabi | `pa` |
| Rajasthani | `raj` | Romanian | `ro` |
| Russian | `ru` | Sanskrit | `sa` |
| Serbian | `sr` | Shona | `sn` |
| Sindhi | `sd` | Sinhala | `si` |
| Slovak | `sk` | Slovenian | `sl` |
| Somali | `so` | Spanish | `es` |
| Sundanese | `su` | Swahili | `sw` |
| Swedish | `sv` | Tagalog | `tl` |
| Tajik | `tg` | Tamil | `ta` |
| Tatar | `tt` | Telugu | `te` |
| Thai | `th` | Tibetan | `bo` |
| Turkish | `tr` | Turkmen | `tk` |
| Ukrainian | `uk` | Urdu | `ur` |
| Uzbek | `uz` | Vietnamese | `vi` |
| Welsh | `cy` | Yiddish | `yi` |
| Yoruba | `yo` | | |


# AI Agent Skills
Source: https://docs.videodb.io/pages/getting-started/agent-skills

Add video and audio perception to your AI agents - capture, upload, search, edit, and stream

Your AI agents can write code and automate tasks brilliantly. But they're missing one critical capability: the ability to work with video and audio - capturing screens, searching through recordings, editing clips, and streaming results.

VideoDB Skills give agents like Claude Code and Codex the power to execute server-side video workflows, turning text-only agents into multimodal collaborators.

***

## Install VideoDB Skills

Get video and audio perception in your agent with one command:

**NPX (Recommended)**

```bash theme={null}
npx skills add video-db/skills
```

**Claude Code Plugin**

```bash theme={null}
/plugin marketplace add video-db/skills
/plugin install videodb@videodb-skills
```

Then run `/videodb setup` to configure your API key and verify connectivity.

> **[VideoDB Skills on GitHub](https://github.com/video-db/skills)**
>
> Complete source code, installation guide, and configuration examples

***

## Prerequisites

**1. VideoDB API Key**

Get a free API key from [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs)

No credit card required. Free tier includes 50 uploads.

**2. System Requirements**

* **Python 3.9+**
* **Platform**: macOS, Linux, Windows (PowerShell)

**3. Set Your API Key**

Export your API key in your shell:

```bash theme={null}
export VIDEO_DB_API_KEY=your-key-here
```

Or add it to a `.env` file in your project root.

***

## What It Does

VideoDB Skills is a perception capability that enables **See → Understand → Act, as an API, for video and audio**. It gives agents like Claude Code, Codex, and Cursor the ability to execute server-side video workflows.

One unified interface for:

* **See** - Capture desktop screens, microphone/system audio, RTSP streams, and ingest files, URLs, and YouTube content
* **Understand** - Visual analysis, transcription, indexing, and searching moments with playable clips
* **Act** - Stream results, trigger alerts, edit timelines, generate subtitles/overlays, and export clips

### Why Use It

**Video Workflows**

Execute video operations without local ffmpeg installation:

* Upload from YouTube, URLs, or local files
* Trim, merge, clip, overlay text/images/audio
* Transcode, reframe, adjust resolution and aspect ratio
* Get instant playable HLS links via built-in CDN

**Real-Time Perception**

Capture and analyze live feeds:

* Desktop screen, microphone, and system audio recording
* Monitor RTSP camera feeds with event detection
* Generate structured context from desktop streams
* Log alerts with timestamps for person detection

**Search & Intelligence**

Find moments by speech, scenes, or metadata:

* "Identify all scenes showing 'phone close-up'"
* "Capture my screen and report activities with insights"
* Timestamped transcripts and subtitles
* Playable evidence clips with exact timestamps

### Quick Start

Ask your agent to execute video tasks:

```text theme={null}
Upload [YouTube URL] and provide a shareable stream link
```

```text theme={null}
Extract clips from 10s-30s and 45s-60s and merge them
```

```text theme={null}
Generate background music and add to this clip
```

```text theme={null}
Add white text on black background subtitles to the original video
```

```text theme={null}
Capture my screen for two minutes and report my activities with insights
```

```text theme={null}
Monitor my IP Camera RTSP stream and log person detection alerts with timestamps
```

### Capabilities

| Capability | What It Does |
| - | - |
| **Capture** | Desktop screen, microphone, and system audio for real-time processing |
| **Upload** | Ingest from YouTube, URLs, or local files |
| **Context** | Generate structured context from RTSP feeds or desktop streams |
| **Search** | Locate moments by speech, scenes, or metadata with playable evidence |
| **Transcripts** | Generate timestamped transcripts |
| **Subtitles** | Auto-generate, style, and burn-in subtitles |
| **Edit** | Trim, merge, clip, overlay text/images/audio; add dubbing/translation |
| **AI Generate** | Create images, video, music, sound effects, voiceovers |
| **Transcode/Reframe** | Adjust resolution, quality, aspect ratio, social crops server-side |
| **Stream** | Obtain instant playable HLS links via built-in CDN |

***

## Example: OpenClaw Monitoring

VideoDB Skills powers [OpenClaw Monitoring](https://github.com/video-db/openclaw-monitoring) - "CCTV for AI agents" that monitors, records, and audits autonomous agent sessions. Every agent run becomes a live stream, replayable recording, and searchable archive.

> **[OpenClaw Monitoring on GitHub](https://github.com/video-db/openclaw-monitoring)**
>
> See how VideoDB Skills enables visual observability for autonomous agents

***

## Next Steps

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Deep dive: channels, permissions, client code, and event handling

> **[Real-time Context](/pages/ingest/capture-sdks/realtime-context)**
>
> How real-time indexing and search works

> **[AI Copilot Examples](/examples-and-tutorials/ai-copilots)**
>
> Explore more AI copilot projects and use cases

> **[Quickstart](/pages/getting-started/quickstart)**
>
> Try desktop perception with a hosted OpenClaw agent


# Core Concepts in 5 Minutes
Source: https://docs.videodb.io/pages/getting-started/core-concepts-in-5-min

The entire VideoDB mental model in one scroll - perception, memory, and action for AI agents.

## The Problem

AI agents can reason about text brilliantly. But show them a 30-minute meeting recording and ask "what did the client say about pricing?" - they fail.

Video files are opaque blobs. Your agent can't query them, can't search them, can't get timestamped answers from them.

## The Platform Loop

Every VideoDB workflow follows the same pattern:

```
See → Understand → Act
```

| Stage | What Happens | Returns |
| :- | :- | :- |
| **See** | Ingest from files, streams, or desktop capture | `Video`, `RTStream`, or `CaptureSession` |
| **Understand** | Create analyzers, reusable artifacts, and indexes. | Retrieval-ready video context |
| **Act** | Trigger alerts. Compose edits. Export streams. | Webhooks, playable URLs, downloadable files |

## Quick Example

```python theme={null}
import videodb

conn = videodb.connect()

# SEE: Ingest
coll = conn.get_collection()
video = coll.upload(url="https://example.com/meeting.mp4")

# UNDERSTAND: Create a transcript artifact and index it
understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}]
)
understanding.wait_until_complete()
transcript = understanding.get_analyzer("transcript")
index = video.index(name="meeting_transcript", source=transcript)
index.wait_until_complete()

# RETRIEVE: Search the completed index
results = video.semantic_search(
    query="pricing discussion",
    index_ids=[index.index_id],
    top_k=5,
)

# ACT: Use the results
for shot in results.get_shots():
    print(f"{shot.start}s - {shot.end}s")
    shot.play()  # Playable proof
```

***

## See: Three Input Types

| Source | Method | Returns |
| :- | :- | :- |
| **Files** | `coll.upload(url="...")` | `Video` |
| **Live streams** | `coll.connect_rtstream(url="...", name="Camera")` | `RTStream` |
| **Desktop capture** | `conn.create_capture_session(...)` | `CaptureSession` → `RTStream` |

```python theme={null}
# Files
coll = conn.get_collection()
video = coll.upload(url="https://youtube.com/watch?v=...")

# Live RTSP
rtstream = coll.connect_rtstream(url="rtsp://camera.local/stream", name="Camera")

# Desktop capture
cap = conn.create_capture_session(end_user_id="user_123")
```

Videos index completed analyzer artifacts. RTStreams continuously index a VLM output descriptor.

***

## Understand: Indexes Are Everything

Indexes are what transform opaque media into searchable knowledge. You create them with prompts.

### Spoken Index

Transcribes audio and makes it searchable:

```python theme={null}
understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}]
)
understanding.wait_until_complete()

transcript = understanding.get_analyzer("transcript")
spoken_index = video.index(name="spoken", source=transcript)
spoken_index.wait_until_complete()
```

### Visual Index

Understands what's happening on screen:

```python theme={null}
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "config": {"prompt": "Describe key activities and events"},
        }
    ]
)
understanding.wait_until_complete()

scene = understanding.get_analyzer("scene")
visual_index = video.index(name="visual", source=scene)
visual_index.wait_until_complete()
```

### Multiple Indexes

Create different perspectives on the same media:

```python theme={null}
# Same video, different questions
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "safety",
            "config": {"prompt": "Identify safety violations"},
        },
        {
            "type": "vlm",
            "name": "summary",
            "config": {"prompt": "Summarize each segment"},
        },
    ]
)
understanding.wait_until_complete()

safety_index = video.index(name="safety", source=understanding.get_analyzer("safety"))
summary_index = video.index(name="summary", source=understanding.get_analyzer("summary"))
safety_index.wait_until_complete()
summary_index.wait_until_complete()
```

Indexes are additive when they reuse an existing artifact. A new analyzer perspective requires another Understand run; remove old indexes without affecting others.

***

## Search Returns Evidence

Search returns timestamps and playable links - not just "found" but verifiable.

```python theme={null}
results = video.semantic_search(
    query="product demo",
    index_ids=[visual_index.index_id],
    top_k=5,
)

for shot in results.get_shots():
    print(f"{shot.start}s - {shot.end}s")  # Timestamps
    shot.play()                            # Play it to verify
```

Every result maps to a playable moment. Your agent can cite its sources.

***

## Act: Events, Alerts, Editing

### Trigger on conditions

```python theme={null}
# Continuously understand and index the live VLM output before attaching an alert.
live_understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe visible activity, including on-screen deadlines and due dates."},
        }
    ],
    store=True,
)
index = rtstream.index(
    name="live_scene",
    source=live_understanding.outputs["scene"],
    use_for=["semantic"],
)

# Create a reusable event
event_id = conn.create_event(
    event_prompt="Detect when a deadline or due date is visible",
    label="visible_deadline"
)

# Wire it to an index. Replace this placeholder with a public endpoint in production.
index.create_alert(event_id=event_id, callback_url="https://example.com")
```

### Compose with code

```python theme={null}
from videodb.editor import Timeline, Track, Clip, VideoAsset

timeline = Timeline(conn)
track = Track()
track.add_clip(0, Clip(asset=VideoAsset(id=video.id), duration=30))
timeline.add_track(track)

stream_url = timeline.generate_stream()
```

***

## Objects at a Glance

| Object | What It Represents |
| :- | :- |
| `Connection` | Your authenticated session |
| `Collection` | Container for organizing media |
| `Video` | Uploaded video |
| `RTStream` | Live stream (RTSP or capture) |
| `Index` | Searchable interpretation layer |
| `SearchResult` | Query results with shots |
| `Shot` | Single timestamped match |
| `Event` | Reusable detection rule |
| `Alert` | Event + delivery config |
| `Timeline` | Programmatic edit composition |

***

## Next Steps

> **[Quickstart](/pages/getting-started/quickstart)**
>
> Try it hands-on

> **[Core Concepts (Full)](/pages/core-concepts/overview)**
>
> Deep dive with examples

> **[Data Model](/pages/core-concepts/data-model)**
>
> All objects and relationships

> **[Indexes & Search](/pages/core-concepts/indexes-and-search)**
>
> How indexing and retrieval work


# Node.js SDK
Source: https://docs.videodb.io/pages/getting-started/node

Install and configure the VideoDB Node.js SDK

## Installation

```bash theme={null}
npm install videodb
# or
yarn add videodb
```

> **[npm](https://www.npmjs.com/package/videodb)**
>
> View package on npm

> **[GitHub](https://github.com/video-db/videodb-node)**
>
> Source code and issues

## Quick Start

```javascript theme={null}
import { connect } from 'videodb';

// Connect using environment variable
const conn = connect();

// Or pass an API key directly instead
// const apiKeyConn = connect({ apiKey: "your-api-key" });
```

## Environment Variables

| Variable | Description |
| :- | :- |
| `VIDEO_DB_API_KEY` | Your API key from [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) |

```bash theme={null}
export VIDEO_DB_API_KEY="your-api-key"
```

## TypeScript Support

Full TypeScript definitions are included. No additional packages needed.

```typescript theme={null}
import { connect, Video } from 'videodb';

const conn = connect();
const media = await conn.uploadURL("default", { url: "..." });

if (!(media instanceof Video)) {
    throw new Error("Expected a video upload");
}

const response = await media.search("query");
```

## Module Formats

Both ESM and CommonJS are supported:

```javascript theme={null}
// ESM
import { connect } from 'videodb';

// CommonJS
const { connect } = require('videodb');
```

## Basic Usage

```javascript theme={null}
import { connect, Video } from 'videodb';

const conn = connect();

// Upload a video
const media = await conn.uploadURL("default", {
    url: "https://www.youtube.com/watch?v=example"
});

if (!(media instanceof Video)) {
    throw new Error("Expected a video upload");
}

const video = media;

const understanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await understanding.waitUntilComplete();
const transcriptAnalyzer = await understanding.getAnalyzer("transcript");

const transcriptIndex = await video.index(transcriptAnalyzer, {
  name: "transcript",
  useFor: ["semantic"],
  fields: { semantic: ["text"] },
});
await transcriptIndex.waitUntilComplete();

const results = await video.semanticSearch("key moments", {
  indexIds: [transcriptIndex.indexId],
  topK: 5,
});
for (const shot of results.getShots()) {
    console.log(`${shot.start}s: ${shot.text}`);
}
```

## Server Side

Use the full SDK on your backend to manage sessions, run AI pipelines, and handle webhooks. Your API key should never be exposed to the browser.

```javascript theme={null}
// Create a capture session and generate a client token
const cap = await conn.createCaptureSession({ endUserId: "user_123" });
const token = await conn.generateClientToken(600);
```

## Client Side

For real-time desktop capture, install the Capture SDK on your client application. It uses short-lived tokens instead of your API key.

> [!NOTE]
>
> Desktop capture currently supports **macOS** and **Windows**.

```bash theme={null}
npm install videodb
```

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Learn how to integrate real-time screen, audio, and camera capture into your application

## Next Steps

> **[Quickstart](/pages/getting-started/quickstart)**
>
> Build your first perception-enabled agent

> **[API Reference](/api-reference/introduction)**
>
> Complete REST API documentation


# Node.js SDK Migration
Source: https://docs.videodb.io/pages/getting-started/node-migration

Upgrade from v0.1.3 to v0.2.0: a cleaner API with direct property access and async/await

If you're upgrading from v0.1.3, this guide walks you through the two breaking changes you need to address. The migration is straightforward (mostly find-and-replace).

## What Changed

| Pattern | Before (v0.1.3) | After (v0.2.0) |
| :- | :- | :- |
| Property Access | `video.meta.id` | `video.id` |
| Async Operations | `waitForJob(video.getTranscript())` | `await video.getTranscript()` |
| Uploads | Returns `UploadJob` | Returns `Video` / `Audio` / `Image` directly |
| Error Handling | `.on('error', callback)` | Standard `try/catch` |

***

## Breaking Change 1: Direct Property Access

In v0.1.3, all properties were wrapped in a `.meta` object. In v0.2.0, properties are accessed directly on the instance.

### Video

```typescript Before (v0.1.3) theme={null}
const video = await collection.getVideo('m-123');

console.log(video.meta.id);           // 'm-123'
console.log(video.meta.name);         // 'My Video'
console.log(video.meta.streamUrl);    // 'https://stream.videodb.io/...'
console.log(video.meta.playerUrl);    // 'https://console.videodb.io/player?...'
console.log(video.meta.length);       // 120.5
console.log(video.meta.collectionId); // 'c-123'

const url = `https://example.com/video/${video.meta.id}`;
```

```typescript After (v0.2.0) theme={null}
const video = await collection.getVideo('m-123');

console.log(video.id);           // 'm-123'
console.log(video.name);         // 'My Video'
console.log(video.streamUrl);    // 'https://stream.videodb.io/...'
console.log(video.playerUrl);    // 'https://console.videodb.io/player?...'
console.log(video.length);       // 120.5
console.log(video.collectionId); // 'c-123'

const url = `https://example.com/video/${video.id}`;
```

### Audio

```typescript Before (v0.1.3) theme={null}
const audio = await collection.getAudio('a-123');

console.log(audio.meta.id);           // 'a-123'
console.log(audio.meta.name);         // 'podcast.mp3'
console.log(audio.meta.length);       // 3600.0
console.log(audio.meta.collectionId); // 'c-123'
```

```typescript After (v0.2.0) theme={null}
const audio = await collection.getAudio('a-123');

console.log(audio.id);           // 'a-123'
console.log(audio.name);         // 'podcast.mp3'
console.log(audio.length);       // 3600.0
console.log(audio.collectionId); // 'c-123'
```

### Image

```typescript Before (v0.1.3) theme={null}
const image = await collection.getImage('img-123');

console.log(image.meta.id);           // 'img-123'
console.log(image.meta.name);         // 'thumbnail.png'
console.log(image.meta.url);          // 'https://...'
console.log(image.meta.collectionId); // 'c-123'
```

```typescript After (v0.2.0) theme={null}
const image = await collection.getImage('img-123');

console.log(image.id);           // 'img-123'
console.log(image.name);         // 'thumbnail.png'
console.log(image.url);          // 'https://...'
console.log(image.collectionId); // 'c-123'
```

### Collection

```typescript Before (v0.1.3) theme={null}
const collection = await conn.getCollection('c-123');

console.log(collection.meta.id);          // 'c-123'
console.log(collection.meta.name);        // 'My Videos'
console.log(collection.meta.description); // 'Collection description'

const videos = await fetchVideos(collection.meta.id);
```

```typescript After (v0.2.0) theme={null}
const collection = await conn.getCollection('c-123');

console.log(collection.id);          // 'c-123'
console.log(collection.name);        // 'My Videos'
console.log(collection.description); // 'Collection description'

const videos = await fetchVideos(collection.id);
```

### Shot (Search Results)

```typescript Before (v0.1.3) theme={null}
const results = await video.search('hello world');

for (const shot of results.shots) {
  console.log(shot.meta.videoId);     // 'm-123'
  console.log(shot.meta.start);       // 10.5
  console.log(shot.meta.end);         // 15.2
  console.log(shot.meta.text);        // 'hello world example'
  console.log(shot.meta.searchScore); // 0.95
}
```

```typescript After (v0.2.0) theme={null}
const results = await video.search('hello world');

for (const shot of results.shots) {
  console.log(shot.videoId);     // 'm-123'
  console.log(shot.start);       // 10.5
  console.log(shot.end);         // 15.2
  console.log(shot.text);        // 'hello world example'
  console.log(shot.searchScore); // 0.95
}
```

> [!TIP]
>
> **Migration shortcut:** Search your codebase for `.meta.` and replace with `.`
> for each class type.

***

## Breaking Change 2: Async/Await Pattern

In v0.1.3, long-running operations returned `Job` objects that required callback registration. In v0.2.0, these are standard async functions. The SDK handles polling internally, errors propagate through `try/catch`, and there's no risk of silent failures from unregistered error handlers.

### Getting Transcripts

```typescript Before (v0.1.3) theme={null}
import { waitForJob } from 'videodb';

// Option 1: Using callbacks
const job = video.getTranscript();

job.on('success', (transcript) => {
  console.log('Transcript:', transcript);
});

job.on('error', (err) => {
  console.error('Failed:', err);
});

await job.start();

// Option 2: Using waitForJob helper
const transcript = await waitForJob(video.getTranscript(true));
```

```typescript After (v0.2.0) theme={null}
// No imports needed - async/await is built-in

// Just await the method directly
// SDK handles polling internally
const transcript = await video.getTranscript();

// Success! Use the result immediately
console.log('Transcript:', transcript);

// Errors? Use standard try/catch (see Error Handling section)




```

### Indexing Spoken Words

```typescript Before (v0.1.3) theme={null}
import { waitForJob } from 'videodb';

const indexJob = video.indexSpokenWords();

indexJob.on('success', (result) => {
  console.log('Indexing complete:', result.success);
});

indexJob.on('error', (err) => {
  console.error('Indexing failed:', err);
});

await indexJob.start();
```

```typescript After (v0.2.0) theme={null}
// No imports needed

// Await the indexing operation directly
const result = await video.indexSpokenWords();

// SDK polls until complete, then returns
console.log('Indexing complete:', result.success);



```

### Uploading Media

```typescript Before (v0.1.3) theme={null}
import { waitForJob } from 'videodb';

const uploadJob = await collection.uploadFile({
  filePath: '/path/to/video.mp4',
  name: 'My Video',
});

uploadJob.on('success', (video) => {
  console.log('Video ID:', video.meta.id);
});

uploadJob.on('error', (err) => {
  console.error('Upload failed:', err);
});

await uploadJob.start();
```

```typescript After (v0.2.0) theme={null}
// No imports needed

// Upload returns the Video object directly
const video = await collection.uploadFile({
  filePath: '/path/to/video.mp4',
  name: 'My Video',
});

// Video is ready to use immediately
console.log('Video ID:', video.id);



```

### Error Handling

```typescript Before (v0.1.3) theme={null}
import { waitForJob } from 'videodb';

const job = video.getTranscript();

job.on('success', (transcript) => {
  console.log('Got transcript:', transcript);
});

job.on('error', (err) => {
  // If you forget this handler, errors are silently logged
  console.error('Transcript failed:', err);
  notifyUser('Could not generate transcript');
});

await job.start();
```

```typescript After (v0.2.0) theme={null}
// No imports needed

// Standard try/catch - errors can't be silently ignored
try {
  const transcript = await video.getTranscript();
  console.log('Got transcript:', transcript);
} catch (err) {
  // Errors bubble up naturally
  console.error('Transcript failed:', err);
  notifyUser('Could not generate transcript');
}
```

***

## Removed Exports

The following exports no longer exist in v0.2.0:

```typescript theme={null}
// These imports will fail in v0.2.0
import {
  Job, // REMOVED
  TranscriptJob, // REMOVED
  UploadJob, // REMOVED
  IndexJob, // REMOVED
  waitForJob, // REMOVED
} from "videodb";
```

> [!WARNING]
>
> If your code imports any of these, remove the imports and refactor to use
> `await` directly on the method calls.

***

## Migration Checklist

### Property Access

**1. Search for .meta. patterns**

Use your IDE or grep to find all occurrences of `.meta.` in your codebase.

**2. Replace with direct access**

* `video.meta.*` → `video.*` - `audio.meta.*` → `audio.*` - `image.meta.*` →
  `image.*` - `collection.meta.*` → `collection.*` - `shot.meta.*` → `shot.*`

### Job Pattern

**1. Remove Job imports**

Delete imports for `Job`, `TranscriptJob`, `UploadJob`, `IndexJob`,
`waitForJob`

**2. Remove callback registrations**

Delete `.on('success', ...)` and `.on('error', ...)` blocks

**3. Remove .start() calls**

Delete any `job.start()` or `await job.start()` calls

**4. Add await to method calls**

Change `const job = video.getTranscript()` to `const transcript = await
        video.getTranscript()`

**5. Update error handling**

Wrap calls in `try/catch` instead of using `.on('error', ...)`

***

## Complete Example

```typescript Before (v0.1.3) theme={null}
import { connect, waitForJob } from 'videodb';

async function processVideo() {
  const conn = connect(process.env.VIDEO_DB_API_KEY);
  const collection = await conn.getCollection();

  // Upload
  const uploadJob = await collection.uploadFile({
    filePath: './video.mp4',
    name: 'My Video',
  });
  const video = await waitForJob(uploadJob);
  console.log('Uploaded:', video.meta.id);

  // Transcript
  const transcriptJob = video.getTranscript();
  const transcript = await waitForJob(transcriptJob);

  // Index
  const indexJob = video.indexSpokenWords();
  await waitForJob(indexJob);

  // Search
  const results = await video.search('keyword');
  for (const shot of results.shots) {
    console.log(`${shot.meta.start}s: ${shot.meta.text}`);
  }
}
```

```typescript After (v0.2.0) theme={null}
import { connect } from 'videodb';

async function processVideo() {
  const conn = connect(process.env.VIDEO_DB_API_KEY);
  const collection = await conn.getCollection();

  // Upload - returns Video directly
  const video = await collection.uploadFile({
    filePath: './video.mp4',
    name: 'My Video',
  });
  console.log('Uploaded:', video.id);

  // Transcript - just await
  const transcript = await video.getTranscript();

  // Index - just await
  await video.indexSpokenWords();

  // Search
  const results = await video.search('keyword');
  for (const shot of results.shots) {
    console.log(`${shot.start}s: ${shot.text}`);
  }
}
```

***

## Need Help?

> **[GitHub Issues](https://github.com/video-db/videodb-node/issues)**
>
> Report bugs or ask questions

> **[Discord Community](https://discord.com/invite/py9P639jGz)**
>
> Get help from the community


# Python SDK
Source: https://docs.videodb.io/pages/getting-started/python

Install and configure the VideoDB Python SDK

## Installation

```bash theme={null}
pip install videodb
```

> **[PyPI](https://pypi.org/project/videodb/)**
>
> View package on PyPI

> **[GitHub](https://github.com/video-db/videodb-python)**
>
> Source code and issues

## Quick Start

```python theme={null}
import videodb

# Connect using environment variable
conn = videodb.connect()

# Or pass API key directly
conn = videodb.connect(api_key="your-api-key")
```

## Environment Variables

| Variable | Description |
| :- | :- |
| `VIDEO_DB_API_KEY` | Your API key from [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) |

```bash theme={null}
export VIDEO_DB_API_KEY="your-api-key"
```

## Requirements

* Python 3.8 or higher
* Works on Linux, macOS, and Windows

## Basic Usage

```python theme={null}
import videodb

conn = videodb.connect()

# Upload a video
coll = conn.get_collection()
video = coll.upload(url="https://www.youtube.com/watch?v=example")

# Create a timed transcript artifact, then make it searchable
understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")

transcript_index = video.index(
    name="transcript",
    source=transcript_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["text"]},
)
transcript_index.wait_until_complete()

# Search with natural language
results = video.semantic_search(
    query="key moments",
    index_ids=[transcript_index.index_id],
    top_k=5,
)
for shot in results.get_shots():
    print(f"{shot.start}s: {shot.text}")
```

## Server Side

Use the full SDK on your backend to manage sessions, run AI pipelines, and handle webhooks. Your API key should never be exposed to the browser.

```python theme={null}
# Create a capture session and generate a client token
cap = conn.create_capture_session(end_user_id="user_123")
token = conn.generate_client_token(expires_in=600)
```

## Client Side

For real-time desktop capture, install the Capture SDK on your client application. It uses short-lived tokens instead of your API key.

> [!NOTE]
>
> Desktop capture currently supports **macOS** and **Windows**.

```bash theme={null}
pip install "videodb[capture]"
```

> **[Capture SDK Overview](/pages/ingest/capture-sdks/overview)**
>
> Learn how to integrate real-time screen, audio, and camera capture into your application

## Next Steps

> **[Quickstart](/pages/getting-started/quickstart)**
>
> Build your first perception-enabled agent

> **[API Reference](/api-reference/introduction)**
>
> Complete REST API documentation


# Quickstart
Source: https://docs.videodb.io/pages/getting-started/quickstart

See media, understand what it contains, and act on timestamped evidence

Video was built for people to watch. VideoDB makes it understandable and actionable for people and AI agents.

With one backend, your application can turn files, live streams, and cameras into structured, timestamped context - then retrieve exact moments and act on playable evidence.

Every VideoDB workflow follows the same loop:

**1. See**

Upload a video, connect a live stream, or capture a desktop.

**2. Understand**

Create reusable artifacts from what was said and what was shown, then index the context you want to retrieve.

**3. Act**

Retrieve matching moments, play the evidence, or trigger an alert.

You will complete this loop with one recorded video, apply it continuously to a live stream, and see how the same foundation extends to desktop capture.

## Set up VideoDB

### Get an API key

1. Go to the [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs).
2. Copy your API key.
3. Add it to your environment:

```bash theme={null}
export VIDEO_DB_API_KEY="your-api-key"
```

### Install the SDK

```bash Python theme={null}
pip install videodb
```

```bash Node.js theme={null}
npm install videodb
```

Run the Node.js examples as ES modules by using an `.mjs` file or setting `"type": "module"` in `package.json`.

***

## See, understand, and search a video

Start with one recorded video and complete the loop. By the end, a natural-language request will return timestamped matches and one stream containing only the relevant moments.

Choose Python or Node.js, then run that language's blocks in order in the same script or interactive session.

### See: upload a video

Connect to your default collection, then upload a public video URL.

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

video = coll.upload(
    url="https://www.youtube.com/watch?v=LPZh9BOjkQs"
)

print(video.id, video.name, video.length)
```

```javascript Node.js theme={null}
import { connect } from "videodb";

const conn = connect();
const coll = await conn.getCollection();

const video = await coll.uploadURL({
    url: "https://www.youtube.com/watch?v=LPZh9BOjkQs"
});

console.log(video.id, video.name, video.length);
```

The media is now available to your application. VideoDB can also ingest local files, object-storage URLs, and other publicly accessible video URLs.

Generate a playable stream for the complete uploaded video. Later, you will compile a second stream containing only matching evidence.

```python Python theme={null}
stream_url = video.generate_stream()
print(stream_url)
```

```javascript Node.js theme={null}
const streamUrl = await video.generateStream();
console.log(streamUrl);
```

### Understand: create visual and spoken artifacts

Next, create reusable, timestamped artifacts from what was said and what was shown. One understanding run can create both: `spoken_words` produces a timed transcript, while the `vlm` prompt controls which visual details become structured context. Understanding and indexing run as background jobs; the wait calls below continue when each result is ready.

```python Python theme={null}
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {
            "type": "vlm",
            "name": "scene",
            "config": {
                "prompt": "Describe the visual content."
            },
        },
    ]
)
understanding.wait_until_complete()
```

```javascript Node.js theme={null}
const understanding = await video.understand([
    { type: "spoken_words", name: "transcript" },
    {
        type: "vlm",
        name: "scene",
        config: {
            prompt: "Describe the visual content."
        }
    }
]);
await understanding.waitUntilComplete();
```

Retrieve the completed analyzer handles:

```python Python theme={null}
transcript_analyzer = understanding.get_analyzer("transcript")
scene_analyzer = understanding.get_analyzer("scene")
```

```javascript Node.js theme={null}
const transcriptAnalyzer = await understanding.getAnalyzer("transcript");
const sceneAnalyzer = await understanding.getAnalyzer("scene");
```

Each artifact contains timestamped scenes. Inspect the first result from each analyzer:

```python Python theme={null}
for analyzer in [transcript_analyzer, scene_analyzer]:
    output = analyzer.get_output()
    first_scene = output["scenes"][0]

    print(f"{analyzer.name}: {len(output['scenes'])} scenes")
    print(first_scene["start"], first_scene["end"])
    print(first_scene["data"]["text"])
    print()
```

```javascript Node.js theme={null}
for (const analyzer of [transcriptAnalyzer, sceneAnalyzer]) {
    const output = await analyzer.getOutput();
    const firstScene = output.scenes[0];

    console.log(`${analyzer.name}: ${output.scenes.length} scenes`);
    console.log(firstScene.start, firstScene.end);
    console.log(firstScene.data.text);
    console.log();
}
```

### Understand: index the artifacts

Artifacts record what VideoDB found. Index them once to make that context retrievable without rerunning the analysis.

Create the transcript index:

```python Python theme={null}
transcript_index = video.index(
    name="quickstart_transcript",
    source=transcript_analyzer,
)
transcript_index.wait_until_complete()

print(transcript_index.index_id, transcript_index.status)
```

```javascript Node.js theme={null}
const transcriptIndex = await video.index(transcriptAnalyzer, {
    name: "quickstart_transcript"
});
await transcriptIndex.waitUntilComplete();

console.log(transcriptIndex.indexId, transcriptIndex.status);
```

Create the visual scene index:

```python Python theme={null}
scene_index = video.index(
    name="quickstart_scene",
    source=scene_analyzer,
)
scene_index.wait_until_complete()

print(scene_index.index_id, scene_index.status)
```

```javascript Node.js theme={null}
const sceneIndex = await video.index(sceneAnalyzer, {
    name: "quickstart_scene"
});
await sceneIndex.waitUntilComplete();

console.log(sceneIndex.indexId, sceneIndex.status);
```

VideoDB derives the index configuration from each artifact. You can choose explicit fields later when an application needs a specific schema for filtering or aggregation.

### Act: find and play matching moments

Now ask for a moment - not a summary. Search both indexes by their returned IDs so the request uses the artifacts created in this run. Each returned `Shot` includes timestamps and the matched index text in its metadata.

```python Python theme={null}
results = video.semantic_search(
    query="an explanation supported by diagrams and on-screen text",
    index_ids=[transcript_index.index_id, scene_index.index_id],
    top_k=5,
)

for shot in results.get_shots():
    print(
        f"{shot.start:.1f}s–{shot.end:.1f}s: "
        f"{shot.metadata['embedded_text']}"
    )
```

```javascript Node.js theme={null}
const results = await video.semanticSearch(
    "an explanation supported by diagrams and on-screen text",
    {
        indexIds: [transcriptIndex.indexId, sceneIndex.indexId],
        topK: 5
    }
);

for (const shot of results.getShots()) {
    console.log(
        `${shot.start.toFixed(1)}s–${shot.end.toFixed(1)}s: ` +
        shot.metadata.embeddedText
    );
}
```

Compile the matching moments into one playable HLS stream:

```python Python theme={null}
evidence_url = results.compile()
print(evidence_url)
```

```javascript Node.js theme={null}
const evidenceUrl = await results.compile();
console.log(evidenceUrl);
```

You now have timestamped results and a stream that plays only the matching evidence - ready to show to a user or pass to another part of your application.

***

## See, understand, and act on a live stream

Files finish processing; streams keep producing new windows. For live media, understanding and indexing run continuously while the RTStream is active.

Copy these blocks in order into one file. The first block opens `main()` in each language, the middle blocks continue it, and the final block closes and runs it.

### See: connect the stream

Connect the sample stream in the default collection and retain it for the continuous workflow.

```python Python theme={null}
import asyncio
import videodb

RTSP_URL = "rtsp://samples.rts.videodb.io:8554/intruder"


async def main():
    conn = videodb.connect()
    coll = conn.get_collection()

    rtstream = coll.connect_rtstream(
        url=RTSP_URL,
        name="Quickstart Live Stream",
        store=True,
    )
```

```javascript Node.js theme={null}
import { connect } from "videodb";

const RTSP_URL = "rtsp://samples.rts.videodb.io:8554/intruder";

async function main() {
    const conn = connect();
    const coll = await conn.getCollection();

    const rtstream = await coll.connectRTStream(
        RTSP_URL,
        "Quickstart Live Stream",
        ["video"],
        30,
        true
    );
```

### Understand: analyze each window

Use one continuous VLM output named `scene` for each five-second window.

```python Python theme={null}
    understanding = rtstream.understand(
        segmentation={"type": "time", "window": "5s"},
        analyzers=[
            {
                "type": "vlm",
                "name": "scene",
                "sampling": {"frame_count": 2},
                "config": {
                    "prompt": "Describe the people, movement, and activity in this stream window."
                },
            }
        ],
        store=True,
    )
```

```javascript Node.js theme={null}
    const understanding = await rtstream.understand({
        segmentation: { type: "time", window: "5s" },
        analyzers: [
            {
                type: "vlm",
                name: "scene",
                sampling: { frame_count: 2 },
                config: {
                    prompt: "Describe the people, movement, and activity in this stream window."
                }
            }
        ],
        store: true
    });
```

### Understand: index the scene output

Create a semantic index that receives each new `scene` record.

```python Python theme={null}
    index = rtstream.index(
        name="quickstart-live-scenes",
        source=understanding.outputs["scene"],
        use_for=["semantic"],
    )
```

```javascript Node.js theme={null}
    const index = await rtstream.index({
        name: "quickstart-live-scenes",
        source: {
            understanding_id: understanding.id,
            output: "scene"
        },
        useFor: ["semantic"]
    });
```

The Node.js index source identifies the continuous understanding run and its `scene` output explicitly.

### Act: connect for live alerts

Open a WebSocket before attaching the alert so matches can reach this client.

```python Python theme={null}
    websocket = conn.connect_websocket(coll.id)
    await websocket.connect()
```

```javascript Node.js theme={null}
    const websocket = await conn.connectWebsocket(coll.id);
    await websocket.connect();
```

### Act: define the event

Give the condition a reusable prompt and label.

```python Python theme={null}
    event_id = conn.create_event(
        event_prompt="Detect when a person appears in the monitored area.",
        label="person_appears",
    )
```

```javascript Node.js theme={null}
    const eventId = await conn.createEvent(
        "Detect when a person appears in the monitored area.",
        "person_appears"
    );
```

### Act: create and receive the alert

Attach the event to the index, then print the first alert and its playable evidence URL.

```python Python theme={null}
    alert_id = index.create_alert(
        event_id=event_id,
        callback_url="https://example.com",
        ws_connection_id=websocket.connection_id,
    )

    async for message in websocket.receive():
        if message.get("channel") == "alert":
            alert = message["data"]
            print(alert["label"], alert["confidence"])
            print(alert["explanation"])
            print(alert["stream_url"])
            break
```

```javascript Node.js theme={null}
    const alertId = await index.createAlert(
        eventId,
        "https://example.com",
        websocket.connectionId
    );

    for await (const message of websocket.receive()) {
        if (message.channel === "alert") {
            const alert = message.data;
            console.log(alert.label, alert.confidence);
            console.log(alert.explanation);
            console.log(alert.stream_url);
            break;
        }
    }
```

The alert includes the event label, confidence, explanation, timestamps, and a playable stream URL for the triggering window.

### Stop the live resources

Disable delivery, stop the continuous jobs and stream, then close the WebSocket.

```python Python theme={null}
    index.disable_alert(alert_id)
    index.stop()
    understanding.stop()
    rtstream.stop()
    await websocket.close()


if __name__ == "__main__":
    asyncio.run(main())
```

```javascript Node.js theme={null}
    await index.disableAlert(alertId);
    await index.stop();
    await understanding.stop();
    await rtstream.stop();
    await websocket.close();
}

main();
```

> [!NOTE]
>
> RTStream understanding is continuous, so the first indexed records and alerts can take several time windows to appear. `callback_url` remains required when WebSocket delivery is enabled. This public sample uses `https://example.com` as a placeholder; replace it with an endpoint you control before using your own stream. Production applications should also run the same cleanup during shutdown or interruption.

***

## Extend the loop to a live desktop

Desktop capture turns a screen, microphone, or system-audio source into an RTStream - with explicit permission from the person using the device. Your backend keeps the API key; the desktop client receives only a short-lived token. Desktop capture currently supports **macOS** and **Windows**.

Install the desktop capture runtime:

```bash Python theme={null}
pip install "videodb[capture]"
```

```bash Node.js theme={null}
npm install videodb
```

### See: create an authorized capture

Create a CaptureSession on your backend and mint a token that expires after ten minutes. Deliver only the session ID and token to the desktop client through your authenticated application channel.

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

capture_session = coll.create_capture_session(
    end_user_id="quickstart-user",
    metadata={"app": "desktop-quickstart"},
)
client_token = conn.generate_client_token(expires_in=600)

# Pass this mapping to the desktop process through your authenticated launcher.
desktop_environment = {
    "VIDEODB_CAPTURE_SESSION_ID": capture_session.id,
    "VIDEODB_CAPTURE_TOKEN": client_token,
}
```

```javascript Node.js theme={null}
import { connect } from "videodb";

const conn = connect();
const coll = await conn.getCollection();

const captureSession = await coll.createCaptureSession({
    endUserId: "quickstart-user",
    metadata: { app: "desktop-quickstart" }
});
const clientToken = await conn.generateClientToken(600);

// Pass this mapping to the desktop process through your authenticated launcher.
const desktopEnvironment = {
    VIDEODB_CAPTURE_SESSION_ID: captureSession.id,
    VIDEODB_CAPTURE_TOKEN: clientToken
};
```

Run the next block in a separate desktop process with that environment. This keeps the short-lived token out of logs, URLs, and source code. Request screen permission, select the default display, and begin streaming it.

```python Python theme={null}
import asyncio
import os
from videodb.capture import CaptureClient


async def capture(session_id, client_token):
    client = CaptureClient(client_token=client_token)

    await client.request_permission("screen_capture")
    channels = await client.list_channels()
    display = channels.displays.default
    display.store = True
    display.is_primary = True

    await client.start_session(
        capture_session_id=session_id,
        channels=[display],
    )

    await asyncio.to_thread(input, "Capture started. Press Enter to stop.")
    await client.stop_session()
    await client.shutdown()


session_id = os.environ["VIDEODB_CAPTURE_SESSION_ID"]
client_token = os.environ["VIDEODB_CAPTURE_TOKEN"]
asyncio.run(capture(session_id, client_token))
```

```javascript Node.js theme={null}
import { CaptureClient } from "videodb/capture";

async function capture(sessionId, clientToken) {
    const client = new CaptureClient({ sessionToken: clientToken });

    await client.requestPermission("screen-capture");
    const channels = await client.listChannels();
    const display = channels.displays.default;

    await client.startSession({
        sessionId,
        channels: [{
            channelId: display.id,
            type: "video",
            store: true,
            isPrimary: true
        }]
    });

    console.log("Capture started. Press Enter to stop.");
    await new Promise((resolve) => process.stdin.once("data", resolve));
    await client.stopSession();
    await client.shutdown();
}

const sessionId = process.env.VIDEODB_CAPTURE_SESSION_ID;
const clientToken = process.env.VIDEODB_CAPTURE_TOKEN;
await capture(sessionId, clientToken);
```

Leave this desktop process running after it prints `Capture started`. Return to the original backend session and continue with Understand and Act; press Enter in the desktop process only after you finish searching the live capture.

### Understand: index the live screen

While the desktop process remains active, fetch the session again on your backend. The captured display is now an RTStream, so it uses the same continuous understanding pipeline as any other live source.

```python Python theme={null}
capture_session = coll.get_capture_session(capture_session.id)
screen = capture_session.get_rtstream("screen")[0]

screen_understanding = screen.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Describe the visible desktop activity."},
        }
    ],
    store=True,
)

screen_index = screen.index(
    name="quickstart-desktop-scenes",
    source=screen_understanding.outputs["scene"],
    use_for=["semantic"],
)
```

```javascript Node.js theme={null}
const activeCapture = await coll.getCaptureSession(captureSession.id);
await activeCapture.refresh();
const screen = activeCapture.getRTStream("screen")[0];

const screenUnderstanding = await screen.understand({
    segmentation: { type: "time", window: "5s" },
    analyzers: [
        {
            type: "vlm",
            name: "scene",
            sampling: { frame_count: 2 },
            config: { prompt: "Describe the visible desktop activity." }
        }
    ],
    store: true
});

const screenIndex = await screen.index({
    name: "quickstart-desktop-scenes",
    source: {
        understanding_id: screenUnderstanding.id,
        output: "scene"
    },
    useFor: ["semantic"]
});
```

### Act: find a captured moment

After a few windows have been indexed, search the live desktop and generate a playable stream for each matching moment.

```python Python theme={null}
matches = screen.search(
    query="source code open in an editor",
    index_id=screen_index.id,
    result_threshold=5,
)

for shot in matches.get_shots():
    print(shot.text)
    print(shot.generate_stream())
```

```javascript Node.js theme={null}
const matches = await screen.search({
    query: "source code open in an editor",
    indexId: screenIndex.id,
    resultThreshold: 5
});

for (const shot of matches.getShots()) {
    console.log(shot.text);
    console.log(await shot.generateStream());
}
```

### Stop live desktop processing

Stop the continuous index before its understanding run. Then return to the separate desktop process and press Enter so it stops capture and shuts down its local client.

```python Python theme={null}
screen_index.stop()
screen_understanding.stop()
```

```javascript Node.js theme={null}
await screenIndex.stop();
await screenUnderstanding.stop();
```

The same live index can also drive events and alerts. When capture stops, stored channels can be exported as regular VideoDB media for playback and longer-term search.

***

## Continue building

You have now completed the loop with recorded and live media, and seen how a desktop becomes another live source: see the media, turn it into timestamped understanding, and act on evidence.

> **[Understanding artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Configure analyzers, prompts, schemas, and multimodal inputs.

> **[Search and retrieval](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Choose semantic search, structured query, aggregation, Search, or Ask.

> **[RTStream understanding](/pages/ingest/live-streams/rtstream-understand-index)**
>
> Build continuous understanding, retrieval, and alert workflows.


# Capture SDK Overview
Source: https://docs.videodb.io/pages/ingest/capture-sdks/overview

Real-time desktop capture for AI agents - stream screen, microphone, and system audio

> [!NOTE]
>
> Desktop capture currently supports **macOS** and **Windows**.

## 1. Backend Setup

### Install

```bash theme={null}
pip install videodb
```

### Create a Capture Session

Your backend creates a session and generates a short-lived token for the desktop client:

```python Python theme={null}
import videodb

conn = videodb.connect()

# Create session for a user
cap = conn.create_capture_session(
    end_user_id="user_abc",
    callback_url="https://your-backend.com/webhooks/videodb",
    metadata={"app": "my-ai-copilot"}
)

# Generate token for desktop client (never share API key)
token = conn.generate_client_token(expires_in=600)

# Send session ID and token to desktop client
print(f"Session: {cap.id}, Token: {token}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();

// Create session for a user
const cap = await conn.createCaptureSession({
    endUserId: "user_abc",
    callbackUrl: "https://your-backend.com/webhooks/videodb",
    metadata: { app: "my-ai-copilot" }
});

// Generate token for desktop client (never share API key)
const token = await conn.generateClientToken(600);

// Send session ID and token to desktop client
console.log(`Session: ${cap.id}, Token: ${token}`);
```

***

## 2. Client Setup

### Install

```bash theme={null}
pip install "videodb[capture]"
```

### Start Capture

The desktop client uses the token to stream screen and audio:

```python Python theme={null}
import asyncio
from videodb.capture import CaptureClient

async def capture(capture_session_id: str, client_token: str):
    client = CaptureClient(client_token=client_token)

    # Request permissions
    await client.request_permission("microphone")
    await client.request_permission("screen_capture")

    # Discover available sources
    channels = await client.list_channels()
    mic = channels.mics.default
    display = channels.displays.default
    if display:
        display.is_primary = True
    system_audio = channels.system_audio.default
    selected = [c for c in [mic, display, system_audio] if c]

    # Start capture
    await client.start_session(
        capture_session_id=capture_session_id,
        channels=selected,
    )

    # Listen for events
    async for ev in client.events():
        print(f"{ev.event}: {ev.payload}")
        if ev.event in ("recording-complete", "error"):
            break

    await client.stop_session()
    await client.shutdown()

# Run the capture
if __name__ == "__main__":
    asyncio.run(capture(
        capture_session_id="cap-xxx",  # From backend
        client_token="token-xxx"        # From backend
    ))
```

```javascript Node.js theme={null}
import { CaptureClient } from 'videodb/capture';

async function capture(captureSessionId, clientToken) {
    const client = new CaptureClient({ sessionToken: clientToken });

    // Request permissions
    await client.requestPermission('microphone');
    await client.requestPermission('screen-capture');

    // Discover available sources
    const channels = await client.listChannels();
    const mic = channels.mics.default;
    const display = channels.displays.default;
    const systemAudio = channels.systemAudio.default;

    const selectedChannels = [mic, display, systemAudio]
        .filter(Boolean)
        .map(c => ({ channelId: c.id, type: c.type, store: true }));

    // Start capture
    await client.startSession({
        sessionId: captureSessionId,
        channels: selectedChannels
    });

    // Listen for events
    client.on('transcript', (data) => console.log(`Transcript: ${data.text}`));
    client.on('recording:stopped', () => client.shutdown());
}

// Run the capture
capture(
    "cap-xxx",      // From backend
    "token-xxx"     // From backend
);
```

***

## 3. Backend Starts AI

When capture begins, your backend receives a webhook and starts AI processing:

> [!NOTE]
>
> Mapping Capture channel categories to RTStream IDs is not yet a supported released-SDK contract, so this backend lookup example is deferred.

***

## 4. What You Get

Capture SDK types guarantee client events. Backend RTStream payload fields depend on the pipeline and are not a fixed Capture SDK envelope; inspect the events your integration receives.

**Build with these:**

* Screen-aware AI agents
* Live meeting copilots
* In-call assistance
* Semantic search and replay

### Architecture

![Diagram showing the architecture of the system](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/capture-sdk/capture-architecture.png?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=e36cf227079d57b2be18e58c1954b29e)

1. **Backend** creates a CaptureSession and mints a short-lived token
2. **Desktop client** uses the token to stream screen + audio (never sees API key)
3. **VideoDB** creates RTStreams (one per channel) when capture starts
4. **Backend** receives the lifecycle webhook and, once it has a supported RTStream ID, starts an RTStream understanding and indexing pipeline
5. **AI events** flow back via WebSocket (real-time) or can be polled

### Two Runtimes

| Backend | Desktop Client |
| :- | :- |
| Holds API key | Receives session token |
| Creates sessions | Captures media |
| Runs AI pipelines | Streams to VideoDB |
| Receives events | Emits local UX events |

> [!TIP]
>
> **Rule of thumb:** Webhooks for correctness (durable, at-least-once). WebSocket for live UI (best-effort).

***

## 5. Example Applications

> **[Claude Pair Programmer](/examples-and-tutorials/ai-copilots/pair-programmer)**
>
> AI coding assistant with screen and audio context

> **[Bloom](/examples-and-tutorials/ai-copilots/bloom)**
>
> Local-first screen recorder with AI indexing

> **[Focusd](/examples-and-tutorials/ai-copilots/focusd)**
>
> AI-powered productivity tracking

> **[Call.md](/examples-and-tutorials/ai-copilots/call-md)**
>
> Real-time meeting intelligence

***

## 6. Core Concepts

### CaptureSession (cap-xxx)

The lifecycle container for one capture run. Created by backend, activated by desktop client.

**States:** `created → starting → active → stopping → stopped → exported`

### RTStream (rts-xxx)

A real-time media stream, one per captured channel. This is where you run AI:

> [!NOTE]
>
> The helpers in this example are RTStream helper APIs. For descriptor-backed V2 understanding and indexing, see [Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index).

```python Python theme={null}
rtstream.start_transcript()
rtstream.index_audio(prompt="Extract key decisions")
rtstream.index_visuals(prompt="Describe what user is doing")
rtstream.search("budget discussion")
```

```javascript Node.js theme={null}
await rtstream.startTranscript();
await rtstream.indexAudio({ prompt: "Extract key decisions" });
await rtstream.indexVisuals({ prompt: "Describe what user is doing" });
await rtstream.search({ query: "budget discussion" });
```

### Channel

A recordable source on the desktop:

| Channel | Description |
| :- | :- |
| `mic:default` | Default microphone |
| `system_audio:default` | System audio output |
| `display:1`, `display:2` | Connected displays |

### Multi-Screen Capture

When multiple monitors are connected, each appears as a separate `display:N` channel. Use `cap.displays` on the backend to inspect available video channels:

```python Python theme={null}
cap = conn.get_capture_session("cap-xxx")

# List all video (display) channels
for d in cap.displays:
    print(f"{d.id}  primary={d.is_primary}")
# display:1  primary=True
# display:2  primary=False
```

`cap.displays` returns a list of video channel objects. Set a selected display's `is_primary` field before capture starts to mark it as primary.

To capture multiple screens, pass all desired display channels to the desktop client:

```python Python theme={null}
channels = await client.list_channels()

# Select both displays
display1 = channels.displays.default
display2 = channels.displays[1]   # display:2
display1.is_primary = True

await client.start_session(
    capture_session_id=cap_id,
    channels=[
        mic,
        display1,
        display2,
        system_audio,
    ],
)
```

Each display produces its own RTStream on the backend. The primary display is used for the default muxed export video; non-primary displays are available as raw channel assets or can be exported separately (see [Storage & Search](/pages/ingest/capture-sdks/storage-and-search)).

***

## Explore More

> **[View All Examples on GitHub](https://github.com/video-db/videodb-capture-quickstart)**
>
> Complete source code with quickstart guides, example apps, and implementation patterns

> **[Real-time Context](/pages/ingest/capture-sdks/realtime-context)**
>
> Events you receive from capture

> **[Storage & Search](/pages/ingest/capture-sdks/storage-and-search)**
>
> Optional persistence and semantic search


# Privacy Controls
Source: https://docs.videodb.io/pages/ingest/capture-sdks/privacy-controls

Consent patterns, permission handling, and privacy-first capture

Desktop capture handles sensitive data. Nothing records without explicit user consent. This page covers permission patterns, storage controls, and privacy-first design.

> [!NOTE]
>
> Desktop capture currently supports **macOS** and **Windows**.

## Quick Example

```python Python theme={null}
from videodb.capture import CaptureClient

client = CaptureClient(client_token=token)

# Request explicit permissions (OS dialogs appear)
await client.request_permission("microphone")
await client.request_permission("screen_capture")

# User must grant permission before capture can start
channels = await client.list_channels()
```

```javascript Node.js theme={null}
import { CaptureClient } from 'videodb/capture';

const client = new CaptureClient({ sessionToken: token });

// Request explicit permissions (OS dialogs appear)
await client.requestPermission('microphone');
await client.requestPermission('screen-capture');

// User must grant permission before capture can start
const channels = await client.listChannels();
```

***

## Trust Model

### Two-Component Architecture

Security is built into the architecture:

| Component | Has API Key | Can See Media |
| :- | :- | :- |
| Backend | Yes | No (receives events only) |
| Desktop Client | No (uses token) | Yes (captures locally) |

* **API key never leaves your backend**
* **Desktop client uses short-lived tokens** (10-15 min expiry recommended)
* **Compromised tokens have limited blast radius**

### Token Pattern

```python Python theme={null}
# Backend (holds API key)
conn = videodb.connect()  # Uses VIDEO_DB_API_KEY

# Generate short-lived token for client
token = conn.generate_client_token(expires_in=600)  # 10 minutes

# Send token to desktop app (never send API key)
```

```javascript Node.js theme={null}
// Backend (holds API key)
const conn = connect();  // Uses VIDEO_DB_API_KEY

// Generate short-lived token for client
const token = await conn.generateClientToken(600);  // 10 minutes

// Send token to desktop app (never send API key)
```

***

## Permission Handling

### Request Before Capture

```python Python theme={null}
# Each permission triggers an OS dialog
await client.request_permission("microphone")
await client.request_permission("screen_capture")
```

```javascript Node.js theme={null}
// Each permission triggers an OS dialog
await client.requestPermission('microphone');
await client.requestPermission('screen-capture');
```

### Required UX Elements

> [!WARNING]
>
> Your desktop client must include:
>
> * **Recording indicator** - Visual cue that capture is active
> * **Pause button** - Let users temporarily stop
> * **Stop button** - Let users end the session

***

## Storage Control

### Ephemeral Mode

Process in real-time without persisting media:

```python Python theme={null}
channels = await client.list_channels()
selected = [
    channel
    for channel in (
        channels.mics.default,
        channels.displays.default,
        channels.system_audio.default,
    )
    if channel
]
for channel in selected:
    channel.store = False

await client.start_session(
    capture_session_id=cap_id,
    channels=selected,
)
```

```javascript Node.js theme={null}
const channels = await client.listChannels();
const selectedChannels = [
    channels.mics.default,
    channels.displays.default,
    channels.systemAudio.default,
]
    .filter(Boolean)
    .map(channel => ({ channelId: channel.id, type: channel.type, store: false }));

await client.startSession({
    sessionId: capId,
    channels: selectedChannels,
});
```

**Use ephemeral mode when:**

* Processing sensitive content
* Storage isn't needed
* Privacy regulations require it
* Building real-time assistants only

### Selective Storage

Store only what you need:

```python Python theme={null}
channels = await client.list_channels()
mic = channels.mics.default
display = channels.displays.default
system_audio = channels.system_audio.default

if mic:
    mic.store = True
if display:
    display.store = False
if system_audio:
    system_audio.store = True

selected = [channel for channel in (mic, display, system_audio) if channel]
await client.start_session(
    capture_session_id=cap_id,
    channels=selected,
)
```

```javascript Node.js theme={null}
const channels = await client.listChannels();
const mic = channels.mics.default;
const display = channels.displays.default;
const systemAudio = channels.systemAudio.default;
const selectedChannels = [
    [mic, true],
    [display, false],
    [systemAudio, true],
]
    .filter(([channel]) => Boolean(channel))
    .map(([channel, store]) => ({ channelId: channel.id, type: channel.type, store }));

await client.startSession({
    sessionId: capId,
    channels: selectedChannels,
});
```

***

## Data Retention

### Default Behavior

* Tell users which selected channels may be stored and for how long.
* Define a retention schedule and access policy in your product before capture starts.
* Keep the records needed to honor consent, access, and deletion requests.

### Data Subject Requests

Provide authenticated ways for people to request access to or deletion of their data, and document how your application maps those requests to its stored media. Coordinate the provider-side lifecycle with your retention process.

> [!NOTE]
>
> Python supports `list_capture_sessions(collection_id="default", status=None)`. It does not support an `end_user_id` filter or `CaptureSession.delete()`, so keep per-user mapping and deletion orchestration in your application until a supported lifecycle API is available.

***

## Compliance Patterns

### GDPR

* Document processing purposes and the data captured for each purpose.
* Provide a clear process for access and deletion requests.
* Minimize collection by selecting only the channels your workflow requires.

### HIPAA and Sensitive Data

* Apply strict access controls and audit access to sensitive captures.
* Minimize captured and stored data, and follow your organization's applicable compliance program.
* Consult your legal, privacy, and security teams for the deployment requirements that apply to your use case.

### General Best Practices

| Practice | Implementation |
| :- | :- |
| Never expose API keys | Use client tokens |
| Default to ephemeral | Only persist when needed |
| Short token lifetimes | 10-15 minutes |
| Support data requests | Define an authenticated application workflow |
| Get consent | Permission dialogs before capture |
| Show indicators | Recording visible to user |

***

## Summary

> [!WARNING]
>
> **Key principles:**
>
> * Nothing records without explicit user permission
> * Storage is optional per-channel (`store: false`)
> * Visible recording indicators are required
> * API key never leaves backend
> * Tokens are short-lived and limited scope

***

## Next Steps

> **[Capture Overview](/pages/ingest/capture-sdks/overview)**
>
> Architecture and quickstart

> **[Real-time Context](/pages/ingest/capture-sdks/realtime-context)**
>
> Events you receive


# Real-time Context
Source: https://docs.videodb.io/pages/ingest/capture-sdks/realtime-context

Client events and backend RTStream event patterns for capture

Capture clients emit typed local events in real-time. Backend RTStream payload fields can vary by pipeline; use webhooks for durable delivery and WebSockets for live UI.

> [!NOTE]
>
> Desktop capture currently supports **macOS** and **Windows**.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
ws = conn.connect_websocket()
await ws.connect()

# Inspect backend events; payload fields are pipeline-specific
async for ev in ws.receive():
    print(ev)
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const ws = await conn.connectWebsocket();
await ws.connect();

// Inspect backend events; payload fields are pipeline-specific
for await (const ev of ws.receive()) {
    console.log(ev);
}
```

***

## Event Types

The following backend payloads are illustrative observations, not a fixed Capture SDK schema. Inspect and validate the payloads your integration receives.

### Transcript Events

Real-time speech-to-text from audio channels:

```json theme={null}
{
  "channel": "transcript",
  "rtstream_id": "rts-xxx",
  "rtstream_name": "mic:default",
  "data": {
    "text": "Let's schedule the meeting for Thursday",
    "is_final": true,
    "start": 1710000001234,
    "end": 1710000002345
  }
}
```

| Field | Description |
| :- | :- |
| `text` | Transcribed speech |
| `is_final` | `true` for final, `false` for interim |
| `start/end` | Timestamps (ms) |

### Visual Index Events

Scene descriptions from screen capture:

```json theme={null}
{
  "channel": "visual_index",
  "rtstream_id": "rts-xxx",
  "rtstream_name": "display:1",
  "data": {
    "text": "User is viewing a Slack conversation with 3 unread messages",
    "start": 1710000012340,
    "end": 1710000018900
  }
}
```

### Audio Index Events

Semantic understanding of audio:

```json theme={null}
{
  "channel": "audio_index",
  "rtstream_id": "rts-xxx",
  "rtstream_name": "mic:default",
  "data": {
    "text": "Discussion about scheduling a team meeting",
    "start": 1710000021500,
    "end": 1710000029200
  }
}
```

### Alert Events

Custom detection rules firing:

```json theme={null}
{
  "channel": "alert",
  "rtstream_id": "rts-xxx",
  "data": {
    "label": "sensitive_content",
    "triggered": true,
    "confidence": 0.92,
    "start": 1710000045100,
    "end": 1710000047800
  }
}
```

***

## RTStream Helper Channels

The helper APIs below are available for compatibility. For descriptor-backed V2 understanding and indexing, see [Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index).

| Channel | Source | Content |
| :- | :- | :- |
| `capture_session` | Session lifecycle | Status changes |
| `transcript` | `start_transcript()` | Speech-to-text |
| `scene_index` | `index_visuals()` | Visual analysis |
| `audio_index` | `index_audio()` | Audio analysis |
| `alert` | `create_alert()` | Alert notifications |

### Connecting to RTStream Helper Events

This example uses RTStream helpers.

```python Python theme={null}
conn = videodb.connect()
ws = conn.connect_websocket()
await ws.connect()

# Pass ws.connection_id when starting AI operations
rtstream.start_transcript(ws_connection_id=ws.connection_id)
rtstream.index_visuals(prompt="...", ws_connection_id=ws.connection_id)
rtstream.index_audio(prompt="...", ws_connection_id=ws.connection_id)
```

```javascript Node.js theme={null}
const conn = connect();
const ws = await conn.connectWebsocket();
await ws.connect();

// Pass the WebSocket connection ID when starting AI operations
await rtstream.startTranscript(ws.connectionId);
await rtstream.indexVisuals({ prompt: "...", socketId: ws.connectionId });
await rtstream.indexAudio({ prompt: "...", socketId: ws.connectionId });
```

***

## Webhooks

Durable delivery for session lifecycle events.

Backend webhook fields and lifecycle payloads are illustrative server behavior, not a fixed Capture SDK schema.

### Illustrative Webhook Envelope

```json theme={null}
{
  "version": "2",
  "event": "capture_session.active",
  "timestamp": "2026-01-20T12:34:56Z",
  "capture_session_id": "cap-xxx",
  "end_user_id": "user_abc",
  "status": "active",
  "data": {}
}
```

### Illustrative Session Lifecycle Events

| Event | Status | Key Data |
| :- | :- | :- |
| `capture_session.created` | `created` | — |
| `capture_session.starting` | `starting` | — |
| `capture_session.active` | `active` | `rtstreams[]` |
| `capture_session.stopping` | `stopping` | — |
| `capture_session.stopped` | `stopped` | — |
| `capture_session.exported` | `exported` | `exported_video_id` |
| `capture_session.failed` | `failed` | `error` object |

### Illustrative `capture_session.active` Payload

This is where you start AI pipelines:

```json theme={null}
{
  "event": "capture_session.active",
  "capture_session_id": "cap-xxx",
  "status": "active",
  "data": {
    "rtstreams": [
      { "rtstream_id": "rts-1", "name": "mic:default", "media_types": ["audio"] },
      { "rtstream_id": "rts-2", "name": "system_audio:default", "media_types": ["audio"] },
      { "rtstream_id": "rts-3", "name": "display:1", "media_types": ["video"] }
    ]
  }
}
```

> [!NOTE]
>
> Resolving Capture RTStreams from category-style webhook values is not yet a supported released-SDK contract, so this runnable lookup example is deferred.

***

## Delivery Semantics

| Method | Guarantee | Handle |
| :- | :- | :- |
| WebSocket | Best-effort | Reconnect on disconnect |
| Webhook | At-least-once | Deduplicate by event ID |

> [!WARNING]
>
> Webhooks may deliver duplicates. Respond 2xx quickly, process asynchronously, implement idempotency.

***

## Next Steps

> **[Capture Overview](/pages/ingest/capture-sdks/overview)**
>
> Architecture and quickstart

> **[Storage & Search](/pages/ingest/capture-sdks/storage-and-search)**
>
> Export and persistence patterns


# Storage & Search
Source: https://docs.videodb.io/pages/ingest/capture-sdks/storage-and-search

Optional persistence, export workflows, and semantic search for captured content

Capture sessions can optionally persist media for later search and playback. Control storage per-channel and access exported assets.

> [!NOTE]
>
> Desktop capture currently supports **macOS** and **Windows**.

## Quick Example

```python Python theme={null}
# After capture_session.exported webhook
cap = conn.get_capture_session("cap-xxx")

# Get the muxed video
video_id = cap.exported_video_id
video = coll.get_video(video_id)

# Search the captured content
understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")
transcript_index = video.index(
    name="transcript",
    source=transcript_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["text"]},
)
transcript_index.wait_until_complete()

results = video.semantic_search(
    query="budget discussion",
    index_ids=[transcript_index.index_id],
    top_k=5,
)
for shot in results.get_shots():
    print(f"{shot.start}s: {shot.text}")
    shot.play()
```

```javascript Node.js theme={null}
// After capture_session.exported webhook
const cap = await conn.getCaptureSession("cap-xxx");

// Get the muxed video
const videoId = cap.exportedVideoId;
const video = await coll.getVideo(videoId);

// Search the captured content
const understanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await understanding.waitUntilComplete();
const transcriptAnalyzer = await understanding.getAnalyzer("transcript");
const transcriptIndex = await video.index(transcriptAnalyzer, {
  name: "transcript",
  useFor: ["semantic"],
  fields: { semantic: ["text"] },
});
await transcriptIndex.waitUntilComplete();

const results = await video.semanticSearch("budget discussion", {
  indexIds: [transcriptIndex.indexId],
  topK: 5,
});
for (const shot of results.getShots()) {
    console.log(`${shot.start}s: ${shot.text}`);
    await shot.play();
}
```

***

## Storage Control

### Per-Channel Storage

Enable/disable storage for each channel:

```python Python theme={null}
# In desktop client
channels = await client.list_channels()
mic = channels.mics.default
display = channels.displays.default
system_audio = channels.system_audio.default

if mic:
    mic.store = True
if display:
    display.store = True
if system_audio:
    system_audio.store = False

selected = [channel for channel in (mic, display, system_audio) if channel]
await client.start_session(
    capture_session_id=cap_id,
    channels=selected,
)
```

```javascript Node.js theme={null}
// In desktop client
const channels = await client.listChannels();
const mic = channels.mics.default;
const display = channels.displays.default;
const systemAudio = channels.systemAudio.default;
const selectedChannels = [
    [mic, true],
    [display, true],
    [systemAudio, false],
]
    .filter(([channel]) => Boolean(channel))
    .map(([channel, store]) => ({ channelId: channel.id, type: channel.type, store }));

await client.startSession({
    sessionId: capId,
    channels: selectedChannels,
});
```

| Setting | Behavior |
| :- | :- |
| `store: true` | Media persisted, available for search and playback |
| `store: false` | Ephemeral - real-time processing only, no persistence |

> [!NOTE]
>
> Export only runs if at least one channel has `store: true`.

***

## What Gets Exported

### Muxed Video

The default "playable recording" containing:

* **Video:** Primary display (set via `primary_video_channel_id`)
* **Audio:** All recorded audio channels mixed together

**Use for:**

* Playback and sharing
* Downstream indexing and search
* Simple "trim and publish" workflows

### Raw Channel Assets

Individual assets for each stored channel:

| Channel | Asset Type |
| :- | :- |
| `display:1` | Raw video |
| `mic:default` | Raw audio |
| `system_audio:default` | Raw audio |

**Use for:**

* Separate audio stems (mic vs system audio)
* Multi-track editing
* Custom muxing strategies
* Picture-in-picture composites

***

## Accessing Exports

### Via Webhook

The `capture_session.exported` webhook includes the muxed video ID:

```json theme={null}
{
  "event": "capture_session.exported",
  "capture_session_id": "cap-xxx",
  "status": "exported",
  "data": {
    "exported_video_id": "m-xxx"
  }
}
```

### Via RTStream

Each RTStream has an `exported_asset_id` after export:

> [!NOTE]
>
> Mapping Capture channel categories to RTStream IDs is not yet a supported released-SDK contract, so this raw-asset lookup example is deferred.

### Using `cap.export()`

You can trigger or check an export programmatically with `cap.export()`:

```python Python theme={null}
cap = conn.get_capture_session("cap-xxx")

result = cap.export(
    video_channel_id=None,      # Optional — defaults to primary video channel
    ws_connection_id=None,       # Optional — for push notification on completion
)
```

| Parameter | Type | Description |
| :- | :- | :- |
| `video_channel_id` | `str` or `None` | The video channel to export. Defaults to the primary video channel when omitted. |
| `ws_connection_id` | `str` or `None` | A WebSocket connection ID. When provided, VideoDB sends a push notification over that connection when the export finishes. |

The returned dict contains:

| Field | Description |
| :- | :- |
| `session_id` | The capture session ID |
| `video_channel_id` | The video channel being exported |
| `export_status` | Current status (`exporting`, `exported`, `failed`) |
| `video_id` | Available once export completes |
| `stream_url` | Playback stream URL (available once exported) |
| `player_url` | Embeddable player URL (available once exported) |

#### Multi-Screen Export

When capturing multiple displays, each display can be exported individually by passing its `video_channel_id`. Use `cap.displays` to discover available video channels:

```python Python theme={null}
cap = conn.get_capture_session("cap-xxx")

for d in cap.displays:
    print(f"{d.channel_id}  primary={d.is_primary}")

# Export a specific (non-primary) display
result = cap.export(video_channel_id="display:2")
print(result)
# {
#   "session_id": "cap-xxx",
#   "video_channel_id": "display:2",
#   "export_status": "exporting"
# }
```

#### Checking Export Status

You can track export completion via webhook or by polling:

```python Python theme={null}
# Option 1: Webhook — your callback_url receives:
# {
#   "event": "capture_session.exported",
#   "capture_session_id": "cap-xxx",
#   "data": { "exported_video_id": "m-xxx" }
# }

# Option 2: Push notification via WebSocket
result = cap.export(ws_connection_id="ws-conn-xxx")

# Option 3: Poll export status
result = cap.export(video_channel_id="display:1")
while result["export_status"] == "exporting":
    time.sleep(5)
    result = cap.export(video_channel_id="display:1")

print(result["video_id"])       # "m-xxx"
print(result["stream_url"])     # Playback URL
print(result["player_url"])     # Embeddable player URL
```

***

## Editing with Raw Assets

Use raw assets when you need control over individual tracks.

### Display Video + Mic Audio Only

> [!NOTE]
>
> This raw-asset composition example is deferred until Capture exposes a supported category-to-RTStream mapping.

***

## Semantic Search

After export, captured content is searchable:

```python Python theme={null}
video = coll.get_video(exported_video_id)

understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")
transcript_index = video.index(
    name="transcript",
    source=transcript_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["text"]},
)
transcript_index.wait_until_complete()

results = video.semantic_search(
    query="action items from the meeting",
    index_ids=[transcript_index.index_id],
    top_k=5,
)
for shot in results.get_shots():
    print(f"{shot.start}s: {shot.text}")
    shot.play()
```

```javascript Node.js theme={null}
const video = await coll.getVideo(exportedVideoId);

const understanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await understanding.waitUntilComplete();
const transcriptAnalyzer = await understanding.getAnalyzer("transcript");
const transcriptIndex = await video.index(transcriptAnalyzer, {
  name: "transcript",
  useFor: ["semantic"],
  fields: { semantic: ["text"] },
});
await transcriptIndex.waitUntilComplete();

const results = await video.semanticSearch("action items from the meeting", {
  indexIds: [transcriptIndex.indexId],
  topK: 5,
});
for (const shot of results.getShots()) {
    console.log(`${shot.start}s: ${shot.text}`);
    await shot.play();
}
```

***

## Which Asset to Use?

| Use Case | Asset |
| :- | :- |
| Quick playback, sharing | Muxed video (`exported_video_id`) |
| Separate mic vs system audio | Raw channel assets |
| Multi-track editing | Raw channel assets |
| Custom audio mix | Raw channel assets |
| Picture-in-picture | Raw channel assets |

***

## Next Steps

> **[Privacy Controls](/pages/ingest/capture-sdks/privacy-controls)**
>
> Consent and redaction patterns

> **[Capture Overview](/pages/ingest/capture-sdks/overview)**
>
> Architecture and quickstart


# Collection Patterns
Source: https://docs.videodb.io/pages/ingest/files-and-collections/collection-patterns

Production-ready patterns for ingesting media at scale. Handle async operations, batch uploads, and error recovery.

## When to Use This

* Uploading many files in a batch
* Building production pipelines
* Handling failures gracefully
* Processing webhook callbacks

***

## Async Upload Pattern

For large files or many uploads, use callbacks instead of waiting:

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Fire-and-forget uploads
for url in video_urls:
    coll.upload(
        url=url,
        callback_url="https://your-backend.com/webhooks/upload"
    )
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect();
const coll = await conn.getCollection();

// Fire-and-forget uploads
for (const url of videoUrls) {
    await coll.uploadURL({
        url,
        callbackUrl: "https://your-backend.com/webhooks/upload"
    });
}
```

***

## Webhook Handler

Handle upload callbacks in your backend:

```python Python theme={null}
from fastapi import FastAPI, Request

app = FastAPI()

@app.post("/webhooks/upload")
async def handle_upload(request: Request):
    event = await request.json()

    if event["success"]:
        video_id = event["data"]["id"]
        # Trigger next step (indexing, processing, etc.)
        await start_indexing(video_id)
    else:
        # Handle failure
        await log_failure(event["message"])

    return {"status": "ok"}
```

### Callback Payloads

**Success:**

```json theme={null}
{
  "success": true,
  "data": {
    "id": "m-xxx",
    "collection_id": "c-xxx",
    "name": "video.mp4",
    "length": "191.14",
    "stream_url": "https://stream.videodb.io/...",
    "player_url": "https://console.videodb.io/player?..."
  }
}
```

**Failure:**

```json theme={null}
{
  "success": false,
  "message": "Download failed."
}
```

***

## Batch Upload with Tracking

Track multiple uploads and wait for all to complete:

```python Python theme={null}
import asyncio
from collections import defaultdict

# Track pending uploads
pending = defaultdict(asyncio.Event)

async def upload_batch(urls):
    for url in urls:
        upload_id = generate_id()
        pending[upload_id] = asyncio.Event()

        coll.upload(
            url=url,
            callback_url=f"https://backend.com/webhooks?id={upload_id}"
        )

    # Wait for all callbacks
    await asyncio.gather(*[e.wait() for e in pending.values()])

# In webhook handler
@app.post("/webhooks")
async def handle(request: Request, id: str):
    event = await request.json()
    if id in pending:
        pending[id].set()
    return {"status": "ok"}
```

```javascript Node.js theme={null}
// Track pending uploads
const pending = new Map();

async function uploadBatch(urls) {
    const promises = urls.map(async (url) => {
        const uploadId = generateId();

        return new Promise((resolve) => {
            pending.set(uploadId, resolve);

            coll.uploadURL({
                url,
                callbackUrl: `https://backend.com/webhooks?id=${uploadId}`
            });
        });
    });

    await Promise.all(promises);
}

// In webhook handler
app.post("/webhooks", (req, res) => {
    const { id } = req.query;
    if (pending.has(id)) {
        pending.get(id)();
        pending.delete(id);
    }
    res.json({ status: "ok" });
});
```

***

## Retry Pattern

Handle transient failures with exponential backoff:

```python Python theme={null}
import time
from functools import wraps

def retry_upload(max_attempts=3, base_delay=1):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == max_attempts - 1:
                        raise
                    delay = base_delay * (2 ** attempt)
                    time.sleep(delay)
        return wrapper
    return decorator

@retry_upload(max_attempts=3)
def upload_with_retry(coll, url):
    return coll.upload(url=url)
```

```javascript Node.js theme={null}
async function retryUpload(coll, url, maxAttempts = 3) {
    for (let attempt = 0; attempt < maxAttempts; attempt++) {
        try {
            return await coll.uploadURL({ url });
        } catch (e) {
            if (attempt === maxAttempts - 1) throw e;
            const delay = 1000 * Math.pow(2, attempt);
            await new Promise(r => setTimeout(r, delay));
        }
    }
}
```

***

## Indexing Callbacks

Chain indexing after upload completes:

This callback-only indexing chain uses a compatibility index API; run the finite Understanding-to-index flow in the worker that owns the video for current retrieval.

```python Python theme={null}
# Upload callback triggers indexing
@app.post("/webhooks/upload")
async def handle_upload(request: Request):
    event = await request.json()

    if event["success"]:
        video_id = event["data"]["id"]
        coll = conn.get_collection(event["data"]["collection_id"])
        video = coll.get_video(video_id)

        # Trigger async indexing
        video.index_spoken_words(
            callback_url="https://backend.com/webhooks/index"
        )

    return {"status": "ok"}

# Index callback
@app.post("/webhooks/index")
async def handle_index(request: Request):
    event = await request.json()

    if event["success"]:
        # Video is now searchable
        await notify_ready(event["data"]["id"])

    return {"status": "ok"}
```

```javascript Node.js theme={null}
// Upload callback triggers indexing
app.post("/webhooks/upload", async (req, res) => {
    const event = req.body;

    if (event.success) {
        const coll = await conn.getCollection(event.data.collection_id);
        const video = await coll.getVideo(event.data.id);

        // Trigger async indexing
        await video.indexSpokenWords(
            "en",
            "sentence",
            false,
            "https://backend.com/webhooks/index"
        );
    }

    res.json({ status: "ok" });
});

// Index callback
app.post("/webhooks/index", async (req, res) => {
    const event = req.body;

    if (event.success) {
        // Video is now searchable
        await notifyReady(event.data.id);
    }

    res.json({ status: "ok" });
});
```

***

## Error Handling

Common error responses and how to handle them:

| Error | Cause | Action |
| :- | :- | :- |
| `Download failed` | URL inaccessible | Verify URL, check permissions |
| `Invalid media type` | Wrong MediaType enum | Match MediaType to file |
| `Something went wrong` | Corrupted file | Re-encode source file |

***

## Next Steps

> **[Upload Video](/pages/ingest/files-and-collections/upload-video)**
>
> Upload methods reference

> **[Live Streams](/pages/ingest/live-streams/rtsp-ingest)**
>
> Ingest from RTSP sources


# Create Collection
Source: https://docs.videodb.io/pages/ingest/files-and-collections/create-collection

Collections group your media for organization and scoped search. Think of them as folders that also enable searching across all videos within.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()

# Create a collection
coll = conn.create_collection(
    name="Q4 Meetings",
    description="All Q4 2024 team meetings"
)
print(coll.id)  # c-xxx
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect();

// Create a collection
const coll = await conn.createCollection(
    "Q4 Meetings",
    "All Q4 2024 team meetings"
);
console.log(coll.id);  // c-xxx
```

***

## Collection Operations

### Create

```python Python theme={null}
coll = conn.create_collection(
    name="Security Footage",
    description="Warehouse camera feeds"
)
```

```javascript Node.js theme={null}
const coll = await conn.createCollection(
    "Security Footage",
    "Warehouse camera feeds"
);
```

### List All

```python Python theme={null}
collections = conn.get_collections()
for c in collections:
    print(f"{c.id}: {c.name}")
```

```javascript Node.js theme={null}
const collections = await conn.getCollections();
for (const c of collections) {
    console.log(`${c.id}: ${c.name}`);
}
```

### Get by ID

```python Python theme={null}
coll = conn.get_collection("c-xxx-xxx")
```

```javascript Node.js theme={null}
const coll = await conn.getCollection("c-xxx-xxx");
```

### Update

```python Python theme={null}
coll = conn.update_collection(
    "c-xxx-xxx",
    "New Name",
    "Updated description"
)
```

```javascript Node.js theme={null}
const coll = await conn.updateCollection(
    "c-xxx-xxx",
    "New Name",
    "Updated description"
);
```

### Delete

```python Python theme={null}
coll.delete()
```

```javascript Node.js theme={null}
await coll.delete();
```

***

## List Media in Collection

```python Python theme={null}
# List videos
videos = coll.get_videos()
for v in videos:
    print(f"{v.id}: {v.name}")

# List audios
audios = coll.get_audios()

# List images
images = coll.get_images()
```

```javascript Node.js theme={null}
// List videos
const videos = await coll.getVideos();
for (const v of videos) {
    console.log(`${v.id}: ${v.name}`);
}

// List audios
const audios = await coll.getAudios();

// List images
const images = await coll.getImages();
```

***

## Collection Search

Search restricts results to videos in that collection - essential for RAG applications:

```python Python theme={null}
understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")
transcript_index = video.index(
    name="transcript",
    source=transcript_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["text"]},
)
transcript_index.wait_until_complete()

results = coll.semantic_search(
    "quarterly results",
    index_ids=[transcript_index.index_id],
    top_k=5,
)
for shot in results.get_shots():
    print(f"{shot.start}s: {shot.text}")
```

```javascript Node.js theme={null}
const understanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await understanding.waitUntilComplete();
const transcriptAnalyzer = await understanding.getAnalyzer("transcript");
const transcriptIndex = await video.index(transcriptAnalyzer, {
  name: "transcript",
  useFor: ["semantic"],
  fields: { semantic: ["text"] },
});
await transcriptIndex.waitUntilComplete();

const results = await coll.semanticSearch("quarterly results", {
  indexIds: [transcriptIndex.indexId],
  topK: 5,
});
for (const shot of results.getShots()) {
    console.log(`${shot.start}s: ${shot.text}`);
}
```

***

## Public Collections

Share collections publicly for read-only access. Anyone with the collection ID can access media and indexes.

### Create Public Collection

```python Python theme={null}
public_coll = conn.create_collection(
    name="Demo Videos",
    description="Public demo collection",
    is_public=True
)
```

```javascript Node.js theme={null}
const publicColl = await conn.createCollection(
    "Demo Videos",
    "Public demo collection",
    true  // isPublic
);
```

### Toggle Visibility

```python Python theme={null}
# Make private
coll.make_private()
print(coll.is_public)  # False

# Make public
coll.make_public()
print(coll.is_public)  # True
```

```javascript Node.js theme={null}
// Make private
await coll.makePrivate();
console.log(coll.isPublic);  // false

// Make public
await coll.makePublic();
console.log(coll.isPublic);  // true
```

### Access Public Collection

Anyone can access with the collection ID:

```python Python theme={null}
# Access VideoDB's OCR Benchmark Collection
public_coll = conn.get_collection("c-c0a2c223-e377-4625-94bf-910501c2a31c")
videos = public_coll.get_videos()
```

```javascript Node.js theme={null}
// Access VideoDB's OCR Benchmark Collection
const publicColl = await conn.getCollection("c-c0a2c223-e377-4625-94bf-910501c2a31c");
const videos = await publicColl.getVideos();
```

***

## What You Can Build

> **[Keyword Search Compilation](/examples-and-tutorials/video-rag/keyword-search)**
>
> Organize videos in collections and search across all of them

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Build searchable video libraries with text and visual queries

> **[Character Clips](/examples-and-tutorials/video-rag/character-clips)**
>
> Search collections to extract clips featuring specific people

***

## Next Steps

> **[Upload Video](/pages/ingest/files-and-collections/upload-video)**
>
> Add media to your collections

> **[Collection Patterns](/pages/ingest/files-and-collections/collection-patterns)**
>
> Batching, retries, and production patterns


# Upload Video
Source: https://docs.videodb.io/pages/ingest/files-and-collections/upload-video

Ingest video, audio, and images from URLs or local files into VideoDB collections

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

video = coll.upload("https://example.com/video.mp4")
print(video.id)  # m-xxx
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect();
const coll = await conn.getCollection();

const video = await coll.uploadURL({ url: "https://example.com/video.mp4" });
console.log(video.id);  // m-xxx
```

***

## Upload Methods

### From URL

Upload directly from any accessible URL (S3, YouTube, public links):

```python Python theme={null}
# Video
video = coll.upload(url="https://youtu.be/a9__D53WsUs")

# Audio
from videodb import MediaType
audio = coll.upload(
    url="https://example.com/podcast.mp3",
    media_type=MediaType.audio
)

# Image
image = coll.upload(
    url="https://example.com/frame.jpg",
    media_type=MediaType.image
)
```

```javascript Node.js theme={null}
import { MediaType } from 'videodb';

// Video
const video = await coll.uploadURL({ url: "https://youtu.be/a9__D53WsUs" });

// Audio
const audio = await coll.uploadURL({
    url: "https://example.com/podcast.mp3",
    mediaType: MediaType.audio
});

// Image
const image = await coll.uploadURL({
    url: "https://example.com/frame.jpg",
    mediaType: MediaType.image
});
```

### From Local File

Upload files from your local filesystem:

```python Python theme={null}
video = coll.upload(file_path="./meeting-recording.mp4")
```

```javascript Node.js theme={null}
const video = await coll.uploadFile({ filePath: "./meeting-recording.mp4" });
```

***

## Media Types

| Type | ID Prefix | Use Case |
| :- | :- | :- |
| Video | `m-xxx` | Primary content, full playback |
| Audio | `a-xxx` | Podcasts, voice recordings |
| Image | `img-xxx` | Thumbnails, frames |

***

## Upload Response

After upload, you receive a media object with:

| Property | Description |
| :- | :- |
| `id` | Unique identifier (m-xxx, a-xxx, img-xxx) |
| `collection_id` | Parent collection ID |
| `name` | File name |
| `length` | Duration in seconds (video/audio) |
| `stream_url` | HLS stream URL (video) |
| `player_url` | Web player URL (video) |

***

## Async Uploads with Callbacks

For production workflows, use callbacks to handle upload completion:

```python Python theme={null}
video = coll.upload(
    url="https://example.com/large-video.mp4",
    callback_url="https://your-backend.com/webhooks/upload"
)
```

```javascript Node.js theme={null}
const video = await coll.uploadURL({
    url: "https://example.com/large-video.mp4",
    callbackUrl: "https://your-backend.com/webhooks/upload"
});
```

Your webhook receives:

```json theme={null}
{
  "success": true,
  "data": {
    "id": "m-xxx",
    "collection_id": "c-xxx",
    "name": "large-video.mp4",
    "stream_url": "https://stream.videodb.io/...",
    "player_url": "https://console.videodb.io/player?url=..."
  }
}
```

***

## What You Can Build

> **[Faceless Video Creator](/examples-and-tutorials/content-factory/faceless-video-creator)**
>
> Upload assets and compose complete AI-generated videos

> **[TikTok Lyric Videos](/examples-and-tutorials/content-factory/tiktok-lyric-video)**
>
> Upload music and create viral vertical clips with synced lyrics

> **[Intro & Outro Automation](/examples-and-tutorials/programmatic-editing/intro-outro)**
>
> Upload brand assets and automatically add to all videos

***

## Next Steps

> **[Create Collection](/pages/ingest/files-and-collections/create-collection)**
>
> Organize uploads into collections

> **[Collection Patterns](/pages/ingest/files-and-collections/collection-patterns)**
>
> Batching, retries, and production patterns


# Real-time APIs
Source: https://docs.videodb.io/pages/ingest/live-streams/realtime-apis

RTStreams offer real-time indexing, transcription, search, and event-based alerts. Convert live video into structured, searchable data.

## Quick Example

The preferred RTStream interface follows the same Understand → Index → Retrieve flow as videos, but runs continuously on new stream windows.

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
rtstream = coll.get_rtstream("rts-xxx")

# 1. Continuously understand the stream with a VLM
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {
                "prompt": "Describe activity and detect unusual behavior"
            },
        }
    ],
    store=True,
)

# 2. Continuously index the VLM output
index = rtstream.index(
    name="activity_monitor",
    source=understanding.outputs["scene"],
    use_for=["semantic"],
)

# 3. Create a reusable event and attach an alert to the index
event_id = conn.create_event(
    event_prompt="Detect when someone enters restricted area",
    label="intrusion_detected",
)

alert_id = index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/webhooks/alerts",
)
```

> [!NOTE]
>
> Initial support for the new interface is VLM-only with time-based segmentation. See [Understand & Index RTStreams](/pages/ingest/live-streams/rtstream-understand-index) for the complete interface.

***

## Visual Indexing Compatibility

The `index_visuals()` helper remains available as an explicit compatibility API. It creates a continuous VLM scene index directly from stream frames.

```python Python theme={null}
scene_index = rtstream.index_visuals(
    prompt="Describe the scene and highlight congestion",
    batch_config={"type": "time", "value": 5, "frame_count": 2},
    name="traffic_monitor",
    ws_connection_id=ws.connection_id  # optional, for real-time events
)
```

```javascript Node.js theme={null}
const sceneIndex = await rtstream.indexVisuals({
    prompt: "Describe the scene and highlight congestion",
    batchConfig: { type: "time", value: 5, frameCount: 2 },
    name: "traffic_monitor",
    socketId: ws.connectionId  // optional
});
```

### `batch_config` options

| Field | Type | Description |
| :- | :- | :- |
| `type` | str | Only `"time"` is supported |
| `value` | int | Window size in seconds |
| `frame_count` | int | Frames to extract per window |

**Examples:**

```python theme={null}
# Every 5 seconds, extract 2 frames
{"type": "time", "value": 5, "frame_count": 2}

# Every 10 seconds, extract 5 frames
{"type": "time", "value": 10, "frame_count": 5}
```

### Managing scene indexes

```python Python theme={null}
# List all indexes
indexes = rtstream.list_scene_indexes()

# Get specific index
scene_index = rtstream.get_scene_index(index_id)

# Poll scenes
scenes = scene_index.get_scenes(
    start=0, end=None, page=1, page_size=100
)
```

```javascript Node.js theme={null}
// List all indexes
const indexes = await rtstream.listSceneIndexes();

// Get specific index
const sceneIndex = await rtstream.getSceneIndex(indexId);

// Poll scenes
const scenes = await sceneIndex.getScenes();
```

***

## Audio Indexing Compatibility

The `index_audio()` helper remains available for compatibility:

```python Python theme={null}
audio_index = rtstream.index_audio(
    prompt="Identify key speakers and main topics",
    batch_config={"type": "word", "value": 50}
)
```

```javascript Node.js theme={null}
const audioIndex = await rtstream.indexAudio({
    prompt: "Identify key speakers and main topics",
    batchConfig: { type: "word", value: 50 }
});
```

### Audio batch\_config

| Type | Value | Description |
| :- | :- | :- |
| `"word"` | count | Segment every N words |
| `"sentence"` | count | Segment every N sentences |
| `"time"` | seconds | Segment every N seconds |

***

## Transcription

Real-time speech-to-text:

```python Python theme={null}
# Start transcription
rtstream.start_transcript(ws_connection_id=ws.connection_id)

# Stop transcription
rtstream.stop_transcript()

# Poll transcripts
transcript = rtstream.get_transcript(
    start=0, page=1, page_size=100
)
```

```javascript Node.js theme={null}
// Start transcription
await rtstream.startTranscript(ws.connectionId);

// Stop transcription
await rtstream.stopTranscript("graceful");

// Poll transcripts
const transcript = await rtstream.getTranscript({
    start: 0, page: 1, pageSize: 100
});
```

***

## Search

Query indexed content with natural language:

```python Python theme={null}
results = rtstream.search(
    query="white car moving fast",
    score_threshold=0.5
)

for shot in results.shots:
    print(f"Match at {shot.start}: {shot.text}")
    shot.play()  # Opens in browser
```

```javascript Node.js theme={null}
const results = await rtstream.search({
    query: "white car moving fast",
    scoreThreshold: 0.5
});

for (const shot of results.shots) {
    console.log(`Match at ${shot.start}: ${shot.text}`);
    await shot.play();
}
```

### Search Results

Each shot contains:

| Attribute | Description |
| :- | :- |
| `start` | Start timestamp |
| `end` | End timestamp |
| `text` | Content description |
| `search_score` | Relevance score (0-1) |
| `stream_url` | Playback URL |

***

## Stream Generation

Use `generate_stream()` to create a playable stream for a time range. You can optionally pass `player_config` to attach metadata to a shareable player URL.

```python Python theme={null}
# Generate stream with player metadata
player_url = rtstream.generate_stream(
    start=1711000000,
    end=1711003600,
    player_config={
        "title": "Live Feed Recording",
        "description": "Camera 1 - Front entrance",
        "slug": "cam1-front"
    }
)

# generate_stream() returns the player_url
# Both URLs are also stored on the instance
print(rtstream.player_url)   # Shareable player URL (same as return value)
print(rtstream.stream_url)   # Raw HLS stream URL (.m3u8)
```

```javascript Node.js theme={null}
// Generate stream with player metadata
const playerUrl = await rtstream.generateStream(
    1711000000,
    1711003600,
    {
        title: "Live Feed Recording",
        description: "Camera 1 - Front entrance",
        slug: "cam1-front"
    }
);

// generateStream() returns the playerUrl
// Both URLs are also stored on the instance
console.log(rtstream.playerUrl);   // Shareable player URL (same as return value)
console.log(rtstream.streamUrl);   // Raw HLS stream URL (.m3u8)
```

### player\_url vs stream\_url

| URL | Format | Use case |
| :- | :- | :- |
| `player_url` | `https://console.videodb.io/player?v=...` | Shareable link with built-in player UI |
| `stream_url` | `https://stream.videodb.io/.../.m3u8` | Raw HLS stream for custom players (HLS.js, Video.js) |

`generate_stream()` **returns `player_url`** and stores both URLs on the instance. Use `player_url` for sharing and embedding, `stream_url` for custom player integrations.

### player\_config

| Key | Type | Description |
| :- | :- | :- |
| `title` | str | Title displayed in the player |
| `description` | str | Description for the player page |
| `slug` | str | URL-friendly slug for the player link |

`player_config` is optional. When provided, the player URL includes the metadata for sharing.

### Embed Code

After generating a stream, you can produce embeddable HTML:

```python Python theme={null}
# Generate embed code (requires generate_stream() first)
rtstream.generate_stream(start=start_ts, end=end_ts)
embed_html = rtstream.get_embed_code()
```

```javascript Node.js theme={null}
// Generate embed code (requires generateStream() first)
await rtstream.generateStream(startTs, endTs);
const embedHtml = rtstream.getEmbedCode();
```

> [!NOTE]
>
> RTStream does **not** support `auto_generate` because `generate_stream()` requires explicit `start` and `end` parameters.

***

## Events and Alerts

Events are reusable detection rules. Alerts wire events to indexes for notifications.

### Create Event

```python Python theme={null}
event_id = conn.create_event(
    event_prompt="Detect pedestrians crossing the zebra",
    label="pedestrian_detected"
)
```

```javascript Node.js theme={null}
const eventId = await conn.createEvent(
    "Detect pedestrians crossing the zebra",
    "pedestrian_detected"
);
```

### Create Alert

```python Python theme={null}
alert_id = scene_index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/webhooks/alerts",
    ws_connection_id=ws.connection_id  # optional
)
```

```javascript Node.js theme={null}
const alertId = await sceneIndex.createAlert(
    eventId,
    "https://your-backend.com/webhooks/alerts",
    ws.connectionId  // optional
);
```

### Alert Delivery

| Method | Latency | Use Case |
| :- | :- | :- |
| Webhook | Under 1s | Server-to-server POST |
| WebSocket | Real-time | Frontend dashboards |

### Manage Alerts

```python Python theme={null}
# List alerts
alerts = scene_index.list_alerts()

# Enable/disable
scene_index.enable_alert(alert_id)
scene_index.disable_alert(alert_id)
```

```javascript Node.js theme={null}
// List alerts
const alerts = await sceneIndex.listAlerts();

// Enable/disable
await sceneIndex.enableAlert(alertId);
await sceneIndex.disableAlert(alertId);
```

***

## WebSocket Events

Receive real-time events by passing `ws_connection_id`:

```python Python theme={null}
ws = conn.connect_websocket()
await ws.connect()

# Pass ws_connection_id to methods
rtstream.start_transcript(ws_connection_id=ws.connection_id)

# WebSocket messages are arbitrary decoded payloads; inspect before accessing fields.
async for ev in ws.receive():
    print(ev)
```

```javascript Node.js theme={null}
const ws = await conn.connectWebsocket();
await ws.connect();

// Pass the WebSocket connection ID to methods
await rtstream.startTranscript(ws.connectionId);

// WebSocket messages are arbitrary decoded payloads; inspect before accessing fields.
for await (const ev of ws.receive()) {
    console.log(ev);
}
```

### Event Channels

The current V2 alert channel is produced by the V2 index in the quick example:

| Channel | Source | Content |
| :- | :- | :- |
| `alert` | `index.create_alert()` | Alert notifications |

The following channel names are available through RTStream helper APIs:

| Channel | Source | Content |
| :- | :- | :- |
| `transcript` | `start_transcript()` | Real-time speech-to-text |
| `scene_index` | `index_visuals()` | Visual analysis |
| `audio_index` | `index_audio()` | Audio analysis |

***

## Next Steps

> **[Stream Lifecycle](/pages/ingest/live-streams/stream-lifecycle)**
>
> Start, stop, and reconnect patterns

> **[RTSP Ingest](/pages/ingest/live-streams/rtsp-ingest)**
>
> Connect RTSP sources


# RTSP Ingest
Source: https://docs.videodb.io/pages/ingest/live-streams/rtsp-ingest

Connect any RTSP/RTMP video source to VideoDB. Cameras, encoders, and live feeds become instantly searchable and actionable.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Connect a live stream
rtstream = coll.connect_rtstream(
    name="Lobby Camera",
    url="rtsp://user:pass@192.168.1.100:554/stream"
)
print(rtstream.id)  # rts-xxx
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect();
const coll = await conn.getCollection();

// Connect a live stream
const rtstream = await coll.connectRTStream(
    "rtsp://user:pass@192.168.1.100:554/stream",
    "Lobby Camera"
);
console.log(rtstream.id);  // rts-xxx
```

***

## Connection Methods

### RTSP URL Format

```
rtsp://[username:password@]host[:port]/path
```

**Examples:**

```
rtsp://admin:pass123@192.168.1.100:554/live
rtsp://camera.example.com:554/stream1
rtsp://user:pass@10.0.0.50/cam/realmonitor
```

### Connect from Camera

```python Python theme={null}
# IP Camera
rtstream = coll.connect_rtstream(
    name="Warehouse Camera 1",
    url="rtsp://admin:password@192.168.1.50:554/live"
)

# Encoder/NVR
rtstream = coll.connect_rtstream(
    name="NVR Channel 3",
    url="rtsp://admin:admin@nvr.local:554/ch3"
)
```

```javascript Node.js theme={null}
// IP Camera
const rtstream = await coll.connectRTStream(
    "rtsp://admin:password@192.168.1.50:554/live",
    "Warehouse Camera 1"
);

// Encoder/NVR
const rtstreamNVR = await coll.connectRTStream(
    "rtsp://admin:admin@nvr.local:554/ch3",
    "NVR Channel 3"
);
```

***

## RTStream Object

After connecting, you receive an RTStream object:

| Attribute | Type | Description |
| :- | :- | :- |
| `id` | str | Unique identifier (rts-xxx) |
| `name` | str | Label you supplied |
| `collection_id` | str | Parent collection |
| `status` | str | `connected`, `stopped`, etc. |
| `sample_rate` | float | Frame rate (default: 1 fps) |
| `audio` | bool | Audio ingestion enabled |

***

## Retrieve Existing Streams

### Get by ID

```python Python theme={null}
rtstream = coll.get_rtstream("rts-xxx")
```

```javascript Node.js theme={null}
const rtstream = await coll.getRTStream("rts-xxx");
```

### List All Streams

```python Python theme={null}
rtstreams = coll.list_rtstreams(
    limit=10,
    offset=0,
    status="connected",
    name="Lobby",
    ordering="-created_at"
)
```

```javascript Node.js theme={null}
const rtstreams = await coll.listRTStreams({
    limit: 10,
    offset: 0,
    status: "connected",
    name: "Lobby",
    ordering: "-created_at"
});
```

| Parameter | Description |
| :- | :- |
| `limit` | Number of results |
| `offset` | Skip N results |
| `status` | Filter by status |
| `name` | Filter by name |
| `ordering` | Sort field (prefix `-` for descending) |

***

## Playback URLs

Generate playback URLs for any time range using Unix timestamps:

```python Python theme={null}
import time

# Get playback URL for last 60 seconds
now = int(time.time())
start = now - 60
stream_url = rtstream.generate_stream(start=start, end=now)
```

```javascript Node.js theme={null}
// Get playback URL for last 60 seconds
const now = Math.floor(Date.now() / 1000);
const start = now - 60;
const playerUrl = await rtstream.generateStream(start, now);
const streamUrl = rtstream.streamUrl;
```

> [!NOTE]
>
> The `start` and `end` parameters expect Unix timestamps (seconds since epoch), not relative time offsets.

***

## Supported Sources

| Source | Format | Notes |
| :- | :- | :- |
| IP Cameras | RTSP | Most common, H.264/H.265 |
| NVR/DVR | RTSP | Per-channel streams |
| Encoders | RTSP/RTMP | OBS, FFmpeg, hardware encoders |
| Streaming Servers | RTSP | Wowza, nginx-rtmp |

***

## Connection Notes

* **Secure Storage** - All video feeds are securely stored and accessible anytime
* **Default Sample Rate** - Streams are ingested at 1 fps by default
* **Network** - Ensure your RTSP source is accessible from VideoDB's servers

***

## What You Can Build

> **[Baby Crib Monitoring](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> Real-time infant monitoring with AI-powered alerts

> **[Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Detect unauthorized access to restricted areas

> **[Traffic Violations](/examples-and-tutorials/live-intelligence/traffic-violations)**
>
> Monitor traffic cameras for red light and stop sign violations

***

## Next Steps

> **[Real-time APIs](/pages/ingest/live-streams/realtime-apis)**
>
> Index, transcribe, and set up alerts

> **[Stream Lifecycle](/pages/ingest/live-streams/stream-lifecycle)**
>
> Start, stop, and reconnect patterns


# Understand & Index RTStreams
Source: https://docs.videodb.io/pages/ingest/live-streams/rtstream-understand-index

Use the new Understand → Index interface with RTStreams for continuous VLM analysis and search.

RTStreams use the same retrieval model as videos:

```text theme={null}
Understand → Index → Retrieve
```

The difference is lifecycle. A video understanding job finishes when the file is processed. An RTStream understanding job is continuous: it keeps processing new windows while the stream and job are running.

> [!NOTE]
>
> Initial RTStream support for the new interface is limited to **VLM visual understanding** with **time-based segmentation**. Other analyzers and segmentation modes will be added later.

***

## What is supported

| Capability | Initial RTStream support |
| - | - |
| Analyzer | `vlm` only |
| Segmentation | `{"type": "time"}` only |
| Sampling | Frames per time window |
| Index source | VLM understanding output |
| Job mode | Continuous |
| Retrieval | RTStream search over indexed windows |
| Alerts | Alerts attached to RTStream indexes |

Unsupported analyzer types such as `spoken_words`, `object_detection`, `ocr`, `faces`, `brands`, `activity_recognition`, and `location_detection` should return a validation error until support is rolled out.

***

## Quick example

```python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
rtstream = coll.get_rtstream("rts-xxx")

# 1. Continuously understand the live stream with a VLM
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "10s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 5},
            "config": {
                "prompt": "Describe the scene, people, activity, and unusual events.",
                "model": "basic",
            },
        }
    ],
    store=True,
)

# 2. Continuously index the VLM output for retrieval
index = rtstream.index(
    name="scene_search",
    source=understanding.outputs["scene"],
    use_for=["semantic"],
)

# 3. Search indexed stream windows
results = rtstream.search(
    query="person entering with a package",
    index_id=index.id,
)

for shot in results.shots:
    print(shot.start, shot.end, shot.text)
    shot.play()
```

***

## 1. Understand the stream

Use `rtstream.understand(...)` to start a continuous understanding job. It returns an `RTStreamUnderstanding` object containing the job ID, status, outputs, and lifecycle methods.

```python theme={null}
understanding = rtstream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[
        {
            "type": "vlm",
            "name": "scene",
            "sampling": {"frame_count": 2},
            "config": {
                "prompt": "Describe what is happening in this time window."
            },
        }
    ],
    store=True,
    ws_connection_id=ws.connection_id,  # optional
)
```

### Parameters

| Field | Description |
| - | - |
| `segmentation.type` | Must be `"time"` in the initial rollout. |
| `segmentation.window` | Duration of each stream window, for example `"5s"` or `"10s"`. |
| `analyzers` | Must contain one `vlm` analyzer in the initial rollout. |
| `analyzers[].name` | Output name. Use `"scene"` unless you need a custom key. |
| `analyzers[].sampling.frame_count` | Number of frames sampled per time window. |
| `analyzers[].config.prompt` | Prompt used to describe or structure each window. |
| `store` | Use `True` when you plan to index the output. |
| `ws_connection_id` | Optional WebSocket connection for real-time updates. |

***

## 2. Get the continuous output

The object returned by `rtstream.understand(...)` already contains the output descriptors:

```python theme={null}
scene_output = understanding.outputs["scene"]
```

To reopen the job later, fetch it by its string ID:

```python theme={null}
understanding = rtstream.get_understanding("und-xxx")
```

For videos, an understanding output can be a finite artifact. For RTStreams, the output is a continuous source descriptor that points to records produced over time.

```json theme={null}
{
  "type": "understanding",
  "asset_type": "rtstream",
  "rtstream_id": "rts-xxx",
  "understanding_id": "und-xxx",
  "output": "scene",
  "extract_type": "vlm",
  "mode": "continuous"
}
```

You can inspect produced records by time range:

```python theme={null}
records = understanding.get_records(
    start=1711000000,
    end=1711000600,
    page=1,
    page_size=100,
)
```

***

## 3. Index the output

Use `rtstream.index(...)` to materialize the continuous VLM output for search. It returns an `RTStreamIndex` object with its ID and lifecycle methods.

```python theme={null}
index = rtstream.index(
    name="front_door_activity",
    source=understanding.outputs["scene"],
    use_for=["semantic"],
)
```

If `use_for` is omitted, VideoDB chooses the default capabilities for the VLM scene output. In the initial RTStream rollout, the primary capability is semantic search over scene descriptions.

***

## 4. Manage the continuous jobs

RTStream understanding and index jobs have continuous lifecycle controls.

```python theme={null}
# Reopen jobs later when you only have their string IDs
understanding = rtstream.get_understanding("und-xxx")
index = rtstream.get_index("idx-xxx")

# Pause or resume understanding
understanding.stop()
understanding.start()

# Pause or resume indexing
index.stop()
index.start()
```

| State | Meaning |
| - | - |
| `running` | The job processes new stream windows. |
| `stopped` | The job stops processing new windows. Existing records remain available. |
| `failed` | The job stopped because of an error. |

Stopping the RTStream itself disconnects the media source. Stopping understanding or indexing only pauses that processing layer.

***

## 5. Retrieve and play results

Search works like existing RTStream search: results are timestamped stream shots.

```python theme={null}
results = rtstream.search(
    query="delivery person at the entrance",
    index_id=index.id,
    score_threshold=0.5,
)

for shot in results.shots:
    print(shot.start, shot.end, shot.text)
    stream_url = shot.generate_stream()
```

Each shot includes Unix timestamps. Use `shot.generate_stream()` or `shot.play()` to create a playable stream for that time range.

***

## 6. Add alerts

Alerts remain attached to RTStream indexes.

```python theme={null}
event_id = conn.create_event(
    event_prompt="Detect when someone enters the restricted area",
    label="restricted_area_entry",
)

alert_id = index.create_alert(
    event_id=event_id,
    callback_url="https://your-backend.com/webhooks/alerts",
    ws_connection_id=ws.connection_id,  # optional
)
```

Manage alerts from the index:

```python theme={null}
alerts = index.list_alerts()
index.disable_alert(alert_id)
index.enable_alert(alert_id)
```

***

## How this differs from videos

| Video | RTStream |
| - | - |
| Finite media asset | Continuous live source |
| `video.understand(...)` eventually completes | `rtstream.understand(...)` keeps running |
| `video.index(...)` indexes a completed or stored artifact | `rtstream.index(...)` continuously indexes new records |
| Timestamps are relative seconds in the video | Timestamps are Unix timestamps |
| Alerts are not part of the initial video index lifecycle | Alerts are supported on RTStream indexes |

***

## Target API routes

The SDK methods map to these RTStream routes:

| SDK method | Route |
| - | - |
| `rtstream.understand(...)` | `POST /rtstream/{stream_id}/understand` |
| `rtstream.list_understanding()` | `GET /rtstream/{stream_id}/understand` |
| `rtstream.get_understanding(id)` | `GET /rtstream/{stream_id}/understand/{understanding_id}` |
| `understanding.start()` / `understanding.stop()` | `PATCH /rtstream/{stream_id}/understand/{understanding_id}/status` |
| `understanding.get_records(...)` | `GET /rtstream/{stream_id}/understand/{understanding_id}/records` |
| `rtstream.index(...)` | `POST /rtstream/{stream_id}/indexes` |
| `rtstream.list_indexes()` | `GET /rtstream/{stream_id}/indexes` |
| `rtstream.get_index(id)` | `GET /rtstream/{stream_id}/indexes/{index_id}` |
| `index.start()` / `index.stop()` | `PATCH /rtstream/{stream_id}/indexes/{index_id}/status` |
| `index.get_records(...)` | `GET /rtstream/{stream_id}/indexes/{index_id}/records` |
| `index.create_alert(...)` | `POST /rtstream/{stream_id}/indexes/{index_id}/alert` |
| `index.list_alerts()` | `GET /rtstream/{stream_id}/indexes/{index_id}/alert` |
| `index.enable_alert(id)` / `index.disable_alert(id)` | `PATCH /rtstream/{stream_id}/indexes/{index_id}/alert/{alert_id}/status` |

***

## Next steps

> **[Real-time APIs](/pages/ingest/live-streams/realtime-apis)**
>
> Existing RTStream transcription, search, alerts, and stream playback APIs.

> **[Events & Real-time](/pages/core-concepts/events-and-realtime)**
>
> Build event rules and deliver alerts over webhooks or WebSocket.


# Stream Lifecycle
Source: https://docs.videodb.io/pages/ingest/live-streams/stream-lifecycle

Manage RTStream lifecycle - start, stop, resume, and handle live stream sources including meeting recordings.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Get existing stream
rtstream = coll.get_rtstream("rts-xxx")

# Control lifecycle
rtstream.stop()   # Pause ingestion
rtstream.start()  # Resume ingestion
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = await connect();
const coll = await conn.getCollection();

// Get existing stream
const rtstream = await coll.getRTStream("rts-xxx");

// Control lifecycle
await rtstream.stop();   // Pause ingestion
await rtstream.start();  // Resume ingestion
```

***

## Lifecycle Control

### Start/Stop

```python Python theme={null}
# Pause ingestion (stream remains configured)
rtstream.stop()

# Resume ingestion
rtstream.start()
```

```javascript Node.js theme={null}
// Pause ingestion
await rtstream.stop();

// Resume ingestion
await rtstream.start();
```

### Status Values

| Status | Description |
| :- | :- |
| `connected` | Actively ingesting |
| `stopped` | Paused, can resume |
| `error` | Connection issue |

***

## Export a Stopped Stream

After stopping a stream, you can export it as a video or audio asset in your collection using `export()`.

```python Python theme={null}
# Export a stopped stream as a video/audio asset
result = rtstream.export(name="my_recording")

# RTStreamExportResult attributes
print(result.video_id)     # "m-xxx"
print(result.stream_url)   # HLS stream URL
print(result.player_url)   # Player URL (None for audio-only)
print(result.duration)     # Duration in seconds

# Generate embed code from export
embed_html = result.get_embed_code()
```

```javascript Node.js theme={null}
// Export a stopped stream as a video/audio asset
const result = await rtstream.export("my_recording");

// RTStreamExportResult attributes
console.log(result.videoId);    // "m-xxx"
console.log(result.streamUrl);  // HLS stream URL
console.log(result.playerUrl);  // Player URL (null for audio-only)
console.log(result.duration);   // Duration in seconds

// Generate embed code from export
const embedHtml = result.getEmbedCode();
```

### Export Parameters

| Parameter | Type | Description |
| :- | :- | :- |
| `name` | str (optional) | Name for the exported asset. Defaults to `"{stream_name} - Recording"` |

### RTStreamExportResult

| Attribute | Description |
| :- | :- |
| `video_id` | The ID of the exported video/audio asset |
| `stream_url` | HLS stream URL for playback |
| `player_url` | Shareable player URL (`None` for audio-only channels) |
| `name` | Name of the exported asset |
| `duration` | Duration of the recording in seconds |

***

## Index Lifecycle

Indexes can also be started/stopped independently:

```python Python theme={null}
scene_index = rtstream.get_scene_index(index_id)

# Pause indexing (stream continues)
scene_index.stop()

# Resume indexing
scene_index.start()
```

```javascript Node.js theme={null}
const sceneIndex = await rtstream.getSceneIndex(indexId);

// Pause indexing
await sceneIndex.stop();

// Resume indexing
await sceneIndex.start();
```

***

## Meeting Recording

Record from Zoom, Google Meet, or Microsoft Teams. A bot joins your meeting, records, and uploads directly to VideoDB.

![Roadmap to AI teammates showing the meeting recording workflow](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/meeting-recorder/roadmap-to-ai-teammates.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=26e0c894e9b01b7c884355d4387dc74d)

### Start Recording

```python Python theme={null}
meeting = conn.record_meeting(
    meeting_url="https://meet.google.com/abc-defg-hij",
    bot_name="Meeting Recorder",
    bot_image_url="https://your-domain.com/bot-avatar.jpg",
    meeting_title="Weekly Standup",
    callback_url="https://your-backend.com/webhooks/meeting",
    callback_data={"internal_id": "123"}
)
print(f"Recording started: {meeting.id}")
```

```javascript Node.js theme={null}
const meeting = await conn.recordMeeting({
    meetingUrl: "https://meet.google.com/abc-defg-hij",
    botName: "Meeting Recorder",
    botImageUrl: "https://your-domain.com/bot-avatar.jpg",
    meetingTitle: "Weekly Standup",
    callbackUrl: "https://your-backend.com/webhooks/meeting",
    callbackData: { internalId: "123" }
});
console.log(`Recording started: ${meeting.id}`);
```

### Recording to Collection

```python Python theme={null}
coll = conn.get_collection("your-collection-id")

meeting = coll.record_meeting(
    meeting_url="https://zoom.us/j/123456789",
    bot_name="Team Recorder",
    meeting_title="Sprint Planning",
    callback_url="https://your-backend.com/webhooks"
)
```

```javascript Node.js theme={null}
const coll = await conn.getCollection("your-collection-id");

const meeting = await coll.recordMeeting({
    meetingUrl: "https://zoom.us/j/123456789",
    botName: "Team Recorder",
    meetingTitle: "Sprint Planning",
    callbackUrl: "https://your-backend.com/webhooks"
});
```

### Track Recording Status

```python Python theme={null}
# Poll status
meeting.refresh()
print(f"Status: {meeting.status}")

# Wait for completion
if meeting.wait_for_status("done", timeout=3600, interval=60):
    print("Recording complete!")
    video = coll.get_video(meeting.video_id)
```

```javascript Node.js theme={null}
// Poll status
await meeting.refresh();
console.log(`Status: ${meeting.status}`);

// Wait for completion
const success = await meeting.waitForStatus("done", 3600, 60);
if (success) {
    console.log("Recording complete!");
    const video = await coll.getVideo(meeting.videoId);
}
```

### Recording Status Values

| Status | Description |
| :- | :- |
| `initializing` | Bot is being set up |
| `processing` | Actively recording |
| `done` | Recording complete |
| `failed` | Recording failed |

### Callback Payload

**Success:**

```json theme={null}
{
  "success": true,
  "message": "Meeting recording completed.",
  "data": {
    "video_id": "m-xxx",
    "speaker_timeline": [
      {"speaker_name": "Alice", "start_time_seconds": 9.94}
    ],
    "stream_url": "...",
    "player_url": "..."
  }
}
```

**Failure:**

```json theme={null}
{
  "success": false,
  "message": "Failed to record meeting."
}
```

### Access Recording

```python Python theme={null}
meeting = coll.get_meeting("meeting-id")

# Get the recorded video
video = coll.get_video(meeting.video_id)

understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
understanding.wait_until_complete()
transcript_analyzer = understanding.get_analyzer("transcript")
transcript_index = video.index(
    name="meeting_transcript",
    source=transcript_analyzer,
    use_for=["semantic"],
    fields={"semantic": ["text"]},
)
transcript_index.wait_until_complete()
```

```javascript Node.js theme={null}
const meeting = await coll.getMeeting("meeting-id");

// Get the recorded video
const video = await coll.getVideo(meeting.videoId);

const understanding = await video.understand([
  { type: "spoken_words", name: "transcript" },
]);
await understanding.waitUntilComplete();
const transcriptAnalyzer = await understanding.getAnalyzer("transcript");
const transcriptIndex = await video.index(transcriptAnalyzer, {
  name: "meeting_transcript",
  useFor: ["semantic"],
  fields: { semantic: ["text"] },
});
await transcriptIndex.waitUntilComplete();
```

***

## Supported Platforms

| Platform | URL Format |
| :- | :- |
| Google Meet | `https://meet.google.com/xxx-xxxx-xxx` |
| Zoom | `https://zoom.us/j/123456789` |
| Microsoft Teams | Teams meeting link |

***

## Meeting Features

* **Brand-able Bot** - Custom name and avatar
* **Speaker Timeline** - Per-speaker timestamps (Google Meet)
* **Webhook Callbacks** - Get notified on completion
* **Collection Storage** - Video lands directly in your collection

![Upcoming features for meeting recorder](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/meeting-recorder/upcoming-features.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=ab2e530833b5d92adad9958143f95e73)

***

## Next Steps

> **[RTSP Ingest](/pages/ingest/live-streams/rtsp-ingest)**
>
> Connect camera streams

> **[Real-time APIs](/pages/ingest/live-streams/realtime-apis)**
>
> Index and search live streams


# Output Formats
Source: https://docs.videodb.io/pages/ingest/transcoding/output-formats

Configure transcoding output - formats, resolutions, quality presets, and aspect ratio control

Configure transcoding output for your use case - streaming, archival, or web delivery.

## Quick Example

```python Python theme={null}
from videodb import TranscodeMode, VideoConfig, AudioConfig, ResizeMode

job_id = conn.transcode(
    source="https://example.com/source.mov",
    callback_url="https://your-backend.com/webhooks",
    mode=TranscodeMode.lightning,
    video_config=VideoConfig(
        resolution=720,
        quality=23,
        framerate=30,
        resize_mode=ResizeMode.fit
    ),
    audio_config=AudioConfig(mute=False)
)
```

```javascript Node.js theme={null}
import { TranscodeMode, ResizeMode } from 'videodb';

const jobId = await conn.transcode(
    "https://example.com/source.mov",
    "https://your-backend.com/webhooks",
    TranscodeMode.lightning,
    {
        resolution: 720,
        quality: 23,
        framerate: 30,
        resizeMode: ResizeMode.fit
    },
    { mute: false }
);
```

***

## Output Formats

| Format | Use Case |
| :- | :- |
| MP4 | Download, archival, playback |
| HLS | Adaptive streaming, web players |

***

## Resolution Options

| Resolution | Dimensions | Use Case |
| :- | :- | :- |
| 360p (SD) | 640 × 360 | Mobile, low bandwidth |
| 720p (HD) | 1280 × 720 | General web delivery |
| 1080p (FHD) | 1920 × 1080 | High quality streaming |

***

## VideoConfig

| Field | Type | Default | Notes |
| :- | :- | :- | :- |
| `resolution` | int | 720 | Height in pixels (auto-calculates width) |
| `quality` | int | 23 | CRF (lower = better quality, larger file) |
| `framerate` | int | Source | Target framerate (max 100 fps) |
| `aspect_ratio` | str | Source | e.g., "16:9", "1:1", None for original |
| `resize_mode` | ResizeMode | crop | How to handle aspect ratio changes |

### Quality Presets (CRF)

| Quality | CRF Range | Use Case |
| :- | :- | :- |
| Visually Lossless | 18–20 | Archival, editing source |
| Good Quality | 21–23 | General delivery |
| Lower Quality | 24–28 | Web previews, thumbnails |

> [!TIP]
>
> Lower CRF = better quality but larger file size. Default 23 is a good balance.

### Framerate

Supported: 30fps and 60fps output.

```python Python theme={null}
VideoConfig(
    resolution=720,
    framerate=30  # or 60
)
```

```javascript Node.js theme={null}
{
    resolution: 720,
    framerate: 30  // or 60
}
```

***

## Aspect Ratio Control

### ResizeMode

| Mode | Behavior | Visual |
| :- | :- | :- |
| `crop` | Crop edges to fill target | No black bars, content may be cut |
| `fit` | Scale uniformly | May letterbox (black bars) |
| `pad` | Add black bars | Content preserved, bars added |

```python Python theme={null}
from videodb import ResizeMode, VideoConfig

# Crop to fill (may lose edges)
VideoConfig(aspect_ratio="16:9", resize_mode=ResizeMode.crop)

# Fit with letterbox
VideoConfig(aspect_ratio="16:9", resize_mode=ResizeMode.fit)

# Pad with black bars
VideoConfig(aspect_ratio="1:1", resize_mode=ResizeMode.pad)
```

```javascript Node.js theme={null}
import { ResizeMode } from 'videodb';

// Crop to fill (may lose edges)
{ aspectRatio: "16:9", resizeMode: ResizeMode.crop }

// Fit with letterbox
{ aspectRatio: "16:9", resizeMode: ResizeMode.fit }

// Pad with black bars
{ aspectRatio: "1:1", resizeMode: ResizeMode.pad }
```

***

## AudioConfig

| Field | Type | Default | Notes |
| :- | :- | :- | :- |
| `mute` | bool | false | `true` removes audio track |

```python Python theme={null}
from videodb import AudioConfig

# Keep audio
AudioConfig(mute=False)

# Remove audio
AudioConfig(mute=True)
```

```javascript Node.js theme={null}
// Keep audio
{ mute: false }

// Remove audio
{ mute: true }
```

***

## Common Configurations

### Web Delivery (720p, optimized)

```python Python theme={null}
VideoConfig(
    resolution=720,
    quality=23,
    framerate=30,
    resize_mode=ResizeMode.fit
)
```

```javascript Node.js theme={null}
{
    resolution: 720,
    quality: 23,
    framerate: 30,
    resizeMode: ResizeMode.fit
}
```

### Social Media Square (1:1)

```python Python theme={null}
VideoConfig(
    resolution=720,
    aspect_ratio="1:1",
    resize_mode=ResizeMode.crop
)
```

```javascript Node.js theme={null}
{
    resolution: 720,
    aspectRatio: "1:1",
    resizeMode: ResizeMode.crop
}
```

### Archival (High Quality)

```python Python theme={null}
VideoConfig(
    resolution=1080,
    quality=18,
    framerate=60
)
```

```javascript Node.js theme={null}
{
    resolution: 1080,
    quality: 18,
    framerate: 60
}
```

### Mobile Preview (Low Bandwidth)

```python Python theme={null}
VideoConfig(
    resolution=360,
    quality=28,
    framerate=30
)
```

```javascript Node.js theme={null}
{
    resolution: 360,
    quality: 28,
    framerate: 30
}
```

***

## Job Output Details

Completed jobs include detailed metadata:

```json theme={null}
{
  "job_id": "xxx",
  "status": "completed",
  "output": "https://transcoded-output.mp4",
  "input_details": {
    "resolution": "1920x1080",
    "framerate": 30,
    "duration": 634.53,
    "size": 263.34,
    "video_codec": "h264",
    "audio_codec": "mp3"
  },
  "output_details": {
    "resolution": "1920x1080",
    "framerate": 30,
    "duration": 635.06,
    "size": 253.84,
    "format": "mp4",
    "video_codec": "h264",
    "audio_codec": "aac"
  },
  "cost": 0.1058
}
```

***

## Next Steps

> **[When to Transcode](/pages/ingest/transcoding/when-to-transcode)**
>
> Decision guide and processing modes

> **[Upload Video](/pages/ingest/files-and-collections/upload-video)**
>
> Ingest media into VideoDB


# When to Transcode
Source: https://docs.videodb.io/pages/ingest/transcoding/when-to-transcode

Decide when transcoding is needed, understand input limits, and choose the right processing mode

Transcoding converts video to standardized formats for playback and processing. Not all uploads need transcoding - use this guide to decide.

## Quick Example

```python Python theme={null}
from videodb import TranscodeMode, VideoConfig

job_id = conn.transcode(
    source="https://example.com/source.mov",
    callback_url="https://your-backend.com/webhooks",
    mode=TranscodeMode.lightning
)
print(f"Job queued: {job_id}")
```

```javascript Node.js theme={null}
import { TranscodeMode } from 'videodb';

const jobId = await conn.transcode(
    "https://example.com/source.mov",
    "https://your-backend.com/webhooks",
    TranscodeMode.lightning
);
console.log(`Job queued: ${jobId}`);
```

***

## When You Need Transcoding

| Scenario | Transcode? | Why |
| :- | :- | :- |
| MOV, MKV, AVI input | Yes | Convert to MP4/HLS for playback |
| 4K source → 720p delivery | Yes | Reduce size and bandwidth |
| High framerate (120fps) → 30fps | Yes | Normalize for web |
| Custom aspect ratio | Yes | Crop or letterbox |
| MP4 already in target format | No | Upload directly |
| Already optimized for web | No | Save processing cost |

***

## Input Limits

| Capability | Limit |
| :- | :- |
| Max Resolution | 4K (3840 × 2160) |
| Max Frame Rate | 100 fps |
| Max File Size | 8 GB |
| Audio Support | AAC, MP3, or mute |

> [!NOTE]
>
> Need larger files or custom pipelines? Contact [Sales](mailto:engg@videodb.io) or reach out on [Discord](https://discord.gg/py9P639jGz).

***

## Processing Modes

Choose based on your latency and cost requirements:

| Mode | Best For | Trade-off |
| :- | :- | :- |
| `lightning` | Real-time workflows | Faster, higher cost |
| `economy` | Batch processing | Slower, lower cost |

```python Python theme={null}
from videodb import TranscodeMode

# Fast - for user-facing flows
job_id = conn.transcode(
    source=url,
    callback_url=callback,
    mode=TranscodeMode.lightning
)

# Economical - for batch jobs
job_id = conn.transcode(
    source=url,
    callback_url=callback,
    mode=TranscodeMode.economy
)
```

```javascript Node.js theme={null}
import { TranscodeMode } from 'videodb';

// Fast - for user-facing flows
const lightningJobId = await conn.transcode(
    url,
    callback,
    TranscodeMode.lightning
);

// Economical - for batch jobs
const economyJobId = await conn.transcode(
    url,
    callback,
    TranscodeMode.economy
);
```

***

## Job Lifecycle

1. **Submit Job** - `conn.transcode()` → Status: `pending`
2. **Processing** - Fetch & encode → Status: `processing`
3. **Complete** - Webhook triggers with output URL

### Check Status

```python Python theme={null}
job = conn.get_transcode_details(job_id)
print(job.status)  # pending | processing | completed | failed
print(job.output)  # URL when completed
```

```javascript Node.js theme={null}
const job = await conn.getTranscodeDetails(jobId);
console.log(job.status);  // pending | processing | completed | failed
console.log(job.output);  // URL when completed
```

***

## Webhook Callbacks

### Success

```json theme={null}
{
  "success": true,
  "data": {
    "job_id": "xxx",
    "output": "https://transcoded-output-url.mp4"
  },
  "message": "Transcode job completed"
}
```

### Failure

```json theme={null}
{
  "success": false,
  "job_id": "xxx",
  "code": "invalid_source",
  "message": "Failed to download source media",
  "reason": "HTTP 403 Forbidden"
}
```

***

## Error Codes

| Code | Meaning | Fix |
| :- | :- | :- |
| `invalid_video_config` | Resolution/FPS/CRF error | Check ≤4K, ≤100 fps |
| `invalid_audio_config` | Conflicting audio params | Adjust audio config |
| `invalid_source` | URL not accessible | Check URL permissions |
| `invalid_media` | Unsupported format | Convert to supported format |
| `internal_server_error` | Unexpected error | Retry or contact support |

***

## Decision Flowchart

```
Source file arrives
        │
        ▼
Is it MP4/HLS at target resolution? ──Yes──► Upload directly
        │
       No
        ▼
Need resolution change? ──────────────────► Transcode
        │
       No
        ▼
Need format conversion? ──────────────────► Transcode
        │
       No
        ▼
Need framerate adjustment? ────────────────► Transcode
        │
       No
        ▼
Upload directly
```

***

## Next Steps

> **[Output Formats](/pages/ingest/transcoding/output-formats)**
>
> MP4/HLS, resolutions, quality settings

> **[Upload Video](/pages/ingest/files-and-collections/upload-video)**
>
> Ingest media into VideoDB


# MCP Authentication and Privacy
Source: https://docs.videodb.io/pages/mcp/auth-and-privacy

How the hosted VideoDB MCP Server authenticates you and what the connector collects

When you connect to the hosted server at `https://mcp.videodb.io/mcp`, your client asks you to log in to your VideoDB account once, in a browser tab.

***

## How Connecting Works

**1. You add the server to your client**

See [Connect the MCP Server](/pages/mcp/connect).

**2. A VideoDB page opens**

It lists what the client will be able to do with your account.

**3. You click Authorize**

You're taken back to your client, connected.

Your client handles the rest. It holds a token from then on and sends it with every tool call.

***

## What the Connector Collects

| | |
| :- | :- |
| **Collected** | The arguments of the tool call being run, and nothing else |
| **Not read or stored** | Conversation content, chat history, memory, your files |
| **Sent to** | The VideoDB API (`api.videodb.io`) only. Nothing goes anywhere else |

***

## Media and Derived Data

Videos you upload, and the transcripts, indexes and streams built from them, live in your VideoDB account under [VideoDB's retention terms](https://videodb.io/legal/privacy): stored until you delete them, with no automatic expiry.

Delete them with `delete_video`, `delete_index`, `delete_collection`, or `remove_video_storage`. See [Security & Privacy](/pages/core-concepts/security-privacy) for retention patterns across the rest of VideoDB.

***

## More

* Privacy policy: [videodb.io/legal/privacy](https://videodb.io/legal/privacy)
* Terms: [videodb.io/legal/terms](https://videodb.io/legal/terms)
* Security questions: [hello@videodb.io](mailto:hello@videodb.io)

***

## Next Steps

> **[MCP Server](/pages/mcp/index)**
>
> What the hosted server is and what it covers

> **[Connect](/pages/mcp/connect)**
>
> Setup for Claude, Claude Code, Cursor and other clients

> **[Tool Reference](/pages/mcp/tools)**
>
> Every tool, grouped by toolset


# Connect the MCP Server
Source: https://docs.videodb.io/pages/mcp/connect

Add the hosted VideoDB MCP Server to Claude, Claude Code, Cursor, or MCP Inspector

Add this URL to your client:

```text theme={null}
https://mcp.videodb.io/mcp
```

You need a VideoDB account. Sign up free at [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) - no credit card.

***

## Claude

Works on web, desktop and mobile.

**1. Open connector settings**

**Settings → Connectors → Add custom connector**.

On Team and Enterprise plans, an admin adds it under **Organization settings → Connectors**.

**2. Paste the server URL**

```text theme={null}
https://mcp.videodb.io/mcp
```

**3. Click Connect**

A VideoDB page opens in your browser.

**4. Log in and authorize**

If you're not already signed in, it takes you to log in (or sign up) at [console.videodb.io](https://console.videodb.io) first. Click **Authorize**. The browser hands you back to Claude with the connector enabled.

***

## Claude Code

```bash theme={null}
claude mcp add --transport http videodb https://mcp.videodb.io/mcp
```

Run `/mcp` inside Claude Code to authorize, then confirm it registered:

```bash theme={null}
claude mcp list
```

***

## Cursor

Open **Settings → Cursor Settings → MCP → Add new global MCP server** and add:

```json theme={null}
{
  "mcpServers": {
    "videodb": {
      "url": "https://mcp.videodb.io/mcp"
    }
  }
}
```

Cursor opens console.videodb.io on first use. Log in and click Authorize there.

***

## MCP Inspector

```bash theme={null}
npx @modelcontextprotocol/inspector
```

Set the transport to **Streamable HTTP**, enter `https://mcp.videodb.io/mcp`, and connect.

***

## Any other client

Add it as a remote server over Streamable HTTP with the same URL. There is nothing else to configure.

***

## Verify It Works

Ask your client:

```text theme={null}
list my videodb collections
```

You get back your collections with their IDs and names.

***

## Troubleshooting

| Symptom | Fix |
| :- | :- |
| The VideoDB page never opens | Allow popups for the client, and check whether a corporate proxy blocks `mcp.videodb.io`. |
| Authorization fails after you log in | Sign out and back in at [console.videodb.io](https://console.videodb.io), then reconnect. |
| Tools are missing after connecting | Disconnect and reconnect so the client re-lists the tools. |
| A long upload or index call returns `status: "processing"` | The job is still running. Re-check it with the read tool named in the response's `retry_with` field. |
| The connection stopped working | Disconnect the connector and connect again. |

***

## Local Server

An older package, `videodb-director-mcp`, runs a local MCP server over stdio. Use it if your client only supports stdio. See [Local MCP Server](/pages/mcp/local-server).


# MCP Server
Source: https://docs.videodb.io/pages/mcp/index

Connect Claude and any MCP client to your VideoDB account - upload, index, search, edit, and stream from a conversation

The VideoDB MCP Server is a remote [MCP](https://modelcontextprotocol.io) server that gives Claude, and any other MCP client, access to a VideoDB account: upload media, index it, search it, cut clips, and get playable stream links from inside a conversation.

| | |
| :- | :- |
| **Endpoint** | `https://mcp.videodb.io/mcp` |
| **Transport** | Streamable HTTP |
| **Auth** | Your MCP client asks you to log in to your VideoDB account once, when it connects |
| **Tools** | 89, in 9 toolsets |

***

## Prerequisites

**1. A VideoDB Account**

Free at [console.videodb.io](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs), no credit card required. You log in there when you connect.

**2. An MCP Client**

Claude (web, desktop, mobile), Claude Code, Cursor, MCP Inspector, or any client that speaks Streamable HTTP.

***

## Toolsets

| Toolset | Tools | Covers |
| :- | :- | :- |
| [collections](/pages/mcp/tools#collections) | 7 | List, create, rename and delete collections; account usage and invoices |
| [media](/pages/mcp/tools#media) | 20 | Upload from URLs, list and inspect videos, audio and images, thumbnails, transcode, delete |
| [transcripts](/pages/mcp/tools#transcripts) | 3 | Get, generate and translate transcripts |
| [indexing](/pages/mcp/tools#indexing) | 15 | Spoken-word and scene indexes, analyzer runs, retrieval-ready v2 indexes |
| [search](/pages/mcp/tools#search) | 7 | Search or ask questions in one video or a whole collection; semantic, structured and aggregate queries |
| [streams](/pages/mcp/tools#streams) | 6 | Clips, supercuts, burned-in subtitles, highlight reels, reframes, MP4 renders |
| [editor](/pages/mcp/tools#editor) | 2 | Render a multi-track timeline and download the result |
| [rtstream](/pages/mcp/tools#rtstream) | 27 | Connect RTSP, RTMP or HLS feeds, index them live, run events and alerts |
| [meetings](/pages/mcp/tools#meetings) | 2 | Send a recorder bot to Zoom, Google Meet or Microsoft Teams |

***

## Example Prompt

```text theme={null}
Upload this YouTube keynote, index it, and find every moment where they demo
the product. Then cut those moments into one highlight reel.
```

***

## What It Does Not Do

| Not available | Use instead |
| :- | :- |
| AI image, video, audio or music generation; dubbing and voice cloning | The [SDK](/pages/act/generative-media/index), which exposes them |
| Local file uploads. `upload_media` takes public URLs and YouTube links | The [SDK](/pages/getting-started/quickstart) or the [console](https://console.videodb.io) |
| Desktop capture, which needs a local capture client | [Agent Skills](/pages/getting-started/agent-skills) or the [Capture SDKs](/pages/ingest/capture-sdks/overview) |

***

## Next Steps

> **[Connect the MCP Server](/pages/mcp/connect)**
>
> Setup for Claude, Claude Code, Cursor, MCP Inspector and other clients

> **[Tool Reference](/pages/mcp/tools)**
>
> Every tool, grouped by toolset, with what it does and what it needs first

> **[Workflows](/pages/mcp/workflows)**
>
> Recipes: upload to highlight reel, search a collection, live alerts

> **[Auth and Privacy](/pages/mcp/auth-and-privacy)**
>
> How login works and what the connector collects


# Local MCP Server (Director)
Source: https://docs.videodb.io/pages/mcp/local-server

The local stdio MCP server, videodb-director-mcp, that runs on your own machine - install it with uvx, pipx or pip and configure Claude Desktop, Cursor or Claude Code

> [!NOTE]
>
> The hosted server at `https://mcp.videodb.io/mcp` is the recommended way to connect. You install nothing, you do not need Python, and your API key never goes into a config file. See [MCP Server](/pages/mcp/index) and [Connect the MCP Server](/pages/mcp/connect).
>
> This page covers the local stdio package, for setups that need it.

## Prerequisite: Ensure Python 3.12 or Later is Installed

Before installing the VideoDB MCP Server, verify that Python 3.12 or later is installed on your system.

* **Check Python version:**

```bash theme={null}
python --version
```

* If the version is below 3.12, update Python from the [official website](https://www.python.org/downloads/).

# Install and Configure VideoDB MCP Server

> Simplest method using `uvx`

## 1. Install `uv`

**macOS:**

```bash theme={null}
brew install uv
```

**For macOS/Linux:**

```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
```

**For Windows:**

```bash theme={null}
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```

You can visit the complete installation steps for uv [here](https://astral.sh/uv).

## 2. Automatic Installation for Clients

To automatically add the MCP Server to Claude, Cursor and Claude Code:

**Install for Claude only**

```bash theme={null}
uvx videodb-director-mcp --install=claude
```

**Install for Cursor only**

```bash theme={null}
uvx videodb-director-mcp --install=cursor

```

**Install for both Claude and Cursor**

```bash theme={null}
uvx videodb-director-mcp --install=all
```

**Install for Claude Code**

```bash theme={null}
claude mcp add videodb-director uvx -- videodb-director-mcp --api-key=<VIDEODB_API_KEY>
```

## 3. Update VideoDB MCP package

To ensure you're using the latest version of the MCP server with `uvx`, start by clearing the cache:

```bash theme={null}
uv cache clean
```

This command removes any outdated cached packages of `videodb-director-mcp`, allowing `uvx` to fetch the most recent version.

If you always want to use the latest version of the MCP server, update your command as follows:

```bash theme={null}
uvx videodb-director-mcp@latest --api-key=<VIDEODB_API_KEY>
```

This ensures that `uvx` pulls the latest release every time you run it.

## Setup Video

[VideoDB MCP Server Setup](https://www.youtube.com/embed/E7y3jRCz2YM)

# Alternative Methods

## 1. Install the VideoDB MCP Server

### a. Using `pipx`

We need to install `pipx` first.

**For macOS:**

```bash theme={null}
brew install pipx
pipx ensurepath
```

**For Windows:**

```bash theme={null}
python -m pip install --user pipx
python -m pipx ensurepath
```

You can now run the MCP Server using:

```bash theme={null}
pipx run videodb-director-mcp --api-key=VIDEODB_API_KEY
```

### b. Install Using `pip`

Install the package using pip:

```bash theme={null}
pip install videodb-director-mcp
```

The MCP server can now be started with the following command:

```bash theme={null}
videodb-director-mcp --api-key=VIDEODB_API_KEY
```

## 2. Configuring the MCP Server in Clients

### Claude Desktop

**a. Open Configuration File**

* **macOS/Linux:**

```bash theme={null}
code ~/Library/Application\ Support/Claude/claude_desktop_config.json
```

* **Windows:**

```bash theme={null}
code $env:AppData\Claude\claude_desktop_config.json
```

**b. Modify Configuration**

**Using `pipx`:**

```json theme={null}
{
  "mcpServers": {
    "videodb-director": {
      "command": "pipx",
      "args": ["run", "videodb-director-mcp", "--api-key=<VIDEODB-API-KEY>"]
    }
  }
}
```

**Using package installed via `pip`:**

```json theme={null}
{
  "mcpServers": {
    "videodb-director": {
      "command": "videodb-director-mcp",
      "args": ["--api-key=<VIDEODB-API-KEY>"]
    }
  }
}
```

### Cursor Editor

**a. Open MCP Settings**

1. Navigate to **Settings** > **Cursor Settings**
2. Click on **MCP**
3. Click on **Add new Global MCP Server**

**b. Add Configuration**

**Using `pipx`:**

```json theme={null}
{
  "mcpServers": {
    "videodb-director": {
      "command": "pipx",
      "args": ["run", "videodb-director-mcp", "--api-key=<VIDEODB-API-KEY>"]
    }
  }
}
```

**Using package installed via `pip`:**

```json theme={null}
{
  "mcpServers": {
    "videodb-director": {
      "command": "videodb-director-mcp",
      "args": ["--api-key=<VIDEODB-API-KEY>"]
    }
  }
}

```

### Claude Code

**a. Add configuration**

To configure VideoDB Director MCP for Claude Code you can run the following command:

**Using `pipx`:**

```bash theme={null}
claude mcp add videodb-director pipx -- run videodb-director-mcp --api-key=<VIDEODB_API_KEY>
```

**Using package installed via `pip`:**

```bash theme={null}
claude mcp add videodb-director videodb-director-mcp -- --api-key=<VIDEODB_API_KEY>
```

**b. Verify configuration**

You can verify if the MCP Server has been added correctly or not by simply running the following command:

```bash theme={null}
claude mcp list
```

## 3. Update the VideoDB Director MCP Package

To ensure you're using the latest version of a package installed via `pipx` or `pip`, run:

```bash theme={null}
pip install --upgrade videodb-director-mcp
```

This will upgrade the package to its latest available version.


# MCP Tools Reference
Source: https://docs.videodb.io/pages/mcp/tools

Every tool exposed by the hosted VideoDB MCP server, grouped by toolset, with its kind, inputs and prerequisites.

The hosted VideoDB MCP server at `https://mcp.videodb.io/mcp` exposes 89 tools, grouped into toolsets by what they operate on. Every tool is annotated with a kind, which is how a client knows whether the call changes your account and whether to ask you first.

| Kind | Meaning |
| - | - |
| read | No change to your account. Lists, lookups, search and ask. |
| create | Makes something new: assets, indexes, streams, jobs. |
| update | Changes something that exists: renames, status toggles, visibility. |
| destructive | Deletes. |

## IDs and conventions

| Prefix | Object |
| - | - |
| `m-` | Video |
| `a-` | Audio |
| `img-` | Image |
| `rts-` | Real-time stream |
| `c-` | Collection |

Every ID input takes one of those prefixed forms: `video_id` is an `m-` ID, `audio_id` an `a-` ID, `image_id` an `img-` ID, `rtstream_id` an `rts-` ID, `collection_id` a `c-` ID. `media_id` accepts a video or an audio ID. Every account has a collection named `default`, so leaving `collection_id` out sends the call there.

Reading the Inputs column: a plain name is required, a trailing `?` means optional, and `?=value` is the default the server applies when you omit it. Tools with more than eight inputs show the first six, and a full table of every field follows the toolset table.

* Inputs and outputs are snake\_case.
* A write that runs long returns `{"status": "processing", ...}` with a `retry_with` field naming the read tool that shows the finished result.
* Large lists are capped. A capped result carries `truncated: true` and `total`.

## collections

Collections hold your media. Account usage and invoices live here too.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `list_collections` | read | none | List all collections in the account with IDs, names, descriptions and visibility. |
| `get_collection` | read | `collection_id?=default` | Get one collection's ID, name, description and public/private visibility. |
| `create_collection` | create | `name`, `description?=""`, `is_public?=false` | Create a new collection to organize videos. |
| `update_collection` | update | `collection_id?=default`, `name?`, `description?`, `is_public?` | Rename a collection, change its description, or switch it between public and private. |
| `delete_collection` | destructive | `collection_id` | Permanently delete a collection and everything in it. |
| `get_account_usage` | read | none | Get the account's current usage and billing summary. |
| `list_invoices` | read | none | List the account's invoices (first 200; `truncated` is set when more exist). |

## media

List, fetch, upload, rename and delete videos, audio and images, plus thumbnails, signed download URLs and transcoding.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `list_assets` | read | `collection_id?`, `asset_type?`, `name_pattern?`, `min_duration?`, `max_duration?`, `min_size?`, +5 more (see below) | List videos, audio and images together in one paginated result, filtered by type, name pattern, duration or file size, and sorted by name, duration or size. |
| `list_videos` | read | `collection_id?=default` | List videos in a collection: IDs, names, durations, and playable stream/player URLs. |
| `list_audios` | read | `collection_id?=default` | List audio files in a collection: IDs, names, durations and sizes. |
| `list_images` | read | `collection_id?=default` | List images in a collection: IDs, names and URLs. |
| `get_video` | read | `video_id`, `collection_id?=default` | Get metadata for one video: name, duration, thumbnail, stream URL, and a player URL for watching it in a browser. |
| `get_audio` | read | `audio_id`, `collection_id?=default` | Get metadata for one audio file: name, duration and size. |
| `get_image` | read | `image_id`, `collection_id?=default` | Get metadata for one image: name and URL. |
| `upload_media` | create | `url`, `collection_id?=default`, `name?`, `description?`, `media_type?`, `callback_url?` | Upload a video, audio file or image from a public URL (direct file links and YouTube URLs both work). |
| `search_youtube` | read | `query`, `result_threshold?=10`, `duration?=medium` | Search YouTube and return titles, URLs and metadata, useful for finding URLs to pass to `upload_media`. |
| `search_videos_by_title` | read | `collection_id?=default`, `query` | Find videos in a collection whose title matches a natural-language query. |
| `update_video` | update | `video_id`, `collection_id?=default`, `name` | Update a video's display name. |
| `generate_thumbnail` | create | `video_id`, `collection_id?=default`, `time_seconds?` | Get the default thumbnail of a video, or generate one from a specific timestamp (in seconds). |
| `list_thumbnails` | read | `video_id`, `collection_id?=default` | List every thumbnail image generated for a video, with image IDs and URLs. |
| `get_media_download_url` | read | `media_id`, `collection_id?=default` | Get a signed, time-limited download URL for an audio file (`a-`) or image (`img-`). |
| `transcode_media` | create | `source`, `callback_url`, `mode?=economy`, `resolution?`, `quality?`, `framerate?`, +3 more (see below) | Start a transcode job for a video at a public URL: change resolution, quality, framerate or aspect ratio, or mute audio. |
| `get_transcode_status` | read | `job_id` | Get the status and output of a transcode job started by `transcode_media`. |
| `delete_video` | destructive | `video_id`, `collection_id?=default` | Permanently delete a video from a collection. |
| `delete_audio` | destructive | `audio_id`, `collection_id?=default` | Permanently delete an audio file from a collection. |
| `delete_image` | destructive | `image_id`, `collection_id?=default` | Permanently delete an image from a collection. |
| `remove_video_storage` | destructive | `video_id`, `collection_id?=default` | Delete the stored media bytes of a video while keeping its record, indexes and transcripts. |

#### `list_assets` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `collection_id` | `string` | no | Scope to one collection. Omitted = every collection in the account. |
| `asset_type` | `('video' \| 'audio' \| 'image')[]` | no | Asset types to include. Omitted = all three. |
| `name_pattern` | `string` | no | Case-insensitive regular expression matched against the asset name, e.g. 'keynote\|demo'. |
| `min_duration` | `number` | no | Minimum duration in seconds (videos and audio). |
| `max_duration` | `number` | no | Maximum duration in seconds (videos and audio). |
| `min_size` | `number` | no | Minimum file size in bytes. |
| `max_size` | `number` | no | Maximum file size in bytes. |
| `sort_by` | `'created_at' \| 'name' \| 'duration' \| 'size'` | no (default `created_at`) | Field to sort on. created\_at is newest-first per type only, since rows carry no timestamp. |
| `sort_order` | `'desc' \| 'asc'` | no (default `desc`) | Sort direction. |
| `page` | `number` | no (default `1`) | Page number (1-based). |
| `page_size` | `number` | no (default `100`) | Items per page. |

#### `transcode_media` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `source` | `string` | yes | Public, downloadable URL of the source video. |
| `callback_url` | `string` | yes | Webhook URL VideoDB calls when the job finishes (required). |
| `mode` | `'economy' \| 'lightning'` | no (default `economy`) | economy = cheaper/slower, lightning = faster. |
| `resolution` | `number` | no | Output height in pixels, e.g. 720 or 1080 (max 4K). |
| `quality` | `number` | no | Encoder CRF quality, lower = better (default 23). |
| `framerate` | `number` | no | Output frames per second. |
| `aspect_ratio` | `string` | no | Output aspect ratio, e.g. "16:9", "9:16", "1:1". |
| `resize_mode` | `'crop' \| 'fit' \| 'pad'` | no | How to reach aspect\_ratio: crop edges, fit (letterbox) or pad with bars. |
| `mute` | `boolean` | no | Strip the audio track. |

**Notes**

* `get_media_download_url` covers audio and images only. For a video, render it with `download_stream`.
* Uploads take a public URL. Local file paths are not supported.
* A write that runs long comes back as `status: "processing"` with a `retry_with`: `upload_media` names `list_videos`, `generate_thumbnail` names `list_thumbnails`, `transcode_media` names `get_transcode_status`.

## transcripts

Speech to text, plus translation of what comes back.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `get_transcript` | read | `media_id?`, `video_id?`, `collection_id?=default`, `timestamped?=false`, `start?`, `end?`, `segmenter?`, `segment_length?` | Get the transcript of a video or audio file, as plain text or with timestamped segments (per word, sentence or time window). |
| `generate_transcript` | create | `media_id?`, `video_id?`, `collection_id?=default`, `force?=false`, `language_code?` | Transcribe a video or audio file, optionally forcing a fresh transcription or hinting the spoken language. |
| `translate_transcript` | create | `video_id`, `collection_id?=default`, `language`, `additional_notes?`, `callback_url?` | Translate a video's transcript into another language and return the translated segments. |

**Prerequisites**

* `translate_transcript` needs an existing transcript. Run `get_transcript` or `generate_transcript` first. It works on videos only.
* `get_transcript` and `generate_transcript` accept either `media_id` or `video_id`. Give exactly one.
* While a transcription is still running, both come back as `status: "processing"` with `retry_with: get_transcript`.

## indexing

Two pipelines: the v1 spoken-word and scene indexes, and the v2 flow where `understand_video` extracts artifacts and `create_index` turns one analyzer's output into a searchable index.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `index_spoken_words` | create | `video_id`, `collection_id?=default`, `language_code?`, `force?=false`, `segmentation_type?`, `callback_url?` | Transcribe and index everything spoken in a video so it becomes searchable with `search_video` / `search_collection` (v1 index). |
| `index_scenes` | create | `video_id`, `collection_id?=default`, `prompt?`, `name?`, `batch_type?=shot`, `batch_value?`, +3 more (see below) | Visually index a video (v1 scene index): samples frames per shot or per time window and describes what is on screen, so visual moments become searchable. |
| `list_scene_indexes` | read | `video_id`, `collection_id?=default` | List the visual (scene) indexes that exist for a video, with IDs, names and status. |
| `get_scene_index` | read | `video_id`, `collection_id?=default`, `scene_index_id` | Return the per-scene records (start, end, description) of one visual scene index. |
| `delete_scene_index` | destructive | `video_id`, `collection_id?=default`, `scene_index_id` | Permanently delete one visual scene index of a video. The video is kept. |
| `understand_video` | create | `video_id`, `collection_id?=default`, `analyzers`, `segmentation?`, `callback_url?` | Run one or more analyzers over a video to extract artifacts: transcript, scene descriptions, on-screen text, objects, brands, activities, locations. |
| `get_understanding` | read | `video_id`, `collection_id?=default`, `understanding_id` | Get the status of one understanding run and of each analyzer in it (pending, running, done, failed). |
| `list_understandings` | read | `video_id`, `collection_id?=default` | List every understanding run of a video with its status and analyzers. |
| `delete_understanding` | destructive | `video_id`, `collection_id?=default`, `understanding_id` | Permanently delete an understanding run and its stored analyzer artifacts. Indexes built from it keep their own data. |
| `get_analyzer_output` | read | `video_id`, `collection_id?=default`, `understanding_id`, `analyzer_name?`, `analyzer_id?` | Return the raw artifact one analyzer produced, such as transcript segments or per-scene descriptions with start/end. |
| `create_index` | create | `video_id`, `collection_id?=default`, `understanding_id`, `analyzer_name?`, `analyzer_id?`, `name?`, +3 more (see below) | Build a retrieval-ready index from one analyzer's output of an understanding run. |
| `get_index` | read | `video_id`, `collection_id?=default`, `index_id?`, `name?` | Get one v2 index manifest by ID or name: status (building/ready/failed), record count, capabilities and field schema. |
| `list_indexes` | read | `video_id`, `collection_id?=default`, `use_for?` | List the v2 indexes of a video (ID, name, status, capabilities, record count). |
| `delete_index` | destructive | `video_id`, `collection_id?=default`, `index_id` | Permanently delete a v2 index. The video and understanding artifacts are kept. |
| `get_index_records` | read | `video_id`, `collection_id?=default`, `index_id`, `limit?=20`, `cursor?` | Page through the records stored in a v2 index (scene\_id, start, end, data). |

#### `index_scenes` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `video_id` | `string` | yes | VideoDB video ID (starts with m-). |
| `collection_id` | `string` | no (default `default`) | VideoDB collection ID. Defaults to "default" (every account has one). |
| `prompt` | `string` | no | Instruction for how to describe scenes. |
| `name` | `string` | no | Name for this scene index. |
| `batch_type` | `'shot' \| 'time'` | no (default `shot`) | Split by detected shots or fixed time windows. |
| `batch_value` | `number` | no | Shot threshold (default 20) or window seconds (default 10). |
| `frame_count` | `number` | no (default `1`) | Frames sampled per scene. |
| `model_name` | `string` | no | Vision model tier or name used to describe scenes, e.g. 'basic', 'pro', 'ultra'. Omit for the default. |
| `callback_url` | `string` | no | Optional webhook URL VideoDB calls when the job finishes (makes the call return immediately). |

#### `create_index` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `video_id` | `string` | yes | VideoDB video ID (starts with m-). |
| `collection_id` | `string` | no (default `default`) | VideoDB collection ID. Defaults to "default" (every account has one). |
| `understanding_id` | `string` | yes | Understanding run ID (from understand\_video / list\_understandings). |
| `analyzer_name` | `string` | no | Analyzer name inside the run, e.g. 'transcript', 'scene', 'objects'. Give this or analyzer\_id. |
| `analyzer_id` | `string` | no | Analyzer ID (an\_...). Give this or analyzer\_name. |
| `name` | `string` | no | Index name; used by semantic\_search index\_names and get\_index. |
| `use_for` | `('semantic' \| 'query' \| 'aggregate')[]` | no | Retrieval capabilities: semantic (embeddings), query (filter/sort), aggregate (group/count). |
| `fields` | `object` | no | Per-field capabilities, e.g. `{description:['semantic'], label:['filter','aggregate']}`. Omit for VideoDB defaults. |
| `callback_url` | `string` | no | Optional webhook URL VideoDB calls when the job finishes (makes the call return immediately). |

**Prerequisites**

* `create_index` needs an `understand_video` run whose analyzer has finished. Check with `get_understanding`.
* `create_index` and `get_analyzer_output` accept either `analyzer_name` or `analyzer_id`. Give exactly one.
* An index declares what it can serve through `use_for`. The values are `semantic`, `query` and `aggregate`, and they map to the search tools below.
* `understand_video` takes `analyzers` as a list of objects, `[{type: "spoken_words"}, {type: "vlm", prompt: "Describe the scene"}]`, and `segmentation` as `{type: "time", seconds: 5}` or `{type: "shot", threshold: 30}`.
* A write that runs long comes back as `status: "processing"` with a `retry_with`: `index_spoken_words` names `search_video`, `index_scenes` names `list_scene_indexes`, `understand_video` names `get_understanding`, `create_index` names `get_index`.

## search

Moment search and question answering, over one video or a whole collection.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `search_video` | read | `video_id`, `collection_id?=default`, `query`, `search_type?`, `index_type?`, `result_threshold?`, +5 more (see below) | Search within one video and get matching moments with start/end timestamps, text and relevance scores. |
| `search_collection` | read | `collection_id?=default`, `query`, `index_type?`, `result_threshold?`, `top_k?`, `mode?`, `include_clip?`, `session_id?` | Search every indexed video in a collection and get matching moments grouped by video, with timestamps and scores. |
| `ask_video` | read | `video_id`, `collection_id?=default`, `question`, `top_k?`, `mode?`, `include_sources?` | Answer a natural-language question from one video's indexed content. |
| `ask_collection` | read | `collection_id?=default`, `question`, `top_k?`, `mode?`, `include_sources?` | Answer a natural-language question from every indexed video in a collection. |
| `semantic_search` | read | `collection_id?=default`, `video_id?`, `query`, `index_names?`, `index_ids?`, `top_k?`, +3 more (see below) | Vector search over one or more indexes in a video or across a collection. |
| `query_index` | read | `collection_id?=default`, `video_id?`, `index_name?`, `index_id?`, `filter?`, `sort?`, `limit?`, `return_fields?` | Structured filter and sort over one index in a video or across a collection. |
| `aggregate_index` | read | `collection_id?=default`, `video_id?`, `index_name?`, `index_id?`, `group_by?`, `metric?`, +3 more (see below) | Counts, groups and facets over one index in a video or across a collection. |

#### `search_video` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `video_id` | `string` | yes | VideoDB video ID (starts with m-). |
| `collection_id` | `string` | no (default `default`) | VideoDB collection ID. Defaults to "default" (every account has one). |
| `query` | `string` | yes | What to look for, e.g. 'moments where pricing is discussed'. |
| `search_type` | `'semantic' \| 'keyword'` | no | Legacy: semantic (default) or keyword matching. Keyword works on videos only. |
| `index_type` | `'spoken_word' \| 'scene'` | no | Legacy: index to search, spoken\_word (default) or scene. |
| `result_threshold` | `number` | no | Legacy: max results (default 5). |
| `scene_index_id` | `string` | no | Legacy: search one scene index (from list\_scene\_indexes). |
| `top_k` | `number` | no | v2: maximum moments to return. |
| `mode` | `'default' \| 'deepsearch'` | no | v2: default = single-step planned retrieval; deepsearch = multi-step investigation with follow-up sessions. |
| `include_clip` | `boolean` | no | v2: also return a playable clip for each moment. |
| `session_id` | `string` | no | v2: continue a deepsearch session (session\_id from a previous response). Only valid with mode=deepsearch. |

#### `semantic_search` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `collection_id` | `string` | no (default `default`) | VideoDB collection ID. Defaults to "default" (every account has one). |
| `video_id` | `string` | no | VideoDB video ID (starts with m-). Omit to run against the whole collection. |
| `query` | `string` | yes | Natural-language query. |
| `index_names` | `string[]` | no | Restrict to these index names. |
| `index_ids` | `string[]` | no | Restrict to these index IDs. |
| `top_k` | `number` | no | Max moments to return (default 10). |
| `score_threshold` | `number` | no | Minimum similarity score. |
| `filter` | `object[] \| object` | no | Structured filter over index fields, e.g. `[{"field": "label", "op": "==", "value": "phone"}]` or `{"activity": "walking"}`. |
| `return_fields` | `string \| string[] \| object` | no | Index fields to hydrate into each moment's metadata; "all" for everything. |

#### `aggregate_index` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `collection_id` | `string` | no (default `default`) | VideoDB collection ID. Defaults to "default" (every account has one). |
| `video_id` | `string` | no | VideoDB video ID (starts with m-). Omit to run against the whole collection. |
| `index_name` | `string` | no | Index name (from list\_indexes). |
| `index_id` | `string` | no | Index ID (from list\_indexes). |
| `group_by` | `string` | no | Aggregatable field to group by; dotted paths allowed, e.g. frames.detections.label. |
| `metric` | `string` | no | count (default), or sum(field), avg(field), min(field), max(field). |
| `filter` | `object[] \| object` | no | Structured filter over index fields, e.g. `[{"field": "label", "op": "==", "value": "phone"}]` or `{"activity": "walking"}`. |
| `sort` | `string \| array` | no | Sort order, e.g. \[\["start", "asc"]]. |
| `limit` | `number` | no | Max groups (default 100). |

**Prerequisites**

* `search_video` and `search_collection` read the v1 indexes: spoken-word search needs `index_spoken_words`, and `index_type='scene'` needs `index_scenes`.
* `ask_video` and `ask_collection` need v2 indexes built by `create_index`.
* `semantic_search`, `query_index` and `aggregate_index` need an index built by `create_index` whose `use_for` includes `semantic`, `query` and `aggregate` respectively. Each takes `index_name` or `index_id`, and each runs over one video with `video_id` or the whole collection without it.
* `filter` takes `[{"field": "label", "op": "==", "value": "phone"}]` or the shorthand `{"activity": "walking"}`. `sort` takes `[["start", "asc"]]`. `return_fields` takes a field name, a list of them, or `"all"`.

## streams

Cut, stitch, subtitle and reframe videos into playable streams, then render them to MP4.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `generate_clip` | create | `video_id`, `collection_id?=default`, `timeline` | Cut a clip, or a supercut of several segments, from a single video without re-encoding. |
| `compile_clips` | create | `clips` | Stitch moments from one or more videos into a single new stream. |
| `add_subtitles` | create | `video_id`, `collection_id?=default`, `font_name?`, `font_size?`, `primary_colour?`, `secondary_colour?`, +17 more (see below) | Burn subtitles into a video from its transcript. |
| `create_highlight_clip` | create | `video_id`, `collection_id?=default`, `prompt`, `content_type?=spoken`, `model_name?=basic` | Pick the moments of a video that match a natural-language prompt and compile them into one stream. |
| `reframe_video` | create | `video_id`, `collection_id?=default`, `start?`, `end?`, `target?=vertical`, `mode?=smart`, `callback_url?` | Re-crop a video, or a segment of it, to a new aspect ratio (vertical, square, landscape or custom dimensions) and save it as a new video in the same collection. |
| `download_stream` | create | `stream_url`, `name` | Render any VideoDB stream URL into a downloadable MP4 file. |

#### `add_subtitles` inputs

| Field | Type | Required | What it is |
| - | - | - | - |
| `video_id` | `string` | yes | VideoDB video ID (starts with m-). |
| `collection_id` | `string` | no (default `default`) | VideoDB collection ID. Defaults to "default" (every account has one). |
| `font_name` | `string` | no | Font family, default Arial. |
| `font_size` | `number` | no | Font size, default 18. |
| `primary_colour` | `string` | no | Text colour. ASS colour string \&HAABBGGRR (alpha, blue, green, red hex), e.g. \&H00FFFFFF is opaque white. Default white. |
| `secondary_colour` | `string` | no | Secondary (karaoke) colour. ASS colour string \&HAABBGGRR (alpha, blue, green, red hex), e.g. \&H00FFFFFF is opaque white. |
| `outline_colour` | `string` | no | Outline colour. ASS colour string \&HAABBGGRR (alpha, blue, green, red hex), e.g. \&H00FFFFFF is opaque white. Default black. |
| `back_colour` | `string` | no | Background/shadow colour. ASS colour string \&HAABBGGRR (alpha, blue, green, red hex), e.g. \&H00FFFFFF is opaque white. Default black. |
| `bold` | `boolean` | no | Bold text. |
| `italic` | `boolean` | no | Italic text. |
| `underline` | `boolean` | no | Underlined text. |
| `strike_out` | `boolean` | no | Strike-through text. |
| `scale_x` | `number` | no | Horizontal scale factor, default 1.0. |
| `scale_y` | `number` | no | Vertical scale factor, default 1.0. |
| `spacing` | `number` | no | Extra spacing between letters in pixels, default 0. |
| `angle` | `number` | no | Rotation angle in degrees, default 0. |
| `border_style` | `'no_border' \| 'opaque_box' \| 'outline'` | no | Border style, default outline. |
| `outline` | `number` | no | Outline thickness in pixels, default 1. |
| `shadow` | `number` | no | Shadow depth in pixels, default 0. |
| `alignment` | `'bottom_left' \| 'bottom_center' \| 'bottom_right' \| 'middle_left' \| 'middle_center' \| 'middle_right' \| 'top_left' \| 'top_center' \| 'top_right'` | no | Position of the subtitles on the frame, default bottom\_center. |
| `margin_l` | `number` | no | Left margin in pixels, default 10. |
| `margin_r` | `number` | no | Right margin in pixels, default 10. |
| `margin_v` | `number` | no | Vertical margin in pixels, default 10. |

**Prerequisites**

* `add_subtitles` needs `index_spoken_words` on the video first.
* `download_stream` takes a stream URL from `generate_clip`, `compile_clips`, `add_subtitles`, `create_highlight_clip` or `get_video`.
* `generate_clip` takes `timeline` as `[start_seconds, end_seconds]` pairs, `[[10, 25], [60, 90]]`. `compile_clips` takes `clips` as a list of objects in playback order.
* `reframe_video` takes `target` as a preset (`vertical`, `square`, `landscape`) or an explicit `{width, height}`. It comes back as `status: "processing"` with `retry_with: list_videos`.

## editor

Multi-track timeline rendering. Tracks hold clips, and each clip carries a video, image, audio, text or caption asset.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `render_timeline` | create | `tracks`, `background?`, `resolution?` | Render a multi-track editor timeline into a new stream in one call. |
| `download_timeline_render` | create | `stream_url` | Request an MP4 download of a stream produced by `render_timeline`. |

**Notes**

* A caption asset with `src: "auto"` pulls from the video's spoken-word index. On an unindexed video the render succeeds with no captions, so run `index_spoken_words` first.
* Text assets sized with `width`/`height` keep those dimensions only under `fit: "contain"`. The default `fit: "crop"` upscales the text box to fill the frame.
* `tracks` carries the whole edit. It is a list of track objects, each holding its clips and their assets.

## rtstream

Ingest live RTSP, RTMP and HLS streams, index them continuously, search them, and get alerts on events. All timestamps are Unix seconds.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `connect_rtstream` | create | `url`, `name`, `collection_id?=default`, `media_types?=["video"]`, `sample_rate?`, `store?`, `enable_transcript?` | Register a live RTSP/RTMP/HLS stream with VideoDB and start ingesting it. |
| `list_rtstreams` | read | `limit?=10`, `offset?=0`, `status?`, `name?`, `ordering?` | List real-time streams in the account with IDs, names, status and media types. |
| `get_rtstream` | read | `rtstream_id` | Get one real-time stream: name, status, sample rate, media types, creation time. |
| `set_rtstream_status` | update | `rtstream_id`, `action` | Start (connect) or stop (disconnect) ingestion of a real-time stream. |
| `get_rtstream_stream` | read | `rtstream_id`, `start`, `end` | Generate a playable stream and player URL for a time window of a real-time stream. |
| `export_rtstream_recording` | create | `rtstream_id`, `name?` | Export the latest completed recording of a real-time stream as a regular VideoDB video or audio asset. |
| `get_rtstream_transcript` | read | `rtstream_id`, `page?=1`, `page_size?=100`, `start?`, `end?`, `since?` | Read the real-time transcript of a stream. |
| `set_rtstream_transcript_status` | update | `rtstream_id`, `action`, `mode?=graceful`, `engine?` | Start or stop real-time transcription on a stream. |
| `index_rtstream_scenes` | create | `rtstream_id`, `prompt?`, `window_seconds?`, `frame_count?`, `model_name?`, `name?` | Start a continuous visual index on a real-time stream: every `window_seconds`, `frame_count` frames are described with the prompt. |
| `index_rtstream_audio` | create | `rtstream_id`, `prompt?`, `segmenter?`, `segment_value?`, `model_name?`, `name?`, `auto_start_transcript?=true` | Start a continuous spoken-word index on a real-time stream: the live transcript is chunked by segmenter and summarized with the prompt. |
| `list_rtstream_scene_indexes` | read | `rtstream_id` | List the scene (visual and audio) indexes on a real-time stream with their IDs, prompts and status. |
| `get_rtstream_scenes` | read | `rtstream_id`, `scene_index_id`, `start?`, `end?`, `page?=1`, `page_size?=100` | Read the records a scene index has produced so far (Unix-second start/end plus description). |
| `set_rtstream_scene_index_status` | update | `rtstream_id`, `scene_index_id`, `action` | Resume or pause a scene index on a real-time stream. |
| `search_rtstream` | read | `rtstream_id`, `query`, `index_type?`, `scene_index_id?`, `result_threshold?`, `score_threshold?` | Semantic search over the indexed records of a real-time stream. |
| `create_event` | create | `event_prompt`, `label` | Define a reusable event, a natural-language condition such as "a person enters the restricted area", that alerts can watch for on any stream index. |
| `list_events` | read | none | List the events defined on the account (ID, label, prompt). |
| `create_rtstream_alert` | create | `rtstream_id`, `index_id`, `index_kind?=scene_index`, `event_id`, `callback_url` | Attach an event to a stream index so VideoDB POSTs to `callback_url` each time the event is detected in a new window. |
| `list_rtstream_alerts` | read | `rtstream_id`, `index_id`, `index_kind?=scene_index` | List the alerts attached to a stream index, with their event, callback and enabled state. |
| `set_rtstream_alert_status` | update | `rtstream_id`, `index_id`, `index_kind?=scene_index`, `alert_id`, `action` | Enable or disable an alert on a stream index without deleting it. |
| `understand_rtstream` | create | `rtstream_id`, `prompt`, `window_seconds?=5`, `frame_count?=2`, `output_name?=scene`, `store?=true` | Start a continuous vision-model analysis on a real-time stream: every `window_seconds`, `frame_count` frames are analyzed with the prompt and stored under `output_name`. |
| `list_rtstream_understandings` | read | `rtstream_id` | List the continuous understanding jobs on a real-time stream with their status, segmentation, analyzers and output descriptors. |
| `set_rtstream_understanding_status` | update | `rtstream_id`, `understanding_id`, `action` | Resume or pause a continuous understanding job on a real-time stream. |
| `get_rtstream_understanding_records` | read | `rtstream_id`, `understanding_id`, `start`, `end`, `output?=scene`, `page?=1`, `page_size?=100` | Read the analysis records an understanding produced for a Unix-second time range (start and end required). |
| `create_rtstream_index` | create | `rtstream_id`, `understanding_id`, `output?=scene`, `name?`, `use_for?` | Materialize an understanding output into a continuously updated searchable index. |
| `list_rtstream_indexes` | read | `rtstream_id` | List the understanding-backed indexes on a real-time stream with their status, source understanding and capabilities. |
| `set_rtstream_index_status` | update | `rtstream_id`, `index_id`, `action` | Resume or pause an understanding-backed index on a real-time stream. |
| `get_rtstream_index_records` | read | `rtstream_id`, `index_id`, `start?`, `end?`, `page?=1`, `page_size?=100` | Read the records materialized into an understanding-backed index, optionally limited to a Unix-second start/end window. |

**Prerequisites**

* Everything here starts with `connect_rtstream`. Indexes, understandings and alerts attach to a stream that is already connected and ingesting.
* `get_rtstream_transcript` needs transcription running, started either by `connect_rtstream` with `enable_transcript` or by `set_rtstream_transcript_status`.
* `search_rtstream` needs a scene index or an understanding-backed index on the stream.
* `create_rtstream_index` needs an `understand_rtstream` job started with `store=true`.
* `create_rtstream_alert` needs an event from `create_event` and an index to attach it to. Alerts cannot be deleted, only disabled with `set_rtstream_alert_status`.
* `list_rtstreams` is account-scoped, not per collection. Each row carries its own `collection_id`.
* Alert tools take `index_kind` to say which index the `index_id` belongs to: `scene_index` for one built by `index_rtstream_scenes` or `index_rtstream_audio`, `index` for one built by `create_rtstream_index`.
* A write that runs long comes back as `status: "processing"` with a `retry_with`: `connect_rtstream` names `list_rtstreams`, `export_rtstream_recording` names `get_video`, `index_rtstream_scenes` and `index_rtstream_audio` name `list_rtstream_scene_indexes`, `understand_rtstream` names `list_rtstream_understandings`, `create_rtstream_index` names `list_rtstream_indexes`, `create_event` names `list_events`, `create_rtstream_alert` names `list_rtstream_alerts`.

## meetings

Send a recorder bot into a meeting and collect the recording.

| Tool | Kind | Inputs | What it does |
| - | - | - | - |
| `record_meeting` | create | `meeting_url`, `collection_id?=default`, `bot_name?`, `bot_image_url?`, `meeting_title?`, `time_zone?=UTC`, `callback_url?` | Send a VideoDB recorder bot to a Zoom, Google Meet or Microsoft Teams meeting URL. |
| `get_meeting` | read | `meeting_id`, `collection_id?=default` | Get the state of a recorder bot: status (initializing, joined, processing, done), the recorded `video_id` once done, and the speaker timeline when available. |

**Prerequisites**

* `record_meeting` comes back as soon as the bot is dispatched, with `status: "processing"` and `retry_with: get_meeting`. Poll `get_meeting` until `status` is `done`. Only then does `video_id` have a value.

## Not exposed

| Feature | Why not |
| - | - |
| AI generation: image, music, sound effect, voice and video generation, dubbing, voice clones | Excluded by connector policy. Use the SDKs. |
| Text generation | The client is already a language model. |
| Local-path uploads and `play()` | These need your filesystem or a browser on your machine. |
| Desktop capture, capture sessions and their client tokens | Capture runs through a native binary on your machine over a long-lived connection. A remote server cannot drive it. |

For generation, use the [generative media guides](/pages/act/generative-media/index). For desktop capture and local files, use [VideoDB Agent Skills](/pages/getting-started/agent-skills), which run on your machine.

> **[MCP Server](/pages/mcp/index)**
>
> What the hosted server is and what it connects to

> **[Connect a Client](/pages/mcp/connect)**
>
> Add the server to Claude, Cursor and other MCP clients

> **[Workflows](/pages/mcp/workflows)**
>
> End-to-end recipes built from these tools


# MCP Workflows
Source: https://docs.videodb.io/pages/mcp/workflows

End-to-end recipes for the hosted VideoDB MCP Server - the prompt to type and the tool chain it runs

You talk to the connector in plain language and the client picks the tools. Each recipe below pairs the prompt you type with the chain it runs, so you know what to expect and what to fix when a step is missing.

***

## Highlight reel from a long video

```text theme={null}
Upload https://www.youtube.com/watch?v=... into my default collection, index
the speech, then cut every moment where they demo the product into one reel.
```

**What runs**

1. `upload_media` - pulls the URL into the collection and returns the `m-` video ID
2. `index_spoken_words` - transcribes and indexes everything spoken
3. `create_highlight_clip` - matches your prompt against the index and compiles the hits into one stream

Indexing has to finish before the search returns anything. `create_highlight_clip` matches spoken content by default, and on an unindexed video it comes back with an empty `shots` list. To control the cut yourself, ask for `search_video` to get timestamps, then `generate_clip` for one video or `compile_clips` to stitch across several.

***

## Transcript Q\&A

```text theme={null}
Transcribe m-abc123 and give me the timestamped transcript. Then set it up so
I can ask questions about it, starting with: what pricing objections came up?
```

**What runs**

1. `generate_transcript`
2. `get_transcript` - plain text, or segments timestamped per word, sentence or time window
3. `understand_video` - runs the transcript analyzer over the video
4. `create_index` - builds a retrieval-ready index from that analyzer's output
5. `ask_video` - answers from the index, with sources

`ask_video` and `semantic_search` read only indexes built by `create_index`, so a transcript on its own is not queryable. While step 3 runs, `get_understanding` reports the status of each analyzer.

***

## Subtitled clip for social

```text theme={null}
From m-abc123, take 2:10 to 2:40, crop it to 9:16, burn in subtitles, and give
me an MP4.
```

**What runs**

1. `reframe_video` - re-crops the 2:10-2:40 segment to vertical and saves it as a new video in the same collection
2. `index_spoken_words` - on that new video, since subtitles come from its own index
3. `add_subtitles` - burns them in and returns a stream URL
4. `download_stream` - renders the stream to an MP4, which needs a file name

Order matters here. `reframe_video` and `add_subtitles` both take a video ID rather than a stream URL, so the reframe goes first and the new video gets its own index. `download_stream` is the only step that accepts a stream URL.

***

## Search across a whole library

```text theme={null}
List my collections, index every video in "Customer calls", then find every
mention of the renewal discount.
```

**What runs**

1. `list_collections` - IDs, names and visibility
2. `list_videos` - the videos in the collection you named
3. `index_spoken_words` - once per video
4. `search_collection` - matching moments grouped by video, with start/end timestamps and relevance scores

`search_collection` reads spoken-word and scene indexes. `ask_collection` answers in prose instead of returning moments, but it reads v2 indexes, so every video needs `understand_video` and then `create_index` first.

***

## Live camera monitoring with alerts

```text theme={null}
Connect rtsp://... as "Lobby cam", watch it for anyone entering after hours,
and POST to https://example.com/hooks/lobby when that happens.
```

**What runs**

1. `connect_rtstream` - registers the RTSP, RTMP or HLS feed and starts ingesting it
2. `index_rtstream_scenes` - describes frames on a rolling window using your prompt
3. `create_event` - the condition to watch for, written in natural language
4. `create_rtstream_alert` - attaches that event to the index with your `callback_url`
5. `search_rtstream` or `get_rtstream_scenes` - search the indexed records, or read them straight through for a time range

Alerts POST to a callback URL you control. See [Alerts and callbacks](/pages/act/live-action/alerts-and-callbacks) for the payload fields. Real-time timestamps are Unix seconds, and `get_rtstream_stream` turns a start/end window into a playable URL.

***

## Record and search a meeting

```text theme={null}
Send a recorder bot to this Google Meet link. Once the recording is ready,
index it and tell me what we agreed on.
```

**What runs**

1. `record_meeting` - sends the bot to the Zoom, Google Meet or Microsoft Teams URL
2. `get_meeting` - poll until `status` is `done` and `video_id` appears (`initializing`, `joined`, `processing`, `done`); the speaker timeline comes back here too
3. `index_spoken_words` - on that `video_id`
4. `search_video` - moments with timestamps

For a prose answer rather than moments, run the Transcript Q\&A chain on the recorded video: `understand_video`, `create_index`, `ask_video`.

***

## Working with long jobs

Uploads, indexing runs and renders that outlast the request budget return `{"status": "processing"}` and a `retry_with` field naming the read tool that will show the result, such as `get_meeting`, `get_transcript`, `get_transcode_status` or `list_videos`. The job keeps running. Ask the client to check with that tool; re-issuing the write starts a second job.

***

## Tips

* Name the collection you mean. `default` always exists and is where things land when you do not say.
* Index before you search. An unindexed video returns nothing from `search_*` and `ask_*`, and a timeline caption on one renders with no captions and no error.
* Every stream URL has a matching player URL that opens in a browser.
* ID prefixes tell you what you are holding: `m-` video, `a-` audio, `img-` image, `rts-` real-time stream, `c-` collection.
* Deletes are permanent. `delete_video`, `delete_collection`, `delete_index` and `delete_scene_index` have no undo. `remove_video_storage` drops the stored bytes but keeps the record, indexes and transcripts.

***

## Next Steps

> **[Tool Reference](/pages/mcp/tools)**
>
> Every tool, grouped by toolset, with what it does and what it needs first

> **[Connect the MCP Server](/pages/mcp/connect)**
>
> Setup for Claude, Claude Code, Cursor, MCP Inspector and other clients

> **[MCP Server Overview](/pages/mcp/index)**
>
> Endpoint, auth, toolsets, and what the connector does not cover

> **[Examples and Tutorials](/examples-and-tutorials/index)**
>
> Longer builds that use the same pieces outside a chat client


# Analyzer Outputs
Source: https://docs.videodb.io/pages/understand/indexing-pipelines/analyzer-outputs

Sample output shapes for VideoDB understanding analyzers.

After an [understanding run](/pages/understand/indexing-pipelines/understanding-artifacts) completes, each analyzer has produced a named **artifact** containing timestamped records. Use `get_output()` to inspect the artifact and see which fields the analyzer produced.

```python theme={null}
understanding.wait_until_complete()

analyzer = understanding.get_analyzer("scene")
output = analyzer.get_output()

print(output["scenes"][0]["data"])
```

Artifact output is organized into **scenes**. Each scene covers a range of video defined by `start` and `end`, in seconds. Scenes may come from visual cuts or fixed time windows. Some analyzers also include frame-level observations inside a scene; those use a single `timestamp`.

The contents of `data` depend on the analyzer. When configuring [index fields](/pages/understand/indexing-pipelines/multiple-indexes), paths are relative to `data`, so use `text` or `frames.detections.label` rather than `scenes.data.text`.

`get_output()` is for inspecting the artifact. To [create an index](/pages/understand/indexing-pipelines/create-an-index), pass the analyzer directly so the output does not need to travel through your client:

```python theme={null}
video.index(source=analyzer)
```

The sections below show the output shape and useful [indexing fields](/pages/understand/indexing-pipelines/multiple-indexes) for each analyzer.

***

## Common output envelope

All analyzer artifacts follow the same high-level shape:

```json theme={null}
{
  "name": "scene",
  "type": "vlm",
  "status": "done",
  "metadata": {},
  "scenes": [
    {
      "scene_id": "scene-0.000-3.420",
      "start": 0.0,
      "end": 3.42,
      "data": {},
      "metadata": {}
    }
  ]
}
```

The `data` object is analyzer-specific. The top-level and scene-level `metadata` objects contain additional analyzer or scene details when available; either object may be empty.

> [!NOTE]
>
> Some fields can vary by model, schema version, or workspace configuration. Built-in analyzers keep stable top-level fields for indexing wherever possible.

The [indexing recommendations](/pages/understand/indexing-pipelines/create-an-index#declare-index-capabilities-with-use_for) use these capabilities:

| Capability | What it does |
| - | - |
| `semantic` | Finds records with similar meaning. |
| `filter` | Restricts results by field values. |
| `aggregate` | Groups or counts field values. |
| `sort` | Orders results by a field. |
| `return_fields` | Includes stored fields in retrieval results; it is a retrieval option rather than an index field group. |

***

## Spoken words: `spoken_words`

Default artifact name: `transcript`

```json theme={null}
{
  "name": "transcript",
  "type": "spoken_words",
  "status": "done",
  "metadata": {},
  "scenes": [
    {
      "scene_id": "scene-0.000-11.762",
      "start": 0.0,
      "end": 11.762,
      "data": {
        "text": "In 1941, in German occupied Poland, the Nazis",
        "words": [
          {
            "text": "In",
            "start": 7.04,
            "end": 7.2,
            "confidence": 0.99560547,
            "speaker": "A"
          },
          {
            "text": "1941,",
            "start": 7.2,
            "end": 8.4,
            "confidence": 0.99902,
            "speaker": "A"
          }
        ]
      },
      "metadata": {}
    }
  ]
}
```

Useful indexing fields:

| Field | Use for |
| - | - |
| `text` | semantic search, retrieval with `return_fields` |
| `words.speaker` | filter, aggregate |
| `words.confidence` | filter, sort |
| `words` | retrieval with `return_fields` |

Each item in `words` contains word-level timing and may include `confidence` and `speaker` values.

***

## Objects: `object_detection`

Default artifact name: `objects`

```json theme={null}
{
  "name": "objects",
  "type": "object_detection",
  "status": "done",
  "metadata": {},
  "scenes": [
    {
      "scene_id": "scene-0.000-8.634",
      "start": 0.0,
      "end": 8.634,
      "data": {
        "frames": [
          {
            "frame_id": "frame-scene-0.000-8.634-0.000",
            "timestamp": 0.0,
            "detections": []
          },
          {
            "frame_id": "frame-scene-0.000-8.634-1.000",
            "timestamp": 1.0,
            "detections": [
              {
                "label": "stop sign",
                "score": 0.6835,
                "box": {
                  "box": [0.2836, 0.378, 0.6965, 0.5823],
                  "unit": "normalized"
                }
              }
            ]
          }
        ]
      },
      "metadata": {}
    }
  ]
}
```

A frame can have an empty `detections` list. Each detection contains a label, confidence score, and bounding-box object. The inner `box` array contains the coordinates, while `unit` describes their coordinate system.

Useful indexing fields:

| Field | Use for |
| - | - |
| `frames.detections.label` | filter, aggregate |
| `frames.detections.score` | filter, sort |
| `frames.detections.box.box` | stored bounding-box coordinates for overlays |
| `frames.detections.box.unit` | stored coordinate unit, such as `normalized` |
| `frames` | complete stored frame detail for retrieval and debugging |

***

## OCR: `ocr`

Default artifact name: `ocr`

`ocr` extracts visible text from frames, such as signs, slides, labels, captions, and UI text.

```json theme={null}
{
  "name": "ocr",
  "type": "ocr",
  "status": "done",
  "metadata": {
    "model": "default",
    "prompt_version": "ocr_v1",
    "schema_version": "v1"
  },
  "scenes": [
    {
      "scene_id": "scene-000001",
      "start": 0.0,
      "end": 3.42,
      "data": {
        "text": "EXIT 25% OFF"
      },
      "metadata": {}
    }
  ]
}
```

Useful indexing fields:

| Field | Use for |
| - | - |
| `text` | semantic search, retrieval with `return_fields` |

***

## Brands: `brand_detection`

Default artifact name: `brands`

`brand_detection` extracts logos, brand names, product labels, and sponsorships.

```json theme={null}
{
  "name": "brands",
  "type": "brand_detection",
  "status": "done",
  "metadata": {
    "model": "default",
    "prompt_version": "brand_detection_v1",
    "schema_version": "v1"
  },
  "scenes": [
    {
      "scene_id": "scene-000002",
      "start": 3.42,
      "end": 7.10,
      "data": {
        "brand_names": ["Nike"]
      },
      "metadata": {}
    }
  ]
}
```

Useful indexing fields:

| Field | Use for |
| - | - |
| `brand_names` | filter, aggregate, retrieval with `return_fields` |

***

## Activity: `activity_recognition`

Default artifact name: `activity`

`activity_recognition` extracts actions and activities.

```json theme={null}
{
  "name": "activity",
  "type": "activity_recognition",
  "status": "done",
  "metadata": {
    "model": "default",
    "prompt_version": "activity_recognition_v1",
    "schema_version": "v1"
  },
  "scenes": [
    {
      "scene_id": "scene-000002",
      "start": 3.42,
      "end": 7.10,
      "data": {
        "activity": "checking phone"
      },
      "metadata": {}
    }
  ]
}
```

Useful indexing fields:

| Field | Use for |
| - | - |
| `activity` | semantic search, filter, aggregate, retrieval with `return_fields` |

***

## Location: `location_detection`

Default artifact name: `location`

`location_detection` extracts setting and place context.

```json theme={null}
{
  "name": "location",
  "type": "location_detection",
  "status": "done",
  "metadata": {
    "model": "default",
    "prompt_version": "location_detection_v1",
    "schema_version": "v1"
  },
  "scenes": [
    {
      "scene_id": "scene-000003",
      "start": 7.10,
      "end": 12.0,
      "data": {
        "location": "outdoor street outside a retail shop in daylight"
      },
      "metadata": {}
    }
  ]
}
```

Useful indexing fields:

| Field | Use for |
| - | - |
| `location` | semantic search, filter, aggregate, retrieval with `return_fields` |

***

## Scene: `vlm`

Default artifact name: `scene`

Here, `scene` is the default artifact name for the `vlm` analyzer. Its output can contain multiple timestamped scene records.

Use `vlm` when you want general scene descriptions or a custom prompt/schema. Without a custom schema, each scene's description is returned in `data.text`:

```json theme={null}
{
  "name": "scene",
  "type": "vlm",
  "status": "done",
  "metadata": {},
  "scenes": [
    {
      "scene_id": "scene-120.000-180.000",
      "start": 120.0,
      "end": 180.0,
      "data": {
        "text": "The scene is a paper-cutout style animation set against a textured wall and a long wooden conference table."
      },
      "metadata": {}
    }
  ]
}
```

When you provide a `schema`, the fields inside `data` follow that schema instead. For example:

```json theme={null}
{
  "scene_description": "string",
  "activity": "string",
  "setting": "string"
}
```

produces scene data shaped like:

```json theme={null}
{
  "scene_description": "A man stands near a parked car while looking at his phone.",
  "activity": "checking phone",
  "setting": "outdoor street"
}
```

Useful indexing fields:

| Field | Use for |
| - | - |
| `text` | semantic search, retrieval with `return_fields` |
| fields from your custom schema | depends on the field type and your retrieval needs |

> [!NOTE]
>
> For object labels, brand names, OCR text, activities, or location signals, use the dedicated analyzers unless your VLM prompt/schema explicitly emits those fields.

***

## How outputs map to indexes

When you [index an artifact](/pages/understand/indexing-pipelines/create-an-index), choose which fields become retrieval fields.

A scene is the retrieval unit. Some artifacts have one scene-level value, such as `activity`, `location`, or OCR `text`. Others contain repeated values, such as `brand_names` or `frames[].detections[]`.

When indexing repeated values:

* one scene can emit multiple filter or aggregate signals
* the index may flatten those signals internally, such as one row per brand, object label, or detection
* retrieval still returns the scene with its `scene_id`, `start`, and `end`

```python theme={null}
scene = understanding.get_analyzer("scene")
objects = understanding.get_analyzer("objects")

video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["scene_description"],
    },
)

video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["frames.detections.label", "frames.detections.score"],
        "aggregate": ["frames.detections.label"],
    },
)
```

General rule:

| Output field shape | Use as |
| - | - |
| descriptions, summaries, transcript text | `semantic` |
| labels, names, enums, booleans | `filter`, `aggregate` |
| confidence scores and other numeric fields | `filter`, `aggregate`, `sort` |
| frames, boxes, raw responses, detailed arrays | requested at retrieval time with `return_fields` |

***

## Next steps

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Turn artifacts into searchable indexes.

> **[Index Fields](/pages/understand/indexing-pipelines/multiple-indexes)**
>
> Choose which fields become searchable, filterable, aggregatable, or sortable.


# Create an Index
Source: https://docs.videodb.io/pages/understand/indexing-pipelines/create-an-index

Turn understanding artifacts into semantic, queryable, and aggregatable indexes.

After you create an understanding, you can turn its artifacts into indexes.

An index stores structured information extracted from the video, including transcript text, scene descriptions, object labels, brands, timestamps, and custom fields, in a retrieval-ready form. Your app or agent can then search, filter, query, and aggregate over that information instead of re-analyzing the video.

```text theme={null}
Understanding → Artifact → Index → search/query/aggregate
```

Understanding and indexing are separate. Understanding extracts reusable facts from the video. Indexing decides which of those facts should become searchable, filterable, aggregatable, or available as result context. You can create multiple indexes from the same artifact for different retrieval needs.

> **[Try the indexing notebooks](https://github.com/video-db/videodb-cookbook/tree/preview/guides/indexing-v2/indexing)**
>
> Explore runnable examples for turning understanding artifacts into indexes.

***

## Quick example

```python theme={null}
# Create artifacts
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words"},
        {"type": "vlm"},
    ],
)

understanding.wait_until_complete()

transcript = understanding.get_analyzer("transcript")
scene = understanding.get_analyzer("scene")

# Index artifacts with defaults
video.index(source=transcript)
video.index(source=scene)
```

VideoDB derives sensible defaults from your data. If `name` is omitted, VideoDB uses the artifact name.

***

## Index sources

`source` accepts three forms:

```python theme={null}
# 1. An analyzer object is the lightest path. Only identifiers travel to the server;
#    the server fetches the analyzer's output from its own store.
analyzer = understanding.get_analyzer("scene")
video.index(name="scene", source=analyzer)

# 2. A reference dict works the same way when you only have the ids.
video.index(
    name="scene",
    source={"understanding_id": "und_123", "analyzer_id": "an_456"},
)

# 3. Your own temporal records.
video.index(name="chapters", source=custom_scenes)
```

Pass analyzer outputs using forms 1 and 2 so the scene payload never round-trips through your client. For custom records, see the [user-provided temporal record format](#index-user-provided-temporal-records).

## Choose an index name

```python theme={null}
location_analyzer = understanding.get_analyzer("location")

index = video.index(
    name="location",
    source=location_analyzer,
)
```

> [!NOTE]
>
> Use the same index name to group the same kind of data across videos in a collection. For example, name the location index on every video `"location"`; you can then target that logical group with `collection.semantic_search(index_names=["location"])`, `collection.query(index_name="location")`, or `collection.aggregate(index_name="location")`.
>
> Indexes that share a name must also share the same field structure. This keeps collection-level retrieval consistent because `"location"` always represents the same fields. If you create another `"location"` index with a different structure, VideoDB returns a `400` showing the existing and incoming structures. See [Collection Search](/pages/understand/search-and-retrieval/collection-search) for index-specific retrieval across videos.

## Index all artifacts

```python theme={null}
for analyzer in understanding.list_analyzers():
    if analyzer.status == "done":
        video.index(name=analyzer.name, source=analyzer)
```

***

## Declare index capabilities with `use_for`

Use `use_for` to say which retrieval capabilities the index should support.

```python theme={null}
video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query", "aggregate"],
)
```

| Value | Meaning | Example |
| - | - | - |
| `semantic` | direct semantic/vector search | "happy reunion outside a shop" |
| `query` | exact structured filtering | `object_labels contains "phone"` |
| `aggregate` | counts, groups, facets, analytics | count scenes by brand |

> [!NOTE]
>
> `semantic` describes an index capability. Use `semantic_search()` for direct vector similarity over semantic fields. Use `search()` for high-level natural-language search that may combine semantic indexes with filters and other signals.

If `use_for` is omitted, the index defaults to `["semantic", "query", "aggregate"]` — semantic search works out of the box. The semantic fields come from the derived defaults below; if nothing qualifies as a default, **all string fields are embedded**. Pass `use_for` without `semantic` (e.g. `["query"]`) to opt out of embeddings. `query` and `aggregate` are interchangeable capabilities.

The `use_for` that `get_index()` reports back is the index's **effective** capability, not a verbatim echo of the request: `aggregate` is present only when a field is actually groupable, `semantic` only when a field was embedded, and an index with no records reports `[]`. This keeps `list_indexes()` from advertising a capability a call would then reject.

***

## Default fields derived from your data

`fields` is optional, **per group**: any group you declare is used verbatim, and any group you omit is filled from your actual scene data.

Well-known field names get product defaults:

| Field / dotted path | Default groups |
| - | - |
| `text`, `scene_description` | `semantic` |
| `language`, `brand_names` | `filter`, `aggregate` |
| `activity`, `location` | `semantic`, `filter`, `aggregate` |
| `frames.detections.label` | `filter`, `aggregate` |
| `frames.detections.score` | `filter`, `sort` |

Every other field is classified by shape:

| Value shape | Default groups |
| - | - |
| prose strings (descriptions, summaries, transcripts) | `semantic`, plus `filter`/`aggregate` so everything stays queryable |
| short strings, booleans, lists of strings/numbers | `filter`, `aggregate` |
| numbers | `filter`, `aggregate`, `sort` |
| nested dicts, arrays of objects (frames, boxes, raw responses) | none; stored with the record and available via `return_fields` |

Two rules to remember:

* `semantic` defaults apply when `use_for` includes `semantic` or is omitted (semantic is on by default); if nothing qualifies, all string fields are embedded.
* An explicitly empty group opts out and is **not** refilled: `fields={"filter": []}` means "no filter fields".

```python theme={null}
# No fields config — the server derives from your data. A scene/VLM analyzer emits a
# nested `outputs` object, so each string leaf becomes its own dotted semantic field
# (nested objects are not auto-filtered/aggregated — declare `fields` for that):
index = video.index(source=analyzer, use_for=["semantic", "query"])
print(index.fields)
# {"semantic": ["outputs.scene_description", "outputs.caption", "outputs.action", ...]}
```

***

## Configure fields

Use `fields` when you want explicit control over which artifact fields become semantic text, text fields, filters, aggregates, or sort keys.

Field names may be **dotted paths into nested data**. For example, `frames.detections.label` reaches inside every frame's detection list. See [Index Fields](/pages/understand/indexing-pipelines/multiple-indexes) for the full pattern.

Fields must exist on the artifact you are indexing (dotted paths are resolved against the scene data; a name that resolves nowhere fails the create with a `400` listing the available fields). Scene descriptions, object detections, and brand detections usually come from separate artifacts, so index them separately:

```python theme={null}
# Scene/VLM artifact: natural-language scene fields
video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query", "aggregate"],
    fields={
        "semantic": ["scene_description", "activity", "setting"],
        "filter": ["activity", "setting"],
        "aggregate": ["activity", "setting"],
    },
)

# Object detection artifact: dotted paths reach the nested detections
video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["frames.detections.label", "frames.detections.score"],
        "aggregate": ["frames.detections.label"],
        "sort": ["frames.detections.score"],
    },
)

# Brand detection artifact, if generated: brand names and logo signals
brands = understanding.get_analyzer("brands")

video.index(
    name="brands",
    source=brands,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["brand_names"],
        "aggregate": ["brand_names"],
    },
)
```

Use the `objects` and `brands` artifacts as the source of truth for object labels and brand names. Keep the `scene` index focused on the fields produced by your VLM prompt/schema.

| Field group | Used for |
| - | - |
| `semantic` | direct vector/semantic search with `semantic_search()`; also usable by high-level `search()`. Answers "find talk **about** X" (meaning). |
| `filter` | structured `query()` filters (exact match, `contains` substring, ranges) |
| `aggregate` | `aggregate()` group-by/count/facet operations |
| `sort` | result ordering |

VideoDB still stores additional artifact fields with indexed records. Use `return_fields` during search to choose which stored fields, such as frames or raw response references, should come back in results.

See [Index Fields](/pages/understand/indexing-pipelines/multiple-indexes) for details.

***

## Index user-provided temporal records

You can also index your own records if they include timestamps. Pass one record dictionary or a list of record dictionaries. VideoDB moves every field other than the reserved fields below into the record's indexable `data`.

| Field | Type | Required | Description |
| - | - | -: | - |
| `start` | number | yes | Start time in seconds. |
| `end` | number | yes | End time in seconds; it must be later than `start`. |
| `scene_id` | string | no | Stable record identifier. VideoDB generates one when omitted. |
| `metadata` | dict | no | Additional scene metadata. |
| any other field | [supported field type](/pages/understand/indexing-pipelines/multiple-indexes#supported-field-value-types) | no | Custom data you can configure for semantic search, filtering, aggregation, sorting, or retrieval. |

Index field paths refer to custom fields directly. For example, use `summary`, not `data.summary`.

```python theme={null}
custom_scenes = [
    {
        "start": 0.0,
        "end": 12.4,
        "summary": "Opening credits with city skyline.",
        "scene_type": "intro",
    },
    {
        "start": 12.4,
        "end": 45.0,
        "summary": "CEO discusses Q4 results.",
        "scene_type": "presentation",
    },
]

video.index(
    name="chapters",
    source=custom_scenes,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["summary"],
        "filter": ["scene_type"],
    },
)
```

***

## Index artifacts from sandbox models

Sandbox Compute affects where the analyzer runs. The indexing interface stays the same.

If an artifact was produced by a sandbox-backed analyzer, pass that artifact to `video.index(...)` exactly like a hosted-model artifact:

```python theme={null}
video.index(
    name="scene",
    source=scene,
    use_for=["semantic", "query"],
    fields={
        "semantic": ["scene_description", "activity", "setting"],
        "filter": ["activity", "setting"],
    },
)
```

Use the sandbox docs for sandbox creation, model selection, and `sandbox_id` validation. Use this page for the artifact-to-index step.

> **[Sandbox Compute](/pages/core-concepts/sandbox-compute)**
>
> Create a sandbox and route supported model jobs to it.

> **[Sandbox Models](/pages/core-concepts/sandbox-models)**
>
> See supported models, tiers, and where to pass model names.

***

## `video.index()` options

| Option | Type | Required | Default | Description |
| - | -: | -: | - | - |
| `source` | analyzer, dict, list | yes | — | Analyzer object, `{understanding_id, analyzer_id}` reference, artifact output dict, or user-provided temporal records. |
| `name` | string | no | artifact/source name | User-facing index name (a per-collection schema contract — see above). |
| `use_for` | list | no | `["semantic", "query", "aggregate"]` | Any of `semantic` (`search` is accepted as an alias), `query`, `aggregate`. The value `get_index()` returns is the index's *effective* capability — see the note above. |
| `fields` | dict | no | derived per group from your data | Field-level indexing configuration; dotted paths supported. |
| `callback_url` | string | no | none | URL called when indexing completes. |

***

## Index status and inspection

Everything is validated synchronously — bad fields, missing sources, and name-contract conflicts fail the create call immediately with a descriptive `400`. What runs asynchronously is the embedding work: every semantic index returns as `building` and flips to `ready` when its vectors are written — typically within seconds for small indexes. Query-only indexes (no semantic embedding) return `ready` immediately.

Indexing is **rows-first**: the structured records are stored before the create call returns, so `query()` and `aggregate()` work on a `building` index right away. `semantic_search()` serves an index only once it is `ready`.

Because the build is asynchronous, the `Index` object has lifecycle helpers so you do not have to poll `get_index()` by hand:

| Method / property | Purpose |
| - | - |
| `index.wait_until_complete(timeout=1800, poll_interval=10)` | Block until the build reaches a terminal status (`ready` or `failed`); raises `TimeoutError` if it does not finish in `timeout` seconds. |
| `index.refresh()` | Re-fetch the manifest and status once, without waiting. |
| `index.is_complete` | `True` when the status is terminal (`ready` or `failed`). |
| `index.is_successful` | `True` only when the status is `ready`. |

```python theme={null}
index = video.index(name="scene", source=analyzer, use_for=["semantic", "query"])

# Wait for embeddings to finish (query-only indexes are already ready and return at once):
index.wait_until_complete(timeout=1800, poll_interval=10)

if index.is_successful:
    print(index.status, index.use_for, index.fields)
else:
    print("indexing failed:", index.error)
```

Common statuses:

| Status | Meaning |
| - | - |
| `building` | Rows are stored and queryable; embeddings are still being written. |
| `ready` | Fully ready, including semantic search. |
| `failed` | Indexing failed; the manifest's `error` carries the reason. |

Use `refresh()` to fetch the latest manifest and update the existing index object:

```python theme={null}
index.refresh()
print(index.status)
```

Use `wait_until_complete()` when your next operation requires a fully built index, such as semantic search. It polls until the index reaches a terminal status: `ready` or `failed`.

```python theme={null}
index.wait_until_complete(timeout=3600, poll_interval=15)

if index.is_successful:
    print("Index is ready for semantic search")
else:
    print(index.error)
```

The default timeout is 1,800 seconds and the default polling interval is 10 seconds. If the timeout expires first, `wait_until_complete()` raises `TimeoutError`. A failed build is a completed build, so the method returns the index with `status == "failed"`; inspect `index.error` for the reason.

Use `video.list_indexes()` and `video.get_index(...)` to reopen or inspect index manifests, field schemas, and record previews. See [View Indexes](/pages/understand/indexing-pipelines/view-indexes).

***

## Next steps

> **[Index Fields](/pages/understand/indexing-pipelines/multiple-indexes)**
>
> Learn how fields become semantic text, filters, facets, and sort keys.

> **[View Indexes](/pages/understand/indexing-pipelines/view-indexes)**
>
> Inspect index status, fields, schema, and records.


# Index Fields
Source: https://docs.videodb.io/pages/understand/indexing-pipelines/multiple-indexes

Choose which artifact fields become semantic text, filters, aggregates, and sort keys.

`fields` controls how an artifact becomes an index.

An artifact may contain many fields. VideoDB stores the additional artifact data with the indexed records, but only selected fields are optimized for retrieval capabilities:

* direct semantic search
* exact/text matching
* structured filters
* aggregation
* sorting

Use `return_fields` at search time to choose which stored fields should come back in results.

***

## Example artifact

A VLM scene artifact may contain scene records like this:

```json theme={null}
{
  "scene_id": "scene-000001",
  "start": 12.4,
  "end": 18.9,
  "data": {
    "scene_description": "A person walks through a retail aisle while holding a phone.",
    "activity": "walking through store",
    "setting": "retail aisle",
    "frames": [{"timestamp": 14.2, "asset": {"type": "s3_object", "key": "..."}}]
  }
}
```

A `fields` config tells VideoDB how to index values that exist on that artifact:

```python theme={null}
fields={
    "semantic": ["scene_description", "activity", "setting"],
    "filter": ["activity", "setting"],
    "aggregate": ["activity", "setting"],
}
```

The `frames` field is still stored with the record. Request it later with `return_fields` when you need it in search results. Object labels and brand names usually live on separate `objects` and `brands` artifacts.

***

## Dotted paths for nested data

Field names may be **dotted paths** that reach inside nested objects, so you do not need to flatten your data. When a path crosses a list, values are collected from **every element**.

An object-detection scene looks like this:

```json theme={null}
{
  "scene_id": "scene-0.000-8.634",
  "start": 0.0,
  "end": 8.634,
  "data": {
    "frames": [
      {
        "frame_id": "frame-scene-0.000-8.634-7.000",
        "timestamp": 7.0,
        "detections": [
          {
            "label": "person",
            "score": 0.5856,
            "box": {
              "box": [0.0002, 0.2332, 0.6967, 0.9947],
              "unit": "normalized"
            }
          },
          {
            "label": "dog",
            "score": 0.5041,
            "box": {
              "box": [0.0018, 0.2346, 0.6908, 0.7378],
              "unit": "normalized"
            }
          }
        ]
      }
    ]
  },
  "metadata": {}
}
```

Index the nested values directly:

```python theme={null}
video.index(
    name="objects",
    source=objects,
    use_for=["query", "aggregate"],
    fields={
        "filter": ["frames.detections.label", "frames.detections.score"],
        "aggregate": ["frames.detections.label"],
        "sort": ["frames.detections.score"],
    },
)
```

`frames.detections.label` becomes the list of every detection label in the scene (`["person", "dog"]`); `frames.detections.score` becomes the list of every score. Bounding boxes remain nested under `frames[].detections[].box`, and the raw `frames` structure stays stored-only; request the objects index data with `return_fields` when you need timestamps or boxes.

Dotted paths work everywhere a field name does: `fields` groups, `query()` filters, `aggregate(group_by=...)`, and targeted semantic refs (`index_names=["scene.outputs.location"]`).

***

## Field groups

| Group | Values | Used for |
| - | - | - |
| `semantic` | list of text-like field names | direct semantic/vector search with `semantic_search()`; also usable by high-level `search()` |
| `filter` | list of field names | structured `query()` filters |
| `aggregate` | list of field names | `aggregate()` count/group/facet operations |
| `sort` | list of field names | result ordering |

> [!NOTE]
>
> Do not add `clip` to `fields`. Clips are generated automatically from `video_id`, `start`, and `end` for moment results. Use `include_clip=False` at retrieval time if you do not need playable clips.

***

## Supported field value types

| Type | Example | Common use |
| - | - | - |
| `string` | `"Nike"` | filter, aggregate |
| `text` | `"A person walks through a store."` | semantic/text retrieval |
| `number` | `0.94` | filter, sort, aggregate |
| `boolean` | `true` | filter, aggregate |
| `string_array` | `["person", "phone"]` | contains filters, facets (an array of objects is reported as `string_array`) |
| `number_array` | `[0.91, 0.88]` | advanced filters, returned fields |
| `object` | `{...}` | stored record detail, returned fields; index inner values with dotted paths |

Nested values you filter or aggregate on should be declared with their dotted path. Declaring them projects the values into the queryable store at ingest time, so retrieval stays fast.

***

## Default fields

`fields` is optional per group. Any group you omit is **derived from your data**: well-known names get product defaults (`text`/`scene_description` → semantic, `language`/`brand_names` → filter + aggregate, `frames.detections.label` → filter + aggregate, `frames.detections.score` → filter + sort), and everything else is classified by shape — prose becomes semantic, scalars and scalar lists become filter + aggregate (numbers also sort), and nested structures stay stored-only.

Groups you declare are used verbatim, and an explicit empty group (`"filter": []`) opts out. See [Create an Index](/pages/understand/indexing-pipelines/create-an-index#default-fields-derived-from-your-data) for the full derivation tables.

```python theme={null}
# Declare only what you want to control; the rest is derived:
video.index(
    name="scene",
    source=analyzer,
    use_for=["semantic", "query"],
    fields={"semantic": ["outputs.description"]},  # filter/aggregate still derived
)
```

***

## User-provided temporal records

If you index your own timestamped records, choose fields the same way:

```python theme={null}
fields={
    "semantic": ["summary"],
    "filter": ["scene_type"],
    "aggregate": ["scene_type"],
}
```

***

## Return fields at retrieval time

Indexing does not decide the result payload. Search does.

Use `return_fields` to choose which stored fields to include in each result:

```python theme={null}
results = video.semantic_search(
    query="happy reunion outside a shop",
    index_names=["scene"],
    return_fields=["scene_description", "activity", "frames"],
)
```

For multi-index search, namespace fields by index name:

```python theme={null}
results = collection.search(
    query="person talking about Nike while holding a phone",
    return_fields={
        "scene": ["scene_description", "frames"],
        "transcript": ["text"],
        "objects": ["object_labels", "frames"],
        "brands": ["brand_names"],
    },
)
```

For debugging or inspection, request all stored fields:

```python theme={null}
results = video.search(
    query="product demo moments",
    return_fields="all",
)
```

Returned fields can make responses larger. Keep them narrow for broad searches and request detailed fields only when you need to display or inspect them.

***

## Filter syntax

Query filters use fields that were indexed in `fields["filter"]`, including dotted paths.

| Filter type | Example |
| - | - |
| equality | `{"activity": "walking through store"}` |
| contains | `{"frames.detections.label": {"contains": "phone"}}` |
| numeric range | `{"frames.detections.score": {">=": 0.9}}` |
| boolean | `{"is_song": True}` |
| multiple conditions | `{"activity": "walking", "setting": "retail aisle"}` |

How filters behave on **list-valued fields** (anything a dotted path collected across a list, or a plain array field):

* equality means element membership: `{"frames.detections.label": "phone"}` matches a scene where **any** detection is a phone
* numeric comparators match **any element**: `{"frames.detections.score": {">=": 0.9}}` matches if any detection scores ≥ 0.9
* `contains` is a per-element substring match: `"car"` matches `["car", "truck"]` but not `["cartoon"]`

***

## Aggregation syntax

Aggregation uses fields that were indexed in `fields["aggregate"]`.

```python theme={null}
collection.aggregate(
    index_name="brands",
    group_by="brand_names",
)

# Dotted paths group by nested values, with one group per detection label
collection.aggregate(
    index_name="objects",
    group_by="frames.detections.label",
    metric="count",
)
```

List-valued fields explode to one group per element (each detection label counts separately). Dict-valued fields (`{"person": 3, "phone": 1}`) explode to one group per key, and numeric metrics (`sum`/`avg`/`min`/`max`) apply to the dict's values.

Common metrics:

| Metric | Meaning |
| - | - |
| `count` | count matching scenes/records |
| `sum(field)` | sum numeric field values |
| `avg(field)` | average numeric field values |
| `min(field)` | minimum value |
| `max(field)` | maximum value |

***

## Best practices

1. Put natural-language descriptions in `fields["semantic"]`.
2. Put exact labels, enums, numbers, and booleans in `fields["filter"]`.
3. Put fields you want to count or facet in `fields["aggregate"]`.
4. Put fields you want to order by in `fields["sort"]`.
5. Keep indexing focused on retrieval capabilities; request display/debug details with `return_fields` during search.
6. Reach into nested data with dotted paths (`frames.detections.label`) instead of reshaping your artifacts. Declare the paths in `fields` so the values are projected for fast retrieval.

***

## Next steps

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Create indexes from artifacts.

> **[View Indexes](/pages/understand/indexing-pipelines/view-indexes)**
>
> Inspect fields, schemas, and records after indexing.


# Understanding Artifacts
Source: https://docs.videodb.io/pages/understand/indexing-pipelines/understanding-artifacts

Turn videos into structured artifacts for search, retrieval, and analysis.

Understanding is the first step in the retrieval pipeline.

```text theme={null}
Understand → Index → Retrieve
```

An **understanding run** applies one or more **analyzers** to a video and creates reusable **artifacts** such as transcripts, object detections, OCR text, brand signals, activity/location signals, or VLM scene descriptions.

You usually start with defaults, then add configuration only when you need more control.

> **[Try the understanding notebooks](https://github.com/video-db/videodb-cookbook/tree/preview/guides/indexing-v2/understanding)**
>
> Explore runnable examples for creating and inspecting understanding artifacts.

***

## Start simple

For most use cases, only provide analyzer types.

```python theme={null}
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words"},
        {"type": "vlm"},
    ],
)

understanding.wait_until_complete()

transcript = understanding.get_analyzer("transcript").get_output()
scene = understanding.get_analyzer("scene").get_output()

print(scene["scenes"][0]["data"])
```

That is enough for a first run. VideoDB automatically chooses the output names, video chunking, frame selection, and default analyzer settings.

Each analyzer type has a default artifact name: `spoken_words` → `transcript`, `object_detection` → `objects`, and `vlm` → `scene`. That name is what you pass to `get_analyzer()`. Override it with `name`. See [Analyzer `type`](#analyzer-type-choose-the-artifact) for the full list.

***

## What should I use?

VideoDB includes built-in analyzers for common video and audio signals.

Use the table below to choose an analyzer based on what you want to retrieve.

| Analyzer | Use when you want to... | Produces |
| - | - | - |
| `spoken_words` | Search spoken words, lectures, meetings, podcasts | Timed transcript text for semantic and full-text search. |
| `vlm` | Search what is visible or happening in the video | Natural-language scene descriptions. |
| `object_detection` | Filter, count, or locate objects | Object labels, counts, confidence scores, and boxes. |
| `ocr` | Search text visible on screen | Signs, slides, labels, captions, and UI text. |
| `brand_detection` | Find logos or brands | Brand names, logos, product labels, and sponsorships. |
| `activity_recognition` | Detect actions or activities | Activity and behavior signals. |
| `location_detection` | Detect setting or location | Place, setting, and scene context. |
| `object_detection` + `spoken_words` + `ocr` + `vlm` | Build the richest scene understanding | Separate object, speech, visible-text, and scene-description artifacts. |

> [!NOTE]
>
> `ocr`, `brand_detection`, `activity_recognition`, and `location_detection` are VLM-backed analyzers with task-specific output shapes. Use the generic `vlm` analyzer when you want to provide your own prompt or schema.

***

## The mental model

An analyzer produces one named artifact. Stored artifacts can be inspected and reused independently.

| Step | What happens | Example |
| - | - | - |
| Choose analyzers | Decide which signals you want to extract | `spoken_words`, `vlm` |
| Run understanding | Create artifacts from the video | `video.understand(analyzers=[...])` |
| Inspect outputs | Fetch artifact output from an analyzer | `understanding.get_analyzer("scene").get_output()` |
| Refine | Add configuration only when defaults are not enough | `name`, `sampling`, `config` |

***

## Inspect and manage runs

`video.understand(...)` returns an understanding run object with an `id`, `status`, and analyzer handles.

```python theme={null}
print(understanding.id, understanding.status)

understanding.refresh()

for analyzer in understanding.list_analyzers():
    print(analyzer.name, analyzer.type, analyzer.status)
```

Use `wait_until_complete()` before fetching analyzer outputs or indexing artifacts. Run-level terminal statuses are `done` and `failed`.

```python theme={null}
understanding.wait_until_complete(timeout=3600, poll_interval=15)
scene = understanding.get_analyzer("scene").get_output()
```

You can also reopen or list previous runs for the same video:

```python theme={null}
same_understanding = video.get_understanding(understanding.id)

for item in video.list_understandings():
    print(item.id, item.status)
```

> [!NOTE]
>
> Only delete an understanding run when you no longer need its stored artifacts.

```python theme={null}
understanding.delete()
```

***

## How `video.understand()` is structured

`video.understand()` has two levels of configuration:

1. **run-level options** apply to the whole understanding run
2. **analyzer options** apply to one item inside `analyzers`

```python theme={null}
understanding = video.understand(
    segmentation={...},      # run-level
    transform={...},         # run-level
    analyzers=[
        {
            "type": "vlm",   # analyzer-level
            "name": "scene",
            "inputs": [...],
            "sampling": {...},
            "config": {...},
        }
    ],
)
```

***

## Run-level options

Run-level options belong directly on `video.understand(...)` and affect the whole understanding run.

| Option | Type | Required | Default | Use when... |
| - | -: | -: | - | - |
| `analyzers` | list | yes | n/a | You want to define what artifacts to create. |
| `segmentation` | dict | no | `{"type": "shot", "threshold": 30}` | You need control over visual time ranges for visual analyzers. |
| `transform` | dict | no | none | You want to resize or preprocess media for the run. |
| `callback_url` | string | no | none | You want async completion notifications. |

### `segmentation`: visual time ranges

`segmentation` controls how VideoDB splits the video into scenes, which are the timestamped ranges that visual analyzers run on and attach their outputs to. Use `time` for fixed-length ranges, or `shot` to split on the video's own scene changes.

| Segmentation type | Options | Best for |
| - | - | - |
| `time` | `seconds` | fixed-length time ranges, live streams, surveillance, long unedited videos |
| `shot` | `threshold` | edited videos with cuts, films, ads, sports clips, trailers |

Time-based segmentation:

```python theme={null}
segmentation={"type": "time", "seconds": 5}
```

| Option | Values | Description |
| - | - | - |
| `type` | `"time"` | Fixed-duration scenes. |
| `seconds` | number | Scene length in seconds. |

Shot-based segmentation:

```python theme={null}
segmentation={"type": "shot", "threshold": 30}
```

| Option | Values | Description |
| - | - | - |
| `type` | `"shot"` | Scenes are based on visual scene changes. |
| `threshold` | number | Sensitivity for detecting cuts; higher values usually create fewer splits. |

### `transform`: media preparation

`transform` applies preprocessing to the media before analyzers run, such as resizing the frames to reduce cost.

| Option | Values | Use when... |
| - | - | - |
| `resolution` | string such as `"480p"`, `"720p"` | You want to reduce cost or latency. |

Example:

```python theme={null}
understanding = video.understand(
    analyzers=[{"type": "vlm"}],
    transform={"resolution": "480p"},
)
```

***

## Analyzer options

Each item in `analyzers` defines one analyzer and the artifact it should produce.

| Field | Type | Required | Default | Use when... |
| - | -: | -: | - | - |
| `type` | string | yes | n/a | You choose what analysis to run. |
| `name` | string | no | type-specific default | You need a custom output key or multiple analyzers of the same type. |
| `inputs` | list of strings | no | none | One analyzer should use artifacts from earlier analyzers. VLM prompts can reference them with `{{inputs.<name>}}`. |
| `sampling` | dict | no | type-specific default | Visual analyzers need specific frame selection. |
| `config` | dict | no | type-specific default | You need model, prompt, or schema control. |

### Analyzer `type`: choose the artifact

`type` chooses which analyzer runs and which artifact shape is produced.

| Type | Default artifact name | Produces |
| - | - | - |
| `spoken_words` | `transcript` | timed spoken text |
| `object_detection` | `objects` | object labels, counts, scores, boxes |
| `vlm` | `scene` | scene descriptions and structured visual reasoning |
| `ocr` | `ocr` | visible text |
| `brand_detection` | `brands` | brand/logo signals |
| `activity_recognition` | `activity` | activities/actions |
| `location_detection` | `location` | setting/location signals |

### Analyzer `name`: output keys

`name` controls the analyzer name used with `get_analyzer(name)` and `inputs`. It is optional.

If you omit it, VideoDB uses the default artifact name for the [analyzer type](#analyzer-type-choose-the-artifact).

You need `name` when you want to:

1. choose a stable analyzer name for output lookup
2. reference another analyzer through `inputs`
3. run multiple analyzers of the same type
4. create readable artifact names for downstream steps

Example with two VLM analyzers:

```python theme={null}
understanding = video.understand(
    analyzers=[
        {
            "type": "vlm",
            "name": "scene_summary",
            "config": {"prompt": "Summarize this scene."},
        },
        {
            "type": "vlm",
            "name": "safety_events",
            "config": {"prompt": "Detect unsafe behavior and hazards."},
        },
    ],
)

understanding.wait_until_complete()
summary = understanding.get_analyzer("scene_summary").get_output()
safety = understanding.get_analyzer("safety_events").get_output()
```

### Analyzer `inputs`: reuse earlier artifacts

`inputs` declares which earlier artifacts a VLM analyzer can use. Reference those artifacts in the VLM prompt with `{{inputs.<name>}}`, where `<name>` is the analyzer `name`.

```python theme={null}
{
    "type": "vlm",
    "name": "scene",
    "inputs": ["objects", "transcript", "ocr"],
    "config": {
        "prompt": (
            "Describe this scene using video frames as primary evidence.\n\n"
            "Spoken words:\n"
            "{{inputs.transcript}}\n\n"
            "Detected objects:\n"
            "{{inputs.objects}}\n\n"
            "Visible text:\n"
            "{{inputs.ocr}}"
        ),
    },
}
```

VideoDB injects compact, scene-aligned context for each referenced input. For example, `{{inputs.transcript}}` renders spoken text for the scene, while `{{inputs.objects}}` renders object labels/counts from the matching object artifact.

Rules:

* Input names must match earlier analyzer `name` values.
* A placeholder such as `{{inputs.transcript}}` must reference an input listed in `inputs`.
* If `inputs` are provided but the prompt has no `{{inputs.*}}` placeholders, VideoDB appends a compact input context block after the prompt.
* Treat injected inputs as evidence/context; write the prompt so frames remain the primary visual source when needed.

### Analyzer `sampling`: frame selection

Sampling is per analyzer. It mainly applies to visual and VLM-backed analyzers such as `object_detection`, `vlm`, `ocr`, `brand_detection`, `activity_recognition`, and `location_detection`.

| Strategy | Required fields | Best for | Example |
| - | - | - | - |
| `uniform` | `frame_count` | a fixed number of representative frames across each scene | `{"strategy": "uniform", "frame_count": 8}` |
| `interval` | `every` | monitoring motion or frequently changing scenes | `{"strategy": "interval", "every": 1}` |

`every` is in seconds. `{"strategy": "interval", "every": 1}` samples one frame per second within each scene. `frame_count` is the total number of frames distributed evenly inside each scene.

Use `{"strategy": "uniform", "frame_count": 1}` when you want one representative frame from the middle of every scene.

How to choose sampling:

| Content / goal | Recommended sampling |
| - | - |
| one representative frame per scene | `uniform` with `frame_count: 1` |
| general scene understanding | `uniform` with 4–8 frames |
| motion-heavy content | `uniform` with more frames |
| regular sampling within each scene | `interval` with the desired number of seconds |
| lower cost/latency | fewer frames or lower resolution |

### Analyzer `config`: models, prompts, and schemas

Use `"default"` or omit `model` unless you need a specific model. For understanding analyzers, model selection is primarily exposed for VLM-backed analyzers and object detection.

#### `spoken_words`

| Config | Values | Description |
| - | - | - |
| `language` | [supported language code](/pages/core-concepts/supported-languages), such as `en`, `hi`, `ja`, or `zh` | Optional language hint. |

#### `object_detection`

| Config | Values | Description |
| - | - | - |
| `model` | `default`, `rtdetr-v2-r50vd` | Object detection model. |
| `sandbox_id` | string | Required when `model` is a sandbox-backed object detection model; omit for hosted/default models. |

> **Supported object labels**
>
> The default object detection model supports the COCO 80-label vocabulary. Outputs use label strings such as `person`, `car`, `cell phone`, and `laptop`.
>
> `person`, `bicycle`, `car`, `motorbike`, `aeroplane`, `bus`, `train`, `truck`, `boat`, `traffic light`, `fire hydrant`, `stop sign`, `parking meter`, `bench`, `bird`, `cat`, `dog`, `horse`, `sheep`, `cow`, `elephant`, `bear`, `zebra`, `giraffe`, `backpack`, `umbrella`, `handbag`, `tie`, `suitcase`, `frisbee`, `skis`, `snowboard`, `sports ball`, `kite`, `baseball bat`, `baseball glove`, `skateboard`, `surfboard`, `tennis racket`, `bottle`, `wine glass`, `cup`, `fork`, `knife`, `spoon`, `bowl`, `banana`, `apple`, `sandwich`, `orange`, `broccoli`, `carrot`, `hot dog`, `pizza`, `donut`, `cake`, `chair`, `sofa`, `pottedplant`, `bed`, `diningtable`, `toilet`, `tvmonitor`, `laptop`, `mouse`, `remote`, `keyboard`, `cell phone`, `microwave`, `oven`, `toaster`, `sink`, `refrigerator`, `book`, `clock`, `vase`, `scissors`, `teddy bear`, `hair drier`, `toothbrush`

#### `vlm`

| Config | Values | Description |
| - | - | - |
| `model` | `default`, `mini`, `basic`, `pro`, or `ultra` | Hosted VLM model tier. |
| `prompt` | string | Instruction for what the VLM should extract. Can include `{{inputs.<name>}}` placeholders for declared `inputs`. |
| `schema` | object | Optional structured output shape. |

Supported hosted tiers:

| Tier | Use when... |
| - | - |
| `mini` | Lowest cost/latency. |
| `basic` | Default balanced option. |
| `pro` | Higher quality reasoning. |
| `ultra` | Highest quality tier. |

#### Specialized VLM analyzers

These analyzers use the same VLM model family as `vlm`, with task-specific output shapes.

| Analyzer | Extracts | Config options |
| - | - | - |
| `ocr` | visible text, signs, slides, labels, UI text | `model` |
| `brand_detection` | logos, brand names, product labels, sponsorships | `model` |
| `activity_recognition` | actions, activities, events, behavior | `model` |
| `location_detection` | setting, place type, indoor/outdoor, scene context | `model` |

| Config | Values | Description |
| - | - | - |
| `model` | `default`, `mini`, `basic`, `pro`, or `ultra` | Hosted VLM model tier used by the analyzer. |

Use the generic `vlm` analyzer instead if you want to control the prompt or schema directly.

> [!NOTE]
>
> Model availability can vary by workspace and deployment. If a model is omitted, VideoDB uses the current default for that analyzer.

***

## Putting it together

This example shows run-level options and analyzer-level options in the same understanding run.

```python theme={null}
understanding = video.understand(
    segmentation={
        "type": "shot",
        "threshold": 30,
    },
    transform={
        "resolution": "480p",
    },
    analyzers=[
        {
            "type": "object_detection",
            "name": "objects",
            "sampling": {
                "strategy": "interval",
                "every": 1,
            },
            "config": {
                "model": "rtdetr-v2-r50vd",
            },
        },
        {
            "type": "spoken_words",
            "name": "transcript",
            "config": {
                "language": "en",
            },
        },
        {
            "type": "vlm",
            "name": "scene",
            "inputs": ["objects", "transcript"],
            "sampling": {
                "strategy": "uniform",
                "frame_count": 8,
            },
            "config": {
                "model": "basic",
                "prompt": (
                    "Describe the scene and activity using the sampled frames, "
                    "spoken words, and detected objects.\n\n"
                    "Spoken words:\n"
                    "{{inputs.transcript}}\n\n"
                    "Detected objects:\n"
                    "{{inputs.objects}}"
                ),
                "schema": {
                    "scene_description": "string",
                    "activity": "string",
                    "setting": "string",
                },
            },
        },
    ],
    callback_url="https://example.com/webhooks/understanding-complete",
)
```

***

## Cost and quality tips

| Goal | What to adjust |
| - | - |
| lower cost | fewer analyzers, fewer sampled frames, lower `resolution` |
| lower latency | fewer frames, shorter scenes, lower model tier |
| better visual detail | more sampled frames, higher resolution, or a higher model tier |
| broader coverage | combine speech, object, OCR, and VLM-backed analyzers |
| more structured output | use analyzers with task-specific output shapes, or provide a VLM `schema` |

***

## Next steps

> **[Analyzer Outputs](/pages/understand/indexing-pipelines/analyzer-outputs)**
>
> See sample output shapes for each analyzer.

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Turn stored artifacts into retrieval-ready indexes.

> **[Search and Retrieval](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Retrieve moments after artifacts have been prepared for search.


# View Indexes
Source: https://docs.videodb.io/pages/understand/indexing-pipelines/view-indexes

List indexes, inspect their fields and capabilities, and preview indexed records.

After you create indexes, use index inspection APIs to see what exists, whether each index is ready, which fields are available, and what records were indexed.

> [!NOTE]
>
> In V2, use `get_index()` plus `index.records()` instead of legacy `get_scene_index()`. Keep using `get_scene_index()` only with legacy scene indexes.

***

## List indexes

Use `list_indexes()` to see the indexes available on a video.

```python theme={null}
indexes = video.list_indexes()

for index in indexes:
    print(index.name, index.status, index.use_for)
```

Example response:

```json theme={null}
[
  {
    "index_id": "idx_scene_123",
    "name": "scene",
    "status": "ready",
    "use_for": ["semantic", "query", "aggregate"],
    "source": {
      "understanding_id": "und_123",
      "artifact": "scene",
      "analyzer": {"name": "scene", "type": "vlm"}
    },
    "record_count": 128
  },
  {
    "index_id": "idx_objects_456",
    "name": "objects",
    "status": "ready",
    "use_for": ["query", "aggregate"],
    "source": {
      "understanding_id": "und_123",
      "artifact": "objects",
      "analyzer": {"name": "objects", "type": "object_detection"}
    },
    "record_count": 128
  }
]
```

You can filter by capability when you only want indexes that support a retrieval mode:

```python theme={null}
semantic_indexes = video.list_indexes(use_for="semantic")
query_indexes = video.list_indexes(use_for="query")
aggregate_indexes = video.list_indexes(use_for="aggregate")
```

***

## Get one index

Use `get_index()` to inspect one index manifest.

```python theme={null}
index = video.get_index(name="scene")

print(index.status)
print(index.use_for)
print(index.fields)
```

You can also fetch by index ID:

```python theme={null}
index = video.get_index(index_id="idx_scene_123")
```

`get_index()` returns metadata and schema. It does not return every indexed record by default.

```json theme={null}
{
  "index_id": "idx_scene_123",
  "name": "scene",
  "status": "ready",
  "use_for": ["semantic", "query", "aggregate"],
  "record_count": 128,
  "source": {
    "understanding_id": "und_123",
    "artifact": "scene",
    "analyzer": {"name": "scene", "type": "vlm"}
  },
  "fields": {
    "semantic": ["scene_description", "activity", "setting"],
    "filter": ["activity", "setting"],
    "aggregate": ["activity", "setting"]
  }
}
```

***

## Inspect field schema

Because V2 indexes can support `query()` and `aggregate()`, users need to know which fields are available for each capability.

```python theme={null}
index = video.get_index(name="scene")

for field, schema in index.field_schema.items():
    print(field, schema.type, schema.groups)
```

Example schema:

```json theme={null}
{
  "scene_description": {
    "type": "text",
    "groups": ["semantic"]
  },
  "activity": {
    "type": "string",
    "groups": ["semantic", "filter", "aggregate"],
    "operators": ["==", "!=", "contains", "in", "exists"]
  },
  "setting": {
    "type": "string",
    "groups": ["semantic", "filter", "aggregate"],
    "operators": ["==", "!=", "contains", "in", "exists"]
  }
}
```

Use this schema to decide which retrieval API to call:

| Field group | Retrieval API |
| - | - |
| `semantic` | `video.semantic_search()` / `collection.semantic_search()`; also used by high-level `search()` |
| `filter` | `video.query()` / `collection.query()`; optional filters in `search()` and `semantic_search()` |
| `aggregate` | `video.aggregate()` / `collection.aggregate()` |
| `sort` | result ordering in `query()` / `search()` |

***

## Preview indexed records

Use record preview when you want to inspect what was indexed.

```python theme={null}
page = index.records(limit=20)

for record in page.records:
    print(record.start, record.end, record.data)
```

Example record:

```json theme={null}
{
  "video_id": "m-123",
  "understanding_id": "und-123",
  "scene_id": "scene-000002",
  "start": 3.42,
  "end": 7.1,
  "data": {
    "scene_description": "A person walks through a retail aisle while holding a phone.",
    "activity": "walking through store",
    "setting": "retail aisle",
    "frames": [
      {"timestamp": 4.8, "asset": {"type": "s3_object", "key": "..."}}
    ]
  }
}
```

Records are paginated:

```python theme={null}
page = index.records(limit=50)
next_page = index.records(limit=50, cursor=page.next_cursor)
```

> [!NOTE]
>
> `index.records()` is for inspection and debugging. For retrieval, filtering, ranking, and analytics, use `search()`, `semantic_search()`, `query()`, and `aggregate()`.

***

## Search APIs remain separate from index inspection

`get_index()` tells you what is possible. It does not duplicate retrieval APIs.

For direct semantic search:

```python theme={null}
results = video.semantic_search(
    query="person holding a phone",
    index_names=["scene"],
)
```

For structured filtering on the scene index:

```python theme={null}
results = video.query(
    index_name="scene",
    filter={"activity": "walking through store"},
)
```

For object filters, query the objects index:

```python theme={null}
results = video.query(
    index_name="objects",
    filter={"object_labels": {"contains": "phone"}},
)
```

For counts and facets, use the index that owns the field:

```python theme={null}
response = collection.aggregate(
    index_name="brands",
    group_by="brand_names",
)
counts = response["results"]
```

***

## Delete an index

Delete by id from the video, or call `delete()` on an `Index` object you already hold:

```python theme={null}
video.delete_index(index_id="idx_scene_123")

# or, given an Index object
index = video.get_index(name="scene")
index.delete()
```

Deleting an index removes the retrieval structures for that index. It does not delete the original video or stored understanding artifacts.

***

## Next steps

> **[Search APIs](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Retrieve from the indexes you inspected.

> **[Index Fields](/pages/understand/indexing-pipelines/multiple-indexes)**
>
> Configure which fields become semantic text, filters, facets, and sort keys.


# Collection Search
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/collection-search

Search across your entire video library with a single query

Search across all videos in a collection with a single query. Apply metadata filters to narrow results.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()

# Search entire collection
results = coll.legacy_search("product announcement")

for shot in results.get_shots():
    print(f"Video: {shot.video_id}")
    print(f"Time: {shot.start}s - {shot.end}s")
    print(f"Content: {shot.text}")
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();

// Search entire collection
const results = await coll.legacySearch("product announcement");

for (const shot of results.shots) {
    console.log(`Video: ${shot.videoId}`);
    console.log(`Time: ${shot.start}s - ${shot.end}s`);
    console.log(`Content: ${shot.text}`);
}
```

***

## Collection vs Video Search

| Scope | Method | Use Case |
| :- | :- | :- |
| Single video | `video.legacy_search()` | Find moments in a specific video |
| Entire library | `coll.legacy_search()` | Find content across all videos |

***

## Search All Videos

### Spoken Content

```python Python theme={null}
# Search spoken content across all videos
results = coll.legacy_search("discusses artificial intelligence")

# Results include video IDs
for shot in results.get_shots():
    print(f"Found in video {shot.video_id} at {shot.start}s")
```

```javascript Node.js theme={null}
// Search spoken content across all videos
const results = await coll.legacySearch("discusses artificial intelligence");

// Results include video IDs
for (const shot of results.shots) {
    console.log(`Found in video ${shot.videoId} at ${shot.start}s`);
}
```

### Visual Content

```python Python theme={null}
from videodb import IndexType

# Search scene indexes across collection
results = coll.search(
    query="person speaking at podium",
    index_type=IndexType.scene
)
```

```javascript Node.js theme={null}
import { IndexTypeValues, SearchTypeValues } from 'videodb';

// Search scene indexes across collection
const results = await coll.search(
    "person speaking at podium",
    SearchTypeValues.semantic,
    IndexTypeValues.scene
);
```

***

## Metadata Filtering

Narrow search to specific categories using metadata filters.

### Index with Metadata

```python Python theme={null}
from videodb import SceneExtractionType

# Add metadata during indexing
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 30},
    prompt="Describe the scene",
    metadata={"category": "news", "topic": "technology"}
)
```

```javascript Node.js theme={null}
// Add metadata during indexing
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 30 },
    prompt: "Describe the scene",
    metadata: { category: "news", topic: "technology" }
});
```

### Search with Filters

```python Python theme={null}
from videodb import IndexType

# Filter by metadata
results = coll.search(
    query="product launch",
    filter=[{"category": "news"}],
    index_type=IndexType.scene
)

# Multiple filters (AND logic)
results = coll.search(
    query="keynote speech",
    filter=[
        {"category": "conference"},
        {"year": "2024"}
    ],
    index_type=IndexType.scene
)
```

```javascript Node.js theme={null}
import { IndexTypeValues, SearchTypeValues } from 'videodb';

// Filter by metadata
const filteredResults = await coll.search("product launch", {
    searchType: SearchTypeValues.semantic,
    indexType: IndexTypeValues.scene,
    filter: [{ category: "news" }],
});

// Multiple filters (AND logic)
const multiFilterResults = await coll.search("keynote speech", {
    searchType: SearchTypeValues.semantic,
    indexType: IndexTypeValues.scene,
    filter: [
        { category: "conference" },
        { year: "2024" },
    ],
});
```

### Metadata Guidelines

| Rule | Limit |
| :- | :- |
| Max key-value pairs | 5 |
| Max key length | 20 characters |
| Max value length | 20 characters |
| Value types | string or int |

***

## Scene-Level Metadata

Tag individual scenes for fine-grained filtering.

```python Python theme={null}
from videodb.scene import Scene

# Create scenes with metadata
scenes = [
    Scene(
        video_id=video.id,
        start=0,
        end=60,
        description="Opening segment with logo",
        metadata={"segment_type": "intro"}
    ),
    Scene(
        video_id=video.id,
        start=60,
        end=300,
        description="Main presentation content",
        metadata={"segment_type": "content"}
    ),
    Scene(
        video_id=video.id,
        start=300,
        end=360,
        description="Q&A session",
        metadata={"segment_type": "qa"}
    )
]

# Index with scene-level metadata
video.index_scenes(scenes=scenes, name="segmented_index")

# Search only Q&A segments
results = video.search(
    query="questions about pricing",
    filter=[{"segment_type": "qa"}],
    index_type=IndexType.scene
)
```

```javascript Node.js theme={null}
import { Scene, IndexTypeValues, SearchTypeValues } from 'videodb';

// Create scenes with metadata
const scenes = [
    new Scene({
        videoId: video.id,
        start: 0,
        end: 60,
        description: "Opening segment with logo",
        metadata: { segment_type: "intro" }
    }),
    new Scene({
        videoId: video.id,
        start: 60,
        end: 300,
        description: "Main presentation content",
        metadata: { segment_type: "content" }
    }),
    new Scene({
        videoId: video.id,
        start: 300,
        end: 360,
        description: "Q&A session",
        metadata: { segment_type: "qa" }
    })
];

// Index with scene-level metadata
await video.indexScenes({ scenes, name: "segmented_index" });

// Search only Q&A segments
const results = await video.search("questions about pricing", {
    searchType: SearchTypeValues.semantic,
    indexType: IndexTypeValues.scene,
    filter: [{ segment_type: "qa" }],
});
```

***

## Use Cases

### Media Archive

```python theme={null}
# Tag videos by topic
for video in news_videos:
    video.index_scenes(
        prompt="Describe the news segment",
        metadata={"channel": "CNN", "category": "politics"}
    )

# Search political news only
results = coll.search(
    query="election coverage",
    filter=[{"category": "politics"}],
    index_type=IndexType.scene
)
```

### Training Library

```python theme={null}
# Tag by skill level
beginner_videos.index_scenes(
    prompt="Describe the tutorial content",
    metadata={"level": "beginner", "topic": "python"}
)

# Find beginner Python content
results = coll.search(
    query="how to define a function",
    filter=[{"level": "beginner"}, {"topic": "python"}],
    index_type=IndexType.scene
)
```

### Surveillance

```python theme={null}
# Tag by camera location
camera_footage.index_scenes(
    prompt="Identify people and vehicles",
    metadata={"location": "entrance", "camera_id": "cam_01"}
)

# Search specific camera
results = coll.search(
    query="person in red jacket",
    filter=[{"location": "entrance"}],
    index_type=IndexType.scene
)
```

***

## Performance Considerations

### Index Before Searching

Collection search only finds content that has been indexed:

```python Python theme={null}
# Index all videos first
for video in coll.get_videos():
    video.index_spoken_words()
    video.index_scenes(prompt="Describe the scene")

# Now search across all
results = coll.legacy_search("quarterly results")
```

```javascript Node.js theme={null}
// Index all videos first
const videos = await coll.getVideos();
for (const video of videos) {
    await video.indexSpokenWords();
    await video.indexScenes({ prompt: "Describe the scene" });
}

// Now search across all
const results = await coll.legacySearch("quarterly results");
```

### Use Metadata Filters

Filters reduce the search space and improve speed:

```python theme={null}
# Fast: filtered search
results = coll.legacy_search(
    query="product demo",
    filter=[{"category": "marketing"}]
)

# Slower: unfiltered search across everything
results = coll.legacy_search("product demo")
```

***

## Search Examples

Explore practical search implementations:

> **[Character Clips](/examples-and-tutorials/video-rag/character-clips)**
>
> Extract specific characters or people from video collections

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine text, visual, and audio search for powerful results

## Next Steps

> **[Accuracy Tips](/pages/understand/quality-and-evaluation/accuracy-tips)**
>
> Improve search precision and recall

> **[Latency and Cost](/pages/understand/quality-and-evaluation/latency-and-cost)**
>
> Optimize for speed and efficiency


# Create an Index
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/create-an-index

Transform video into searchable data with spoken word and visual indexes

Indexes turn raw video into structured, searchable data. Create a spoken word index for dialogue and narration, or a scene index for visual content.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
video = coll.get_video("m-xxx")

# Index spoken content (dialogue, narration)
video.index_spoken_words()

# Index visual content (scenes, objects, actions)
scene_index_id = video.index_scenes(
    prompt="Describe what's happening in the scene"
)

# Search both
results = video.legacy_search("car chase through the city")
results.play()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();
const video = await coll.getVideo("m-xxx");

// Index spoken content (dialogue, narration)
await video.indexSpokenWords();

// Index visual content (scenes, objects, actions)
const sceneIndexId = await video.indexScenes({
    prompt: "Describe what's happening in the scene"
});

// Search both
const results = await video.legacySearch("car chase through the city");
await results.play();
```

***

## Spoken Word Index

Transcribes audio into timestamped text using automatic speech recognition (ASR).

```python Python theme={null}
video.index_spoken_words()
```

```javascript Node.js theme={null}
await video.indexSpokenWords();
```

**What it captures:**

* Dialogue and conversations
* Narration and voiceovers
* Lectures and presentations
* Interviews and podcasts

### Language Support

Major languages are auto-detected. For others, pass the language code:

```python Python theme={null}
# Auto-detect (English, Spanish, French, German, Italian, Portuguese, Dutch)
video.index_spoken_words()

# Explicit language code
video.index_spoken_words(language_code="hi")  # Hindi
video.index_spoken_words(language_code="ja")  # Japanese
video.index_spoken_words(language_code="zh")  # Chinese
```

```javascript Node.js theme={null}
// Auto-detect (English, Spanish, French, German, Italian, Portuguese, Dutch)
await video.indexSpokenWords();

// Explicit language code
await video.indexSpokenWords("hi");  // Hindi
await video.indexSpokenWords("ja");  // Japanese
await video.indexSpokenWords("zh");  // Chinese
```

| Language | Code |
| :- | :- |
| English (Global) | `en` |
| English (US/UK/AU) | `en_us`, `en_uk`, `en_au` |
| Spanish | `es` |
| French | `fr` |
| German | `de` |
| Hindi | `hi` |
| Japanese | `ja` |
| Chinese | `zh` |
| Korean | `ko` |
| Russian | `ru` |

***

## Scene Index

Analyzes video frames using vision models to describe visual content.

```python Python theme={null}
scene_index_id = video.index_scenes(
    prompt="Describe the scene in detail"
)
```

```javascript Node.js theme={null}
const sceneIndexId = await video.indexScenes({
    prompt: "Describe the scene in detail"
});
```

**What it captures:**

* Objects and people
* Actions and activities
* Environments and settings
* Visual transitions

### Prompt Shapes the Index

The prompt you provide determines what gets indexed:

```python Python theme={null}
# Focus on people
video.index_scenes(prompt="Describe the people and their actions")

# Focus on environment
video.index_scenes(prompt="Describe the location and setting")

# Focus on specific objects
video.index_scenes(prompt="Identify all vehicles and their colors")
```

```javascript Node.js theme={null}
// Focus on people
await video.indexScenes({ prompt: "Describe the people and their actions" });

// Focus on environment
await video.indexScenes({ prompt: "Describe the location and setting" });

// Focus on specific objects
await video.indexScenes({ prompt: "Identify all vehicles and their colors" });
```

### Extraction Configuration

Control how frames are sampled - choose between frame segmentation (regular intervals) and scene segmentation (automatic transitions):

![Comparison of frame segmentation and scene segmentation extraction types](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/scene-extraction-type.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=05d44b8c0a2ae53395f638015a60b6b2)

```python Python theme={null}
from videodb import SceneExtractionType

# Time-based: every N seconds
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 10, "frame_count": 2},
    prompt="Describe the scene"
)

# Shot-based: detect visual transitions
video.index_scenes(
    extraction_type=SceneExtractionType.shot_based,
    extraction_config={"threshold": 20, "frame_count": 1},
    prompt="Describe the scene"
)
```

```javascript Node.js theme={null}
// Time-based: every N seconds
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 10, frame_count: 2 },
    prompt: "Describe the scene"
});

// Shot-based: detect visual transitions
await video.indexScenes({
    extractionType: 'shot',
    extractionConfig: { threshold: 20, frame_count: 1 },
    prompt: "Describe the scene"
});
```

![Time-based extraction example showing consistent frame sampling at regular intervals](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/time-based-extraction.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=ad5e70511eed02ed94e367e473a35a43)

| Method | Best For |
| :- | :- |
| Time-based | Consistent sampling, dynamic content |
| Shot-based | Edited videos with clear scene changes |

***

## Managing Indexes

### List All Scene Indexes

```python Python theme={null}
indexes = video.list_scene_index()
for idx in indexes:
    print(f"{idx.id}: {idx.name} - {idx.status}")
```

```javascript Node.js theme={null}
const indexes = await video.listSceneIndex();
for (const idx of indexes) {
    console.log(`${idx.id}: ${idx.name} - ${idx.status}`);
}
```

![List of scene indexes showing id, name, and status](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/list_scene_index.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=69c1616c1805a831c89ac2b87c655f80)

### Get Index Details

```python Python theme={null}
scene_index = video.get_scene_index(scene_index_id)
for scene in scene_index:
    print(f"{scene.start}-{scene.end}: {scene.description}")
```

```javascript Node.js theme={null}
const sceneIndex = await video.getSceneIndex(sceneIndexId);
for (const scene of sceneIndex) {
    console.log(`${scene.start}-${scene.end}: ${scene.description}`);
}
```

### Delete an Index

```python Python theme={null}
video.delete_scene_index(scene_index_id)
```

```javascript Node.js theme={null}
await video.deleteSceneIndex(sceneIndexId);
```

***

## Async Processing with Callbacks

For long videos, use callbacks to get notified when indexing completes:

```python Python theme={null}
scene_index_id = video.index_scenes(
    prompt="Describe the scene",
    callback_url="https://your-backend.com/webhooks/index-complete"
)
```

```javascript Node.js theme={null}
const sceneIndexId = await video.indexScenes({
    prompt: "Describe the scene",
    callbackUrl: "https://your-backend.com/webhooks/index-complete"
});
```

***

## What You Can Build

> **[Keyword Search Compilation](/examples-and-tutorials/video-rag/keyword-search)**
>
> Index spoken words, then search to create highlight reels

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine spoken word and scene indexes for powerful queries

> **[Baby Crib Monitoring](/examples-and-tutorials/live-intelligence/baby-crib-monitoring)**
>
> Scene indexing enables real-time infant monitoring

> **[Intrusion Detection](/examples-and-tutorials/live-intelligence/intrusion-detection)**
>
> Index camera feeds to detect unauthorized access

***

## Next Steps

> **[Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Extraction strategies for video + audio

> **[Multiple Indexes](/pages/understand/indexing-pipelines/multiple-indexes)**
>
> Layer different perspectives on the same media


# Indexes & Search
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/indexes-and-search

Indexes turn raw media into searchable knowledge. Search returns playable evidence - timestamps and stream URLs you can verify.

## Quick Example

```python theme={null}
import videodb

conn = videodb.connect()
video = conn.get_collection().upload(url="https://example.com/video.mp4")

# Create an index with a prompt
index_id = video.index_scenes(
    prompt="Extract key decisions and action items"
)

# Search returns playable evidence
results = video.search("budget approval", index_id=index_id)
for shot in results.shots:
    print(f"{shot.start}s: {shot.text}")
    shot.play()  # Opens in browser
```

![Comparison showing video search results with and without indexing, demonstrating how indexing finds exact moments](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/with-without-indexes.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=6e5a1579b89cd65578bf954757c7b3f7)

***

## Index Types

![Scene index visual example showing search query 'Show me the bomb explosion scene' with matching video frames and timestamps](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/scene-index.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=619bc32606efd38518d328a996eb0c9d)

| Type | Method | Use Case |
| :- | :- | :- |
| **Visual** | `video.index_scenes()` | Describe scenes, detect objects |
| **Spoken** | `video.index_spoken_words()` | Transcripts, spoken content |
| **Audio** | `rtstream.index_audio()` | Audio analysis, topics |

### Key Properties

* **Prompt-driven** - You define what to extract with natural language
* **Additive** - Multiple indexes on the same media
* **Non-destructive** - Add/remove without affecting source

```python theme={null}
# Multiple indexes = multiple perspectives
safety_index = video.index_scenes(prompt="Identify safety issues")
summary_index = video.index_scenes(prompt="Summarize each segment")
transcript = video.index_spoken_words()
```

***

## Search Scopes

```python theme={null}
# Single video
results = video.legacy_search("product demo")

# Single stream
results = rtstream.search("intrusion")

# Collection-wide
results = coll.search("quarterly results", index_type="scene")
```

***

## Playable Evidence

Every search result includes playable proof:

```python theme={null}
for shot in results.shots:
    shot.start        # 45.2 (seconds)
    shot.end          # 52.8
    shot.text         # "CEO announces Q3 results"
    shot.search_score # 0.87
    shot.play()       # Verify the result
```

***

## Multi-Index Search

Search across visual and spoken content:

```python theme={null}
results = video.search(
    query="budget discussion",
    index_type=["scene", "spoken_word"]
)
```

* **Union** - Broader recall (results from any index)
* **Intersection** - Higher precision (matches in all indexes)

***

## Best Practices

1. **Be specific in prompts** - "Identify safety violations" beats "describe what you see"
2. **Use specific queries** - "red car entering lot" beats "car"
3. **Verify with playback** - Always validate with `shot.play()`
4. **Create focused indexes** - One index per question or domain

## What You Can Build

> **[Keyword Search Compilation](/examples-and-tutorials/video-rag/keyword-search)**
>
> Create highlight reels from keyword-based search

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine visual and spoken indexes for powerful queries

> **[Character Clips](/examples-and-tutorials/video-rag/character-clips)**
>
> Extract clips featuring specific people from video libraries

***

## Next Steps

> **[Indexing Guide](/pages/understand/indexing-pipelines/create-an-index)**
>
> Detailed indexing patterns

> **[Semantic Search](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Advanced search techniques


# Frame Processing Primitives
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/multimodal-indexing

Video carries information in multiple modalities: what's said, what's shown, and how it changes over time. Multimodal indexing extracts meaning from all of these through extraction strategies for combining visual and spoken content analysis.

## Quick Example

```python Python theme={null}
from videodb import SceneExtractionType

# Index spoken content
video.index_spoken_words()

# Index visual content with extraction strategy
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 5, "frame_count": 3},
    prompt="Describe the scene, people, and any visible text"
)
```

```javascript Node.js theme={null}
// Index spoken content
await video.indexSpokenWords();

// Index visual content with extraction strategy
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 5, frame_count: 3 },
    prompt: "Describe the scene, people, and any visible text"
});
```

***

## Extraction Strategies

### Time-Based Extraction

Split video into fixed intervals. Simple and predictable.

![Time-based extraction example showing consistent frame sampling at regular intervals](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/time-based-extraction.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=ad5e70511eed02ed94e367e473a35a43)

```python Python theme={null}
from videodb import SceneExtractionType

video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={
        "time": 10,           # Scene length in seconds
        "frame_count": 2      # Frames to analyze per scene
    },
    prompt="Describe what's happening"
)
```

```javascript Node.js theme={null}
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: {
        time: 10,           // Scene length in seconds
        frame_count: 2      // Frames to analyze per scene
    },
    prompt: "Describe what's happening"
});
```

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `time` | int | 10 | Interval in seconds |
| `frame_count` | int | 1 | Frames per scene |
| `select_frames` | list | `["first"]` | Which frames: `"first"`, `"middle"`, `"last"` |

> [!NOTE]
>
> Use either `frame_count` or `select_frames`, not both.

**Best for:**

* Surveillance and monitoring
* Live streams
* Content with no clear scene boundaries
* Consistent sampling across long videos

### Shot-Based Extraction

Detect visual transitions (cuts, fades) to identify natural scene boundaries.

![Shot-based extraction example showing automatic detection of visual transitions and scene boundaries](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/shot-based-extraction.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=f007bdd7bd6afdc2946a1cade7718860)

```python Python theme={null}
from videodb import SceneExtractionType

video.index_scenes(
    extraction_type=SceneExtractionType.shot_based,
    extraction_config={
        "threshold": 20,      # Sensitivity (lower = more sensitive)
        "frame_count": 1      # Frames per detected shot
    },
    prompt="Describe the scene"
)
```

```javascript Node.js theme={null}
await video.indexScenes({
    extractionType: 'shot',
    extractionConfig: {
        threshold: 20,      // Sensitivity (lower = more sensitive)
        frame_count: 1      // Frames per detected shot
    },
    prompt: "Describe the scene"
});
```

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `threshold` | int | 20 | Detection sensitivity |
| `frame_count` | int | 1 | Frames per shot |

**Best for:**

* Movies and TV shows
* Edited content with clear cuts
* Music videos
* Commercials

***

## Prompt Engineering

The prompt shapes what gets extracted. Think of it as telling the vision model what to look for.

### Basic Prompts

```python theme={null}
# General description
prompt = "Describe what's happening in this scene"

# Object-focused
prompt = "Identify all objects and people visible"

# Action-focused
prompt = "Describe the activities and movements"
```

### Domain-Specific Prompts

```python Python theme={null}
# Retail / E-commerce
video.index_scenes(
    prompt="Identify products, brands, and pricing visible on screen"
)

# Sports
video.index_scenes(
    prompt="Describe the play, players involved, and outcome"
)

# Security
video.index_scenes(
    prompt="Identify people, vehicles, and any unusual activity"
)

# Education
video.index_scenes(
    prompt="Describe the topic being taught and any diagrams or text shown"
)
```

```javascript Node.js theme={null}
// Retail / E-commerce
await video.indexScenes({
    prompt: "Identify products, brands, and pricing visible on screen"
});

// Sports
await video.indexScenes({
    prompt: "Describe the play, players involved, and outcome"
});

// Security
await video.indexScenes({
    prompt: "Identify people, vehicles, and any unusual activity"
});

// Education
await video.indexScenes({
    prompt: "Describe the topic being taught and any diagrams or text shown"
});
```

### Structured Output Prompts

Guide the model to produce consistent, parseable output:

```python theme={null}
prompt = """
Describe this scene with the following structure:
- Setting: Where is this taking place?
- People: Who is present and what are they doing?
- Objects: What notable items are visible?
- Action: What is happening?
"""
```

***

## Frame Selection Strategy

More frames = more detail but higher cost. Choose based on your content.

### Static Content (1 frame)

For content where a single frame captures the scene:

```python Python theme={null}
# One frame is enough for static shots
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 10, "frame_count": 1},
    prompt="Describe the scene"
)
```

```javascript Node.js theme={null}
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 10, frame_count: 1 },
    prompt: "Describe the scene"
});
```

### Motion and Activity (3-5 frames)

For understanding movement and temporal changes:

```python Python theme={null}
# Multiple frames to capture motion
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 5, "frame_count": 5},
    prompt="Describe the activity and how it progresses"
)
```

```javascript Node.js theme={null}
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 5, frame_count: 5 },
    prompt: "Describe the activity and how it progresses"
});
```

### Key Moment Selection

Select specific frames within each scene:

```python Python theme={null}
# First and last frames only
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 10, "select_frames": ["first", "last"]},
    prompt="Describe how the scene changes from start to end"
)
```

```javascript Node.js theme={null}
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 10, select_frames: ["first", "last"] },
    prompt: "Describe how the scene changes from start to end"
});
```

***

## Combining Modalities

Index both spoken and visual content, then search across both:

```python Python theme={null}
from videodb import IndexType, SearchType

# Index both modalities
video.index_spoken_words()
video.index_scenes(prompt="Describe the visual content")

# Search spoken content
spoken_results = video.search(
    query="discusses climate change",
    index_type=IndexType.spoken_word
)

# Search visual content
visual_results = video.search(
    query="shows melting glaciers",
    index_type=IndexType.scene
)
```

```javascript Node.js theme={null}
import { IndexTypeValues, SearchTypeValues } from 'videodb';

// Index both modalities
await video.indexSpokenWords();
await video.indexScenes({ prompt: "Describe the visual content" });

// Search spoken content
const spokenResults = await video.legacySearch(
    "discusses climate change",
    SearchTypeValues.semantic,
    IndexTypeValues.spoken
);

// Search visual content
const visualResults = await video.legacySearch(
    "shows melting glaciers",
    SearchTypeValues.semantic,
    IndexTypeValues.scene
);
```

***

## Extraction Examples

### Traffic Monitoring

```python Python theme={null}
# Detect vehicle colors (single frame sufficient)
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 1, "frame_count": 1},
    prompt="Identify the color and type of each vehicle"
)

# Detect stopped vehicles (need multiple frames)
video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 4, "frame_count": 5},
    prompt="Identify if any vehicle has stopped or is moving slowly"
)
```

```javascript Node.js theme={null}
// Detect vehicle colors (single frame sufficient)
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 1, frame_count: 1 },
    prompt: "Identify the color and type of each vehicle"
});

// Detect stopped vehicles (need multiple frames)
await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 4, frame_count: 5 },
    prompt: "Identify if any vehicle has stopped or is moving slowly"
});
```

### Educational Content

```python Python theme={null}
# Combine visual and spoken indexing
video.index_spoken_words()

video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 30, "select_frames": ["first", "middle", "last"]},
    prompt="Describe diagrams, equations, or visual aids shown"
)
```

```javascript Node.js theme={null}
// Combine visual and spoken indexing
await video.indexSpokenWords();

await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 30, select_frames: ["first", "middle", "last"] },
    prompt: "Describe diagrams, equations, or visual aids shown"
});
```

***

## Next Steps

> **[Multiple Indexes](/pages/understand/indexing-pipelines/multiple-indexes)**
>
> Layer different perspectives on the same media

> **[Accuracy Tips](/pages/understand/quality-and-evaluation/accuracy-tips)**
>
> Improve precision and recall


# Multiple Indexes
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/multiple-indexes

Layer different perspectives on the same video for precise retrieval

A single video can have multiple indexes, each capturing a different aspect. Layer them together for precise, multi-dimensional retrieval.

## Quick Example

```python Python theme={null}
from videodb import IndexType

# Create two indexes with different focus
car_index = video.index_scenes(
    prompt="Identify the color and model of each vehicle",
    name="vehicle_identification"
)

motion_index = video.index_scenes(
    prompt="Describe the movement pattern - speeding, stopping, turning",
    name="motion_analysis"
)

# Search each index separately
car_results = video.search(
    query="red sedan",
    index_type=IndexType.scene,
    index_id=car_index
)

motion_results = video.search(
    query="driving recklessly",
    index_type=IndexType.scene,
    index_id=motion_index
)
```

***

## Why Multiple Indexes?

Different prompts extract different information. A single index optimized for one use case may miss details relevant to another.

| Index Focus | Prompt Example | Use Case |
| :- | :- | :- |
| Objects | "Identify all objects and their colors" | Inventory, cataloging |
| Actions | "Describe activities and movements" | Activity detection |
| People | "Describe people, clothing, expressions" | Person identification |
| Environment | "Describe the setting and location" | Scene classification |
| Text | "Read any visible text or signage" | OCR, document extraction |

***

## Creating Focused Indexes

### Same Video, Different Perspectives

```python Python theme={null}
from videodb import SceneExtractionType

# Environment index
env_index = video.index_scenes(
    prompt="Describe the environment - indoor/outdoor, lighting, weather",
    name="environment"
)

# People index
people_index = video.index_scenes(
    prompt="Describe the people - count, clothing, actions",
    name="people"
)

# Object index
object_index = video.index_scenes(
    prompt="List all visible objects and their positions",
    name="objects"
)
```

```javascript Node.js theme={null}
// Environment index
const envIndex = await video.indexScenes({
    prompt: "Describe the environment - indoor/outdoor, lighting, weather",
    name: "environment"
});

// People index
const peopleIndex = await video.indexScenes({
    prompt: "Describe the people - count, clothing, actions",
    name: "people"
});

// Object index
const objectIndex = await video.indexScenes({
    prompt: "List all visible objects and their positions",
    name: "objects"
});
```

### Different Extraction Configs

```python Python theme={null}
from videodb import SceneExtractionType

# Fast sampling for color detection (1 frame per second)
color_index = video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 1, "frame_count": 1},
    prompt="Identify vehicle colors",
    name="colors"
)

# Slower sampling for motion analysis (5 frames over 4 seconds)
motion_index = video.index_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 4, "frame_count": 5},
    prompt="Describe vehicle movement patterns",
    name="motion"
)
```

```javascript Node.js theme={null}
// Fast sampling for color detection (1 frame per second)
const colorIndex = await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 1, frame_count: 1 },
    prompt: "Identify vehicle colors",
    name: "colors"
});

// Slower sampling for motion analysis (5 frames over 4 seconds)
const motionIndex = await video.indexScenes({
    extractionType: 'time',
    extractionConfig: { time: 4, frame_count: 5 },
    prompt: "Describe vehicle movement patterns",
    name: "motion"
});
```

***

## Combining Search Results

Search multiple indexes and combine results for precise retrieval.

### Intersection (AND)

Find segments that match criteria in both indexes:

![Intersection method showing segments that match criteria in both search indexes combined with AND logic](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/search/intersection-method.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=5d74e5d08d318739f8338c2ea34e4cd4)

```python Python theme={null}
def get_intersection(results1, results2):
    """Find overlapping timestamps from two search results"""
    shots1 = [(s.start, s.end) for s in results1.get_shots()]
    shots2 = [(s.start, s.end) for s in results2.get_shots()]

    intersection = []
    for s1 in shots1:
        for s2 in shots2:
            start = max(s1[0], s2[0])
            end = min(s1[1], s2[1])
            if start < end:
                intersection.append((start, end))

    return intersection

# Find "red sedan" that is "driving recklessly"
car_results = video.search("red sedan", index_type=IndexType.scene, index_id=car_index)
motion_results = video.search("reckless driving", index_type=IndexType.scene, index_id=motion_index)

combined = get_intersection(car_results, motion_results)
stream_url = video.generate_stream(combined)
```

### Union (OR)

Find segments that match criteria in either index:

![Union method showing segments that match criteria in either search index combined with OR logic](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/search/union-method.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=bb560a75aecf2e5fa9c185689486a973)

```python Python theme={null}
def get_union(results1, results2):
    """Merge timestamps from two search results"""
    all_shots = [(s.start, s.end) for s in results1.get_shots()]
    all_shots += [(s.start, s.end) for s in results2.get_shots()]

    # Sort and merge overlapping intervals
    all_shots.sort()
    merged = []
    for start, end in all_shots:
        if merged and start <= merged[-1][1]:
            merged[-1] = (merged[-1][0], max(merged[-1][1], end))
        else:
            merged.append((start, end))

    return merged

# Find any scene with "red sedan" OR "motorcycle"
car_results = video.search("red sedan", index_type=IndexType.scene, index_id=car_index)
bike_results = video.search("motorcycle", index_type=IndexType.scene, index_id=car_index)

combined = get_union(car_results, bike_results)
```

***

## Custom Annotations

Bring your own scene descriptions from external pipelines or manual tagging.

![Custom scene annotation example showing manually created scene descriptions and timestamps](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/custom_scene.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=b8f56c55dbdfc70de9a2704cc466aba8)

```python Python theme={null}
from videodb.scene import Scene

# Create scenes with custom descriptions
scenes = [
    Scene(
        video_id=video.id,
        start=0,
        end=60,
        description="Opening credits with cityscape aerial shot"
    ),
    Scene(
        video_id=video.id,
        start=60,
        end=180,
        description="Interview with CEO discussing Q4 results"
    ),
    Scene(
        video_id=video.id,
        start=180,
        end=300,
        description="Product demo showing new features"
    )
]

# Index custom scenes
custom_index = video.index_scenes(scenes=scenes, name="manual_chapters")

# Search the custom index
results = video.search(
    query="product demo",
    index_type=IndexType.scene,
    index_id=custom_index
)
```

### External Vision Pipeline

Integrate descriptions from your own models:

```python Python theme={null}
# Extract scenes
scene_collection = video.extract_scenes(
    extraction_type=SceneExtractionType.time_based,
    extraction_config={"time": 10, "select_frames": ["middle"]}
)

# Process with your own model
for scene in scene_collection.scenes:
    for frame in scene.frames:
        # Call your custom vision model
        frame.description = your_model.describe(frame.url)

    # Aggregate frame descriptions into scene description
    scene.description = aggregate_descriptions(scene.frames)

# Index the processed scenes
custom_index = video.index_scenes(
    scenes=scene_collection.scenes,
    name="custom_model_index"
)
```

```javascript Node.js theme={null}
// Extract scenes
const sceneCollection = await video.extractScenes({
    extractionType: 'time',
    extractionConfig: { time: 10, select_frames: ["middle"] }
});

// Process with your own model
for (const scene of sceneCollection.scenes) {
    for (const frame of scene.frames) {
        // Call your custom vision model
        frame.description = await yourModel.describe(frame.url);
    }

    // Aggregate frame descriptions into scene description
    scene.description = aggregateDescriptions(scene.frames);
}

// Index the processed scenes
const customIndex = await video.indexScenes({
    scenes: sceneCollection.scenes,
    name: "custom_model_index"
});
```

***

## Use Cases

### Investigation Workflow

```python theme={null}
# Index 1: Vehicle identification
vehicle_index = video.index_scenes(prompt="Identify vehicle make, model, color")

# Index 2: Person detection
person_index = video.index_scenes(prompt="Describe people - clothing, actions")

# Index 3: Location context
location_index = video.index_scenes(prompt="Identify location markers, signs, landmarks")

# Query: "Person in red jacket near blue Toyota"
person_results = video.search("red jacket", index_id=person_index)
vehicle_results = video.search("blue Toyota", index_id=vehicle_index)
combined = get_intersection(person_results, vehicle_results)
```

### Sports Highlights

```python theme={null}
# Index 1: Player actions
player_index = video.index_scenes(prompt="Describe player actions and jersey numbers")

# Index 2: Game events
event_index = video.index_scenes(prompt="Identify goals, fouls, penalties, celebrations")

# Query: "Player #10 scoring"
player_results = video.search("player 10", index_id=player_index)
event_results = video.search("goal", index_id=event_index)
combined = get_intersection(player_results, event_results)
```

***

## Next Steps

> **[Natural Language Query](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Search your indexes with plain English

> **[Accuracy Tips](/pages/understand/quality-and-evaluation/accuracy-tips)**
>
> Improve precision and recall


# Natural Language Query
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/natural-language-query

Search videos using plain English questions and get relevant results

Ask questions in plain English. VideoDB uses semantic search to understand intent and return relevant video segments.

## Quick Example

```python Python theme={null}
import videodb

conn = videodb.connect()
coll = conn.get_collection()
video = coll.get_video("m-xxx")

# Natural language query
results = video.legacy_search("when does the speaker discuss climate change?")

# Play matching segments
results.play()
```

```javascript Node.js theme={null}
import { connect } from 'videodb';

const conn = connect();
const coll = await conn.getCollection();
const video = await coll.getVideo("m-xxx");

// Natural language query
const results = await video.legacySearch("when does the speaker discuss climate change?");

// Get stream URL
const streamUrl = await results.compile();
console.log(streamUrl);
```

***

## How It Works

1. **Query Understanding** - Your query is transformed into a vector embedding
2. **Similarity Matching** - Embeddings are compared against indexed content
3. **Relevance Scoring** - Results are ranked by semantic similarity
4. **Timestamp Retrieval** - Matching segments are returned with timestamps

***

## Search Types

### Semantic Search (Default)

Understands meaning and intent, not just keywords.

```python Python theme={null}
from videodb import SearchType

# Semantic search (default)
results = video.legacy_search("How do I fix a leaky faucet?")

# Explicit semantic search
results = video.search(
    query="How do I fix a leaky faucet?",
    search_type=SearchType.semantic
)
```

```javascript Node.js theme={null}
import { SearchTypeValues } from 'videodb';

// Semantic search (default)
const defaultResults = await video.legacySearch("How do I fix a leaky faucet?");

// Explicit semantic search
const semanticResults = await video.legacySearch(
    "How do I fix a leaky faucet?",
    SearchTypeValues.semantic
);
```

**Best for:**

* Questions ("What causes...?", "How do you...?")
* Conceptual queries ("explain the theory")
* Fuzzy matching ("something about cars")

### Keyword Search

Exact substring matching. Finds literal occurrences.

```python Python theme={null}
from videodb import SearchType

results = video.search(
    query="API",
    search_type=SearchType.keyword
)
```

```javascript Node.js theme={null}
import { SearchTypeValues } from 'videodb';

const results = await video.legacySearch(
    "API",
    SearchTypeValues.keyword
);
```

**Best for:**

* Technical terms
* Proper nouns
* Exact phrases

### Comparison

| Feature | Semantic Search | Keyword Search |
| :- | :- | :- |
| Query | Natural language | Exact terms |
| Matching | By meaning | By substring |
| Example | "How to repair pipes?" | "plumbing repair" |
| Scope | Single video or collection | Single video only |

***

## Index Types

Specify which index to search.

```python Python theme={null}
from videodb import IndexType

# Search spoken content (default)
results = video.search(
    query="discusses machine learning",
    index_type=IndexType.spoken_word
)

# Search visual content
results = video.search(
    query="person running through a park",
    index_type=IndexType.scene
)

# Search specific scene index
results = video.search(
    query="red car",
    index_type=IndexType.scene,
    index_id="scene-index-xxx"
)
```

```javascript Node.js theme={null}
import { IndexTypeValues, SearchTypeValues } from 'videodb';

// Search spoken content (default)
const spokenResults = await video.legacySearch(
    "discusses machine learning",
    SearchTypeValues.semantic,
    IndexTypeValues.spoken
);

// Search visual content
const visualResults = await video.legacySearch(
    "person running through a park",
    SearchTypeValues.semantic,
    IndexTypeValues.scene
);

```

***

## Tuning Results

### Result Threshold

Limit the number of results returned:

```python Python theme={null}
results = video.search(
    query="funny moments",
    result_threshold=10  # Return top 10 matches
)
```

```javascript Node.js theme={null}
const results = await video.legacySearch(
    "funny moments",
    SearchTypeValues.semantic,
    IndexTypeValues.spoken,
    10  // resultThreshold - Return top 10 matches
);
```

### Score Threshold

Filter out low-relevance results:

```python Python theme={null}
results = video.legacy_search(
    query="product demo",
    score_threshold=0.3  # Only results with score >= 0.3
)
```

```javascript Node.js theme={null}
const results = await video.search("product demo", {
    searchType: SearchTypeValues.semantic,
    indexType: IndexTypeValues.spoken,
    scoreThreshold: 0.3,
});
```

### Dynamic Score Percentage

Adaptive filtering based on score distribution:

```python Python theme={null}
results = video.search(
    query="key insights",
    dynamic_score_percentage=50  # Keep top 50% of score range
)
```

```javascript Node.js theme={null}
const results = await video.search("key insights", {
    searchType: SearchTypeValues.semantic,
    indexType: IndexTypeValues.spoken,
    dynamicScorePercentage: 50,
});
```

The dynamic threshold is calculated as:

```
dynamic_threshold = max_score - (range × percentage)
```

***

## Search Parameters Reference

| Parameter | Type | Default | Description |
| :- | :- | :- | :- |
| `query` | str | required | Natural language query |
| `search_type` | SearchType | semantic | `semantic` or `keyword` |
| `index_type` | IndexType | spoken\_word | `spoken_word` or `scene` |
| `result_threshold` | int | 5 | Max results to return |
| `score_threshold` | float | 0.2 | Minimum relevance score |
| `dynamic_score_percentage` | float | None | Adaptive score filter |
| `index_id` | str | None | Specific scene index ID |
| `sort_docs_on` | str | None | Sort results by `"score"` or `"start"` (timestamp) |

![Layers and parameters of semantic search showing how queries are transformed into vectors and matched against indexed content](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/search/semantic-search-parameters.webp?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=1fbfe6405d24035f422db2e255cb5c2e)

***

## Query Examples

### Sort by Timestamp

```python Python theme={null}
# Sort results by timestamp instead of relevance
results = coll.search(query="morning sunlight", sort_docs_on="start")
```

```javascript Node.js theme={null}
// Search collection content
const results = await coll.search("morning sunlight", {
    sortDocsOn: "start",
});
```

### Spoken Content Queries

```python theme={null}
# Question format
video.legacy_search("What are the main benefits of solar energy?")

# Topic lookup
video.legacy_search("discussion about renewable energy")

# Speaker search
video.legacy_search("when the CEO mentions revenue")
```

### Visual Content Queries

```python theme={null}
# Object detection
video.search("red car on the highway", index_type=IndexType.scene)

# Action detection
video.search("person running", index_type=IndexType.scene)

# Scene description
video.search("sunset over the ocean", index_type=IndexType.scene)
```

### Multimodal Queries

Combine spoken and visual search for precise results:

```python Python theme={null}
from videodb import IndexType

# Search spoken content
spoken_results = video.search(
    query="talks about the solar system",
    index_type=IndexType.spoken_word
)

# Search visual content
visual_results = video.search(
    query="shows planets or galaxies",
    index_type=IndexType.scene
)

# Find intersection (both conditions met)
spoken_times = [(s.start, s.end) for s in spoken_results.get_shots()]
visual_times = [(s.start, s.end) for s in visual_results.get_shots()]
```

```javascript Node.js theme={null}
import { IndexTypeValues, SearchTypeValues } from 'videodb';

// Search spoken content
const spokenResults = await video.legacySearch(
    "talks about the solar system",
    SearchTypeValues.semantic,
    IndexTypeValues.spoken
);

// Search visual content
const visualResults = await video.legacySearch(
    "shows planets or galaxies",
    SearchTypeValues.semantic,
    IndexTypeValues.scene
);

// Find intersection (both conditions met)
const spokenTimes = spokenResults.shots.map(s => [s.start, s.end]);
const visualTimes = visualResults.shots.map(s => [s.start, s.end]);
```

***

## What You Can Build

> **[Keyword Search Compilation](/examples-and-tutorials/video-rag/keyword-search)**
>
> Create highlight reels from specific keywords or phrases

> **[Multimodal Search](/examples-and-tutorials/video-rag/multimodal-search)**
>
> Combine spoken and visual search for precise results

> **[Character Clips](/examples-and-tutorials/video-rag/character-clips)**
>
> Extract clips featuring specific people using search

***

## Next Steps

> **[Timestamps, Clips, Streams](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> What you get back from search

> **[Collection Search](/pages/understand/search-and-retrieval/collection-search)**
>
> Search across your entire library


# Timestamps, Clips, Streams
Source: https://docs.videodb.io/pages/understand/legacy-indexing-search/timestamps-clips-streams

Understand search results and generate playable video clips

Search returns structured results with timestamps, descriptions, and relevance scores. Generate playable clips or stream URLs from any result.

## Quick Example

```python Python theme={null}
results = video.legacy_search("car chase scene")

# Access individual shots
for shot in results.get_shots():
    print(f"{shot.start}s - {shot.end}s: {shot.text}")
    print(f"Score: {shot.search_score}")

# Play all matching segments
results.play()

# Generate a stream URL
stream_url = results.compile()
print(stream_url)
```

```javascript Node.js theme={null}
const results = await video.legacySearch("car chase scene");

// Access individual shots
for (const shot of results.shots) {
    console.log(`${shot.start}s - ${shot.end}s: ${shot.text}`);
    console.log(`Score: ${shot.searchScore}`);
}

// Get stream URL
const streamUrl = await results.compile();
console.log(streamUrl);
```

***

## Search Result Structure

Each search returns a `SearchResult` object containing matching shots:

### Shot Attributes

| Attribute | Type | Description |
| :- | :- | :- |
| `start` | float | Start timestamp in seconds |
| `end` | float | End timestamp in seconds |
| `text` | str | Content description or transcript |
| `search_score` | float | Relevance score (0-1) |
| `stream_url` | str | Direct playback URL |

### Accessing Results

```python Python theme={null}
results = video.legacy_search("introduction to the topic")

# Get all shots
shots = results.get_shots()

# Access first result
first_shot = shots[0]
print(f"Starts at: {first_shot.start}")
print(f"Ends at: {first_shot.end}")
print(f"Content: {first_shot.text}")
print(f"Relevance: {first_shot.search_score}")
```

```javascript Node.js theme={null}
const results = await video.legacySearch("introduction to the topic");

// Get all shots
const shots = results.shots;

// Access first result
const firstShot = shots[0];
console.log(`Starts at: ${firstShot.start}`);
console.log(`Ends at: ${firstShot.end}`);
console.log(`Content: ${firstShot.text}`);
console.log(`Relevance: ${firstShot.searchScore}`);
```

***

## Playback Options

### Play Results

```python Python theme={null}
results = video.legacy_search("product demo")
results.play()  # Opens matching segments in your default browser
```

```javascript Node.js theme={null}
const results = await video.legacySearch("product demo");
const streamUrl = await results.compile();
console.log(streamUrl); // Use this URL in a video player
```

### Play Individual Shot

```python Python theme={null}
results = video.legacy_search("funny moments")
shots = results.get_shots()

# Play just the first match
shots[0].play()
```

```javascript Node.js theme={null}
const results = await video.legacySearch("funny moments");
const shots = results.shots;

// Play just the first match
await shots[0].play();
```

***

## Generate Stream URLs

Create playable URLs for embedding or sharing.

### From Search Results

```python Python theme={null}
results = video.legacy_search("highlight reel")
stream_url = results.compile()
# Returns: https://stream.videodb.io/v3/...
```

```javascript Node.js theme={null}
const results = await video.legacySearch("highlight reel");
const streamUrl = await results.compile();
// Returns: https://stream.videodb.io/v3/...
```

### From Custom Timestamps

```python Python theme={null}
# Generate stream from specific time ranges
timestamps = [
    (10.5, 25.0),   # First segment
    (45.0, 60.0),   # Second segment
    (120.0, 135.5)  # Third segment
]

stream_url = video.generate_stream(timestamps)
```

```javascript Node.js theme={null}
// Generate stream from specific time ranges
const timestamps = [
    [10.5, 25.0],   // First segment
    [45.0, 60.0],   // Second segment
    [120.0, 135.5]  // Third segment
];

const streamUrl = await video.generateStream(timestamps);
```

### Stream URL Format

Generated URLs are HLS streams that work in any video player:

```
https://stream.videodb.io/v3/published/manifests/{manifest-id}.m3u8
```

***

## Working with Timestamps

### Extract Timestamps from Results

```python Python theme={null}
results = video.legacy_search("key moments")

# Get as list of tuples
timestamps = [(shot.start, shot.end) for shot in results.get_shots()]
# [(5.2, 15.0), (45.5, 52.3), (120.0, 145.8)]
```

```javascript Node.js theme={null}
const results = await video.legacySearch("key moments");

// Get as list of arrays
const timestamps = results.shots.map(shot => [shot.start, shot.end]);
// [[5.2, 15.0], [45.5, 52.3], [120.0, 145.8]]
```

### Merge Overlapping Segments

```python Python theme={null}
def merge_timestamps(timestamps):
    """Merge overlapping time ranges"""
    if not timestamps:
        return []

    sorted_ts = sorted(timestamps)
    merged = [list(sorted_ts[0])]

    for start, end in sorted_ts[1:]:
        if start <= merged[-1][1]:
            merged[-1][1] = max(merged[-1][1], end)
        else:
            merged.append([start, end])

    return merged

# Merge results from multiple searches
ts1 = [(10, 20), (15, 25)]
ts2 = [(22, 30), (50, 60)]
merged = merge_timestamps(ts1 + ts2)
# [[10, 30], [50, 60]]
```

```javascript Node.js theme={null}
function mergeTimestamps(timestamps) {
    if (timestamps.length === 0) return [];

    const sorted = [...timestamps].sort((a, b) => a[0] - b[0]);
    const merged = [[...sorted[0]]];

    for (const [start, end] of sorted.slice(1)) {
        const last = merged[merged.length - 1];
        if (start <= last[1]) {
            last[1] = Math.max(last[1], end);
        } else {
            merged.push([start, end]);
        }
    }

    return merged;
}

// Merge results from multiple searches
const ts1 = [[10, 20], [15, 25]];
const ts2 = [[22, 30], [50, 60]];
const merged = mergeTimestamps([...ts1, ...ts2]);
// [[10, 30], [50, 60]]
```

***

## Embedding Streams

### HTML Embed

```html theme={null}
<video controls>
  <source src="https://stream.videodb.io/v3/published/manifests/{id}.m3u8" type="application/x-mpegURL">
</video>
```

### React Component

```jsx theme={null}
import Hls from 'hls.js';

function VideoPlayer({ streamUrl }) {
  const videoRef = useRef(null);

  useEffect(() => {
    if (Hls.isSupported()) {
      const hls = new Hls();
      hls.loadSource(streamUrl);
      hls.attachMedia(videoRef.current);
    }
  }, [streamUrl]);

  return <video ref={videoRef} controls />;
}
```

### VideoDB Player

Use the built-in player with the console URL:

```
https://console.videodb.io/player?url={stream_url}
```

***

## Filter and Sort Results

### By Score

```python Python theme={null}
results = video.legacy_search("important moments")

# Get high-confidence results only
shots = results.get_shots()
high_confidence = [s for s in shots if s.search_score > 0.5]
```

```javascript Node.js theme={null}
const results = await video.legacySearch("important moments");

// Get high-confidence results only
const shots = results.shots;
const highConfidence = shots.filter(s => s.searchScore > 0.5);
```

### By Duration

```python Python theme={null}
results = video.legacy_search("long segments")

# Get segments longer than 10 seconds
shots = results.get_shots()
long_segments = [s for s in shots if (s.end - s.start) > 10]
```

```javascript Node.js theme={null}
const results = await video.legacySearch("long segments");

// Get segments longer than 10 seconds
const shots = results.shots;
const longSegments = shots.filter(s => (s.end - s.start) > 10);
```

***

## Next Steps

> **[Collection Search](/pages/understand/search-and-retrieval/collection-search)**
>
> Search across your entire library

> **[Accuracy Tips](/pages/understand/quality-and-evaluation/accuracy-tips)**
>
> Improve search precision


# Accuracy Tips
Source: https://docs.videodb.io/pages/understand/quality-and-evaluation/accuracy-tips

Better search results come from better indexing and smarter queries. This guide covers techniques to improve precision and recall.

## Quick Wins

1. **Use specific prompts** during indexing
2. **Choose the right retrieval mode** (semantic search vs structured query)
3. **Tune thresholds** based on your use case
4. **Combine multiple indexes** for layered search

***

## Understanding Precision and Recall

| Metric | Definition | Goal |
| :- | :- | :- |
| **Precision** | % of returned results that are relevant | Fewer false positives |
| **Recall** | % of relevant content that was returned | Fewer missed results |

**The trade-off:** Higher precision often means lower recall, and vice versa. Tune based on your priority.

***

## Indexing for Accuracy

### Specific Prompts Beat Generic Ones

```python Python theme={null}
# Pass one of these VLM analyzer configurations to video.understand(...).
# Too generic - low precision
generic_scene = {"type": "vlm", "name": "scene", "config": {"prompt": "Describe the scene"}}

# Better - focused on what matters
vehicle_scene = {
    "type": "vlm",
    "name": "scene",
    "config": {"prompt": "Identify all vehicles with their color, make, and model"},
}

# Best - structured for your retrieval needs
structured_scene = {
    "type": "vlm",
    "name": "scene",
    "config": {"prompt": """Describe this scene with:
- People: count, actions, clothing colors
- Vehicles: type, color, direction of travel
- Environment: indoor/outdoor, time of day
"""},
}
```

```javascript Node.js theme={null}
// Pass one of these VLM analyzer configurations to video.understand(...).
const genericScene = { type: "vlm", name: "scene", config: { prompt: "Describe the scene" } };

const vehicleScene = {
    type: "vlm",
    name: "scene",
    config: { prompt: "Identify all vehicles with their color, make, and model" }
};

const structuredScene = {
    type: "vlm",
    name: "scene",
    config: { prompt: `Describe this scene with:
- People: count, actions, clothing colors
- Vehicles: type, color, direction of travel
- Environment: indoor/outdoor, time of day` }
};
```

### Match Extraction to Content Type

| Content | Extraction | Reasoning |
| :- | :- | :- |
| Static shots | 1 frame/scene | Single frame captures all info |
| Action/motion | 3-5 frames/scene | Need temporal context |
| Quick cuts | Shot-based | Respect natural boundaries |
| Continuous | Time-based, short intervals | Capture changes |

![Time-based extraction example showing consistent frame sampling at regular intervals](https://mintcdn.com/videodb/6KL5X6-sIPSRpEUt/assets/indexing/time-based-extraction.avif?fit=max\&auto=format\&n=6KL5X6-sIPSRpEUt\&q=85\&s=ad5e70511eed02ed94e367e473a35a43)

```python Python theme={null}
# For interviews (mostly static)
interview_analyzer = {
    "type": "vlm",
    "name": "interview",
    "sampling": {"frame_count": 1},
    "config": {"prompt": "Describe the speaker and topic"},
}
interview_segmentation = {"type": "time", "seconds": 30}

# For action content (need motion)
action_analyzer = {
    "type": "vlm",
    "name": "action",
    "sampling": {"frame_count": 4},
    "config": {"prompt": "Describe the activity and movements"},
}
action_segmentation = {"type": "time", "seconds": 5}
```

```javascript Node.js theme={null}
// Pass each analyzer and segmentation pair to video.understand(...).
const interviewAnalyzer = {
    type: "vlm",
    name: "interview",
    sampling: { frame_count: 1 },
    config: { prompt: "Describe the speaker and topic" }
};
const interviewSegmentation = { type: "time", seconds: 30 };

const actionAnalyzer = {
    type: "vlm",
    name: "action",
    sampling: { frame_count: 4 },
    config: { prompt: "Describe the activity and movements" }
};
const actionSegmentation = { type: "time", seconds: 5 };
```

***

## Query Strategies

### Semantic Search vs Structured Query

| Query Type | Use Semantic Search | Use Structured Query |
| :- | :- | :- |
| Questions | ✓ "How does the engine work?" | |
| Concepts | ✓ "explains machine learning" | |
| Exact fields | | ✓ `event_type == "goal"` |
| Technical categories | | ✓ `camera_view == "road_ahead"` |
| Numeric metadata | | ✓ `score >= 0.8` |

```python Python theme={null}
# Semantic for conceptual queries
results = video.semantic_search(
    query="explains the benefits",
    index_names=["scene"],
    top_k=10,
)

# Structured query for exact indexed fields
results = video.query(
    index_name="events",
    filter=[{"field": "event_type", "op": "==", "value": "quarterly_earnings"}],
    limit=10,
)
```

```javascript Node.js theme={null}
// Semantic for conceptual queries
const semanticResults = await video.semanticSearch(
    "explains the benefits",
    { indexNames: ["scene"], topK: 10 }
);

// Structured query for exact indexed fields
const exactResults = await video.query({
    indexName: "events",
    filter: [{ field: "event_type", op: "==", value: "quarterly_earnings" }],
    limit: 10
});
```

### Threshold Tuning

| Parameter | Higher Value | Lower Value |
| :- | :- | :- |
| `score_threshold` | ↑ Precision, ↓ Recall | ↓ Precision, ↑ Recall |
| `top_k` | Allows more returned moments | Limits the returned moments |

```python Python theme={null}
# High precision (strict)
results = video.semantic_search(
    query="CEO announcement",
    index_names=["scene"],
    score_threshold=0.5,
    top_k=3,
)

# High recall (inclusive)
results = video.semantic_search(
    query="CEO announcement",
    index_names=["scene"],
    score_threshold=0.1,
    top_k=20,
)
```

```javascript Node.js theme={null}
// High precision (strict)
const highPrecisionResults = await video.semanticSearch(
    "CEO announcement",
    { indexNames: ["scene"], topK: 3, scoreThreshold: 0.5 }
);

// High recall (inclusive)
const highRecallResults = await video.semanticSearch(
    "CEO announcement",
    { indexNames: ["scene"], topK: 20, scoreThreshold: 0.1 }
);
```

***

## Evaluating Search Quality

### Set Up Ground Truth

Create test queries with known correct answers:

```python theme={null}
test_cases = [
    {
        "query": "six",
        "expected_timestamps": [(4.0, 5.0), (14.0, 15.0), (24.0, 25.0)]
    },
    {
        "query": "introduction",
        "expected_timestamps": [(0.0, 30.0)]
    }
]
```

### Measure Precision and Recall

```python theme={null}
def evaluate_search(results, expected):
    """Calculate precision and recall"""
    returned = set((s.start, s.end) for s in results.get_shots())
    expected_set = set(expected)

    true_positives = len(returned & expected_set)
    false_positives = len(returned - expected_set)
    false_negatives = len(expected_set - returned)

    precision = true_positives / (true_positives + false_positives) if returned else 0
    recall = true_positives / (true_positives + false_negatives) if expected_set else 0

    return {"precision": precision, "recall": recall}

# Evaluate
results = video.semantic_search(query="six", index_names=["scene"])
metrics = evaluate_search(results, [(4.0, 5.0), (14.0, 15.0), (24.0, 25.0)])
print(f"Precision: {metrics['precision']:.2f}, Recall: {metrics['recall']:.2f}")
```

***

## Advanced Techniques

### Multi-Index Search

Layer indexes for precise filtering:

```python theme={null}
# Search vehicles index
vehicle_results = video.semantic_search(
    query="red car",
    index_ids=[vehicle_index.index_id],
)

# Search motion index
motion_results = video.semantic_search(
    query="speeding",
    index_ids=[motion_index.index_id],
)

# Intersect exact timestamped shots for high precision.
motion_timestamps = {(shot.start, shot.end) for shot in motion_results.get_shots()}
final_shots = [
    shot for shot in vehicle_results.get_shots()
    if (shot.start, shot.end) in motion_timestamps
]
```

### Metadata Filtering

Pre-filter before semantic search:

```python theme={null}
# Narrow search space with metadata
results = coll.semantic_search(
    query="product demo",
    filter=[{"field": "category", "op": "==", "value": "marketing"}],
    index_names=["scene"],
)
```

### Post-Processing with LLMs

For complex queries, use an LLM to refine results:

```python theme={null}
# Get broad results first
results = video.semantic_search(
    query="numbers",
    index_names=["scene"],
    score_threshold=0.1  # Low threshold for high recall
)

# Refine with LLM
refined = llm.filter(
    results,
    criteria="Keep only results showing numbers greater than 10"
)
```

***

## Common Pitfalls

| Problem | Cause | Fix |
| :- | :- | :- |
| Missing relevant results | Threshold too high | Lower `score_threshold` |
| Too many irrelevant results | Threshold too low | Raise `score_threshold` |
| Semantic search misses an exact word or phrase | Exact terms need full-text matching | Use an [FTS index](/examples-and-tutorials/video-rag/keyword-search) |
| Poor visual search results | Generic prompt | Use specific, structured prompts |
| Inconsistent results | Wrong extraction config | Match extraction to content type |

***

## Iterative Improvement

1. **Start broad** - Low thresholds, high recall
2. **Evaluate** - Check precision on sample queries
3. **Refine** - Adjust thresholds, improve prompts
4. **Test** - Validate against ground truth
5. **Repeat** - Iterate until satisfied

***

## Next Steps

> **[Latency and Cost](/pages/understand/quality-and-evaluation/latency-and-cost)**
>
> Optimize for speed and efficiency

> **[Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts)**
>
> Better extraction strategies


# Latency and Cost
Source: https://docs.videodb.io/pages/understand/quality-and-evaluation/latency-and-cost

Trade-offs between understanding speed, retrieval performance, and API costs

Understanding and retrieval have trade-offs between speed, quality, and cost. This guide helps you optimize for your priorities with the Understand → Index → Retrieve workflow.

## Quick Reference

| Factor | Lower Cost | Higher Quality |
| :- | :- | :- |
| Frame count | 1 frame/scene | 3-5 frames/scene |
| Scene interval | 30+ seconds | 5-10 seconds |
| Segmentation | Time-based | Shot-based |
| Prompt complexity | Simple | Detailed |

***

## Indexing Cost Factors

### Frame Count and Scene Interval

Configure sampling and finite-video time segmentation during Understand. Fewer windows and frames reduce the amount of visual analysis; select the configuration that preserves the context your retrieval needs.

```python Python theme={null}
# Economical: one frame every 30 seconds
economical_understanding = video.understand(
    segmentation={"type": "time", "seconds": 30},
    analyzers=[
        {
            "type": "vlm",
            "name": "economical_scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe the scene."},
        }
    ],
)
economical_understanding.wait_until_complete()
economical_scene = economical_understanding.get_analyzer("economical_scene")
economical_index = video.index(name="economical_scene", source=economical_scene)
economical_index.wait_until_complete()

# Detailed: five frames every 10 seconds
detailed_understanding = video.understand(
    segmentation={"type": "time", "seconds": 10},
    analyzers=[
        {
            "type": "vlm",
            "name": "detailed_scene",
            "sampling": {"frame_count": 5},
            "config": {
                "prompt": "Describe the activity, how it progresses, and its content category.",
                "schema": {"description": "string", "category": "string"},
            },
        }
    ],
)
detailed_understanding.wait_until_complete()
detailed_scene = detailed_understanding.get_analyzer("detailed_scene")
detailed_index = video.index(
    name="detailed_scene",
    source=detailed_scene,
    use_for=["semantic", "query"],
    fields={"semantic": ["description"], "filter": ["category"]},
)
detailed_index.wait_until_complete()
```

```javascript Node.js theme={null}
// Economical: one frame every 30 seconds
const economicalUnderstanding = await video.understand(
    [
        {
            type: "vlm",
            name: "economical_scene",
            sampling: { frame_count: 1 },
            config: { prompt: "Describe the scene." },
        },
    ],
    { segmentation: { type: "time", seconds: 30 } },
);
await economicalUnderstanding.waitUntilComplete();
const economicalScene = await economicalUnderstanding.getAnalyzer("economical_scene");
const economicalIndex = await video.index(economicalScene, { name: "economical_scene" });
await economicalIndex.waitUntilComplete();

// Detailed: five frames every 10 seconds
const detailedUnderstanding = await video.understand(
    [
        {
            type: "vlm",
            name: "detailed_scene",
            sampling: { frame_count: 5 },
            config: {
                prompt: "Describe the activity, how it progresses, and its content category.",
                schema: { description: "string", category: "string" },
            },
        },
    ],
    { segmentation: { type: "time", seconds: 10 } },
);
await detailedUnderstanding.waitUntilComplete();
const detailedScene = await detailedUnderstanding.getAnalyzer("detailed_scene");
const detailedIndex = await video.index(detailedScene, {
    name: "detailed_scene",
    useFor: ["semantic", "query"],
    fields: { semantic: ["description"], filter: ["category"] },
});
await detailedIndex.waitUntilComplete();
```

| Config | Scenes/hour | Frames/hour | Relative Cost |
| :- | :- | :- | :- |
| 30s, 1 frame | 120 | 120 | 1x |
| 10s, 1 frame | 360 | 360 | 3x |
| 10s, 3 frames | 360 | 1,080 | 9x |
| 5s, 5 frames | 720 | 3,600 | 30x |

### Scene Interval

Shorter intervals create more artifact records and more visual-analysis work. Use a longer interval when broad topic coverage is sufficient; use a shorter interval when activity changes quickly.

```python theme={null}
long_interval = {"type": "time", "seconds": 60}  # One artifact window per minute
short_interval = {"type": "time", "seconds": 5}  # Twelve artifact windows per minute
```

***

## Indexing Latency

### Time-Based vs Shot-Based

| Method | Latency | Best For |
| :- | :- | :- |
| Time-based | Predictable | Long-form content |
| Shot-based | Variable | Edited content |

Shot segmentation adds boundary detection work, but can produce more natural units for edited content. Use `segmentation={"type": "shot", "threshold": ...}` when those boundaries matter.

### Async Processing

Understanding and semantic indexing are background jobs. Wait for the finite understanding before retrieving its analyzer, and wait for the index before semantic retrieval:

```python Python theme={null}
results = video.semantic_search(
    query="Describe the main topic",
    index_ids=[economical_index.index_id],
    top_k=5,
)
```

```javascript Node.js theme={null}
const topicResults = await video.semanticSearch(
    "Describe the main topic",
    { indexIds: [economicalIndex.indexId], topK: 5 },
);
```

For long-running applications, persist returned understanding and index IDs in your application, resume with the documented handles/status where appropriate, and avoid submitting duplicate jobs. The finite V2 examples do not require a callback API to avoid blocking an application worker.

***

## Search Latency

### Single Video vs Collection

| Scope | Latency | Use Case |
| :- | :- | :- |
| `video.semantic_search()` | Faster | Known video |
| `collection.semantic_search()` | Slower | Discovery |

### Reduce Search Space

Select the semantic index and apply indexed filters when your schema supports them:

```python Python theme={null}
results = coll.semantic_search(
    query="product demo",
    index_names=["detailed_scene"],
    filter=[{"field": "category", "op": "==", "value": "marketing"}],
    top_k=10,
)
```

```javascript Node.js theme={null}
const filteredResults = await coll.semanticSearch(
    "product demo",
    {
        indexNames: ["detailed_scene"],
        filter: [{ field: "category", op: "==", value: "marketing" }],
        topK: 10,
    },
);
```

### Limit Results

```python Python theme={null}
results = video.semantic_search(
    query="highlights",
    index_ids=[detailed_index.index_id],
    top_k=5,
)
```

```javascript Node.js theme={null}
const highlightResults = await video.semanticSearch(
    "highlights",
    { indexIds: [detailedIndex.indexId], topK: 5 },
);
```

***

## Optimization Strategies

### Tiered Indexing

The two distinct analyzer/index pairs above keep a low-cost discovery view separate from a detailed retrieval view. Search the appropriate returned index ID for the user request; do not merge the jobs unless one artifact is sufficient for the lesson or product behavior.

### Index on Demand

Create the artifact required by the requested feature, then index it once:

```python theme={null}
transcript_understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}]
)
transcript_understanding.wait_until_complete()
transcript = transcript_understanding.get_analyzer("transcript")
transcript_index = video.index(name="transcript", source=transcript)
transcript_index.wait_until_complete()
```

### Batch Processing

For large libraries, schedule bounded background work and retain the returned IDs so a completed analyzer artifact is indexed instead of recomputed:

```python theme={null}
for library_video in coll.get_videos():
    understanding = library_video.understand(
        analyzers=[
            {
                "type": "vlm",
                "name": "scene",
                "config": {"prompt": "Describe the scene."},
            }
        ]
    )
    understanding.wait_until_complete()
    scene = understanding.get_analyzer("scene")
    scene_index = library_video.index(name="scene", source=scene)
    scene_index.wait_until_complete()
```

***

## Cost Estimation

### Factors

1. **Video duration** - Longer videos create more time windows or shots.
2. **Frame sampling** - More frames increase visual-analysis work.
3. **Segmentation interval** - Shorter intervals create more artifact records.
4. **Collection size** - More videos require more background jobs.

### Example Calculations

| Video | Config | Scenes | Frames | Cost Factor |
| :- | :- | :- | :- | :- |
| 1 hour | 60s, 1 frame | 60 | 60 | 1x |
| 1 hour | 30s, 1 frame | 120 | 120 | 2x |
| 1 hour | 10s, 3 frames | 360 | 1,080 | 18x |

***

## Recommendations by Use Case

### Media Archive (Cost-Sensitive)

Use a named transcript artifact and a low-frequency visual artifact when the library needs broad coverage:

```python theme={null}
archive_understanding = video.understand(
    segmentation={"type": "time", "seconds": 60},
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {
            "type": "vlm",
            "name": "archive_scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe the main content."},
        },
    ],
)
archive_understanding.wait_until_complete()
```

### Security Monitoring (Quality-First)

For a finite review video, use a short time interval and enough frames to capture activity progression:

```python theme={null}
security_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[
        {
            "type": "vlm",
            "name": "security_scene",
            "sampling": {"frame_count": 3},
            "config": {"prompt": "Identify people, vehicles, and suspicious activity."},
        }
    ],
)
security_understanding.wait_until_complete()
```

### E-commerce (Balanced)

Use shot segmentation when the video is edited around product reveals:

```python theme={null}
commerce_understanding = video.understand(
    segmentation={"type": "shot", "threshold": 20},
    analyzers=[
        {
            "type": "vlm",
            "name": "product_scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Identify products, brands, and pricing."},
        }
    ],
)
commerce_understanding.wait_until_complete()
```

***

## Monitoring

Measure the complete lifecycle rather than only request submission:

```python theme={null}
import time

start = time.time()
understanding = video.understand(
    analyzers=[{"type": "vlm", "name": "scene", "config": {"prompt": "Describe the scene."}}]
)
understanding.wait_until_complete()
scene = understanding.get_analyzer("scene")
index = video.index(name="measured_scene", source=scene)
index.wait_until_complete()
indexing_time = time.time() - start

start = time.time()
results = video.semantic_search(query="query", index_ids=[index.index_id], top_k=10)
search_time = time.time() - start
print(f"Indexing took {indexing_time:.2f}s; retrieval took {search_time:.3f}s")
```

***

## Next Steps

> **[Accuracy Tips](/pages/understand/quality-and-evaluation/accuracy-tips)**
>
> Improve search precision

> **[Create an Index](/pages/understand/indexing-pipelines/create-an-index)**
>
> Get started with indexing


# Search Patterns and Results
Source: https://docs.videodb.io/pages/understand/search-and-retrieval/collection-search



## Find moments from a natural-language request

Use intelligent search when the user knows the goal but the application should not have to select indexes or retrieval methods.

```python theme={null}
response = collection.search(
    query="Find scenes where a red car enters a parking garage at night",
    top_k=10,
    return_fields=["scene", "objects"],
)

for shot in response:
    print(shot.video_id, shot.start, shot.end)
```

The same pattern works inside one video:

```python theme={null}
response = video.search(
    query="Find the moment the presenter reveals the final design",
    top_k=10,
)
```

## Search by meaning

Use semantic search when the application should control which semantic indexes are searched.

```python theme={null}
results = video.semantic_search(
    query="a cyclist riding through heavy rain at night",
    index_names=["scene"],
    top_k=20,
    score_threshold=0.7,
)
```

Add exact conditions when a semantic match must satisfy additional criteria:

```python theme={null}
results = collection.semantic_search(
    query="an athlete using a phone near the finish line",
    index_names=["scene"],
    filter=[
        {"field": "brand_names", "op": "contains", "value": "Nike"},
        {"field": "object_labels", "op": "contains", "value": "cell phone"},
    ],
    top_k=20,
)
```

## Filter search results

Filters are supported by default-mode `search()`, `semantic_search()`, `query()`, and `aggregate()`. A top-level list combines conditions with AND.

```python theme={null}
filter=[
    {"field": "brand_names", "op": "contains", "value": "Nike"},
    {"field": "max_score", "op": ">=", "value": 0.8},
]
```

Use an explicit OR or NOT when the conditions should be combined differently:

```python theme={null}
filter={
    "or": [
        {"field": "brand_names", "op": "contains", "value": "Nike"},
        {"field": "brand_names", "op": "contains", "value": "Adidas"},
    ]
}
```

```python theme={null}
filter={
    "not": {"field": "location_type", "op": "==", "value": "indoor"}
}
```

### Filter operators

| Operator | Example | Meaning |
| - | - | - |
| `==` | `{"field": "emotion", "op": "==", "value": "joy"}` | Field equals the value |
| `contains` | `{"field": "label", "op": "contains", "value": "car"}` | Text or an array contains the value |
| `in` | `{"field": "brand_names", "op": "in", "value": ["Nike", "Adidas"]}` | Field matches one of the values |
| `>=` | `{"field": "max_score", "op": ">=", "value": 0.8}` | Numeric lower bound |
| `<=` | `{"field": "max_score", "op": "<=", "value": 0.9}` | Numeric upper bound |
| `exists` | `{"field": "text", "op": "exists", "value": True}` | Field exists |

See [How index and field targeting works](/pages/understand/search-and-retrieval/natural-language-query#how-index-and-field-targeting-works) for selector and filter scoping examples.

## Find every exact match

Use query for deterministic filtering and sorting on one index. This works well for review queues, inventories, dataset preparation, and compliance workflows.

```python theme={null}
results = collection.query(
    index_name="objects",
    filter=[
        {"field": "label", "op": "contains", "value": "cell phone"},
        {"field": "max_score", "op": ">=", "value": 0.8},
    ],
    limit=100,
    sort=[("max_score", "desc")],
)
```

## Turn indexed signals into analytics

Use aggregation for counts, groups, and facets rather than moment playback.

```python theme={null}
response = collection.aggregate(
    index_name="brands",
    group_by="brand_names",
    metric="count",
    limit=100,
)

for row in response["results"]:
    print(row)
```

Apply a filter before grouping when the analysis should cover only part of the indexed data:

```python theme={null}
response = collection.aggregate(
    index_name="scene",
    filter=[
        {"field": "location_type", "op": "==", "value": "indoor"},
    ],
    group_by="emotion",
    metric="count",
    limit=100,
)

indoor_emotions = response["results"]
```

Aggregation returns its rows in `response["results"]`, so playback methods do not apply to them. Read optional migration information with `response.get("warnings", [])`.

## Investigate and refine

Use DeepSearch for questions that benefit from multiple retrieval steps or follow-up requests.

```python theme={null}
response = collection.search(
    query="Trace how the crowd changes before, during, and after the winning goal",
    mode="deepsearch",
    top_k=10,
)

followup = collection.search(
    query="Only include moments where the home supporters are visible",
    mode="deepsearch",
    session_id=response.session_id,
    top_k=10,
)
```

## Answer with supporting moments

Use Ask when the application should present a synthesized answer. Request sources when users should be able to inspect the supporting video moments.

```python theme={null}
answer = collection.ask(
    question="How does the storefront change from daytime to night?",
    include_sources=True,
)

print(answer.answer)

for source in answer.sources:
    print(source.video_id, source.start, source.end)
```

## Inspect a result

Moment-returning methods provide `Shot` objects. A shot identifies where the result came from and where it occurs in the video.

```python theme={null}
shot = response.shots[0]

print(shot.video_id)
print(shot.video_title)
print(shot.start, shot.end)
print(shot.search_score)
print(shot.text)
```

Use `return_fields` when the interface should also show stored index evidence:

```python theme={null}
response = collection.search(
    query="Find someone discussing Nike while holding a phone",
    return_fields=["scene", "transcript", "objects", "brands"],
)

for shot in response:
    evidence = shot.metadata.get("indexes", {})
    print(evidence.get("transcript", []))
    print(evidence.get("objects", []))
    print(evidence.get("brands", []))
```

## Play one matching moment

Call `play()` to open an individual shot in the VideoDB player:

```python theme={null}
shot = response.shots[0]
shot.play()
```

Use `generate_stream()` when the application needs the playable HLS URL:

```python theme={null}
stream_url = shot.generate_stream()
print(stream_url)
```

`generate_stream()` returns an existing stream URL when one is already available and generates one otherwise.

## Compile several results

Direct semantic search and query return a `SearchResult`. Compile all of its shots into one stream:

```python theme={null}
results = video.semantic_search(
    query="sunset over a city skyline",
    index_names=["scene"],
)

stream_url = results.compile()
```

High-level Search and DeepSearch return a `SearchResponse`. Compile the nested `SearchResult` when the response contains shots:

```python theme={null}
response = collection.search(
    query="Find the best wildlife moments",
)

if response.response_type in {"shots", "deepsearch"}:
    stream_url = response.results.compile()
```

You can also play the compiled result directly:

```python theme={null}
response.results.play()
```

## Embed a result

Generate an iframe for an individual moment:

```python theme={null}
shot = response.shots[0]
iframe = shot.get_embed_code()
```

Or embed a compiled set of results:

```python theme={null}
iframe = response.results.get_embed_code()
```

The embed helper generates the required player URL when it is not already available.

## Build common search experiences

| Experience | Useful result workflow |
| - | - |
| Search results page | Iterate over shots and show title, timestamps, score, and a play action |
| Evidence panel | Request `return_fields` and render `shot.metadata["indexes"]` |
| Highlight reel | Compile a `SearchResult` into one stream |
| Video Q\&A | Call `ask(include_sources=True)` and make each source playable |
| Investigation workspace | Continue DeepSearch with `session_id` and preserve prior results |
| Analytics dashboard | Render aggregate rows as counts, groups, or facets |

## Video and collection scope

Use `video.*` when the search should stay inside one known video. Use `collection.*` when a result may come from any indexed video in the collection.

```python theme={null}
video.search(query="Find the red car entering the garage")
collection.search(query="Find red cars across the traffic footage")
```

The result and playback workflows remain the same at both scopes.

## Next steps

> **[Intelligent Search](/pages/understand/search-and-retrieval/intelligent-search)**
>
> Learn when to use Search, DeepSearch, and Ask.

> **[Results and Evidence](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> Explore Shot fields, returned index data, and stream generation in detail.


# Intelligent Search
Source: https://docs.videodb.io/pages/understand/search-and-retrieval/intelligent-search

Use Search, DeepSearch, and Ask when you want VideoDB to plan retrieval from natural language.

Use intelligent search when you know the goal, but do not want to manually choose indexes and retrieval methods.

VideoDB interprets the request, plans retrieval over available indexes, and returns timestamped evidence or a grounded answer.

| Method | Best for | Returns |
| - | - | - |
| `search()` | Natural-language retrieval across one or more signals | Matching moments or aggregate rows |
| `search(mode="deepsearch")` | Complex investigation, refinement, and follow-up questions | Matching moments plus session state |
| `ask()` | A synthesized answer grounded in indexed video | An answer and optional source moments |

## How intelligent search works

Intelligent search sits above the direct retrieval methods. Your application describes the goal, and VideoDB handles the retrieval plan.

```text theme={null}
Natural-language request
        ↓
Interpret the intent and available index schemas
        ↓
Use one or more of: semantic search · structured query · aggregation
        ↓
Return moments, analytics, or a grounded answer
```

The plan can draw on several indexed signals, such as spoken words, scene descriptions, objects, brands, OCR, and custom structured fields. Results retain their source video and timestamps, so the output remains connected to playable evidence.

## Ways to use intelligent search

The right intelligent method depends on the outcome the user wants. Consider a collection of customer interviews.

### Find the relevant moments

```python theme={null}
response = collection.search(
    query="Find moments where customers object to pricing",
)
```

Use Search when the result should be ranked, timestamped moments or an aggregate selected from the request.

### Investigate and refine

```python theme={null}
response = collection.search(
    query="Find the strongest pricing objections and identify which ones mention competitors",
    mode="deepsearch",
)
```

Use DeepSearch when the task benefits from multiple retrieval steps, clarification, or follow-up questions.

### Explain the findings

```python theme={null}
answer = collection.ask(
    question="What are the main pricing objections?",
    include_sources=True,
)
```

Use Ask when the application needs a synthesized answer with optional source moments.

| User intent | Result |
| - | - |
| “Find product demonstrations” | Ranked, timestamped moments |
| “How many scenes contain each brand?” | Aggregate rows |
| “Investigate negative reactions, then refine the result” | DeepSearch results and session state |
| “What are the main customer objections?” | A grounded answer with optional sources |

## Search responses

`search()` returns a `SearchResponse` because the planned result can take more than one form.

| Attribute | Description |
| - | - |
| `response_type` | Usually `"shots"`; can be `"aggregate"` when the request asks for grouped analytics. |
| `results` | A `SearchResult` for moments, or aggregate rows for an aggregate response. |
| `shots` | Convenience list of `Shot` objects for moment responses. |
| `trace` | Optional planner trace for debugging. |

Compile matching moments into one stream through the nested `SearchResult`:

```python theme={null}
if response.response_type == "shots":
    stream_url = response.results.compile()
```

## DeepSearch: multi-step investigation

Use DeepSearch when one retrieval step is unlikely to be enough, or when a user needs to refine the result through follow-up requests.

```python theme={null}
response = collection.search(
    query="find the strongest examples of customers objecting to pricing",
    mode="deepsearch",
    top_k=10,
    return_fields="all",
)

print(response.session_id)
```

Continue the investigation with the returned session ID:

```python theme={null}
followup = collection.search(
    query="only keep moments where a competitor is mentioned",
    mode="deepsearch",
    session_id=response.session_id,
    top_k=10,
)
```

DeepSearch responses can include:

| Attribute | Description |
| - | - |
| `session_id` | ID used to continue the investigation. |
| `waiting_for` | Current DeepSearch waiting state. |
| `clarification` | A question from DeepSearch when it needs more detail before continuing. |
| `results` / `shots` | Retrieved timestamped moments. |

For example, DeepSearch might ask:

> Which product line or time period should I focus on?

Answer by continuing the same DeepSearch session:

```python theme={null}
if response.clarification:
    print(response.clarification)

followup = collection.search(
    query="Focus on the enterprise plan discussed in the Q4 interviews",
    mode="deepsearch",
    session_id=response.session_id,
)
```

> [!NOTE]
>
> DeepSearch supports `top_k`, `session_id`, and `return_fields`. Filters, sorting, score thresholds, planner traces, and index selectors are not supported in DeepSearch mode.

## Ask: answers grounded in video

Use `ask()` when the desired output is an answer rather than a list of matching moments.

```python theme={null}
answer = collection.ask(
    question="What objections do customers raise about pricing?",
    top_k=15,
    include_sources=True,
)

print(answer.answer)
```

When `include_sources=True`, the response includes the timestamped moments used as evidence:

```python theme={null}
for source in answer.sources:
    print(source.video_id, source.start, source.end)
    print(source.stream_url or source.generate_stream())
```

Ask is available at both scopes:

```python theme={null}
video.ask(
    question="What happens after the demo?",
    include_sources=True,
)

collection.ask(
    question="Which videos mention Nike?",
    include_sources=True,
)
```

## Choose an intelligent method

| Your goal | Use |
| - | - |
| Find relevant moments from a natural-language request | `search()` |
| Investigate a complex request over multiple steps | `search(mode="deepsearch")` |
| Continue refining a previous investigation | DeepSearch with `session_id` |
| Receive a concise answer with optional evidence | `ask()` |
| Control the exact retrieval operation or index | [Direct retrieval methods](/pages/understand/search-and-retrieval/natural-language-query#direct-retrieval-methods) |

## Next steps

> **[Direct Retrieval](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Use semantic search, query, and aggregation directly.

> **[Results and Evidence](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> Work with shots, timestamps, returned fields, and streams.


# Search and Retrieval
Source: https://docs.videodb.io/pages/understand/search-and-retrieval/natural-language-query

Find exact moments, query and aggregate indexed data, and answer questions across your video library.

VideoDB gives you two ways to retrieve indexed video:

* Use **direct retrieval methods** when you know how the data should be searched, filtered, or grouped.
* Use **intelligent search** when you want VideoDB to interpret a natural-language request and choose the retrieval strategy.

> **[Try the search notebook](https://github.com/video-db/videodb-cookbook/blob/preview/guides/indexing-v2/search/search_guide.ipynb)**
>
> Explore a runnable guide to searching and retrieving indexed video.

## Direct retrieval methods

| Method | Use it when you need | Returns |
| - | - | - |
| `semantic_search()` | Moments similar in meaning to a text query | `SearchResult` with timestamped `Shot` objects |
| `query()` | Exact filters, conditions, or sorting on one index | `SearchResult` with timestamped `Shot` objects |
| `aggregate()` | Counts, groups, facets, or metrics on one index | Aggregate rows as a `list` or `dict` |

## Let VideoDB choose the retrieval strategy

You do not always need to select a direct method yourself. Intelligent search interprets the request and uses the relevant retrieval capabilities internally.

* `search()` plans retrieval for a natural-language request.
* `search(mode="deepsearch")` performs deeper, multi-step retrieval and supports follow-up sessions.
* `ask()` retrieves evidence and synthesizes a grounded answer.

### One request, multiple video signals

Consider this request:

> Find moments where someone discusses Nike while holding a phone outside a store.

A useful answer may depend on several indexed signals:

| Signal | What it contributes |
| - | - |
| Transcript | Whether someone discusses Nike |
| Object detections | Whether a phone is present |
| Brand detections | Whether Nike appears in the scene |
| Scene understanding | Whether the person is outside a store |
| Temporal data | The exact start and end of the matching moment |

With intelligent search, the application sends the goal rather than manually coordinating each retrieval step:

```python theme={null}
response = collection.search(
    query="Find moments where someone discusses Nike while holding a phone outside a store",
    top_k=10,
    return_fields=["scene", "transcript", "objects", "brands"],
)

for shot in response:
    print(shot.video_id, shot.start, shot.end)
    print(shot.generate_stream())
```

The result keeps the timestamped evidence and can hydrate the relevant index rows through `return_fields`.

> **[Intelligent Search](/pages/understand/search-and-retrieval/intelligent-search)**
>
> Learn when to use Search, DeepSearch, and Ask.

## Index selection at a glance

With direct retrieval, you choose the indexes. With intelligent search, VideoDB selects them based on your request and the available index schemas.

| Method | Accepted selectors | Selection rule |
| - | - | - |
| `semantic_search()` | `index_names` and/or `index_ids` | Select zero, one, or many indexes. If neither is provided, VideoDB searches all semantic indexes in scope. |
| `query()` | `index_name` or `index_id` | Select exactly one index using either `index_name` or `index_id`. |
| `aggregate()` | `index_name` or `index_id` | Select exactly one index using either `index_name` or `index_id`. |
| `search()` (`default` and `deepsearch` modes) | None | Index selectors are not accepted. VideoDB chooses the indexes and retrieval strategy. |
| `ask()` (`default` and `deepsearch` modes) | None | Index selectors are not accepted. VideoDB chooses the indexes used as answer context. |

Plural selectors (`index_names`, `index_ids`) support multi-index retrieval. Singular selectors (`index_name`, `index_id`) identify one index.

### How index and field targeting works

An index name searches all applicable fields inside that index. Add a field path when you want to target one specific field.

```python theme={null}
# Search all semantic fields inside the scene index
results = collection.semantic_search(
    query="a busy street at night",
    index_names=["scene"],
)

# Search only outputs.location inside the scene index
results = collection.semantic_search(
    query="a busy street at night",
    index_names=["scene.outputs.location"],
)
```

Filter fields follow the same targeting pattern. Suppose both the `transcript` and `ocr` indexes contain a `text` field.

An unqualified field checks `text` in every index in scope where that field is present:

```python theme={null}
response = collection.search(
    query="Find moments about emergency exits",
    filter=[
        {"field": "text", "op": "contains", "value": "EXIT"},
    ],
)
```

Qualify the field when the condition should use only text detected on screen by the `ocr` index:

```python theme={null}
response = collection.search(
    query="Find moments about emergency exits",
    filter=[
        {"field": "ocr.text", "op": "contains", "value": "EXIT"},
    ],
)
```

| Form | Meaning |
| - | - |
| `index_name` | Search all applicable fields inside that index. |
| `field` | Apply the condition to every index in scope that contains the field. |
| `index_name.field` | Target one field inside one specific index. |

For semantic refs, an exact index-name match wins first. Otherwise, VideoDB treats the first segment as the index name and the remainder as its field path.

For `query()` and `aggregate()`, the singular `index_name` or `index_id` already limits the operation to one index.

## Direct retrieval in detail

Use the direct methods when your application should control the retrieval operation and index selection.

### Find similar moments with `semantic_search()`

Use semantic search when you know the meaning you want to match and which semantic indexes should be searched.

```python theme={null}
results = video.semantic_search(
    query="a customer reacting positively to a product",
    index_names=["scene"],
    top_k=10,
    score_threshold=0.7,
)

for shot in results:
    print(shot.start, shot.end, shot.search_score)
```

You can select indexes by name or ID. If you omit both selectors, VideoDB searches all semantic indexes in the video or collection scope.

```python theme={null}
results = collection.semantic_search(
    query="person presenting at a podium",
    index_ids=["idx_scene_123"],
    top_k=20,
)
```

Add a filter when semantic similarity should also satisfy an explicit condition:

```python theme={null}
results = collection.semantic_search(
    query="customer discussing a product launch",
    index_names=["scene"],
    filter=[
        {"field": "brand_names", "op": "contains", "value": "Nike"},
    ],
    top_k=20,
)
```

> [!NOTE]
>
> Filters must use fields supported by the indexes available to the search. Inspect an index's `field_schema` before relying on a field or operator in production.

### Apply exact conditions with `query()`

Use query when you know the index and the exact conditions to apply. Query does not interpret natural language or calculate semantic similarity.

```python theme={null}
results = collection.query(
    index_name="objects",
    filter=[
        {"field": "label", "op": "contains", "value": "cell phone"},
    ],
    limit=100,
    sort=[("max_score", "desc")],
)
```

Select exactly one index using `index_name` or `index_id`.

```python theme={null}
results = video.query(
    index_id="idx_objects_123",
    filter=[{"field": "label", "op": "==", "value": "person"}],
    limit=50,
)
```

### Count and group with `aggregate()`

Use aggregation when you need analytics rather than matching moments. `group_by` is required and must reference an aggregatable field.

```python theme={null}
response = collection.aggregate(
    index_name="brands",
    group_by="brand_names",
    metric="count",
    limit=100,
)

for row in response["results"]:
    print(row)
```

Filters can narrow the rows included in the aggregation:

```python theme={null}
response = collection.aggregate(
    index_name="objects",
    filter=[{"field": "max_score", "op": ">=", "value": 0.8}],
    group_by="label",
    metric="count",
    limit=100,
)

counts = response["results"]
```

Unlike semantic search and query, aggregation returns an envelope rather than `Shot` objects. Read the aggregate rows from `response["results"]` and optional migration warnings with `response.get("warnings", [])`.

## Which direct method should I use?

| If you know\... | Use |
| - | - |
| The meaning you want to match | `semantic_search()` |
| The exact condition to apply | `query()` |
| What you want to count or group | `aggregate()` |
| Only the goal, not the retrieval strategy | [Intelligent Search](/pages/understand/search-and-retrieval/intelligent-search) |

## Video and collection scope

The same methods are available at both scopes:

```python theme={null}
video.semantic_search(...)
collection.semantic_search(...)

video.query(...)
collection.query(...)

video.aggregate(...)
collection.aggregate(...)
```

Use a video method for one video. Use a collection method to search every indexed video in a collection.

## Next steps

> **[Intelligent Search](/pages/understand/search-and-retrieval/intelligent-search)**
>
> Let VideoDB plan retrieval or synthesize an answer.

> **[Results and Evidence](/pages/understand/search-and-retrieval/timestamps-clips-streams)**
>
> Work with shots, timestamps, returned fields, and streams.

> **[Collection Search](/pages/understand/search-and-retrieval/collection-search)**
>
> Apply retrieval methods across a video library.


# Results, Evidence, and Returned Fields
Source: https://docs.videodb.io/pages/understand/search-and-retrieval/timestamps-clips-streams

Work with timestamped search results, supporting evidence, and playable video streams.

Moment-returning APIs provide timestamped `Shot` objects for the matching parts of a video.

Each `Shot` identifies the source video and exact time range, so an application can inspect the evidence, play an individual moment, or combine several results into one stream. Use `return_fields` to include the stored index rows associated with each result.

```text theme={null}
Search request
      ↓
Timestamped Shot
      ├── source video and time range
      ├── relevance score and matched text
      ├── hydrated index evidence
      └── playable stream
```

| Result layer | What it provides |
| - | - |
| Identity | `video_id` and `video_title` identify the source. |
| Time | `start` and `end` locate the exact moment. |
| Relevance | `search_score` and optional text explain the match strength and content. |
| Evidence | `metadata["indexes"]` contains requested index rows. |
| Playback | `stream_url`, `player_url`, or `generate_stream()` makes the moment playable. |

***

## Basic result

```python theme={null}
response = video.search(
    query="find moments where a red car drives away",
)

for shot in response:
    print(shot.start, shot.end, shot.search_score)
    print(shot.generate_stream())
```

A `Shot` includes:

| Field | Type | Description |
| - | -: | - |
| `video_id` | string | Source video ID. |
| `video_length` | float | Source video duration in seconds. |
| `video_title` | string | Source video title. |
| `start` | float | Start time in seconds. |
| `end` | float | End time in seconds. |
| `text` | string or `None` | Optional matched text-like value. |
| `search_score` | float | Relevance score, when applicable. |
| `scene_index_id` | string or `None` | Index ID associated with the match, when available. |
| `scene_index_name` | string or `None` | Index name associated with the match, when available. |
| `metadata` | dict or `None` | Result metadata plus hydrated rows under `metadata["indexes"]`. |
| `stream_url` | string or `None` | Playable stream URL for this shot when already generated. |
| `player_url` | string or `None` | Player URL for this shot when already generated. |

Example:

```python theme={null}
Shot(
    video_id="m-123",
    video_length=120.5,
    video_title="Store walkthrough",
    start=3.42,
    end=7.10,
    text="Just grabbing my Nike bag...",
    search_score=0.87,
    scene_index_id="idx-123",
    scene_index_name="scene",
    metadata={
        "indexes": {
            "objects": [
                {"label": "person", "count": 1, "max_score": 0.93},
                {"label": "cell phone", "count": 1, "max_score": 0.91},
            ]
        }
    },
    stream_url="https://stream.videodb.io/v3/published/manifests/result.m3u8",
    player_url="https://console.videodb.io/player/...",
)
```

***

## APIs that return shots

These APIs return `Shot` objects:

```python theme={null}
# High-level v2 search: returns SearchResponse
response = video.search(query="happy reunion")
shots = response.shots

# Direct semantic search: returns SearchResult
semantic_results = video.semantic_search(query="happy reunion", index_names=["scene"])
shots = semantic_results.get_shots()

# Structured query: returns SearchResult
query_results = video.query(
    index_name="objects",
    filter=[{"field": "label", "op": "contains", "value": "cell phone"}],
    limit=100,
)
shots = query_results.get_shots()
```

`aggregate()` does not return shots; it returns aggregate rows. `ask()` returns an answer plus optional source shots.

***

## Play a result

You can play an individual shot:

```python theme={null}
response.shots[0].play()
```

Or use the URL directly when present:

```python theme={null}
url = response.shots[0].stream_url
```

If a stream URL is not already present, generate it from the shot timestamps:

```python theme={null}
url = response.shots[0].generate_stream()
```

***

## Return hydrated index rows

`return_fields` selects which indexes to hydrate for each matched scene. Hydrated rows are available in `shot.metadata["indexes"]`, not `shot.fields`.

For a single index:

```python theme={null}
results = video.semantic_search(
    query="person holding a phone",
    index_names=["scene"],
    top_k=10,
    return_fields=["scene"],
)

for shot in results:
    scene_rows = shot.metadata.get("indexes", {}).get("scene", [])
    print(scene_rows)
```

For high-level search across multiple indexes:

```python theme={null}
response = collection.search(
    query="person holding a phone",
    top_k=10,
    mode="default",
    return_fields=["scene", "objects"],
)

for shot in response:
    indexes = shot.metadata.get("indexes", {})
    print(indexes.get("scene", []))
    print(indexes.get("objects", []))
```

A hydrated `Shot` then looks like:

```python theme={null}
Shot(
    video_id="m-123",
    video_length=120.5,
    video_title="Store walkthrough",
    start=3.42,
    end=7.10,
    text="A person holds a phone while speaking.",
    search_score=0.91,
    scene_index_id="idx_scene_123",
    scene_index_name="scene",
    metadata={
        "indexes": {
            "scene": [
                {"scene_description": "A person holds a phone while speaking."}
            ],
            "objects": [
                {"label": "person", "count": 1, "max_score": 0.93},
                {"label": "cell phone", "count": 1, "max_score": 0.91},
            ],
        }
    },
    stream_url="https://stream.videodb.io/...",
    player_url="https://console.videodb.io/player/...",
)
```

For debugging, request all stored index rows for each matched scene:

```python theme={null}
response = video.search(
    query="product demo moments",
    top_k=10,
    return_fields="all",
)
```

Accepted `return_fields` values:

| Value | Meaning |
| - | - |
| `None` | No additional hydration. |
| `"all"` or `"*"` | Hydrate all available indexes for the matched scene. |
| string | Hydrate one index by name or ID. |
| list | Hydrate selected indexes by name or ID. |
| dict | Dict keys select indexes; values are accepted for compatibility but are not field projections. |

> [!NOTE]
>
> `return_fields` currently selects indexes, not individual fields. A hydrated index value is a list because one scene can map to multiple rows, such as several object labels or brand mentions.

***

## Compile many results into one stream

For `semantic_search()` or `query()` results, compile the `SearchResult` directly:

```python theme={null}
results = video.semantic_search(query="product demo", index_names=["scene"])
stream_url = results.compile()
```

For high-level `search()`, compile the nested `SearchResult`:

```python theme={null}
response = video.search(query="product demo")
stream_url = response.results.compile()
```

Or build a stream from `Shot` timestamps:

```python theme={null}
timestamps = [(shot.start, shot.end) for shot in response]
stream_url = video.generate_stream(timestamps)
```

Generated streams are playable HLS URLs.

```text theme={null}
https://stream.videodb.io/v3/published/manifests/{manifest-id}.m3u8
```

***

## Ask sources

`ask()` returns an answer and, when `include_sources=True`, supporting `Shot` objects.

```python theme={null}
answer = collection.ask(
    question="Which brands are mentioned?",
    include_sources=True,
)

print(answer.answer)

for source in answer.sources:
    print(source.video_id, source.start, source.end)
    print(source.generate_stream())
```

***

## Result controls

| Option | Type | Default | Description |
| - | -: | - | - |
| `return_fields` | string, list, dict, or `"all"` | none | Select indexes to hydrate under `shot.metadata["indexes"]`. |
| `top_k` | int | API-specific | Maximum returned moments for search-style APIs. |
| `score_threshold` | float | none | Minimum semantic score for `semantic_search()`. |
| `limit` | int | API-specific | Maximum rows for `query()` and `aggregate()`. |

Hydrated index rows can make responses larger. Keep `return_fields` narrow for broad search previews and use `"all"` mainly for debugging or final inspection.

***

## Next steps

> **[Search APIs](/pages/understand/search-and-retrieval/natural-language-query)**
>
> Retrieve indexed moments.

> **[Hybrid and Collection Search](/pages/understand/search-and-retrieval/collection-search)**
>
> Combine results across indexes and videos.