Latency and Cost

Understanding and retrieval have trade-offs between speed, quality, and cost. This guide helps you optimize for your priorities with the Understand → Index → Retrieve workflow.

Quick Reference

Factor	Lower Cost	Higher Quality
Frame count	1 frame/scene	3-5 frames/scene
Scene interval	30+ seconds	5-10 seconds
Segmentation	Time-based	Shot-based
Prompt complexity	Simple	Detailed

Indexing Cost Factors

Frame Count and Scene Interval

Configure sampling and finite-video time segmentation during Understand. Fewer windows and frames reduce the amount of visual analysis; select the configuration that preserves the context your retrieval needs.

# Economical: one frame every 30 seconds
economical_understanding = video.understand(
    segmentation={"type": "time", "seconds": 30},
    analyzers=[
        {
            "type": "vlm",
            "name": "economical_scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe the scene."},
        }
    ],
)
economical_understanding.wait_until_complete()
economical_scene = economical_understanding.get_analyzer("economical_scene")
economical_index = video.index(name="economical_scene", source=economical_scene)
economical_index.wait_until_complete()

# Detailed: five frames every 10 seconds
detailed_understanding = video.understand(
    segmentation={"type": "time", "seconds": 10},
    analyzers=[
        {
            "type": "vlm",
            "name": "detailed_scene",
            "sampling": {"frame_count": 5},
            "config": {
                "prompt": "Describe the activity, how it progresses, and its content category.",
                "schema": {"description": "string", "category": "string"},
            },
        }
    ],
)
detailed_understanding.wait_until_complete()
detailed_scene = detailed_understanding.get_analyzer("detailed_scene")
detailed_index = video.index(
    name="detailed_scene",
    source=detailed_scene,
    use_for=["semantic", "query"],
    fields={"semantic": ["description"], "filter": ["category"]},
)
detailed_index.wait_until_complete()

// Economical: one frame every 30 seconds
const economicalUnderstanding = await video.understand(
    [
        {
            type: "vlm",
            name: "economical_scene",
            sampling: { frame_count: 1 },
            config: { prompt: "Describe the scene." },
        },
    ],
    { segmentation: { type: "time", seconds: 30 } },
);
await economicalUnderstanding.waitUntilComplete();
const economicalScene = await economicalUnderstanding.getAnalyzer("economical_scene");
const economicalIndex = await video.index(economicalScene, { name: "economical_scene" });
await economicalIndex.waitUntilComplete();

// Detailed: five frames every 10 seconds
const detailedUnderstanding = await video.understand(
    [
        {
            type: "vlm",
            name: "detailed_scene",
            sampling: { frame_count: 5 },
            config: {
                prompt: "Describe the activity, how it progresses, and its content category.",
                schema: { description: "string", category: "string" },
            },
        },
    ],
    { segmentation: { type: "time", seconds: 10 } },
);
await detailedUnderstanding.waitUntilComplete();
const detailedScene = await detailedUnderstanding.getAnalyzer("detailed_scene");
const detailedIndex = await video.index(detailedScene, {
    name: "detailed_scene",
    useFor: ["semantic", "query"],
    fields: { semantic: ["description"], filter: ["category"] },
});
await detailedIndex.waitUntilComplete();

Config	Scenes/hour	Frames/hour	Relative Cost
30s, 1 frame	120	120	1x
10s, 1 frame	360	360	3x
10s, 3 frames	360	1,080	9x
5s, 5 frames	720	3,600	30x

Scene Interval

Shorter intervals create more artifact records and more visual-analysis work. Use a longer interval when broad topic coverage is sufficient; use a shorter interval when activity changes quickly.

long_interval = {"type": "time", "seconds": 60}  # One artifact window per minute
short_interval = {"type": "time", "seconds": 5}  # Twelve artifact windows per minute

Indexing Latency

Time-Based vs Shot-Based

Method	Latency	Best For
Time-based	Predictable	Long-form content
Shot-based	Variable	Edited content

Shot segmentation adds boundary detection work, but can produce more natural units for edited content. Use segmentation={"type": "shot", "threshold": ...} when those boundaries matter.

Async Processing

Understanding and semantic indexing are background jobs. Wait for the finite understanding before retrieving its analyzer, and wait for the index before semantic retrieval:

results = video.semantic_search(
    query="Describe the main topic",
    index_ids=[economical_index.index_id],
    top_k=5,
)

const topicResults = await video.semanticSearch(
    "Describe the main topic",
    { indexIds: [economicalIndex.indexId], topK: 5 },
);

For long-running applications, persist returned understanding and index IDs in your application, resume with the documented handles/status where appropriate, and avoid submitting duplicate jobs. The finite V2 examples do not require a callback API to avoid blocking an application worker.

Search Latency

Single Video vs Collection

Scope	Latency	Use Case
`video.semantic_search()`	Faster	Known video
`collection.semantic_search()`	Slower	Discovery

Reduce Search Space

Select the semantic index and apply indexed filters when your schema supports them:

results = coll.semantic_search(
    query="product demo",
    index_names=["detailed_scene"],
    filter=[{"field": "category", "op": "==", "value": "marketing"}],
    top_k=10,
)

const filteredResults = await coll.semanticSearch(
    "product demo",
    {
        indexNames: ["detailed_scene"],
        filter: [{ field: "category", op: "==", value: "marketing" }],
        topK: 10,
    },
);

Limit Results

results = video.semantic_search(
    query="highlights",
    index_ids=[detailed_index.index_id],
    top_k=5,
)

const highlightResults = await video.semanticSearch(
    "highlights",
    { indexIds: [detailedIndex.indexId], topK: 5 },
);

Optimization Strategies

Tiered Indexing

The two distinct analyzer/index pairs above keep a low-cost discovery view separate from a detailed retrieval view. Search the appropriate returned index ID for the user request; do not merge the jobs unless one artifact is sufficient for the lesson or product behavior.

Index on Demand

Create the artifact required by the requested feature, then index it once:

transcript_understanding = video.understand(
    analyzers=[{"type": "spoken_words", "name": "transcript"}]
)
transcript_understanding.wait_until_complete()
transcript = transcript_understanding.get_analyzer("transcript")
transcript_index = video.index(name="transcript", source=transcript)
transcript_index.wait_until_complete()

Batch Processing

For large libraries, schedule bounded background work and retain the returned IDs so a completed analyzer artifact is indexed instead of recomputed:

for library_video in coll.get_videos():
    understanding = library_video.understand(
        analyzers=[
            {
                "type": "vlm",
                "name": "scene",
                "config": {"prompt": "Describe the scene."},
            }
        ]
    )
    understanding.wait_until_complete()
    scene = understanding.get_analyzer("scene")
    scene_index = library_video.index(name="scene", source=scene)
    scene_index.wait_until_complete()

Cost Estimation

Factors

Video duration - Longer videos create more time windows or shots.
Frame sampling - More frames increase visual-analysis work.
Segmentation interval - Shorter intervals create more artifact records.
Collection size - More videos require more background jobs.

Example Calculations

Video	Config	Scenes	Frames	Cost Factor
1 hour	60s, 1 frame	60	60	1x
1 hour	30s, 1 frame	120	120	2x
1 hour	10s, 3 frames	360	1,080	18x

Recommendations by Use Case

Media Archive (Cost-Sensitive)

Use a named transcript artifact and a low-frequency visual artifact when the library needs broad coverage:

archive_understanding = video.understand(
    segmentation={"type": "time", "seconds": 60},
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {
            "type": "vlm",
            "name": "archive_scene",
            "sampling": {"frame_count": 1},
            "config": {"prompt": "Describe the main content."},
        },
    ],
)
archive_understanding.wait_until_complete()

Security Monitoring (Quality-First)

For a finite review video, use a short time interval and enough frames to capture activity progression:

security_understanding = video.understand(
    segmentation={"type": "time", "seconds": 5},
    analyzers=[
        {
            "type": "vlm",
            "name": "security_scene",
            "sampling": {"frame_count": 3},
            "config": {"prompt": "Identify people, vehicles, and suspicious activity."},
        }
    ],
)
security_understanding.wait_until_complete()

E-commerce (Balanced)

Use shot segmentation when the video is edited around product reveals:

commerce_understanding = video.understand(
    segmentation={"type": "shot", "threshold": 20},
    analyzers=[
        {
            "type": "vlm",
            "name": "product_scene",
            "sampling": {"frame_count": 2},
            "config": {"prompt": "Identify products, brands, and pricing."},
        }
    ],
)
commerce_understanding.wait_until_complete()

Monitoring

Measure the complete lifecycle rather than only request submission:

import time

start = time.time()
understanding = video.understand(
    analyzers=[{"type": "vlm", "name": "scene", "config": {"prompt": "Describe the scene."}}]
)
understanding.wait_until_complete()
scene = understanding.get_analyzer("scene")
index = video.index(name="measured_scene", source=scene)
index.wait_until_complete()
indexing_time = time.time() - start

start = time.time()
results = video.semantic_search(query="query", index_ids=[index.index_id], top_k=10)
search_time = time.time() - start
print(f"Indexing took {indexing_time:.2f}s; retrieval took {search_time:.3f}s")

Start Here

Core Concepts

Ingest

Understand

Act

Automate

MCP Server

Open Source Frameworks

Quick Reference

Indexing Cost Factors

Frame Count and Scene Interval

Scene Interval

Indexing Latency

Time-Based vs Shot-Based

Async Processing

Search Latency

Single Video vs Collection

Reduce Search Space

Limit Results

Optimization Strategies

Tiered Indexing

Index on Demand

Batch Processing

Cost Estimation

Factors

Example Calculations

Recommendations by Use Case

Media Archive (Cost-Sensitive)

Security Monitoring (Quality-First)

E-commerce (Balanced)

Monitoring

Next Steps

Accuracy Tips

Create an Index

​Quick Reference

​Indexing Cost Factors

​Frame Count and Scene Interval

​Scene Interval

​Indexing Latency

​Time-Based vs Shot-Based

​Async Processing

​Search Latency

​Single Video vs Collection

​Reduce Search Space

​Limit Results

​Optimization Strategies

​Tiered Indexing

​Index on Demand

​Batch Processing

​Cost Estimation

​Factors

​Example Calculations

​Recommendations by Use Case

​Media Archive (Cost-Sensitive)

​Security Monitoring (Quality-First)

​E-commerce (Balanced)

​Monitoring

​Next Steps

Accuracy Tips

Create an Index

Quick Reference

Indexing Cost Factors

Frame Count and Scene Interval

Scene Interval

Indexing Latency

Time-Based vs Shot-Based

Async Processing

Search Latency

Single Video vs Collection

Reduce Search Space

Limit Results

Optimization Strategies

Tiered Indexing

Index on Demand

Batch Processing

Cost Estimation

Factors

Example Calculations

Recommendations by Use Case

Media Archive (Cost-Sensitive)

Security Monitoring (Quality-First)

E-commerce (Balanced)

Monitoring

Next Steps