TwelveLabs Integration

Human Monitoring is Expensive, Exhausting, and Doesn’t Scale

From baby monitors to warehouse cameras or endless CCTV feeds, many companies still depend on human eyes to monitor live video—a costly, tedious, and highly error-prone approach. Fatigue inevitably sets in, accuracy declines, and scaling up means hiring more personnel rather than enhancing systems. But in an AI-driven era, monitoring shouldn’t be manual, miss critical details, or become a bottleneck. Imagine receiving instant alerts when packages are stolen, notifying parents the moment a baby attempts to climb out of a crib, proactively highlighting safety risks before they escalate, or doctors instantly knowing when ICU patients need immediate attention. This is precisely the responsive monitoring VideoDB delivers through its real-time infrastructure. VideoDB now provides first party integration with TwelveLabs’s advanced Pegasus 1.2 AI model for precise frame understanding.

Live video to Instant Action

Real-time video analysis isn’t merely a nice-to-have—it’s a fundamental shift in capability.

Safety and Security: Transforming reactive measures into proactive alerts, potentially saving lives during emergencies like flash floods or security breaches.
Enterprise Productivity: Converting passive meeting archives into interactive, searchable knowledge repositories, vastly enhancing collaboration.
Content Platforms: Automatically tagging, chaptering, and moderating content in unprecedented detail, elevating user experiences.

The opportunities are immense, yet technical barriers have historically prevented many development teams from unlocking this potential.

Reality of Building Video AI

If you’ve ever attempted to build a robust video understanding system, you know the pain firsthand. You’re often stuck playing the role of a systems integrator, wrestling with:

API Spaghetti: Managing credentials, juggling rate limits, and navigating diverse SDKs across multiple video processing, AI modeling, and storage services.
Scaling Nightmares: Each component scales independently, causing bottlenecks and inefficiencies—particularly with resource-intensive GPU workloads.
Latency Issues: The delays involved in interactions between storage, AI models, and your applications undermine genuine real-time capabilities.

These challenges can quickly stall innovation, turning promising ideas into lengthy, frustrating engineering endeavors.

Simplifying Video AI Infrastructure with VideoDB

VideoDB is a purpose-built infrastructure for AI driven video management. It provides a unified, AI-native infrastructure handling the complete lifecycle of video content—from ingestion and indexing to alert management—all via a streamlined, developer-friendly API. Leveraging VideoDB, you can effortlessly ingest multiple real-time video streams, manage customizable indexes uniquely tailored for specific analyses, and simultaneously analyze diverse aspects of a single video stream. Additionally, VideoDB’s targeted alerting system triggers automated webhooks, ensuring rapid responses to critical events. And now, we’ve equipped that powerful infrastructure with an even more powerful intelligence.

Introducing the TwelveLabs Integration: Your Frame Understanding, Supercharged

We’re excited to introduce TwelveLabs model configurations for VideoDB’s VLM workflow when the selected model is available for your backend and project.

VideoDB and TwelveLabs integration architecture

Configure the VLM analyzer with config.model to use TwelveLabs’ sophisticated video understanding models, such as Pegasus, within the VideoDB workflow.

Real-World Use Cases: From Theory to Action

With VideoDB + TwelveLabs, sophisticated real-time video monitoring is just minutes away:

Real-World Use Cases in Action

🌊 Flash Flood Detection

Click image to open interactive notebook →

Imagine a camera monitoring a dry riverbed in a flood-prone area. With TwelveLabs integrated into VideoDB, you can continuously detect critical events—like rapidly rising floodwaters—in real-time. The moment Pegasus recognizes the signs of a flash flood, VideoDB immediately triggers life-saving alerts. This isn’t just video analysis; it’s proactive, intelligent disaster prevention.

👶🏻 Baby Crib Monitoring

Click image to open interactive notebook →

After a long day, parents deserve restful, worry-free sleep. With VideoDB’s real-time monitoring powered by TwelveLabs’ Pegasus, you’ll instantly know if your baby tries climbing out of the crib or needs immediate attention. Sleep easy, knowing VideoDB and TwelveLabs have you covered—every second of the night.

How It Works: One Line of Code to Unlock a New World

Ready to see how simple this is? To understand a video stream with TwelveLabs’ powerful Pegasus model, specify it as config.model for the VLM analyzer.

import videodb

conn = videodb.connect()
coll = conn.get_collection()
rtsp_url = "rtsp://your-stream-url"  # Replace with your camera stream.
flood_stream = coll.connect_rtstream(
    name="Flood Detection Stream",
    url=rtsp_url,
    store=True,
)

flood_understanding = flood_stream.understand(
    segmentation={"type": "time", "window": "5s"},
    analyzers=[{
        "type": "vlm",
        "name": "flood_scene",
        "sampling": {"frame_count": 3},
        "config": {
            "prompt": "Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.",
            "model": "twelvelabs-pegasus-1.2",
        },
    }],
    store=True,
)
flood_scene_output = flood_understanding.outputs.get("flood_scene")
flood_index = flood_stream.index(
    source=flood_scene_output,
    name="Flash_Flood_Detection_Index",
    use_for=["semantic"],
)
print("Continuous index ID:", flood_index.id)

import { connect } from "videodb";

const conn = connect();
const coll = await conn.getCollection();
const rtspUrl = "rtsp://your-stream-url"; // Replace with your camera stream.
const floodStream = await coll.connectRTStream(
    rtspUrl,
    "Flood Detection Stream",
    ["video"],
    30,
    true,
);

const floodUnderstanding = await floodStream.understand({
    segmentation: { type: "time", window: "5s" },
    analyzers: [{
        type: "vlm",
        name: "flood_scene",
        sampling: { frame_count: 3 },
        config: {
            prompt: "Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.",
            model: "twelvelabs-pegasus-1.2",
        },
    }],
    store: true,
});

const floodIndex = await floodStream.index({
    source: {
        understanding_id: floodUnderstanding.id,
        output: "flood_scene",
    },
    name: "Flash_Flood_Detection_Index",
    useFor: ["semantic"],
});
console.log("Continuous index ID:", floodIndex.id);

When the selected model is available for your backend and project, configure it in the VLM analyzer and use it in VideoDB’s indexing pipeline.

Provider routing for model names is backend-dependent; confirm the configured model is available for your project.

Try It Now: Interactive Notebooks

Ready to build your own real-time video intelligence system? Open these interactive notebooks directly in Google Colab and start experimenting with TwelveLabs’ Pegasus model:

Flash Flood Detection Notebook

Build a real-time flood detection system that monitors dry riverbeds and sends instant alerts

Baby Crib Monitoring Notebook

Create an AI-powered baby monitor that detects escape attempts and keeps parents informed

Overview

Agentic Workflows

Video Search and Understanding

Live Intelligence

Content Factory

Programmatic Editing

Safety & Compliance

Live video to Instant Action

Reality of Building Video AI

Simplifying Video AI Infrastructure with VideoDB

Introducing the TwelveLabs Integration: Your Frame Understanding, Supercharged

Real-World Use Cases: From Theory to Action

Real-World Use Cases in Action

🌊 Flash Flood Detection

👶🏻 Baby Crib Monitoring

How It Works: One Line of Code to Unlock a New World

Try It Now: Interactive Notebooks

Flash Flood Detection Notebook

Baby Crib Monitoring Notebook

​Live video to Instant Action

​Reality of Building Video AI

​Simplifying Video AI Infrastructure with VideoDB

​Introducing the TwelveLabs Integration: Your Frame Understanding, Supercharged

​Real-World Use Cases: From Theory to Action

​Real-World Use Cases in Action

​🌊 Flash Flood Detection

​👶🏻 Baby Crib Monitoring

​How It Works: One Line of Code to Unlock a New World

​Try It Now: Interactive Notebooks

Flash Flood Detection Notebook

Baby Crib Monitoring Notebook

Live video to Instant Action

Reality of Building Video AI

Simplifying Video AI Infrastructure with VideoDB

Introducing the TwelveLabs Integration: Your Frame Understanding, Supercharged

Real-World Use Cases: From Theory to Action

Real-World Use Cases in Action

🌊 Flash Flood Detection

👶🏻 Baby Crib Monitoring

How It Works: One Line of Code to Unlock a New World

Try It Now: Interactive Notebooks