Skip to main content
YouTube, Netflix, Zoom, Twitch. The entire video industry was built for one thing: putting pixels on human eyeballs. That’s a 70-year-old assumption. And it’s the reason AI agents can’t use video natively.

​
The Playback Paradigm

Video infrastructure was designed around a simple model: 
Source → Encode → Distribute → Decode → Display
Every piece of the stack optimizes for this:
  • Codecs minimize bandwidth for sequential playback
  • CDNs cache content for low-latency delivery
  • Players buffer and render frames at the right framerate
  • Protocols (HLS, DASH) adapt quality to network conditions
The end goal: a human watches a video from start to finish.

​
What Playback Gives You

When you press play on a YouTube video:
  1. The CDN delivers compressed chunks
  2. Your device decodes frames in real-time
  3. Frames render at 24/30/60 fps
  4. Audio syncs with video
  5. You scrub the timeline to navigate
This works brilliantly for entertainment. But notice what it doesn’t give you:
  • No way to query content
  • No structured access to “what happened”
  • No timestamp-level retrieval
  • No semantic understanding
  • No event detection
The video just… plays.

​
What Perception Needs

AI agents don’t watch. They query. 
# Agent question: "What did they say about the timeline?"
results = video.search("timeline discussion")

# Agent needs: timestamped, verifiable answers
for shot in results.shots:
    evidence = f"{shot.start}s: {shot.text}"
    playable_url = shot.stream_url
Perception requires:
CapabilityPlayback ModelPerception Model
Access patternSequentialRandom
Query type”Play from 10:00""Find mentions of X”
OutputPixels on screenStructured data + evidence
LatencySeconds to bufferMilliseconds to query
ScaleOne viewer at a timeThousands of queries/second

​
The YouTube Gap

You can’t ask YouTube:
  • “What videos in my library mention competitor pricing?”
  • “Show me every safety incident from last month”
  • “When did this person appear in any of our recordings?”
YouTube has the content. But it has no semantic layer - no way to query what’s inside. You can search titles and descriptions. You can’t search content.

​
The Zoom Gap

You can’t ask Zoom:
  • “What was the action item from yesterday’s call?”
  • “Show me the moment the client expressed concern”
  • “When was the slide about Q4 projections shown?”
Zoom has recordings. But they’re files - opaque blobs waiting for someone to watch them.

​
The Enterprise Gap

Enterprise video is even worse. Security footage, training recordings, customer calls, manufacturing feeds. All captured. None queryable. The common workflow:
  1. Something happens
  2. Someone requests a recording
  3. A human watches it (at 1x speed)
  4. They manually note timestamps
  5. Days later, you have an answer
This doesn’t scale. And it definitely doesn’t work for AI.

​
Perception-First Architecture

What if video infrastructure was built for perception? 
Source → Ingest → Index → Query → Evidence
Every piece optimizes for understanding:
  • Ingest normalizes media from any source
  • Indexing extracts semantic meaning with prompts
  • Query returns timestamped, relevant moments
  • Evidence provides playable verification
The end goal: an agent queries content and gets grounded answers.

​
From “Play” to “Answer”

PlaybackPerception
”Play the recording""What happened at 2pm?"
"Skip to 10:00""Find the product demo"
"Watch this video""Search across all videos"
"Download the file""Give me the relevant clips”
Perception turns video from a thing you watch into a thing you query.

​
Real-time, Not Batch

The playback model assumes recordings. You capture, then watch. Perception works in real-time: 
# Live stream
rtstream.index_visuals(prompt="Detect intruders")

# Real-time alerts
{"channel": "alert", "label": "intruder", "confidence": 0.94}
No recording. No waiting. Events detected as they happen.

​
The Platform Shift

For 70 years, video infrastructure optimized for:
  • High visual fidelity
  • Low latency playback
  • Global distribution
  • Human consumption
The next era optimizes for:
  • Semantic understanding
  • Instant queryability
  • Real-time event detection
  • Machine consumption
Video infrastructure is being rebuilt - not for playback, but for perception.

​
What’s Next

Episodic Memory for Agents

Why agents need to remember experiences

Quickstart

Build perception-first