> ## Documentation Index
> Fetch the complete documentation index at: https://docs.videodb.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Use Case: Conference Slide Extraction
> Extract and search slide content from conference videos by combining spoken content with visual understanding for instant information retrieval.
When watching a talk or presentation, it's common to take notes or share interesting points with others. Often, the content on the slides captures our attention. At VideoDB, we follow many top engineering processes and regularly take notes from talks and conferences, which we share internally on our Slack channel.
However, when trying to recall specific parts of these talks, only a few keywords might come to mind. To address this, we built an internal tool that stores all these talks in VideoDB, allowing us to find and share the content on the screen in text form using a search query.
Let's explore the problem:
What was on the screen when the speaker discussed the "hard and fast rule" in the following video?
This notebook is a step towards creating a Slack bot that posts valuable engineering practices from top tech talks daily. Stay Tuned!
## Introduction
In this tutorial, we'll explore an advanced yet accessible technique for retrieving visual information from video content based on what the speaker was discussing. Specifically, we'll focus on finding information on slides in a video recording of a speech.
As video content continues to grow in volume and importance, being able to quickly find specific information within videos becomes crucial. Imagine being able to locate a particular statistic mentioned in an hour-long presentation without watching the entire video. That's the power of multimodal video search!
This approach combines structured transcript queries with visual understanding to create a robust, multimodal search pipeline. Don't worry if these terms sound complex - we'll break everything down step by step!
## Setup
### Installing packages
```
!pip install videodb
```
### API Keys
Before proceeding, ensure access to [VideoDB](https://videodb.io). If not, sign up for API access on the respective platforms.
Get your API key from [VideoDB Console](https://console.videodb.io/auth?utm_source=docs_videodb_io\&utm_medium=docs_link\&utm_campaign=console_auth\&utm_content=docs_link\&id=docs) (Free for first 50 uploads, No credit card required).
### Step 1: Connect to VideoDB
Gear up by establishing a connection to VideoDB.
```python theme={null}
import videodb
# Set your API key
api_key = "your_api_key"
# Connect to VideoDB
conn = videodb.connect(api_key=api_key)
coll = conn.get_collection()
```
### Step 2: Upload the Video
Next, let's upload our sample video:
```python theme={null}
# Upload a video by URL
video = coll.upload(url="https://www.youtube.com/watch?v=IEe-5VOv0Js")
```
### Step 3: Understand Spoken Content and Slides
Build the multimodal workflow in two clear stages:
1. Create a queryable transcript from spoken content.
2. Compare slide configurations and prompts before analyzing the full video.
### Understand Spoken Content
```python theme={null}
import re
transcript_understanding = video.understand(
analyzers=[{"type": "spoken_words", "name": "transcript"}],
)
transcript_understanding.wait_until_complete()
transcript_analyzer = transcript_understanding.get_analyzer("transcript")
transcript_output = transcript_analyzer.get_output()
sentence_records = []
sentence_words = []
for scene in transcript_output["scenes"]:
for word in scene["data"]["words"]:
sentence_words.append(word)
if re.search(r"""[.!?]+["'”’)\]]*$""", word["text"]):
sentence_records.append(
{
"start": sentence_words[0]["start"],
"end": sentence_words[-1]["end"],
"text": " ".join(item["text"] for item in sentence_words),
}
)
sentence_words = []
if sentence_words:
sentence_records.append(
{
"start": sentence_words[0]["start"],
"end": sentence_words[-1]["end"],
"text": " ".join(item["text"] for item in sentence_words),
}
)
transcript_index = video.index(
name="conference_transcript_sentences",
source=sentence_records,
use_for=["query"],
fields={"filter": ["text"]},
)
transcript_index.wait_until_complete()
print(f"✅ Transcript index is ready with {len(sentence_records)} timed sentences!")
```
This function transcribes the speech in the video and indexes it, making it searchable.
### Find the Right Configuration for Slide Understanding
Build the slide workflow in two stages:
1. Compare slide configurations and prompts on extracted scenes.
2. Analyze the full video with the selected configuration.
To learn more about video understanding, explore [Understanding Artifacts](/pages/understand/indexing-pipelines/understanding-artifacts) and [Analyzer Outputs](/pages/understand/indexing-pipelines/analyzer-outputs).
1. **Compare Slide Segmentation Configurations**
Start with the default scene extraction so there is a visual baseline for comparison.
```python theme={null}
from IPython.display import Image, display
import requests
# Helper function to preview extracted scenes
def display_scenes(scenes, images=True):
for scene in scenes:
print(f"{scene.id} : {scene.start}-{scene.end}")
if images:
for frame in scene.frames:
image = Image(requests.get(frame.url, stream=True).content)
display(image)
print("----")
scene_collection_default = video.extract_scenes()
display_scenes(scene_collection_default.scenes)
```
Conference videos can contain subtle slide changes that the default configuration misses. Lower the shot threshold and compare the extracted scenes again.
```python theme={null}
from videodb import SceneExtractionType
scene_collection = video.extract_scenes(
extraction_type=SceneExtractionType.shot_based,
extraction_config={"threshold": 10},
)
display_scenes(scene_collection.scenes)
```
#### Test the Slide Prompt on Sample Scenes
Test the prompt on a few scenes before analyzing the entire video. This keeps the experiment focused while you check that the analyzer describes slide content and returns `None` when a scene does not show a slide.
```python theme={null}
slide_prompt = (
"Give the content written on the slides, output None if it isn't the slides."
)
for scene in scene_collection.scenes[20:23]:
description = scene.describe(slide_prompt)
print(f"{scene.id} : {scene.start}-{scene.end}")
print(description)
print("-----")
```
The lower threshold and sample prompt results provide the configuration for the full slide-analysis run.
### Analyze the Video With the Finalized Configuration
The final slide run uses the selected configuration:
1. Segment the video into shots with threshold 10.
2. Sample one representative frame of each shot.
3. Return a structured slide description for every timed segment.
```python theme={null}
def display_scene_index(scene_index):
for scene in scene_index:
print(f"{scene['start']} - {scene['end']}")
print(scene["description"])
print("----")
slide_prompt = (
"Give the content written on the slides, output None if it isn't the slides."
)
slide_segmentation = {"type": "shot", "threshold": 10}
slide_sampling = {"strategy": "uniform", "frame_count": 1}
slide_schema = {"description": "string"}
slide_understanding = video.understand(
segmentation=slide_segmentation,
analyzers=[
{
"type": "vlm",
"name": "slides",
"sampling": slide_sampling,
"config": {"prompt": slide_prompt, "schema": slide_schema},
}
],
)
slide_understanding.wait_until_complete()
slides_analyzer = slide_understanding.get_analyzer("slides")
slides_output = slides_analyzer.get_output()
scene_index = [
{
"start": float(segment["start"]),
"end": float(segment["end"]),
"description": segment["data"]["description"],
}
for segment in slides_output["scenes"]
]
scene_index.sort(key=lambda scene: (scene["start"], scene["end"]))
display_scene_index(scene_index)
```
### Step 4: Search Pipeline Implementation
The heart of this approach is the search pipeline, which combines a structured transcript query with slide-understanding artifact records.
This pipeline does the following:
1. Runs a structured transcript query with a text filter
2. Extracts time ranges from the returned transcript records
3. Filters slide-understanding artifact records by their overlap with those ranges
4. Returns the matching slide descriptions and their time ranges
```python theme={null}
def simple_filter_scenes(time_ranges, scene_dicts):
def is_in_range(scene, range_start, range_end):
scene_start = scene["start"]
scene_end = scene["end"]
return scene_start < range_end and range_start < scene_end
filtered_scenes = []
for start, end in time_ranges:
filtered_scenes.extend(
scene
for scene in scene_dicts
if is_in_range(scene, start, end)
)
# Remove duplicates while preserving order
seen = set()
return [
scene
for scene in filtered_scenes
if not (
(scene["start"], scene["end"]) in seen
or seen.add((scene["start"], scene["end"]))
)
]
```
```python theme={null}
def search_pipeline(query, video):
search_result = video.query(
index_id=transcript_index.index_id,
filter=[{"field": "text", "op": "contains", "value": query}],
limit=100,
sort=[("start", "asc")],
)
time_ranges = [(shot.start, shot.end) for shot in search_result.get_shots()]
final_result = simple_filter_scenes(time_ranges, scene_index)
result_text = "\n\n".join(item["description"] for item in final_result
if item.get("description", "").lower().strip() != "none")
result_timeline = [(item.get("start"), item.get("end")) for item in final_result]
return result_text, result_timeline
```
### Step 5: Viewing the Search Results
Finally, let's use our search pipeline:
```python theme={null}
from videodb import play_stream
query = "hard and fast rule"
result_text, result_timeline = search_pipeline(query, video)
# Truncate result_text before printing if it's too long
max_chars = 500
if len(result_text) > max_chars:
print(result_text[:max_chars] + "...\n")
else:
print(result_text)
# Get the video's actual length from the video object
video_length = video.length
# Clip the end times in result_timeline to not exceed video_length
clipped_result_timeline = [
(start, min(end, video_length)) for start, end in result_timeline
]
stream_link = video.generate_stream(clipped_result_timeline)
play_stream(stream_link)
```
It returns slide-understanding artifact records whose time ranges overlap the matching structured transcript results.
Here's the result for this particular search query "hard and fast rule":
### The content written on the slide is
```
IT'S ALL IN THE DETAILS
- Prefer American English for naming
- Avoid payment-industry jargon
- Timestamp fields should use
- Amount properties should also provide a currency
- API resources with IDs are top-level
- New API resources should be retrieved and listed one way
- API resource mutations should be reflected in API responses
- Use nested structures for future extensibility
- Prefer enums to booleans for new properties
- Use a type field for polymorphic objects
- Use verbs for properties with side effects
- Use top-level namespaces for product APIs
- Evaluate new features in the Dashboard before building an API
- Use simple, unambiguous language
- Always paginate unbounded lists
- Iterate on designs with beta users with the feature behind a gate
```
Here are some other query outputs using the same search pipeline:
### Search for "stripe api review"
### API REVIEW CHECKLIST
```
API Review: [Insert Title Here]
Ziec: Gavel jar, link for API review join creation
Gavel block
To ping PM when Q/A and other stakeholders have
Summary
(Please include a short description of the change you would like to make...)
```
### Search for "Friction Log"
### Internal Terminal Dogfooding Instructions
```
https://go/terminal-dogfooding-instructions
Stripe! Thanks a ton for your help in dogfooding ahead of our Terminal GA launch!
We’re very close to launching Terminal in public beta and then GA.
And we could use your help! There are a ton of different use cases of these integrations to test and polish, and we want to stress test what we've made and ship these paths and accompanying docs and dashboard flows in a developer-friendly as possible in the time we have before the Terminal GA launch.
If you’re arriving at this doc after having signed up to dogfood, continue to Steps below.
If you haven’t signed up yet, please signup here, and we’ll get back to you when you have a slot your test.
Steps
1. If you’re dogfooding remotely and haven’t received instructions on ordering hardware or attending demos, please email terminal-dogfooding@stripe.com or ping in #terminal-dogfooding
2. If you're dogfooding the iOS or Android SDK, you'll need to set up your environment as described in the links...
```
## Conclusion
This document has outlined a sophisticated approach to multimodal video search, combining structured transcript queries with visual understanding. The pipeline finds specific visual content, such as slide information, based on what was discussed.
This technique has broad applications beyond just searching for slides in speeches. It could be adapted for various use cases where visual information needs to be retrieved based on audio content, such as:
* Finding product demonstrations in long-form video content
* Identifying key moments in educational videos
* Searching for specific visual elements in recorded meetings or presentations
As video content continues to grow in importance and volume, tools and techniques like these will become increasingly valuable for efficient information retrieval and analysis.
## Further Resources
To learn more about understanding artifacts and analyzer outputs, explore the following guides:
Create reusable timed artifacts from video understanding runs.
Inspect the timestamped records produced by each analyzer.
## Get Support
If you have any questions or feedback, feel free to reach out to us:
Connect with the VideoDB community for support and discussions
Explore our open source projects and contribute
Visit our main website for more information
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.
