VideoDB Documentation

Build with VideoDB

Eleven Labs x VideoDB: Adding AI Generated voiceovers to silent footage

Case: Automatically Creating a David Attenborough-style Voiceover for Silent Footage of the Underwater World

⁠

Overview

Voiceovers are the secret sauce that turns silent footage into captivating stories. They add depth, emotion, and excitement, elevating the viewing experience to new heights. With VideoDB's cutting-edge technology, creating dynamic voiceovers for your videos is easier than ever before.

Creating AI content and merging outputs from different tools can be tough and time-consuming. Doing it manually is even harder! What if you could do it all- in just a few lines of code?

VideoDB's cool tech helps you generate intelligent AI content and combine outputs from various tools quickly and automatically. No more long and tedious processes. Just simple, efficient, and awesome results!

Let’s try adding an automatically generated voiceover to

this silent footage⁠

in the style and voice of Sir David Attenborough. We shall do this using the powerful tech provided by Open AI, elevenlabs and VideoDB

⁠

Setup

📦 Installing packages

%pip install openai

%pip install videodb

🔑 API Keys

Before proceeding, ensure access to

VideoDB⁠

OpenAI⁠

, and

ElevenLabs⁠

API key. If not, sign up for API access on the respective platforms.

Get your API key from

VideoDB Console⁠

. ( Free for first 50 uploads, No credit card required ) 🎉

import os

os.environ["OPENAI_API_KEY"] = ""

os.environ["ELEVEN_LABS_API_KEY"] = ""

os.environ["VIDEO_DB_API_KEY"] = ""

🎙️ ElevenLab's Voice ID

You will also need ElevenLab's VoiceID of a Voice that you want to use.

For this demo, we will be using

David Attenborough's Voice.⁠

ElevenLabs has a large variety of voices to choose from (browse them

here⁠

). Once finalized, copy the Voice ID from ElevenLabs and link it here.

voiceover_artist_id = "VOICEOVER_ARTIST_ID"

⁠

Implementation

🌐 Step 1: Connect to VideoDB

Connect to VideoDB using your API key to establish a session for uploading and manipulating video files.

from videodb import connect

# Connect to VideoDB using your API key

conn = connect()

🎥 Step 2: Upload Video

# Upload a video by URL (replace the url with your video)

video = conn.upload(url='https://youtu.be/RcRjY5kzia8')

🔍 Step 3: Analyze Scenes and Generate Scene Descriptions

Start by analyzing the scenes within your Video using VideoDB's scene indexing capabilities. This will provide context for generating the script prompt.

video.index_scenes()

Let's view the description of first scene from the video

scenes = video.get_scenes()

print(f"{scenes[0]['start']} - {scenes[0]['end']}")

print(scenes[0]["response"])

Output:

0 - 9.033333333333333

The image presents a close-up, textured pattern reminiscent of organic forms. Dominated by a cool color palette of blue and turquoise hues, it gives the impression of looking at a magnified cluster of cells, crystalline structures, or perhaps a zoomed-in portion of aquatic flora. The shapes appear irregular yet symmetrical, like petals or leaves clustered tightly together. Light and shadow play across the surfaces, creating a dynamic interplay that suggests depth and complexity. The overall effect is one of natural beauty, with a soothing, almost hypnotic visual rhythm. The image is abstract enough to allow for multiple interpretations, depending on the viewer's perspective.

🎤 Step 4: Generate Voiceover Script with LLM

Combine scene descriptions with the script prompt, instructing LLM to create a voiceover script in David Attenborough's style.

This script prompt can be refined and tweaked to generate the most suitable output. Check out

these examples⁠

to explore more use cases.

import openai

client = openai.OpenAI()

script_prompt = "Here's the data from a scene index for a video about the underwater world. Study this and then generate a synced script based on the description below. Make sure the script is in the language, voice and style of Sir David Attenborough \n \n"

full_prompt = script_prompt

# run this prompt for each scene

for scene in scenes:

full_prompt += f"- {scene}\n"

openai_res = client.chat.completions.create(

model="gpt-3.5-turbo",

messages=[{"role": "system", "content": full_prompt}],

)

voiceover_script = openai_res.choices[0].message.content

# If you have ElevenLab's paid plan remove the :2500 limit on

# voiceover script.

voiceover_script = voiceover_script[:2500]

🎤 Step 5: Generate Voiceover Audio with ElevenLabs

Utilize the generated script to synthesize AI-generated voiceover narration in David Attenborough's voice using ElevenLabs API.

import requests

# Call ElevenLabs API to generate voiceover

url = f"https://api.elevenlabs.io/v1/text-to-speech/{voiceover_artist_id}"

headers = {

"xi-api-key": os.environ.get("ELEVEN_LABS_API_KEY"),

"Content-Type": "application/json"

}

payload = {

"model_id": "eleven_monolingual_v1",

"text": voiceover_script,

"voice_settings": {

"stability": 0.5,

"similarity_boost": 0.5

}

elevenlabs_res = requests.request("POST", url, json=payload, headers=headers)

# Save the audio file

audio_file = "audio.mp3"

CHUNK_SIZE = 1024

with open(audio_file, 'wb') as f:

for chunk in elevenlabs_res.iter_content(chunk_size=CHUNK_SIZE):

if chunk:

f.write(chunk)

🎬 Step 6: Add Voiceover to Video with VideoDB

In order to use the voiceover generated above, let's upload the audio file (voiceover) to VideoDB first

audio = conn.upload(file_path=audio_file)

Finally, add the AI-generated voiceover to the original footage using VideoDB's

timeline feature⁠

⁠

from videodb.timeline import Timeline

from videodb.asset import VideoAsset, AudioAsset

# Create a timeline object

timeline = Timeline(conn)

# Add the video asset to the timeline for playback

video_asset = VideoAsset(asset_id=video.id)

timeline.add_inline(asset=video_asset)

# Add the audio asset to the timeline for playback

audio_asset = AudioAsset(asset_id=audio.id)

timeline.add_overlay(start=0, asset=audio_asset)

🪄 Step 7: Review and Share

Preview the video with the integrated voiceover to ensure it functions correctly. Once satisfied, generate a stream of the video and share the link for others to view and enjoy this wholesome creation!

from videodb import play_stream

stream_url = timeline.generate_stream()

play_stream(stream_url)

Output:

⁠

🎉 Conclusion

By leveraging advanced AI technologies, you can enhance the storytelling and immersive experience of your video content. Experiment with different prompts and scene analysis techniques to further improve the quality and accuracy of the voiceovers. Enjoy creating captivating narratives with AI-powered voiceovers using VideoDB!

For more such explorations, refer to the

Build with VideoDB⁠

and join the VideoDB community on

GitHub⁠

Discord⁠

for support and collaboration 🙌🏼

We're excited to see your creations, so we welcome you to share your creations via

Discord⁠

LinkedIn⁠

Twitter.⁠

⁠

Overview