Tutorial

Learn how to build real-time video processing apps with Streamlit-WebRTC step by step.

Basic Usage

Create app.py with the content below:

app.py

from streamlit_webrtc import webrtc_streamer

webrtc_streamer(key="sample")

Unlike other Streamlit components, webrtc_streamer() requires the key argument as a unique identifier. Set an arbitrary string to it.

Then run it with Streamlit and open http://localhost:8501/:

streamlit run app.py

You see the app view, so click the "START" button.

Then, video and audio streaming starts. If asked for permissions to access the camera and microphone, allow it.

Adding Video Processing

Next, edit app.py as below and run it again:

app.py

import av
from streamlit_webrtc import webrtc_streamer


def video_frame_callback(frame):
    img = frame.to_ndarray(format="bgr24")

    flipped = img[::-1, :, :]

    return av.VideoFrame.from_ndarray(flipped, format="bgr24")


webrtc_streamer(key="example", video_frame_callback=video_frame_callback)

Now the video is vertically flipped.

As shown in this example, you can edit the video frames by defining a callback that receives and returns a frame and passing it to the video_frame_callback argument (or audio_frame_callback for audio manipulation).

The input and output frames are instances of av.VideoFrame (or av.AudioFrame when dealing with audio) from the PyAV library.

You can inject any kinds of image (or audio) processing inside the callback.

Pass Parameters to the Callback

You can also pass parameters to the callback.

In the example below, a boolean flip flag is used to turn on/off the image flipping:

app.py

import av
import streamlit as st
from streamlit_webrtc import webrtc_streamer

flip = st.checkbox("Flip")


def video_frame_callback(frame):
    img = frame.to_ndarray(format="bgr24")

    flipped = img[::-1, :, :] if flip else img

    return av.VideoFrame.from_ndarray(flipped, format="bgr24")


webrtc_streamer(key="example", video_frame_callback=video_frame_callback)

Pull Values from the Callback

Sometimes we want to read the values generated in the callback from the outer scope.

Note that the callback is executed in a forked thread running independently of the main script, so we have to take care of the following points and need some tricks for implementation like the example below:

• Thread-safety - Passing the values between inside and outside the callback must be thread-safe.
• Using a loop to poll the values - During media streaming, while the callback continues to be called, the main script execution stops at the bottom as usual. So we need to use a loop to keep the main script running and get the values from the callback in the outer scope.

The following example passes the image frames from the callback to the outer scope and continuously processes them in a loop. In this example, simple image analysis (calculating the histogram) is done on the image frames:

app.py

import threading

import cv2
import streamlit as st
from matplotlib import pyplot as plt
from streamlit_webrtc import webrtc_streamer

lock = threading.Lock()
img_container = {"img": None}


def video_frame_callback(frame):
    img = frame.to_ndarray(format="bgr24")
    with lock:
        img_container["img"] = img

    return frame


ctx = webrtc_streamer(key="example", video_frame_callback=video_frame_callback)

fig_place = st.empty()
fig, ax = plt.subplots(1, 1)

while ctx.state.playing:
    with lock:
        img = img_container["img"]
    if img is None:
        continue
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    ax.cla()
    ax.hist(gray.ravel(), 256, [0, 256])
    fig_place.pyplot(fig)

threading.Lock is one standard way to control variable accesses across threads. A dict object img_container here is a mutable container shared by the callback and the outer scope and the lock object is used at assigning and reading the values to/from the container for thread-safety.

Callback Limitations

The callbacks are executed in forked threads different from the main one, so there are some limitations:

• Streamlit methods (st.* such as st.write()) do not work inside the callbacks.
• Variables inside the callbacks cannot be directly referred to from the outside.
• The global keyword does not work expectedly in the callbacks.
• You have to care about thread-safety when accessing the same objects both from outside and inside the callbacks as stated in the section above.

Ready for Production?

When you're ready to deploy your app, see the Deployment Guide for:

• HTTPS configuration requirements
• STUN/TURN server setup
• Platform-specific deployment instructions
• Troubleshooting common issues