Multimodal (Video + Audio)

Vision alone gets you a long way. Audio — the HDMI-embedded audio plus the 3.5 mm mic input — lets the agent hear the scene too. Combined, you have a near-complete perception stack.

This page covers capturing audio alongside video and presenting both to a multimodal LLM.

The two audio sources

The Pengo mixes these into a single stereo USB audio stream:

1. HDMI-embedded audio — whatever your source device sends over HDMI (game sound, video playback, mic-in on the source itself, etc.)
2. 3.5 mm microphone input — a mono mic plugged into the front jack of the Pengo (mic-level only — not line-level)

If you want only the mic, set the HDMI source to output no audio (mute the game console, for example). If you want only HDMI audio, leave the mic jack unplugged.

Capturing audio with PyAV

PyAV can open the USB audio device directly.

import av

audio_container = av.open("hw:1,0", format="alsa", options={
    "sample_rate": "48000",
    "sample_format": "s16",
})

# Read 1 second of audio at a time
sample_rate = 48000
channels = 2
samples_per_chunk = sample_rate * 1  # 1 second

for frame in audio_container.decode(audio=0):
    # frame is an av.AudioFrame with .to_ndarray() returning (channels, samples)
    pcm = frame.to_ndarray()
    # ... send to STT, save to disk, etc.

Transcribing with Whisper

Local Whisper (faster-whisper) is fast enough for live transcription:

pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("base.en", device="cuda", compute_type="float16")

def transcribe(pcm_int16, sample_rate=48000):
    # pcm_int16 is a numpy array shape (channels, samples), int16
    audio = pcm_int16.mean(axis=0)        # to mono
    audio = audio.astype("float32") / 32768.0
    segments, _ = model.transcribe(audio, language="en", vad_filter=True)
    return " ".join(s.text for s in segments)

Capturing both audio and video in lock-step

The simplest production pattern: write a small ring buffer that the agent loop reads from.

import threading, queue, time
import av
import numpy as np

video_q: queue.Queue = queue.Queue(maxsize=2)
audio_q: queue.Queue = queue.Queue(maxsize=8)

def video_thread():
    container = av.open("/dev/video0", format="v4l2", options={
        "input_format": "mjpeg", "framerate": "30", "video_size": "1920x1080",
    })
    for frame in container.decode(video=0):
        try:
            video_q.put_nowait(frame)
        except queue.Full:
            pass

def audio_thread():
    container = av.open("hw:1,0", format="alsa", options={
        "sample_rate": "48000", "sample_format": "s16",
    })
    for frame in container.decode(audio=0):
        try:
            audio_q.put_nowait(frame)
        except queue.Full:
            pass

threading.Thread(target=video_thread, daemon=True).start()
threading.Thread(target=audio_thread, daemon=True).start()

# Agent loop
while True:
    v = video_q.get()
    # accumulate ~2 s of audio
    audio_frames = []
    while not audio_q.empty():
        audio_frames.append(audio_q.get_nowait())
    # ... send to multimodal LLM with v as image and concatenated audio as
    # an audio attachment (or as transcribed text) ...

Multimodal API input shape

Claude (Anthropic) accepts audio as base64-encoded WAV/MP3/etc. in the content array alongside images:

import base64
import httpx

def make_payload(jpeg_bytes, wav_bytes, prompt):
    return {
        "model": "claude-sonnet-5",
        "max_tokens": 1024,
        "messages": [{
            "role": "user",
            "content": [
                {"type": "image",
                 "source": {"type": "base64",
                            "media_type": "image/jpeg",
                            "data": base64.b64encode(jpeg_bytes).decode()}},
                {"type": "audio",
                 "source": {"type": "base64",
                            "media_type": "audio/wav",
                            "data": base64.b64encode(wav_bytes).decode()}},
                {"type": "text", "text": prompt},
            ],
        }],
    }

OpenAI's GPT-4o accepts audio similarly via input_audio content parts.

Lower-cost alternative: transcribe then text

If you don't need raw audio in the prompt, transcribe locally first with faster-whisper, then send transcript + image as text + image. Much cheaper, fewer tokens, faster.

transcript = transcribe(pcm)
payload = {
    "model": "claude-haiku-4-5",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "image", "source": {...}},
            {"type": "text", "text":
             f"Here's what's currently on screen, and a transcript of the "
             f"last few seconds of audio:\n\nTranscript:\n{transcript}\n\n"
             f"Question: ..."}
        ],
    }],
}

Latency budgets

For a "responsive" agent loop:

Stage	Typical latency
Frame capture (PyAV, MJPG, 1080p30)	33 ms (1 frame)
Resize + JPEG encode	5–15 ms
API round-trip (Claude / OpenAI)	400–1500 ms
Tool call + execution	varies

So end-to-end you're looking at ~500–1500 ms per cycle. Fine for "ask every 2 s" usage. For sub-second reaction times you'd need a smaller local model.

---

Next: Example scripts → for full copy-paste code.