Skip to content

AI Agent Integration

The Pengo HDMI Grabber shows up to Linux as a standard UVC webcam (/dev/video0). That means any Python tool that can read a webcam can read frames off it — including the pipelines that feed multimodal LLMs like Claude and GPT-4V.

This section covers, end-to-end:

  1. How to grab frames from the device in Python (multiple ways)
  2. How to feed those frames to a vision LLM as part of an agent loop
  3. How to combine video + audio into a multimodal stream
  4. Working example scripts you can copy and adapt

Why this is interesting

Most agent "vision" setups use one of:

  • Screenshots of the agent's own desktop (Playwright, scrot, etc.)
  • Webcam input (the agent's face / a phone camera)
  • External cameras mounted on the room

A Pengo + a small headless computer gives you a fourth option that's especially powerful: let the agent see any HDMI output. Plug in:

  • A second computer's display output (peer-to-peer agent debugging)
  • A game console (visual game testing, QA bots)
  • A camera pointed at hardware (e.g. a 3D printer, a telescope, a lab rig)
  • A Raspberry Pi running an LCD / e-paper display
  • A TV tuner, a microscope with HDMI output, an oscilloscope

…and the agent sees exactly what any human monitor would show. This is cheaper and more flexible than rigging up network cameras, and uses zero custom drivers.

What this section assumes

  • Linux (any modern distro with kernel ≥ 5.x)
  • Python ≥ 3.10
  • Pengo Grabber enumerated as /dev/video0 (change paths in scripts if yours is different)
  • Some kind of vision-capable API key (Anthropic, OpenAI, or a local model like Llama 3.2 Vision via Ollama)

Pages in this section

Page Contents
Vision pipeline Reading frames in Python with PyAV, OpenCV, ffmpeg-python, GStreamer; converting to the formats vision APIs want; rate-limiting
Multimodal (video + audio) Mixing video frames with audio from the mic input / HDMI-embedded audio for full multimodal input
Example scripts A complete, runnable vision agent script you can adapt — frame loop → Claude API → optional tool call → reply

Start with the Vision pipeline →.