Vision-Agents

[VisionAgents]

Open Vision Agents by Stream

[Listed on TakoAPI]

[build] [PyPI version] [PyPI - Python Version] [License] [Discord] [X (Twitter)]

Multi-modal AI agents that watch, listen, and understand video.

Vision Agents give you the building blocks to create intelligent, low-latency video experiences powered by your models, your infrastructure, and your use cases.

Key Highlights

• Video AI: Built for real-time video AI. Combine YOLO, Roboflow, and others with Gemini/OpenAI in real-time.
• Low Latency: Join quickly (500ms) and maintain audio/video latency under 30ms using Stream's edge network.
• Open: Built by Stream, but works with any video edge network.
• Native APIs: Native SDK methods from OpenAI (create response), Gemini (generate), and Claude ( create message) — always access the latest LLM capabilities.
• SDKs: SDKs for React, Android, iOS, Flutter, React Native, and Unity, powered by Stream's ultra-low-latency network.

Getting Started

Step 1: Install via uv

uv add vision-agents

Step 2: (Optional) Install with extra integrations

uv add "vision-agents[getstream, openai, elevenlabs, deepgram]"

Step 3: Obtain your Stream API credentials

Get a free API key from Stream. Developers receive 333,000 participant minutes per month, plus extra credits via the Maker Program.

Follow the quickstart guide to build your first agent.

See It In Action

https://github.com/user-attachments/assets/d1258ac2-ca98-4019-80e4-41ec5530117e

This example shows you how to build golf coaching AI with YOLO and Gemini Live. Combining a fast object detection model (like YOLO) with a full realtime AI is useful for many different video AI use cases. For example: Drone fire detection, sports/video game coaching, physical therapy, workout coaching, just dance style games etc.

# partial example, full example: examples/02_golf_coach_example/golf_coach_example.py
agent = Agent(
    edge=getstream.Edge(),
    agent_user=agent_user,
    instructions="Read @golf_coach.md",
    llm=gemini.Realtime(fps=10),
    processors=[ultralytics.YOLOPoseProcessor(model_path="yolo11n-pose.pt", device="cuda")],
)

Features

Out-of-the-Box Integrations

LLMs: OpenAI · Gemini · xAI · OpenRouter · Hugging Face · Kimi AI · MiniMax · Telnyx

Realtime: OpenAI Realtime · Gemini Live · AWS Nova Sonic · Qwen · Inworld

STT: Deepgram · AssemblyAI · Fast-Whisper · Fish Audio · Wizper · Mistral Voxtral · Telnyx

TTS: ElevenLabs · Cartesia · Deepgram · AWS Polly · Pocket · Kokoro · Inworld · Fish Audio · Telnyx

Vision: Ultralytics · Roboflow · Moondream · TwelveLabs · NVIDIA · Decart

Avatars: LemonSlice

Turn Detection: Vogent · Smart Turn

Other: Twilio · Telnyx · TurboPuffer

Documentation

Check out the full docs at VisionAgents.ai.

Quickstart: Voice AI · Video AI

Guides: MCP & Function Calling · Video Processors · Phone Calling · RAG · Testing

Production: HTTP Server · Deployment · Kubernetes · Horizontal Scaling · Prometheus Metrics

Examples

Community Highlights

More involved demos built by the community and the Stream team - full applications that go beyond the in-repo examples and show what's possible with Vision Agents in production.

Got a demo you'd like featured? Open a PR or reach out on Discord.

• Sales Assistant Demo - a real-time AI meeting coach that lives on your desktop as a translucent macOS overlay. Built on Vision Agents and Flutter.
• Crashout Buddy - an emotionally aware voice agent demo built on Vision Agents and Stream Video.
• Cricket DRS AI — AI-powered Decision Review System for 🏏 Women's Cricket using Gemini Live vision, YOLO pose detection, and real-time voice verdicts by @jaya6400.

Development

See DEVELOPMENT.md

Want to add your platform or provider? See Create Your Own Plugin or reach out to nash@getstream.io.

Current Limitations

• Video AI struggles with small text — models may hallucinate scores, signs, etc.
• Context degrades on longer sessions (~30s+) for continuous video understanding
• Most use cases need a mix of specialized models (YOLO, Roboflow) with larger LLMs
• Real-time models require audio/text to trigger responses — video alone won't prompt output

Star History

[Star History Chart]