Hippo2 is a real-time video streaming application that detects faces using MediaPipe, recognizes known individuals, and intelligently triggers actions based on detected activities. It integrates with OpenAI's GPT for natural language action processing and Pinecone for vector-based action indexing.

• Real-Time Face Detection: Uses MediaPipe for fast, accurate face detection in video streams
• Face Recognition: Compares detected faces against a database of known face encodings
• Action Detection: Analyzes video frames to detect and classify user actions
• AI-Powered Reasoning: Uses OpenAI GPT to interpret detected actions and determine responses
• Vector Indexing: Stores and retrieves action patterns using Pinecone vector database
• WebSocket Support: Real-time bidirectional communication with connected clients
• Web Interface: Interactive HTML-based dashboard for monitoring and control

• Python: 3.8 or higher
• Camera/Webcam: Required for real-time video input
• Internet Connection: Required for OpenAI and Pinecone API calls

git clone https://github.com/your-username/hippo2.git
cd hippo2

python -m venv venv

On Windows:

venv\Scripts\activate

On macOS/Linux:

source venv/bin/activate

pip install -r requirements.txt

Key dependencies:

• fastapi — Web framework
• uvicorn — ASGI server
• python-socketio — WebSocket support
• opencv-python — Video processing
• face-recognition — Face encoding and comparison
• mediapipe — Face detection
• openai — GPT integration
• pinecone-client — Vector database
• python-dotenv — Environment variable management
• aiofiles — Async file operations

Create a .env file in the project root with the following keys:

OPENAI_API_KEY=your_openai_api_key_here
PINECONE_API_KEY=your_pinecone_api_key_here

How to get API keys:

• OpenAI API Key: Visit platform.openai.com → API keys → Create new secret key
• Pinecone API Key: Visit console.pinecone.io → API keys → Copy your key

1. Create a faces/ directory in the project root if it doesn't exist
2. Add .jpg or .png images of known individuals in subdirectories:

faces/
├── john_doe/
│   ├── photo1.jpg
│   └── photo2.jpg
├── jane_smith/
│   ├── photo1.jpg
│   └── photo2.jpg
└── alice_johnson/
    └── photo1.jpg

Each subdirectory name becomes the identity label for that person.

python server.py

The server will start on http://localhost:5000 by default.

Open your browser and navigate to:

http://localhost:5000

The application uses WebSocket to stream video frames and receive real-time action updates:

• Endpoint: ws://localhost:5000/socket.io
• Namespace: socket.io

• FACES_DIR: Directory containing known face images (default: faces/)
• ACTION_BUFFER_SIZE: Number of recent actions to keep in memory (default: 15)
• GPT_CHECK_INTERVAL: Interval (seconds) between GPT action evaluations (default: 3.0)
• WORKATO_WEBHOOK_URL: External webhook for triggering external actions (optional)

Handles real-time video streaming and action notifications.

const socket = io("http://localhost:8000");

socket.on("frame_processed", (data) => {
  console.log("Detected faces:", data.faces);
  console.log("Detected actions:", data.actions);
});

socket.emit("send_frame", frameData);

1. Video Capture: Video frames are captured from the camera
2. Face Detection: MediaPipe detects faces in each frame
3. Face Recognition: Detected faces are compared against known encodings
4. Action Analysis: GPT analyzes detected activities from the video context
5. Action Storage: Relevant actions are indexed in Pinecone for future reference
6. Response Triggering: External actions (webhooks) are triggered based on detected patterns
7. Real-Time Updates: WebSocket sends detected faces and actions to the client

hippo2/
├── server.py              # Main application server
├── .env                   # API keys (do not commit)
├── .gitignore             # Git ignore rules
├── README.md              # This file
├── requirements.txt       # Python dependencies
├── faces/                 # Directory for known face images
│   └── [person_name]/
│       └── [photos]
└── templates/
    └── index.html         # Web dashboard

Install with:

pip install fastapi uvicorn python-socketio opencv-python face-recognition mediapipe openai pinecone-client python-dotenv aiofiles

Or use the requirements file:

pip install -r requirements.txt

• Real-time face detection and recognition
• Action detection with GPT integration
• WebSocket support for live streaming
• Pinecone vector indexing for actions
• Web dashboard interface

Last Updated: December 2025
Version: 1.0.0