Home-ops

Home-Ops uses Gemini 3 Flash live video to fix products. If broken, it autonomously buys replacements using USDC.

Comment

Inspiration

A few years ago, my bike needed a simple repair. I found a popular YouTube tutorial and followed it step-by-step, but because the video didn't match my exact model, I ended up breaking the bike completely.

More recently, my table fan stopped working. I spent hours watching repair videos, trying to diagnose the sound and the wiring. In the end, the videos were too generic, I couldn't fix it, and I had to buy a new one anyway. I realized I had wasted hours of my life for nothing.

We realized the problem isn't a lack of information—it's a lack of context. Static videos cannot see my specific broken fan or my bike. They can't tell me if I'm turning the wrong screw. We built Home-ops to solve this: an AI agent that uses live video to see exactly what I see, diagnose the specific problem in real-time, and honestly tell me, "This is fixable, here is how," or "It's dead, let me buy you a new one."

What it does

Home-ops is an intelligent home manager that closes the loop between "Broken" and "Fixed."

1.See & Diagnose: The user points their camera at a broken device. Using Gemini 3 Flash Preview, the app analyzes the live video stream to understand the issue (e.g., "The drill motor is sparking" or "The screen is cracked").

2.The Decision Engine: The Agent calculates the feasibility of repair.

-If Fixable: It provides step-by-step, interactive repair instructions.

-If Broken: It switches to commerce mode.

3.Autonomous Purchase: If a replacement is needed, the Agent identifies the correct model and executes an instant purchase using USDC (USD Coin). This removes the friction of checkout forms and credit cards, allowing the AI to act as an independent economic agent.

How we built it

Frontend: Built with Flutter for a smooth, native experience on Android/iOS and Desktop.

AI Brain: We used Gemini 3 Flash Preview (via Google AI Studio). Its multimodal capabilities allow it to process video frames in real-time, enabling "eyes-on" diagnostics that text-only models can't do.

Backend: Firebase Firestore handles our real-time inventory and user data.

Payments: We integrated USDC (Circle Programmable Wallets) to enable the agent to hold funds and execute transactions programmatically on the blockchain.

Data Generation: We wrote custom Python scripts to generate realistic "broken vs. working" product scenarios and inventory data to test the agent's reasoning.

Challenges we ran into

Prompt Hallucinations & Product Mismatch: Early on, the AI would often "hallucinate" products that didn't match the visual input. For example, when the camera saw a broken table fan, the AI might suggest buying a ceiling fan or even a completely different appliance.

The Fix: We had to refine our System Instructions to enforce strict "Visual Grounding." We added a validation step where the AI must first describe the object in detail (e.g., "I see a 3-blade table fan with a white base") before it is allowed to search for a replacement product. This reduced mismatches by 80%.

Accomplishments that we're proud of

Real-Time Video Diagnostics: We successfully integrated Gemini 3 Flash Preview to analyze live video streams. The app doesn't just "see" a static image; it understands the context of a broken appliance in real-time.

Autonomous Commerce with USDC: We moved beyond simple chatbots by integrating Circle Programmable Wallets. The agent can now hold its own funds and execute USDC transactions on the blockchain without human intervention.

Solving the "Hallucination" Problem: We engineered a robust system prompt that forces the AI to visually verify an object before suggesting a replacement, reducing product mismatches by over 80%.

Cross-Platform Agent: Building a Flutter app that runs seamlessly on both mobile (for the user) and desktop (for admin/monitoring) with a single codebase.

What we learned

Commerce Empowers AI: We learned that an AI without a wallet is just a chatbot. By building commerce capabilities (USDC) directly into the application, we transformed the AI from a passive advisor into an active problem-solver that can physically resolve the user's issue. Trust in Autonomous Systems: We discovered that users are hesitant to let AI spend money automatically. Building a "human-in-the-loop" confirmation flow was critical to making the autonomous commerce feature feel safe and usable.

What's next for Home-ops

Health & Pharmacy Integration: We plan to upgrade the computer vision capabilities to scan medical bills and handwritten prescriptions. The agent will then cross-reference the doctor's notes and autonomously order the correct medicines using the existing USDC payment flow.

IoT & Predictive Maintenance: We are building a demo to connect Home-ops directly to Smart Home devices (like smart fridges and washing machines). This will allow the agent to detect glitches and order repairs before the user even notices the appliance is broken.

Universal Home OS: upgrading the "Home-ops" agent from just a repair tool into a full-service household manager that handles groceries, utilities, and health supplies automatically.

Built With

Share this project:

Updates