Tarvis — Devpost About the Project

What Inspired Us

We wanted to build technology that helps people reach new heights by improving the tools they already use every day. Productivity apps and smart home dashboards often feel disconnected, while AI tools can be generic and unreliable for real‑world action. Tarvis was inspired by the idea of a practical JARVIS + TARS assistant that keeps users focused, reduces friction, and connects software insights to physical environments.

What We Built

Tarvis is a life management platform that unifies:

Productivity (tasks and workflow)
Habits (streaks and consistency)
Automation (devices, scenes, and triggers)
Robot control (safe movement + telemetry)
Live robot camera streaming (real‑time visibility)

The system is designed around a dual‑mode AI model. Root mode executes deterministic, whitelisted commands for safety‑critical actions, while LLM mode enables flexible assistance through a structured tool system.

How We Built It

Frontend (React + TypeScript + Vite):

Desktop dashboard UI with real‑time updates
Structured panels for tasks, habits, automation, and robot control
Live MJPEG camera feed embedded on the Robot page

Backend (FastAPI + SQLAlchemy + WebSockets):

REST APIs for tasks, habits, devices, and scenes
WebSocket broadcasts for live updates and Pi telemetry
Capability plugin system that registers AI tools at runtime

Hardware Layer (Raspberry Pi):

Robot and environment clients connect over WebSocket
Commands from the base station trigger physical actions
MJPEG stream server exposes the robot’s live camera feed

This architecture creates a full stack loop: UI ↔ backend ↔ hardware, with AI tools acting as a safe, modular execution layer.

Challenges We Faced

Safety constraints for AI: We had to prevent free‑form AI from issuing risky physical commands, which led to a strict root/LLM separation and tool allowlists.
Real‑time coordination: Synchronizing UI updates, backend state, and Pi telemetry required careful WebSocket handling.
Hardware + software integration: Streaming video and device control introduced network reliability and latency considerations.
Extensibility: We wanted to add new capabilities without touching the core system, which required a plugin architecture and consistent tool schemas.

What We Learned

Building AI systems that interact with hardware requires explicit safety boundaries.
Modular plugin systems are essential for scaling AI tooling without creating a monolith.
Real‑time UX improves trust and clarity, but demands robust WebSocket design.
A clean, focused UI can make advanced systems feel approachable and useful.