SeniCare

An AI-powered 60-second webcam + voice check-in that detects early risk signals, auto-triages, and generates clinician-ready summaries for faster intervention.

Comment

Inspiration

We wanted to build something genuinely useful for clinics that care for seniors, especially older patients who live alone, are remote, or do not have someone consistently watching over their day-to-day health.

The core idea was a daily "lightweight check-in" that feels simple and non-intimidating for the senior, while still giving clinicians a reliable signal to spot changes early and prioritize follow-up.

We were also inspired by how multimodal AI can combine voice interaction with camera-based biomarkers in a single flow, creating a practical way for doctors to stay informed about a patient’s condition without requiring constant in-person visits.

What it does

SeniCare is a role-based web app designed to help clinics monitor seniors who live alone or remotely through a simple daily check-in.

  • Seniors complete a guided daily check-in.
  • Clinicians view triage, trends, and summaries to quickly identify who needs attention.

Senior flow

User signs in, starts the check-in, and grants camera/mic access. -> Gemini Live provides the intro prompt and guides the interaction. -> The app records a 10-second face clip and uploads it. -> The backend runs facial symmetry analysis and optional video heart-rate estimation (VHR / open-rppg) -> Gemini asks screening questions, and responses are captured and transcribed. -> The backend validates all required data, completes the check-in, and returns triage (Green, Yellow, Red). -> The UI shows a completion confirmation.

Clinician flow

  • The dashboard shows the senior list, latest statuses, and triage distribution.
  • A report endpoint generates a clinician-facing summary for each senior.

How we built it

Frontend

  • React + Vite + Tailwind
  • useAuth manages register, login, and JWT session handling.
  • useCheckin orchestrates the real-time check-in pipeline (Gemini session, camera capture/upload, screening submission, completion).
  • SeniorCheckin uses a minimal state-driven UI (idle, running, mascot, complete) with modular components.

Backend

  • FastAPI with modular routes (auth, checkins, screenings, dashboard, reports, etc.).
  • MongoDB via PyMongo for users, check-ins, and screenings.
  • JWT auth + role-based access checks (clinician-only dashboard and report routes).

AI + signal processing

  • Gemini Live for voice interaction and ephemeral token flow.
  • MediaPipe / OpenCV for facial symmetry analysis.
  • open-rppg + ffmpeg preprocessing for heart-rate estimation.
  • Gemini summary generation for clinician reports in structured JSON.

Challenges we ran into

  • Synchronizing real-time phases across voice and video
    • Ensuring camera scan timing aligns with the AI prompt timing.
  • Browser media reliability
    • Permission gating, device availability, and handling browser differences for speech and media.
  • Data completeness
    • Preventing premature completion when only partial data exists.
    • Added validation logic for missing transcript, screening, or facial outputs.
  • Signal quality variability
    • Lighting, camera quality, and clip encoding affected analysis consistency.

Accomplishments that we're proud of

  • Built a true end-to-end multimodal pipeline, not just mock screens.
  • Implemented role-specific product surfaces (senior experience and clinician analytics/reporting).
  • Added robust backend checks for incomplete or abandoned check-ins.
  • Kept the senior UI intentionally minimal and accessible while preserving full workflow behavior.
  • Integrated biomarker extraction and conversational screening into one check-in journey.

What we learned

  • Reliable health-adjacent UX depends heavily on state orchestration and clear transition rules.
  • A simple UI still requires complex backend sequencing to be trustworthy.
  • Structured prompts and strict output formatting are critical for clinician-facing LLM features.
  • Media workflows need fallback paths and graceful degradation to handle real-world variability.

What's next for SeniCare

  • Add longitudinal baseline and trend modeling per senior (not just per-check-in triage).
  • Replace browser-dependent speech handling with a more robust STT pipeline.
  • Expand alerting from a test route to production-grade messaging and escalation flows.
  • Add automated test coverage for check-in orchestration and edge cases.
  • Harden for production with stronger audit trails, privacy controls, and compliance-oriented safeguards.
Share this project:

Updates