SiteOps

“Real-Time Control for Safer Construction.”

Comment

SiteOps: Making Construction Sites Safer with AI Inspiration

When I first saw the Infrastructure track at HackHarvard, I began reflecting on the conditions of construction workers in Singapore. Many of them are transported in open trucks instead of proper vehicles, underpaid for long hours, and often work in unsafe environments where injuries or deaths go unreported. I wanted to create something that could protect them, something that would make their workplaces safer and ensure that their efforts were valued and respected.

As I explored further, I realized that this issue was not just local. In the United States alone, 1,075 construction workers lost their lives in 2023, the highest number in more than a decade. Thousands more were injured, and billions of dollars were spent on medical and compensation costs. I began to understand that the problem of safety in construction is truly global. Regardless of the country, every worker deserves to return home safely after a long day of work.

I wanted to figure out a way to track workers and detect their bodily movements, whether they were standing, resting, or had fallen down. A similar detection system existed in a game I used to play growing up, Call of Duty. One feature that always fascinated me was the Predator Missile killstreak, which revealed bounding boxes over every player on the battlefield and gave a complete overview of the environment. Back then, I thought it was just a cool visual effect. Now, I saw it as something much more meaningful — a concept that could be reimagined for good.

That realization became the spark for SiteOps. What began as a fascination with computer vision turned into a mission to build technology that saves lives. SiteOps uses AI, tracking, and real-time data to give construction managers the same kind of awareness once seen in games, but this time to protect, not to harm. Every line of code in this project is written with one belief in mind: every worker deserves a safe workplace and the chance to go home at the end of the day.

Ultimately, to push our infrastructure forward, we must first ensure that the people who build it are safe, protected, and free from fear in their workspace.

What it does:

SiteOps is an AI-powered construction safety and operations platform that builds upon the foundations laid by current project management tools. While most competitors focus on work tracking, resource management, and administrative planning, SiteOps integrates real-time safety intelligence directly into these workflows. Existing platforms often require interfacing with third-party monitoring systems for safety analytics, but these integrations are rarely comprehensive or responsive. SiteOps bridges that gap by combining operational management with proactive, AI-driven safety monitoring in a single unified platform.

Core Platform Features:

Integrated Ticketing System: Every task, inspection, or safety incident is managed through tickets structured like initiatives, epics, and subtasks. Each ticket includes a built-in chat interface for on-site communication, priority tagging, and document uploads such as permits or blueprints.

People Management Page: A centralized dashboard that tracks every worker’s status, showing who is active, on leave, or unavailable. Managers can also access associated documents such as medical certificates or work permits.

Machinery and Equipment Registry: A real-time list of all active and idle machinery on site, allowing supervisors to cross-reference with proximity alerts and maintenance records.

Proximity Detection: YOLOv8n detects workers and vehicles in real time. If a worker gets too close to operating machinery, SiteOps triggers an instant alert to prevent struck-by incidents.

Fall Detection: Automatically detects when a worker has fallen or is lying prone, immediately alerting supervisors and logging the event with GPS coordinates.

Live Headcount Monitoring: Continuously compares expected versus actual workers on site. Unaccounted or unauthorized personnel trigger alerts, ensuring accountability and emergency readiness.

AI Site Agent: Powered by Google Gemini 2.5 Flash, the system continuously analyzes all activity, identifying patterns such as repeated fall detections or recurring equipment faults. When anomalies are detected, the AI autonomously generates relevant tickets or inspection requests.

Conversational Assistant: A natural language chat interface that allows managers to ask questions like “Who’s currently on site?” or “Any high-priority safety alerts today?” Responses are grounded in live data through a Retrieval-Augmented Generation (RAG) pipeline, ensuring accuracy and context awareness.

By merging computer vision, AI reasoning, and operational management, SiteOps transforms construction oversight into a connected ecosystem. It ensures that safety is not an afterthought managed by separate systems, but an integral part of everyday project operations.

How we built it

Computer Vision Pipeline (Python):

YOLOv8n for real-time person and vehicle detection

MiDaS for monocular depth estimation to assess relative distances

ByteTrack for consistent multi-object tracking and identity preservation across frames

OpenCV for video processing at 720p and 20 FPS

AI Intelligence Layer (Google Gemini 2.5 Flash):

Retrieval-Augmented Generation (RAG) pipeline feeding live construction data from Supabase

Two operating modes: Reactive, which responds to user queries, and Proactive, which autonomously monitors and analyzes site activity

Domain-specific instruction tuning to ensure context-aware, accurate safety recommendations

Web Application (Next.js 15 + React 19):

Real-time dashboard displaying live camera feeds, active alerts, and ticket management

Integrated chat interface to interact with the AI Site Agent for live updates and safety insights

Dynamic analytics page showing alert patterns, site statistics, and worker activity

Backend Infrastructure:

Supabase for PostgreSQL database, authentication, and real-time data synchronization

FastAPI for WebSocket event streaming and API endpoints

NDJSON logging for structured event recording and auditing

Auto-refresh safety monitoring with configurable time intervals

The key innovation is the RAG architecture, which ensures that every AI response is grounded in live database state. When SiteOps reports multiple fall in

Challenges we ran into

Model Selection Issues: Early Gemini models returned 404 errors before identifying the correct one, Gemini 2.5 Flash.

Chat History Validation: Gemini required chat histories to begin with a user message, causing initial validation failures.

Environment Variable Caching: Next.js cached outdated Supabase variables, requiring restarts during debugging.

Real-time Performance: Balancing accuracy and speed demanded fine-tuning YOLOv8n thresholds to reduce false positives.

RAG Optimization: We developed a keyword-triggered system to balance context size and response latency for Gemini queries.

Accomplishments that we're proud of

I am proud to have run my first computer vision model ever. I had never built an app with a CV model or even an app in general, as I am not a software engineer. While I understand how models work conceptually, implementing them into an application was a new challenge. I am proud that most of the functions I wanted to implement are now part of the app in one way or another.

What we learned

I learned a lot about simple heuristics that CV models can use and about the real-world statistics surrounding construction worker safety in America. The data was truly eye-opening and motivated me to pursue this project further.

I also don’t have extensive experience with frontend development, so I made use of Cursor to assist me while building the interface. This was honestly my first time vibecoding — learning how to integrate AI directly into my workflow. My background is mostly in data analysis and research, so I’m used to manually validating everything myself. However, I realized that in the context of building MVPs, leveraging AI makes a lot of sense. It was a huge productivity boost, even though it took some getting used to. At times, the AI seemed to understand what I wanted before I even typed it.

Learning basic TypeScript was also an interesting experience, especially when debugging. I also became more familiar with how model thresholds work and what the YOLO models are capable of. Although I understood the theory behind them, I had never used one hands-on before.

I also stumbled upon the concept of adversarial attacks. While testing, I tried simulating trucks by displaying pictures of vehicles on my phone, but the model’s confidence was very low due to reflections and poor image quality. After researching and asking ChatGPT about this, I learned how adversarial attacks can affect models and how I could make datasets more robust by generating manipulated versions of images with different noise and distortions. I found this incredibly cool and insightful.

What's next for SiteOps

SiteOps is currently in a very early stage. Moving forward, I plan to integrate more sophisticated detection models and support multiple cameras for enhanced situational awareness. The current MiDaS model provides depth perception but cannot yet measure exact distances between objects. With access to multiple cameras, we could triangulate precise coordinates for both workers and machinery and visualize them on a site map for better monitoring.

I also want to implement zone-based safety systems, where managers can draw digital boundaries directly on the map. Each worker’s mobile app could use their camera feed and GPS to verify whether they are operating within their assigned zones, helping improve accountability and reduce risk. Similarly, heavy machinery can be assigned specific operational zones to ensure workers do not enter unsafe areas.

Beyond this, I aim to add fire hazard detection by integrating IR sensors to identify overheating or combustion risks early and sound automated alarms.

Lastly, for the project management side, I plan to develop analytical dashboards showing alert trends and comparing safety performance across sites of similar type and scale. This would help project managers assess how safe their sites are relative to others and maintain accountability for worker safety.

Built With

  • bytetrack
  • bytetrack-(object-tracking)-ai/ml:-google-gemini-2.5-flash-(rag-based-chat-assistant-and-autonomous-site-monitoring-agent)-infrastructure:-supabase-for-database-and-storage
  • coco-ssd
  • fastapi
  • github
  • google-gemini-2.5-flash
  • google-generative-ai-api-development-tools:-github
  • midas
  • midas-(depth-estimation)
  • next.js-15
  • npm
  • opencv
  • postgresql
  • python
  • react-19
  • shadcn/ui
  • shadcn/ui-components
  • supabase
  • supabase-(postgresql-+-storage-+-real-time-subscriptions)-computer-vision:-yolov8n-(ultralytics)
  • tailwindcss
  • tensorflow.js
  • tensorflow.js-(coco-ssd-for-browser-based-object-detection)-backend:-python
  • turbopack
  • typescript
  • vercel-ready-deployment-apis:-supabase-rest-api
  • websocket
  • yolov8n
Share this project:

Updates