IRL Damage

IRL Damage
The Testing Process :)
Full Hardware
Wire Diagram
Region of Interest for HUD
Extracted Mask Using Canny Edge Detection
Edge Detection on ROI Snapshot
Limb Distinction Map
Critical Pressure Timeline over Prev. Session
Limb State Timeline over Prev. Session

Inspiration

While modern games feature incredibly realistic visuals, most still lack physical feedback. We were inspired to bridge that gap by translating visual in-game damage into live haptic response.

What it does

The project uses computer vision to detect where the player takes damage and delivers limb-specific haptic feedback corresponding to the affected body part.

How we built it

Our system uses computer vision to continuously monitor the in-game HUD of Escape from Tarkov and determine which part of the player’s body has taken damage.

A Python-based CV pipeline captures a specific region of the screen containing the player health model and analyzes limb colors (green, yellow, red) using HSV color segmentation, contour analysis, and temporal smoothing to ensure stable detection.

Since the Tarkov body HUD is a unique shape (not symmetrical, difficult to manually recreate the shapes for the limb outlines), we decided to utilize an edge detection algorithm to get the HUD outline for us, using kernels detecting values over a certain threshold as specific edges. We specifically decided to use Canny edge detection, as the gaussian blur reduces the noise from the ever-changing background, especially when there are fine details like foliage that we don't want the code to pick up.

Each limb, including the head, chest, torso, arms, and legs, is independently tracked by utilizing the edge detection over 7 separate files and classified in near-real time.

When damage is detected, the system sends a signal over serial communication to an Arduino, which controls a relay module. Each relay corresponds to a specific body region and dynamically enables or disables electrical flow to pads connected to a TENS machine. The TENS unit remains powered, but feedback is only delivered when the relay for a damaged limb is activated.

This creates a closed-loop feedback system where visual in-game damage is translated into physical haptic feedback, significantly increasing immersion and reinforcing player awareness and reaction.

Challenges we ran into

At the start of the project, we were not even sure whether translating in-game damage into physical feedback was technically possible. This uncertainty required early experimentation across both software and hardware before committing to a final design.

None of us had prior experience working with Arduino or relay modules, which made hardware integration one of our biggest challenges. Documentation for relay control varied widely, and we had to learn through trial, testing, and careful electrical safety checks to correctly map Arduino outputs to relay channels.

On the software side, computer vision color detection proved difficult due to lighting variation, compression artifacts, and background interference in the game UI. We addressed this by switching to HSV color space, tuning saturation thresholds, and adding temporal smoothing to reduce false positives.

Finally, testing was limited by hardware constraints. Only one team member had a Windows machine capable of running Escape from Tarkov, while the rest of the team used macOS. This required us to carefully coordinate testing, split responsibilities, and validate changes asynchronously.

Accomplishments that we're proud of

We're incredibly proud of successfully integrating hardware and software to create a fully functional haptic feedback system. Getting the Arduino to communicate reliably with the relay module was a major milestone—especially since half of us had no prior experience with embedded systems or relay control. We learned how to safely wire these machine outputs through relays, map Arduino pins to specific body regions, and ensure that electrical signals were properly isolated and controlled.

What we learned

This project taught us how to bridge the gap between virtual and physical systems. We gained hands-on experience with Arduino programming, relay module control, and the intricacies of serial communication between Python and embedded hardware. We learned that hardware integration requires patience, careful debugging, and a strong focus on electrical safety, as we saw early on that with the wrong setting and voltage, you could seriously hurt yourself.

What's next for IRL Damage

Moving forward, we want to expand IRL Damage beyond Escape from Tarkov to support multiple games with different HUD designs. This would involve training a machine learning model to generalize damage detection across various visual styles, or building a configuration system that lets users calibrate the CV pipeline for their specific game.