Hackbox Kinect

Demo
Hand Debug view

Inspiration

We wanted to make something that could network different devices together and provide somewhat of an accessibility features. We also had made previous hacking projects with the raspberrypi that enabled it to directly interface with other RF devices. Both of use wanted to gain experience working with some soft of computer vision and/or object detection as we have never touched it before.

What it does

The project is a HID input device using a kinect sensor to interact with a computer. It takes your hand position from the sensor and maps it to computer input so that you can manipulate the mouse and other gestures with just your hands or voice. Additionally, we also developed a audio interface to push audio to a LED matrix over udp wifi

How we built it

We originally had made an idea for networked nodes that could run independent of each other, however we ended up changed the priority of what we were working on to focus more on the object detection, gesture recognition, and hand orientation of the datapoints. We had also build a led matrix from parts available online and wanted to have an additional visualization.

Challenges we ran into

We had a significant amount of the starting time that was supposed to be dedicated to programming was actually used to plan out the details of the project. We didnt start coding till around 3:30. We spent the first 3-4 hours planning and doing research online so we ended up working on the more critical parts like the image recognition. Additionally

Accomplishments that we're proud of

We had got the image recognition working and spent a significant amount of time dealing with the datapoints that came out from openCV/mediapipe and stabilizing them enough to use as user input.

What we learned

Taking in positional data and smoothing it out to use for precise motion is really difficult. As well as manually registering certain movements for gestures and when to enter states to start tracking specific hand movements.