SignRPG

Title Screen
Gameplay

Inspiration

Knowing ASL has always been crucial for communicating with the hearing impaired, yet it has such a large barrier to entry. Few classes teach ASL, and there often lacks a motivation to learn for the average person. Recently, I wanted to get more into Mediapipe and hand detection, so I thought: Why not combine these and make a fun game to learn ASL?

What it does

SignRPG is a 2D fantasy RPG game where you play as a wizard, casting spells and navigating the world through ASL. The game utilizes your laptop's built-in webcam to track your hands and uses AI to predict the ASL letter. These predictions are fed into the fantasy game to not only control your character movement, but also cast spells against a horde of enemies!

This game is designed to encourage the average person to learn ASL through an interactive and fun environment, aiming to lower the barrier to entry.

How I built it

My stack consisted of a Python and Flask backend that handled image capture, landmarking, and AI prediction of ASL letters before being passed through a WebSocket to our React frontend.

Backend: A camera or webcam is accessed through OpenCV, and then passed through Mediapipe to detect the positions of all hand landmarks on your left and right hands. These points are then processed and passed through an AI based static gesture model to predict the current ASL letter being signed. The current webcam frame, with landmarks, is combined with this prediction and sent to the frontend through a Websocket.

Frontend: The frontend website was built using React and Flask, then packaged with Vite. The React components are updated every time there is a new update through the Websocket. The game itself is split into three main parts: player/enemy sprites, world generation, and spell casting. Each of these is split into its own React component for modularity, and dynamically updated.

World generation is done through preset arrays of tile numbers, and constantly repeated for a semi-infinite map. Enemy sprites are randomly generated, and will constantly head towards the player. Spell casting is done through a "spell buffer" that tracks the most recent ASL letters. Once there is an exact match, that spell is cast, the buffer is cleared, and the enemies are defeated.

Challenges I ran into

The AI model I loaded using Tensorflow was very slow in predicting once I started to add more objects into the game, which drastically lowered the game FPS. It took me a while, but it turns out that I could load the model without any of the training features and optimize my prediction function!
Trying to test the game without knowing any ASL. I had to slowly learn some ASL letters throughout the hackathon in order to test.

Accomplishments that I'm proud of

I came in with barely any experience with Mediapipe, WebSockets, and game development. I'm proud to say that I was able to create a fun game while learning all of these new skills myself!

What I learned

Never be afraid to do something yourself!

My original team was fragmented due to a series of unfortunate circumstances, but I still insisted on doing a solo project. Initially, I was scared of tackling an entire full-stack project alone and worried that nothing would work. Throughout the hackathon, I found that working alone is a special type of experience, and extremely rewarding once things start to click!

What's next for SignRPG

I want to incorporate more levels and increase the customizability to encourage users to explore and learn even more ASL!

In addition to this, ASL is filled with various shorthand signs. I want to incorporate this into our game to broaden the possibilities, increase the pace of gameplay, and expand the repertoire of ASL signs that can be practiced.