Sign Sense

Sign Sense revolutionizes sign language education by gamifying and revolutionizing the industry with DIY features that allow users to practice sign language with real-time feedback.

Comment

Sign Sense: Revolutionizing Sign Language Education

Inspiration

Sign Sense was inspired by the challenges deaf children face when learning sign language through traditional methods, especially during the early stages of development. Recognizing the need for a more engaging and effective approach, we set out to create an interactive learning experience that makes the process both enjoyable and impactful.

Inspired by popular language learning tools like Duolingo, Sign Sense does what these big corporations are scared to do: actually focus on teaching and using all resources available to do so without looking for profit. We spare no expense for our users, and are not focused on gathering a profit, but rather simply in educating the future youth.

What it does

Sign Sense goes above and beyond by integrating a one-of-a-kind, state-of-the-art real-time feedback mechanism. Using machine learning, AI, and various elements of data science, Sign Sense delivers personalized guidance and feedback, catering specifically to the needs of the deaf community. It's a gamified app designed to revolutionize sign language education for deaf children.

It offers:

  • Engaging learning experiences through GIF's and Interactive Videos.
  • Implementing a Health Bar for tracking progress through the lessons.
  • Using statistics it promotes healthy learning through Goals and Progress by minutes.
  • A never-seen-before DIY feature allows real-time practice with feedback through a highly trained AI Model

How we built it

Our development process was thorough and challenging:

Frontend

  1. We started with React for web development
  2. Transitioned to Swift for backend integration possibilities
  3. Explored Flutter, when Swift was not working for cross-platforms
  4. Finally settled on Svelte for its performance and ease of use with both Mobile and Web development

Backend

  1. We gathered different datasets that we deemed appropriate.
  2. We then had to classify and label these different classes based on multi-class classification decision boundaries.
  3. We then split our samples into the appropriate batches for the model: split the data into training, testing, and validation.
  4. These were then trained using Yolov5 model on AWS Sagemaker and fine tuned as needed.
  5. Comparing these results, we chose our best model and used python scripting to create a fast api to integrate it with our Svelte frontend.

The core of our application relies on machine learning models for sign language recognition, which required extensive dataset training and optimization. This process was particularly challenging and time-consuming:

  1. Dataset: This took a lot of research time and required a thorough look at all the samples to ensure that the model would be trained well. This included, but was not limited to checking different signers, lighting conditions, and backgrounds.

  2. Model Training and Optimization: Using PyTorch, we experimented with deep learning architectures. The training process was computationally intensive, and took hours to train on Google Cloud. It also took several iterations to get a model with high accuracy which meant fine-tuning the hyperparameters, epochs, and batch sizes.

  3. Integration with Svelte: One of our biggest challenges was integrating the Python-based machine learning models with our Svelte frontend. We solved this by:

- Creating an API using FastAPI through python scripting.

  • Exporting our trained models to .onnx format

  • Invoked different API requests from user input such as with the user's camera

  • Process the request and display what the model predicted to the user

Challenges we ran into

  1. Dataset Training: One of our biggest obstacles was training our machine learning models with limited computational power and GPU resources. The time constraints of the hackathon added even more pressure for this situation.

  2. Accuracy Issues: Initially, our models weren't achieving the accuracy levels we needed for reliable sign language recognition, sometimes entirely disregarding certain letters. This causes us to run multiple iterations to try to optimize our model.

  3. Integration Difficulties: We faced significant challenges in deciding between web-based and mobile-based systems. Each platform (React, Swift, Flutter, and Svelte) presented their own integration challenges with our machine learning models.

Accomplishments that we're proud of

  1. Learning and deploying in 4 entirely new languages: We successfully navigated through React, Swift, Flutter, and Svelte, deploying functional components in each for the first time.

  2. High-accuracy hand gesture recognition model: We trained a machine learning model that can accurately detect hand gestures and return the correct letters of the alphabet. This represents our DIY feature.

  3. Engaging educational and gamified app: We developed an application that not only serves its educational purpose but also keeps users engaged through effective gamification. We're confident in the app's ability to maintain user retention throughout the learning process.

  4. First-ever Learning DIY feature: We created a DIY feature that allows for real-time sign language practice using AI and Machine Learning models.

  5. Mobile implementation of machine learning models: Successfully implementing our model to work well on a mobile device.

What we learned

  1. The importance of design and finalizing on a specific language or system before starting development.
  2. What techniques are best for optimizing machine learning models
  3. The demands of training a M.L model - the extensive computational power and time.
  4. Insights into the challenges of sign language education and the needs of the deaf community

What's next for Sign Sense

  1. Expanding our sign language dataset to improve recognition accuracy and cover a wider range of signs such as many common phrases
  2. Branching out to different sign languages such as Auslan, British Sign Language, and Indian Sign Language.
  3. Developing more game type features, such as daily quests, rewards, and streaks.
  4. Incorporate AR/VR glasses that integrate with our M.L model to generate sign language translation in real-time.
  5. Deploy the app for both Android and iOS platforms. For iOS development, revisit Swift for the application.
Share this project:

Updates