ConnectASL

Inspiration

“The most important thing about communication is to hear what isn’t being said.” Although most people would see this quote in a figurative way, it’s also important to look at it quite literally. There are many people who aren't able to either speak or hear. Such people often struggle to communicate due to the lack of platforms that are disability-friendly. Currently, 430 million people in the world require rehabilitation for disabling hearing loss. We really wanted to find a way to support all of these people.

What it does

Our goal was to create software that gives such people an opportunity to communicate efficiently. We came up with a translator that converted sign language to both text and speech aimed to help mute, deaf, or aurally challenged people and giving them a way to communicate easily with others.

How I built it

ConnectASL is an innovative and functional website. Upon being granted access to the user’s camera, our website interprets any sign language they do, using TensorflowJS along with Handpose and Fingerpose. Then, with a small database that we created, it displays what the sign language says. A text-to-speech function allows other people to hear, in real-time, what the user is signing.

Challenges I ran into

We overcame many challenges during the creation of this project for our hackathon; most were technical or related to our code. However, we were able to solve all of these problems and overcome our obstacles relatively easily and without too much struggle! Our teamwork and preparation came in handy during this hackathon. Other challenges included having to finish the entire project in such a short period of time and having to learn some basic sign language in order to be able to test the site.

Accomplishments that I'm proud of

Our most prominent accomplishments are being able to make our website fully accessible, efficient functionality (with real-time movement tracking), and extensive, interactive UI that not only translates sign language but also teaches users how to use sign language. These components were all a huge part of our project.

What I learned

We learned many things while making our project. Our teamwork and ability to work so well together was a huge advantage throughout the hackathon, ensuring that we were efficient while still having a blast. All of our skills improved as we learned and coded this project, and we only grew from our mistakes. We (Andrew) even took this opportunity to learn some sign language so that we would be able to test if our website worked.

What's next for ConnectASL

If we were to improve or grow our website further, we would add more functionality and an even more user-friendly interface. It would be amazing to be able to add features such as using text-to-speech on speakers such as Alexa / Google Home, and speaking chatbots that could reply to the messages sent in sign language. Any feature that makes our project more accessible, unique, and engaging would make it better, and in turn: change the future of AI forever.

Built With

• css
• fingerpose
• handpose
• html
• javascript
• mediapipe
• react.js
• tensorflowjs