BrailleBuddy

Inspiration

160 million people around the world are deaf-blind. Normal daily communication is a challenge for them, which makes it harder for them to form relationships. In addition, deaf-blind people should be able to access the internet's resources and learn as much as they can, but this is currently a challenge with the technology available today.

What it does

Braille Buddy has three features. First is a speech to braille translator. This feature takes in speech input from the non deaf-blind user. That speech input is then converted into braille through a linear encoding. The linear encoding is transcribed to the deaf-blind user through a series of vibrations, where a character is represented through six short vibrations and a space through one single one. Second is the braille to speech. A "custom" keyboard was created using six cells in a 2x3 grid that be used to replicate an actual Braille character to create a message. The deaf blind user can type in a character and then submit that character to the message, which is confirmed through its linear-encoded vibration. If they messed up, they can remove the character from the message. When the message is finished being created, the deaf blind user can click a speak button to read the message to the other person through a text-to-speech AI. Third is a braille to search query. This feature uses the same "keyboard" as that from the braille to speech feature. However, instead of making the message to speech, the message is actually a query that is sent to GPT-4o. The GPT-4o API runs that query and then sends the result back to the user, who then can have the response read to them through the vibrations.

How we built it

We built the app using Android Studio and Java to build the frontend and backend. For the text-to-speech and vice versa functionalities, we used Google Cloud AI. For queries, Claude AI was used.

Challenges we ran into

We initially wanted to used localized haptic feedback to tell the user the braille character. However, we could not implement this easily. We also were challenged with figuring out how to run the text-to-braille function asynchronously along with the speech-to-text to decrease time, which took a lot of time to figure out. In addition, we had a fun time figuring out the UI, because we wanted to make this app as simple and accessible for a deaf blind person.

Accomplishments that we're proud of

We are proud of creating an innovative app that actually can make a positive difference in this world. In particular, we are proud of developing a viable linear encoding system that can be used for Braille, as well as a Braille keyboard that can allow for deaf blind people to communicate.

What we learned

We learned how to develop with new languages and platforms on the fly. From a technical standpoint, we learned how to create an app using Android Studio, create and utilize API calls, as well as utilize AI tools.

What's next for BrailleBuddy

We would like to make this app available for iOS devices, possibly through a cross platform development software. We would also like to implement full device support, which would allow for deaf-blind people to use their devices for long-range communication through calls and messaging apps.