Inspiration
According to the National Commission for Disabilities, people with disabilities in Indonesia have limited access to formal education. Only about 30% can access basic education, 17% complete junior high school, 11% complete senior high school, and just 2.87% finish higher education.
This lack of education impacts their job opportunities in the formal sector, despite laws requiring at least 1% of employees in government and private companies to be people with disabilities.
In 2022, the Central Statistics Agency reported that out of 17 million people with disabilities of productive age, only 7.6 million were employed. Most of them, about 75%, work in the informal sector or are self-employed, while only 25% work in the formal sector.
What it does
Convy isn't just another video app. It’s built with heart, designed to make communication fun and easy for the hearing impaired, using the latest tech to bring everyone together. Our video conference platform uses cutting-edge machine learning to translate sign language in real time.
How we built it
We build this application on top of the tech stack of ReactJs for the client side and Python for the back-end to developing the machine learning model, as well Firebase for the database, including Firebase Firestore, Firebase Authentication, and Firebase Storage. Along side that, we also implemented WebRTC via PeerJs to develop the real-time video conferenc. With this tech stack, we accomplished the feature to do a realtime sign language and gesture detection in our video conferencing application. For details :
- ReactJs : Client side application.
- Tailwindcss : UI Design.
- PeerJs : WebRTC to enable realtime media (video & audio) sharing via Peer-to-Peer.
- Firebase : Cloud services to handle database & data storing, user authentication, and storage bucket.
- Python : Utilize Flask to create API that communicates with the client and run the trained & weighted machine learning model.
- Mediapip Holistic : Face, hands, and pose detection for training machine learning model.
Challenges we ran into
we ran into challenges such as training the data, since the time is limited we can not train a lot of datas, thus for this current version the dataset is very limited. We are planning on training the model on bigger dataset, so it can be used in various context.
Accomplishments that we're proud of
We are proud that we achieved 100% accuracy in the model summary after training it more than 1000 epochs. This allows our application to provide the best quality of sign language detection while doing the video call, although it is still still depends on the clients device and internet connection.
What we learned
This was a fun project. My team and I has learned a lot of valuable lessons, from technical hard skills to soft skills. We learned about how to be more aware with an idea, to make sure its full-prove-ness. We also learned how to better do a project management. Furthermore, from this project we also learned how to build a realtime video call app that also has a realtime machine learning detection.
What's next for Convy
There is still a lot possibilities and room of improvement for Convy. In the future, we would like to add more features such as multi-participant in one room and also multi-sign language detection in a meeting conference. We also want to implement Share Screen functionality to the app.
Log in or sign up for Devpost to join the conversation.