Transtreaming

Border-less Communication

Comment 2

Inspiration

During this pandemic, our day to day communication is even more digitalized with people self-isolating themselves. The importance of remote meetings and communications has increased more than ever. Keeping in mind this importance we should make efforts to make this communication better and better. With a diverse range of people speaking different languages and working together their communication should be smooth. The native language is the best way to transmit your ideas so an application that enables meeting attendees to speak in their native languages is essentials. This need and its importance motivated us to work on this idea.

What it does

The application transcribes user audio in any language and then translates it to the desired destination language. The translated text is shown at the chatbox of both attendees. For example, Take two attendees A and B. If attendee A has chosen English as his/her language and attended B has chosen german then in attendee A chatbox all the conversation will be displayed in English and in attendee B chatbox all the conversation will be displayed in german.

How We built it

First off we integrated Agora web SDK for a realtime video calling feature. Then we transcribed the user audio at the client-side of the application. After the audio is transcribed to text on the client-side, the transcribed text is sent to the server using a socket, and the text is then translated and send to the client partner application on his/her socket channel. The translated text that is returned is shown on the chatbox of the attendee that has chosen that specific language.

Challenges we ran into

1: Differentiate between the transcribed text of both the attendees as each transcribed text has to be translated into a different language.

2: Deciding the distributions of the functionality on the client-side and the server-side.

Accomplishments that I'm proud of

1: Successful implementation of realtime translated subtitles.

What We learned

1: Explore the agora web SDK and how to integrate with any modern web framework.

2: Explore the different NLP libraries.

3: Understood in a bit more detail on how sockets work.

2: How audio transcribing is done.

3: How to work with socket and APIs to make a real-time application.

What's next for Transtreaming

Really excited about the future of Transtreaming as they're always so much room for improvements. This application can be used in many use cases like helping diverse people communicate in a meeting. For that, we need to integrate this on websites and applications that offer remote meeting services. If we move one step forward we can extend the idea to the computer vision side by detecting hand directions and converting them text for people who are unable to hear.

Presentation video of Transtreaming

https://youtu.be/jrSjLrxAKhc

Live Demo of Transtreaming

https://youtu.be/ZqaPyCA1OMY

GitHub Repos

1: Project Description Repo: https://github.com/zilehuda/transtreaming

2- Backend Repo: https://github.com/zilehuda/transtreaming-europa

3- Frontend Repo: https://github.com/zilehuda/transtreaming-jupiter

Share this project:

Updates