UnSpoken

The application overview showing the different API services and how they are connected.
A live feed of audio recordings posted near your location. This allows users to share moments and enrich human connections.
A map view of all the audio recordings posted. This allows users to listen and live experiances vicariously.
This page records audio clips using the Agora Cloud Recording feature. It then uses Symbl to get the formatted text and sentiment score.
Listening to unspoken thoughts on the map view.
ClickUp board that we used to brainstorm ideans, streamline our development and to ensure deadlines.
This is the SkyFlow users table. We save users to SkyFlow on register and authenticate them during login.
These are the two tables we used in SkyFlow. We used users to save user information and posts to save audio recordings.
The accquire API call we used to initiate a cloud recording.
This backend server was programmed in GO and made the Agora, Symbl and WebHookie API calls.
These are the two webhooks that we made to grab information from Symbl and update the SkyFlow database when a post is made.

Inspiration

The world was more inviting pre-pandemic. It felt safer to engage in small talk with those around us. We're looking to recapture that experience with a platform focused on thoughtful communication. It's easy to put content online to seek attention, but hard to find a place to put those unspoken thoughts.

Have a reminiscent memory of a location? Share your perspective and help give insights into the world around us.
Think of a funny joke and just want to share it? Users filtering by comedy will definitely appreciate it.
Want to save a memory just for yourself? Mark it as private, so it will only appear in your feed.

UnSpoken is inspired by the idea that most of human history was passed orally, and so much can be gleaned by paying attention to what people say.

What it does

Unspoken allows users to record their thoughts or stories, and save them with a title, location, tags, and whether it's public or private. The posts are available through a map view, populated with pins showing where they were saved. Users can also access posts through a feed view, showing them posts near them. When a post is created, UnSpoken automagically generates a transcription of the recording!

Posts are saved using Agora's Cloud Recording service which saves the audio file to an Amazon S3 bucket. When the post is saved in SkyFlow, it triggers a WebHookie that calls the Symbl API to get the formatted text and the sentiment analysis of the audio recording. The WebHookie updates the SkyFlow post with the results of the Symbl API. This process allows users to listen to audio experiences while also having a transcription and a sentiment of the audio recording.

How we built it

The application was built with Google's flutter framework. This meant that the front-end could be programmed in the Dart programming language and be deployed for both android, IOS and Web. Unspoken allows users to record their thoughts or stories geo-located, and it is processed by Agora to provide .m3u8 and .mp4 files. The .m3u8s can be used for playback, and the .mp4s are sent to Symbl to be processed for the formatted transcription. The back-end is written in Go, and interfaces with Agora via the Cloud Recording SDK to start and stop calls when it receives a POST message from the front-end. After a recording has stopped, the files are saved to an S3 bucket, and the links recorded in SkyFlow. When the SkyFlowID and url are saved, a webhook is called to the back-end to authenticate via Symbl.ai After authentication, the url is submitted, and after the conversation id is returned, the full transcript and a sentiment score is requested and saved to SkyFlow.

Challenges we ran into

In particular it has been really hard to coordinate efforts remotely during this hackathon, but ClickUp has some cool features that made life easier and took the busy work out of organizing tasks and keeping the boards clean.
Harness was a hurdle for us. We initially tried to deploy our backend via kubernetes, and due to pathing issues and docker we decided to instead deploy to the cloud via GCP.
It was definitely a struggle getting WebHookie to integrate into our backend at first, but I think this is because we didn't use it from the beginning. It would have been great to use the .yaml file in our configurations.
The most unexpected challenge was suffering a hard drive failure where we were saving videos to clip together for our submission. We were able to get an emulator recording, but the vlog format journal app development and features were unable to be recovered by the deadline.

Accomplishments that we're proud of

We were so happy we got subscriptions set up through WebHookie, it felt absolutely amazing to finally get the backend subscribed. Neither of us are particularly experienced in web development, and getting an application built using vue.js was a great learning experience. Although we did not build out the application on the web, it was exciting to use Harness to deploy to the Google Cloud Platform.

What we learned

It was really fun to learn about Go, already having experience in C/C++ made it easier to understand, but it was still a process.
We learned a lot about how applications can be layered, especially how cloud native apps work. We're interested in learning more about deploying applications, especially via kubernetes.
It was tough to learn about how information is handled over the web. WebHookie helped a lot, and with SkyFlow made us rethink how we stored and passed data.

What's next for unSpoken

Next we're excited to implement features to allow users to filter based on tags. We will use Symbl.ai to generate tags based on computed topics, and create a moderation layer before posts are available on the feed. As we continue development, we plan to implement a social aspect, allowing users to leave voice messages in response to those they've heard. Users will be able to receive messages at their leisure and choose whether or not to reply without fear of reprisal. As two users communicate more, we will unlock other features, allowing them to actually perform an audio call, messaging, sending images, and video calls. These will only be available as decided by an algorithm based on the polarity score analysis of topics provided by Symbl.

As tools and technologies continue to grow, we see this being an integral part of virtual platforms, allowing people to leave messages and notes around the world.