Contextify

Inspiration

During meetings or professional conversations, we often encounter technical terms, unfamiliar courses, or references from fields outside our expertise. This can make it challenging to stay focused and fully grasp the ongoing discussion. Additionally, staying updated on global events or news while managing these conversations can be overwhelming. This inspired us to develop Contextify, an app that provides real-time information and context while others introduce new topics. By delivering insights simultaneously, it helps users quickly capture and understand knowledge, making conversations more productive and engaging.

What it does

Contextify is an AI-powered tool that transcribes speech into text using OpenAI’s Whisper and enriches it with insights via GPT-4o. It identifies key entities like names, companies, and technical terms while integrating the Bing News API to provide real-time relevant updates. Contextify turns conversations into actionable insights, enhancing comprehension and saving time in meetings, research, and multilingual environments.

How we built it

The frontend of Contextify was developed as a macOS app using SwiftUI, providing a modern and intuitive user interface.

The backend, built with FastAPI and Python, integrates advanced tools like Whisper for real-time transcription, GPT-4o for contextual entity extraction, and the Wikipedia API for instant summaries. Additionally, we incorporated Bing News API for real-time updates and graph libraries for data visualization.

Challenges we ran into

Implementing real-time speech-to-text with Whisper was challenging due to low initial accuracy and significant background noise in audio recorded from smartphone microphones. We had to carefully adjust the audio frequency, channels, and bit depth to align with Whisper’s requirements. Noise reduction was also crucial to improve transcription quality. After extensive testing and fine-tuning, we achieved an accuracy rate of 70%, demonstrating the importance of proper audio preprocessing.

Accomplishments that we're proud of

We are incredibly proud to have transformed an idea into a fully functional app within just 24 hours. The process tested not only our technical skills but also our mental and physical endurance. Balancing the challenges of overcoming fatigue while debugging and resolving technical issues was no small feat. This journey embodies the true spirit of a hackathon—pushing boundaries, staying resilient, and turning creativity into reality under intense time constraints.

What we learned

During this hackathon, we gained valuable experience in building an application that bridges complex backend functionalities with a user-friendly frontend interface.

We enhanced our skills in real-time transcription with Whisper and entity extraction using GPT-4o, while effectively integrating Wikipedia and Bing News APIs for real-time data. On the frontend, we mastered SwiftUI and Xcode to create an intuitive macOS app. This project strengthened our problem-solving, API management, and teamwork under tight deadlines.

What's next for Contextify

Moving forward, Contextify aims to further enhance the accuracy of voice recognition technology and refine course-related information, such as credits (Units), to deliver a more seamless and efficient query experience for students. At the same time, Contextify remains dedicated to addressing critical communication challenges in hybrid and remote work environments. By resolving misunderstandings caused by specialized jargon and technical terms, Contextify will integrate cutting-edge features to identify and interpret workplace terminology in real time. These advancements will solidify Contextify’s role as a vital tool for bridging communication gaps, boosting productivity, and fostering a more connected and effective workplace.