Camera languages

Inspiration

As a Filipino and working alongside my Mexican teammate, we’ve both experienced firsthand the challenges of learning a new language. Despite years of classes and long Duolingo streaks, we noticed that many learners (including ourselves), struggled to move beyond basic interactions and lost motivation. Through some research, we discovered that language retention improves dramatically when learners are immersed in the culture and real-world usage of a language. This insight inspired LingoLens—an app we created to make language learning more immersive, engaging, and practical, connecting learners not just with words, but with the people and culture behind them.

What it does

LingoLens is an interactive language-learning app that helps users build vocabulary through real-world object recognition. The app selects words from a database and prompts users to find those objects in their surroundings using their phone’s camera. By associating words with real-life contexts, users are encouraged to actively engage with their environment, reinforcing memory and improving retention. Audio feedback generated with AI voices further enhances pronunciation and listening skills.

How we built it

We built LingoLens as a native iOS application using Apple’s frameworks to take advantage of iPhone-specific features such as the camera. We used MongoDB to store and manage our word database, Google Gemini for AI-powered language and vision capabilities, and ElevenLabs to generate natural-sounding voice feedback for spoken language prompts. Development was done collaboratively using Live Share to stay organized and work efficiently as a team. Challenges we ran into One of our biggest challenges was working with the ElevenLabs API. Our account was unexpectedly flagged for suspicious activity and temporarily shut down, which cost us a significant amount of time as we searched for workarounds. Additionally, Google Gemini API keys did not function correctly on one of our devices due to location recognition issues. Fortunately, Bernardo was able to step in and resolve this so we could continue development.

Accomplishments that we’re proud of

We’re especially proud of building our first iOS app and ending the hackathon with a product we were genuinely excited about. Overcoming technical setbacks and working through challenges as a team was a major achievement. This was also our first time using Apple frameworks to access iPhone-specific features like the camera—something not possible on a virtual machine.

We made a strong effort to stay organized throughout the project, and using Live Share for the first time proved extremely beneficial. For me (Nate), this was my very first hackathon, and I was nervous going into it. Thanks to Bernardo's support, it turned into an incredibly positive and educational experience.

What we learned

We became much more comfortable working with the iOS framework and iPhone-specific features. We learned how to integrate video backgrounds, connect and test apps directly on physical devices, and better understand the overall app development workflow.

I (Nate) also learned how to properly manage and securely store API keys, how different frameworks fit together, and what it really takes to build a functional application from start to finish.

What’s next for LingoLens

Next, we plan to expand support for more languages, improve the quality and reliability of ElevenLabs voice integration, and add features such as missions, leaderboards, and enhanced gameplay mechanics to further motivate and engage users.

Built With

• avfoundation
• elevenlabs
• gemini
• mongodb
• swift
• swiftui
• xcode