Allergy Vision

Inspiration

Our inspiration came from seeing videos online of people using Meta Smart Glasses to cheat at chess - telling them what move to make in real time as the board updated. We thought we could apply a similar concept to create a tool which would be more useful in everyday life, and thus we came up with the idea of detecting allergies.

What it does

Using an object detection algorithm, we are able to detect foods as the user raises them to their mouth using the camera built into the Meta Smart Glasses, and determine whether said food contains any of their pre-specified allergens. If it does, an audio warning will play describing the allergen within the food and advising the user not to eat said food.

How we built it

We use the meta smart glasses connected to a phone that is calling another laptop on the desktop version of WhatsApp. Whatsapp and Instagram are the only two ways to livestream the video feed from the glasses. A screen recording of the WhatsApp call is taken by the python and put into Yolo (an object detection model that we trained with custom foods. Once an item of food is detected, the name of the food along with the allergies the user has are uploaded alongside a prompt to the ChatGPT API explaining that it should give short recommendations on whether the user should consume the food. It also has a true or false flag in the text response which is removed from the final output which will determine if it is necessary to put audio through the glasses.

As there is no API or interface to connect to the glasses we downloaded a program that let us add a virtual microphone to the laptop which lets us have the laptop audio be sent through WhatsApp. This means that when Python runs the text to speech the audio is sent through Whatsapp and then played through the glasses.

Challenges we ran into

Challenge 1 - Figuring out the best way to stream the glasses to a laptop so the object detection algorithm could work. Can’t screen share an instagram livestream that uses the glasses. Watching Instagram live stream from different account has 20 second delay. Challenge 2 - The virtual microphone was not sending audio to WhatsApp. Challenge 3 - We first tried using a default Yolo food database, which we found wasn’t accurate enough so we had to make our own. Challenge 4 - classes in the custom model were wrong as in foods were being detected with the wrong label Challenge 5 - the window used to display the video feed and the chat gpt response were stretched.

Accomplishments that we're proud of

We succeeded in our goal of creating a tool that can accurately detect foods in real time and warn users of potential allergens. We successfully created our own database of custom foods. We managed to solve the vast majority of the issues we encountered, including the issue with the classes and the issues we had trying to stream the glasses feed to a laptop.

What we learned

We learned how to best work with the video feed from the meta glasses and how to add a virtual microphone. We also learned how to build the detection model ourselves when our first method did not work.

What's next for Allergy Vision

Since we trained the object detection model on foods we had available, the model is fundamentally limited in its scope for detecting a wide range of foods. In the future, the model could be trained to detect a much wider variety of foods. Also, another improvement could be to somehow make the product self contained within the Meta Glasses and not require a laptop to stream to at all.