Image Captioner plugin

Inspiration

Some images on the internet do not have proper alt attributes, which excludes people that rely on assistive technologies, such as screen readers, to view images. We wanted to create software that generates alt attributes for images on web pages.

What it does

This software creates buttons next to images, that can be pressed using a screen reader. When one of these buttons is pressed, it adds an alt tag to its associated image to describe it using a machine-learning model.

How we built it

We created a browser extension that adds buttons beside images on web pages. When clicked, such a button makes a request to our Python server, which opens the image using the image link and generates a caption for it using a machine-learning model. The generated caption is then passed back to the extension, which adds it to the image as an alt attribute.

Challenges we ran into

Understanding how to use the machine learning model was a huge obstacle that was overcome.

Accomplishments that we're proud of

We made the thing work!!!!!!!

What we learned

We learned how to make an image to caption generator, how to make a Python server using Django, how to clean and isolate HTML data, and more.

What's next for the Image Captioner plugin

This caption generator does not work with all photos and could be more accurate. To accomplish this, we use a more sophisticated model for the caption generator.

Built With

• django
• javascript
• machine-learning-models
• python