Pupil

Inspiration

It’s 2018, and access to the internet is crucial for research, logistics management, online commerce, data storage, etc. However, the web is still only partially accessible to many users with vision difficulties. Screen readers partially mitigate the challenge of using the web by converting web page content into actionable elements that are read aloud for users. However, the functionality of screen readers (e.g. Google ChromeVox) is limited to only narrating actions and reading webpage text, relying on developers to add alternate descriptions to images and other non-textual elements, leaving users with vision impairments unable to view embedded images.

What it does

Our Hack@Brown group aimed to solve a web accessibility problem with Pupil, a chrome extension that utilizes visual recognition to automatically caption web images with alternate descriptions in real-time. These descriptions are injected in the source code of the webpage to be read off to the user by whatever screen reader they already have.

How we built it

We built the extension with JavaScript, HTML, CSS, JQuery, and the Microsoft Vision API. We also utilized chrome’s cloud browser storage feature to save the status (on/off) of the plugin between sessions.

Challenges we ran into

Challenges for this project included learning the chrome extension method of message passing between a script we run on the client and our extension itself, using browser storage to save whether a user has turned the extension on/off between sessions, and adhering to Microsoft Computer Vision API’s image restrictions.

Accomplishments we are proud of

We are extremely happy to have been able to complete this project in less than a day. We are also proud that this chrome extension has a great social impact of helping those who are visually impaired have a better, more accessible web experience.

What we learned

Web Development (HTML, CSS, jQuery, JavaScript), Microsoft Cognitive Services API, Chrome Extension Development, Teamwork, Healthy hacking

What's next for Pupil

Future features would be to upgrade the current tag building system. Having a home-grown image captioner would allow for more images to be captioned than Microsoft’s API allows for, and it would allow the system to work offline. This improvement could potentially lead to pupil being an tool that can improve accessibility across platforms, not just the chrome web browser.

Built With

html
javascript
jquery
json
microsoft-cognitive-services-api

Submitted to

Hack@Brown 2018
- Winner pMD - Best User Experience

Created by

I worked on sending the ajax requests for Microsoft Vision API, html parsing with jquery, limiting API requests, and handling input data issues.

Sathya Anisetti
I helped code the API call, page crawling, the Manifest JSON, and helped debug/test by using example websites.

Pranav1234567
Designed motion graphic animations for looping presentation, established brand identity with logo design and visual design language. Contributed to accessibility research and UI/UX development.

samrochelle
I worked on message passing between the the page crawler and the background script. Essentially, before adding captions to images, the crawler checks with the background script whether the extension is turned on.

Sean Rayment