A.I. | Autism Inspire

Assisting visual communication for early autism treatment by deep sketch captioning

Comment 1

Inspiration

In 2018, about 1 in 59 children is diagnosed with autism. Treatment are expensive and over-booked, parents couldn’t afford more accompany time.

Visual communication is more effective than text for intellectual and social development of children with autism.

What it does

With our "sketch captioning" technology, they can join conversation with AI via drawing! Demo Site

How I built it

  • We built a CNN(Convolutional Neural Network) model served on SPELL.
  • We built a webpage with webflow for interaction including (user) drawing sketches and making conversation.

  • We used GCP(Google Cloud Platform) service to synthesis text-to-speech read aloud to users.

  • We interviewed with multiple autism-related institutions and physicians in China to understand the pain points of early autism treatment.

  • We validated our ideas on a medical basis under professional guidance

Challenges I ran into

  • We co-developed with SPELL team to overcome difficulties in this newly-released product.
  • We maintained interactiveness of our web app while keep it good-looking with webflow.
  • We encountered two new technologies (SPELL, webflow) and learned to leverage them efficiently.

Accomplishments that I'm proud of

  • We played as a team.
  • We were challenged. We once hesitated. We kept staggering. Yet we stuck to our purpose and made it.

What I learned

  • We learned new technologies!
  • We learned to cooperate in intense working environment!
  • We learned to forsake minor features for the major purpose.

What's next for A.I.

  • We will extend from sketch classification (as shown in demo) to sketch captioning, allowing more expressive conversations catered to our users’ special needs.

Built With

Share this project:

Updates