Ukulele Hero

Inspiration

Nick brought his Ukulele to TreeHacks, because he heard everyone at Stanford can play Ukulele. Raphael is sad because he cannot play the Ukulele. Raphael and Nick decide to make Ukulele Hero so that Raphael is less sad.

What it does

We stream the phone microphone's input to the cloud, where it is plugged into a machine learning model that recognizes what notes compose the chord you're playing, a problem that is at the limits of what's possible today, one that we weren't able to find any convenient libraries or even appealing-looking research on.

The model makes a prediction for what was played and sends it back to the phone, which uses it in Ukulele Hero to know whether you played the right notes.

How we built it

To create training data for the machine learning model, we recorded all the possible strummed notes on the Ukulele and randomly add segments of those note recordings together to get chords, from which we extract several FFT's as features. We trained our model on the Google Cloud Platform using these FFT's as our X and n-hot vectors as our Y.

Challenges we ran into

We spent a huge portion of Saturday trying to hack up a guitar tuner and some earpods (now airpods) to extract the sound the ukulele is making without any interference -- basically trying to wire the ukulele directly to our phone audio jack. It did not work out.

As I write, our machine learning model data is being regenerated after we found a bug in the generation. Hopefully it'll finish by expo-time!

Accomplishments that we're proud of

We brought a Ukulele across the country, cut up a bunch of things that would deem us crazy anywhere but a hackathon, and have a seriously non-trivial technical side to our project that we wrote ourselves.