WebScore

WebScore Logo
Demonstration of Tutorial

Inspiration

Anybody who is just getting into playing music knows it can be intimidating. Sheet music just isn't something we're born understanding, neither is music theory. The only truly intuitive way to learn is by playing, and by watching others play. We figured that if there was a tool that could replicate the environment of watching another person play piano, we could make learning piano accessible, too.

What it does

WebScore is an application that takes any song in the world and automatically converts it into a playable piano visualization (like Synthesia or Falling-Notes). First, it downloads the track and converts it to an MP3, then transforms the audio into a MIDI file, which preserves the song’s tempo, rhythm, and note structure. From there, WebScore generates an animated piano-roll visualization, letting you see and play the notes as they are played- no sheet music or theory required.

How we built it

We built the project in two main functions: a user interface server and an API server. The user interface server is what the user actually interacts with, enters the URL into, and views the falling note display. The user interface relies on the API server to provide the URL for audio-to-MIDI to MIDI conversion. The API has three endpoints: the first will take a URL and use yt-dlp to retrieve the audio from YouTube. It's going through a proxy to avoid being flagged as a robot, however. The second endpoint takes in an audio file and returns a MIDI file. It does this by running it through an open-source ML model created by Spotify to turn audio into MIDI. In case you don't know what MIDI is, it's a standard protocol for musical data. The third endpoint just combines the functionality of the first two for use in the UI. We deployed everything (except the proxy) on DigitalOcean, on virtual machines running Ubuntu. We used Bash, Python, TypeScript, React, and Vite for programming.

Challenges we ran into

One of the biggest issues came with downloading music from YouTube. Because we deployed our project onto DigitalOcean's cloud environment, our requests to YouTube were flagged as being from a bot (which they sort of are). The way we got around this was by deploying a SOCKS server on a server outside of this cloud environment- A.K.A., one of our computers on our home network, and proxied requests to YouTube through that machine. YouTube would then interpret our requests as regular traffic because it was coming from a residential IP. It was also a challenge to align the falling note display with the audio it produced. We needed to make sure the notes would hit the visual keyboard at the same time they would ring out.

Accomplishments that we're proud of

We're proud of overcoming that request issue, and that we deployed basically everything on virtual machines in the cloud. You can actually visit webscore.graysmal.com and interact with the project. Same with the api through api.webscore.graysmal.com. It also took a lot of thought to connect the components, and we overcame hiccups along every part of the journey.

What we learned

The greatest part of this project was that everyone in the group put their hands on something new. We learned about system design and how to make a project that's more than just one monolithic file. We learned about APIs, networking, and different communication protocols. We also learned how to break a big problem into several smaller, more manageable problems that we can delegate. We also learned about deploying systems in a cloud environment.