Inspiration
As high schoolers, music plays a large role in our mental health and maintaining healthy relationships with our peers. It’s a medium for us to share our interests and bond through concerts, shared earbuds, and midnight jam outs. However, we noticed how difficult it was for people who listen to one kind of song to share their interest with others. People tend to form their own listening circles, sometimes leaving others out. Our goal was to create a platform that bridged listening circles through expanding the genres the user listens to by slowly introducing them to new genres. Introducing Tunesphere.
What it does
Tunesphere leverages a Natural Language Processing (NLP) and Computer Vision (CV) based workflow to recommend songs that are both from diverse genres, but still similar to the user’s liked songs. The Spotify API allows us to gauge a sense of the users current listening habits, and create a listening plan that introduces them to different genres. Similar songs are predicted using our NLP model based on 3 criteria: tune, rhythm, and lyrics. Lyrics are further evaluated by sentiment. For example, if the user is a Pop junkie, and mainly listens to happy songs, the models will recommend happy songs from other genres, like R&B. The CV aspect involves analyzing the user’s reaction to the new songs we introduce to them to further enhance the model. When the user is seen as happy, we recommend more similar songs, and if the user is not happy we generate a new set of songs.
How we built it
Tunesphere is a web application coded in ReactJS. Our models include a Bidirectional Encoder Representations from Transformers (BERT) NLP model to obtain sentiment from lyrics of songs that we obtain from the Spotify API. We use this to recommend songs with similar sentiment. Our computer vision model is a VGG19 convolutional neural which analyzes sentiment of the user which allows us to determine which of our recommendations the user liked. We coded a backend using flask to run our NLP model and are hosting it on railway. We prepartitioned the spotify song lyrics in order to speed up our recommendation processes.
Challenges we ran into
Incorporating the machine learning models into the React application were at the forefront of our problems. At first, we got errors saying that the model could not be loaded because of a NO-CORS security protocol error. Then, we transitioned our database URL containing the model files to make sure that security protocol would not be needed. Another error with implementing the model with React was that the model could not read in the JSON file due to the wrong path.
Accomplishments that we're proud of
Some accomplishments that we are proud of are implementing highly-accurate sentimental analysis by analyzing the lyrics of different songs using BERT. Additionally, we were able to prioritize user privacy by not storing any information, like user data, including the user’s video, and all this information is deleted even from the local machine every time the user’s image frames is passed through a model and a new song is returned by the Spotify API.
What we learned
This project was a major undertaking for our entire group. This was the first time we ran a BERT NLP model, and we were excited that we could utilize BERT for song sentiment just from the lyrics. Additionally, we learned that to incorporate a Tensorflow model into a React.js application that we would have to host the model in an online server.
What's next for Tunesphere
Research has shown that performing sentiment analysis on video is extremely difficult, so we aim to explore deeper neural networks to better predict emotion from a user’s video.
Furthermore, in case a user would not like their video to be used, we will have them take a short quiz upon launching the application and based on that quiz and their history of liked songs we will recommend songs from different genres that they are most likely to like. We would also look on expanding our model to work on music on different languages and build a social network where you can share with your friends the music you listen to.
Log in or sign up for Devpost to join the conversation.