Ai-Factor

Inspiration

The X-Factor show with Simon Cowell and more.

What it does

AI-Factor lets you stream yourself through Theta EdgeCloud live streaming. AI judges will be ready with commentary throughout, each with their own personality.

How we built it

During the research phase, we discovered interesting technologies like object detection, stem splitting, beat tracking. Not all of them could be implemented by deadline. To demonstrate the possibilities, we went with object detection on a per frame basis

Challenges we ran into

Video streaming technology is super complicated. Theta EdgeCloud makes it easy to stream within minutes through available apps like OBS or FFMPEG. But to inject our own special-sauce, we had to delve deeply into very complicated protocols like RTC, RTMP/RTSP. Running out of Theta EdgeCloud quotas, ingestors and edge resouces, which made it challenging to fully test. We picked AI models already available on Theta EdgeCloud to continue developing even when it was not possible to complete model deployment. We could not finish integrating all components and finish livestreaming testing as access to ingestors were limited.

Accomplishments that we're proud of

It consist of many moving parts and we are proud of making eachwork together.

What we learned

Sooo much! Its a huge world of very impressive technologies ready for a taking. Like https://www.lalal.ai/ for Stem-splitting, try it and be impressed.

What's next for Ai-Factor

Optimize, optimize, more features and better recognition of objects. Audio stem-spliting to separate instruments and track beats and feed into AI for feedback when you are off-beat. Realtime audio processing

Built With

• .net
• azure
• c#
• docker
• ffmpeg
• llama
• react