3DREADME

Inspiration

After reading MLH's category for Most Creative Use of GitHub, I dug into the GitHub API and knew that I wanted to challenge myself to create something fun and completely unique to the platform. I became infatuated with GitHub's capability to display STL models in the README of a repository and knew that I wanted to find a way to make the repository fun to interact with. I fell in love with the idea of user-generated 3D models, and am proud to have created a project that allows a user to interact with GitHub in a way I'm confident they never have before!

What it does

3DREADME is a text-to-model generative AI tool built entirely within GitHub! 3DREADME will take any prompt and create a 3D model for the described object using point-cloud generation and ASCII STL conversion. User model prompting and display is all driven through GitHub, taking advantage of GitHub's powerful API to create a unique and interactive repository!

How we built it

3DREADME is powered by a unique backend that not only utilizes Google Colaboratory's GPU capabilities to drive the text-to-point-cloud, but wraps the model in an Ngrok and Flask server that not only hooks into the GitHub repository, but uses the GitHub API to make replies and update the README.

Challenges we ran into

By far the largest challenge in 3DREADME was leveraging the text-to-point-cloud and discovering how to convert it into an ASCII STL for GitHub README readability. The solution to this problem became a lot more mathematically involved than I had initially thought it might be. The OpenAI model that produces the point-cloud outputs thousands of single vertices with no vectors. The first step in solving the problem was assigning vectors to each of the vertices, which I was able to accomplish by estimating their values by proximity.

After obtaining approximate vectors for each of the vertices and after lots of experimentation, I discovered that using the Ball Pivoting Algorithm for model face generation was the best algorithm for this use-case, producing a water-tight model in each model generation that closely matched the visual solidity of the original point-cloud.

After going through these initially daunting tasks, the 3D model then goes through triangle decimation to reduce the polygon count and fit within the GitHub README size limit, then ASCII conversion for the repository to parse it within the codeblock.

Accomplishments that we're proud of

I'm really excited to have accomplished a lot of firsts with this project, especially as a solo hacker!

First time working with the GitHub API and Webhooks
First time hosting an OpenAI Model
First time working with point-clouds and STLs
First time writing a backend with Ngrok and Flask

I'm incredibly proud to have seen this idea go from conception to reality, and I've had a ton of fun playing around with the finished product -- I hope you do too!

What we learned

I learned that GitHub is a lot more powerful than I originally thought. The API and Webhook made it very easy to see the ideas I had for the project come to fruition. I also learned that Ngrok is a very valuable tool for deploying a reliable backend in a relatively short amount of time, especially within a Google Colab instance, and I look forward to using it at HackGT XI!