VibeCAD

VibeCAD: Design for Everyone
Hand gesture manipulation of 3D models
Sketch to CAD
Edit CAD using voice

Inspiration

The barrier of entry for designing and modeling solutions is too high. From ideation to building a proof of concept, users face so many hurdles that it becomes impossible to pursue their idea. We want to break the learning curve associated with 3d design and allow anyone who has an idea to become a designer, regardless of experience and skill.

What it does

VibeCAD is CAD for the everyday user. It takes your rough sketches and transforms them into 3D CAD models using AI. Users can move around in 3D space and manipulate objects using intuitive hand gestures. Just like using Cursor, you can let AI make edits to the CAD model using your voice.

Being able to load multiple models into a 3d space and manipulate them via physical gestures enables users to rapidly prototype, experiment, and explore to bring their idea to life.

How we built it

We used Gemini API to turn sketch photos into SCAD, a CAD-as-code language. Then, we exported the SCAD code as STL, then rendered it on the browser with ThreeJS. We used Google MediaPipe for hand landmark detection on the browser, then fiddled with the landmarks to identify specific command gestures. We used Chrome's Web Speech API for speech transcription.

Challenges we ran into

We spent a lot of time researching which libraries we could make use of to make sure our imagination was feasible within the hackathon (thankfully it was). Hand gesture detection often got confused between similar gestures.

Accomplishments that we're proud of

Using three.js to support object files not natively supported Training MediaPipe to recognize and update data correctly with gestures Creating a robust pipeline to turn sketches into models using SCAD

This was a technologically challenging project to pull off, and we are proud to have created an end-to-end working demo deployed to the public!

What's next for VibeCAD

More robust CAD generation using ML. Implementing collaborative sessions where users can design together. Improving workspace and model manipulation to make it more intuitive Allowing multi-model editing and exports