Worldly

Worldly takes in any footage and converts it into fully explorable 3D VR Worlds in a Meta Quest Headset.

Inspiration

All of us are super passionate about the recent advancements in World Models and Machine Learning Research. When World Labs released their world foundation model API (Prompt → 3D world, Released ~2 weeks ago: worldlabs.ai/blog/announcing-the-world-api), we immediately saw the potential for TreeHacks, but we wanted to take it further. What if we could make this tech accessible to anyone with a drone, a VR headset, or even just a phone?

We thought back to all our favorite memories and wished we could revisit that place exactly how it was in the videos we took on our phones. Thinking about literally stepping into those moments inspired us to build Worldly. The capabilities are what pushed us to go further: training simulations in reconstructed real-world environments, preserving historical sites before they change, enabling remote collaboration in actual physical spaces, or simply going back your favorite vacation spot whenever you want.

That is Worldly.

What it does

Worldly lets you revisit any location you've ever been to in fully immersive VR.

Here's how it works:

Capture with your drone → Using our custom iOS app, you fly your DJI drone anywhere and record footage directly from the controller's live feed. Our app captures your footage and optimizes it for our World Model.
Automated Processing → The second you stop recording, your footage automatically uploads to our backend server and enters our processing pipeline.
AI-powered 3D reconstruction → We feed your footage into the World Labs API, which uses their world foundation model to reconstruct the entire environment as a photorealistic 3D Gaussian splat.
Instant VR access → The generated world automatically appears in your Worldly dashboard and Meta Quest library. You can click on any world, put on your headset, and you're standing inside that location and can explore it in all directions.

It also works with phones for smaller indoor spaces. You can record with your camera, and the same pipeline transforms it into an explorable VR world. We've already captured four locations across Stanford: outside the NVIDIA building, the church, inside the TreeHacks venue, and Hoover Tower.

How we built it

DJI Drone Integration (iOS/Swift + ReplayKit) → We built a native iOS app that captures live drone footage directly from the DJI controller app using ReplayKit. The app crops the center viewport for the best angle, cleans it, and automatically uploads it to our server.

Backend Server/Connections (Node.js, Flask): Our Node.js backend receives uploaded footage and orchestrates the entire 3D world pipeline. It manages video preprocessing, API calls, file conversion, and world deployment automatically.

World Model API (Python): We integrated the newly released World Labs Marble world-model API, which uses AI to reconstruct real-world geometry from video. It analyzes camera movement, depth, and scene structure to generate a photorealistic 3D Gaussian splat in SPZ format. We input a custom prompt along with the drone footage to create a hyperrealistic world.

WebXR Viewer in Meta Quest (three.js, Vite, Meta Immersive Kit, TypeScript): We built a custom web-based renderer for Meta Quest that loads PLY Gaussian splat files directly, renders them interactively in real-time, and enables full spatial navigation inside the Meta Quest

World Dashboard/Web app (C#, HTML, JSON): We built a web dashboard that tracks all your captured worlds, shows processing status, and lets you launch any world directly into VR with one click. It's your personal library of places you've been.

Challenges we ran into

WorldLabs API → Since the API is so new, we faced a lot of errors with I/O. We had to be very specific with the types of footage we input into the API, as well as the output format. The file format output was not compatible with the headset, so we had to integrate conversion into our custom world map viewer. We also had to carefully orchestrate our backend to queue requests efficiently and handle processing delays. Some reconstructions took 10-15 minutes, so we built a status tracking system to keep us informed.
DJI SDK → Only worked on Android, but we needed iOS compatibility. We pivoted to controlling the drone manually while using ReplayKit to automatically screen-record the footage from the DJI controller app. This workaround gave us clean video capture without requiring SDK access.
VR Issues →The PLY formatted output was super niche and lacked proper WebXR support, so we had to build our own custom WebXR viewer from scratch.

Accomplishments that we're proud of

We are super proud of being one of the first devs to build on top of the World Labs API. We see this technology going so far and are super excited to have worked with it. We are also proud that we are about to turn our drone footage into 3D worlds. We weren't sure it would work or generate anything viewable, and it was such a reassuring moment when we saw what we created.

What we learned

We learned a lot. For all of us, this was one of our first times working with Unity and AR/VR, so it was a really fun and exciting experience to build on top of Meta Quest.

What's next for Worldly

The applications for Worldly are endless, as World Models improves, so will Worldly:

Multi-user worlds →Let multiple people explore the same captured space together in VR
Social features → Share your worlds publicly, explore places others have captured around the world
More VR support →More intuitive Menu/Exploration, expand beyond Meta Quest to Apple Vision Pro, smartphone-based VR headsets
Higher resolution captures → Use better models, optimize for 4K+ drone footage to increase splat quality

Vision/Real World Use Cases:

Real-time collaboration → Capture a space and have your remote team members "walk through" it instantly
Families → Create "albums" of real-world memories for families everywhere
Historical preservation → Partner with museums and historical sites to create permanent VR archives of important locations
Industrials → Property tours, construction site monitoring, spatial planning
Training → Military, emergency response, search/rescue
Tourism → Preview destinations before booking, revisit your favorite vacation spots