MomentScape

banner

Inspiration:

As international students, being away from home means missing out on the places, people, and moments that shaped us. A photo or a video call can help, but they often leave us longing for something more—a way to "feel" like we're back home, standing in familiar places, surrounded by memories. We dreamed of a way to bridge that distance, to step into a moment we miss and experience it beyond a screen. MomentScape was born from this need: to turn cherished memories into immersive experiences we can explore, so no matter where we are, home feels just a little bit closer.

What it does

Image Upload: Users begin by selecting and uploading an image from their device, capturing a memory they want to experience more deeply. Feature Extraction: Once uploaded, the image is processed by an intelligent model that identifies important features of the scene, like walls, floors, and other textures, creating a structured description of the environment. 3D Model Creation: Using advanced 3D modeling tools, MomentScape builds a basic 3D structure based on the image, establishing the layout and key elements of the scene. Texture Generation: The environment details are then enhanced with high-quality textures that replicate the look and feel of the original image, adding realism to walls, floors, and other surfaces. Environment Assembly: All elements are brought together into an interactive 3D space that users can explore on either a VR headset or a standard screen, allowing them to move freely and immerse themselves in the memory.

How we built it

Frontend & Backend Communication: We created a user-friendly interface for image uploads, which sends images from the frontend to the backend via Flask. Flask acts as the bridge, allowing us to process the image with different models and assemble all components into the final 3D environment. Feature Extraction with ChatGPT: When an image is uploaded, it’s first analyzed by a ChatGPT image detection model. This model extracts key features—like descriptions of walls, floors, and ceilings—which are stored in a JSON prompt. This step helps define the unique textures and environment details for each memory. 3D Model Generation with InstantMesh-3D: The image is simultaneously processed by InstantMesh-3D, which generates a basic 3D mesh of the space in GLTF format. This mesh serves as the structural foundation for the environment. Texture Creation Using a Diffusion Model: The JSON prompt (containing details from ChatGPT’s analysis) is passed to a texture diffusion model, which generates high-quality textures for walls, ceilings, and floors. These textures, along with the 3D model, are saved locally to be used in the final scene. Environment Assembly with A-Frame: Finally, we use A-Frame to combine the textures and 3D model, creating a complete, immersive environment. This allows users to explore the space freely in VR or on a desktop, making the memory truly come to life. flowchart

Challenges we ran into

Building MomentScape came with a fair share of technical hurdles. One significant challenge was managing 3D model file conversions, as we needed a seamless way to generate and store these models in GLTF format while preserving quality and compatibility. Real-time generation of immersive environments proved demanding as well; balancing the need for detail with fast processing required optimization across multiple tools. Additionally, ensuring compatibility of files between different components—especially as we sent data back and forth between frontend, backend, and VR systems—added another layer of complexity. Despite these challenges, we remained committed to creating an experience that would allow users to explore their memories in an authentic and immersive way.

Accomplishments that we're proud of

Creating a Unique Experience: MomentScape brings memories to life in a way no existing application does, allowing users to step into their photos and explore them in 3D. Building a Full-Stack Application in 24 Hours: We successfully developed a complex, end-to-end application, from front-end to back-end, all within the intense time constraints of a 24-hour hackathon. Effective Team Collaboration: We combined our individual strengths, problem-solving skills, and creativity to bring this project to life in a fast-paced environment. Innovative 3D Model Creation from Images: From extracting key features in images to constructing a full 3D model, we tackled challenging technology integration to make each memory feel immersive. Advanced Texture Mapping: We developed a way to apply layered 2D textures on 3D models, achieving realistic environments that enhance the user experience. Virtual Reality Exploration with A-Frame: We’re proud to offer users the option to explore their memories in VR, adding a unique, immersive touch to the experience.

What we learned

3D Model Generation: For many of us, this was our first experience working with 3D models, from generating initial structures to refining them with detailed textures. We learned how to convert 2D images into 3D environments and tackle the challenges of realism in virtual spaces. Using Flask for Full-Stack Development: Building the backend with Flask taught us the importance of seamless data flow between front-end and back-end, especially when handling complex processes like feature extraction and model generation. Diffusion Models for Textures: Working with diffusion models for texture generation expanded our understanding of AI-based texture synthesis, showing us how machine learning can create realistic, adaptive elements for 3D environments. WebXR and A-Frame for VR Integration: Integrating WebXR and A-Frame to make the environments VR-compatible was a new challenge, but it helped us understand the nuances of virtual reality development and how to create immersive user experiences. Adapting Quickly to New Tools: Through this project, we learned to rapidly pick up new technologies and integrate them into a cohesive product under tight deadlines.

What's next for MomentScape

Enhanced Texture and 3D Model Generation: We plan to integrate more advanced texture generation and 3D modeling techniques, improving the realism and accuracy of recreated environments for an even richer user experience. Support for Outdoor Environments: Currently, MomentScape focuses primarily on indoor scenes. Our next goal is to support outdoor images, enabling users to explore memories in natural landscapes and larger open spaces. Expanded Platform Compatibility: We aim to optimize MomentScape for different devices, making it accessible not just on VR headsets and desktops, but also mobile devices for more flexible exploration. User Customization Options: Allowing users to adjust elements within the memory environment, such as lighting or textures, to personalize the experience even further.