GenForge

• 

  original scene made in blender

• 

  night scene by genAI

• 

  NerF model after training

• 

  Pipeline in ComfyUI

• 

  input scene for NYC

• 

  snow in NYC made by genAI

• 

  another input to the model

• 

  which proves genAI works in funny ways!

Inspiration

We wanted to make a tool which could visualise a car in sunny Bay Area in San Francisco to be in snowy condition in NYC at night using GenAI algorithms. We wanted to explore relighting with GenAI on 3D models.

What it does

Generate an output image with atmospheric changes by using generative AI on sample images and a user prompt.

How we built it

We started building 3D models in Blender, followed by trying to explore the concept of reconstruction by NerF (neural radiance fields) but after training 40 multi-view images of the scene from Blender, the generated NerF was very blurry and noisy. Thus, we switched to using our input image for ComfyUI, dynamically generating different backgrounds/lighting conditions/extra elements like starry skies through GenAI nodes and samplers. We used a pre-trained deliberateCyber_v50 model. Finally, we incorporated all of these features into ComfyUI, so that the user can just pick a theme by providing prompts in the tex field and get the perfect image by adjusting parameters like "noise" on their own.

Challenges we ran into

We initially planned to integrate NerF with any GenAI model, finally producing an altered image, but faced difficulties understanding particular details about the format of inputs going through NerF (tried NerFStudio and instant-ngp from Nvidia) and ComfyUI (whether we will be able to view an intractable 3D model at ComfyUI as it supports 2D images). NerF Studio also took a long time to train on our dataset and produced a very colourful yet noisy image which could not be a viable input to ComfyUI. This motivated us to cut out NerF from our pipeline entirely due to time constraints.

Accomplishments that we're proud of

We have explored Blender, NerF and ComfyUI and it was a lot of fun designing and brainstorming despite the challenges we faced.

What we learned

Learnt how to design an entire pipeline and mitigate challenges we faced and make progress through perseverance. We also noticed how sometimes the design pipeline needs more time than we anticipate.

What's next for GenForge

Make it work for 3D models. Add changes in seasons as one of the generated scenes that affect object looks and lighting (like orange trees in the fall), while keeping the objects in scene as intact as possible

Built With

• blender
• comfyui
• python