Advancements in 3D Generative Modeling and Reconstruction

The field of 3D generative modeling and reconstruction is witnessing significant advancements, particularly in the integration of diffusion models and novel representations for 3D data. A common theme across recent research is the exploration of methods to directly generate or reconstruct 3D objects and textures from 2D inputs, leveraging the strengths of pretrained 2D models while addressing the unique challenges of 3D data. Innovations include the combination of heat and denoising diffusion for 3D mesh generation, the use of Gaussian splats for efficient 3D representation, and the development of metrics for evaluating multi-view consistency in generated images. These approaches aim to improve the fidelity, consistency, and computational efficiency of 3D generative models, making them more applicable to real-world tasks such as 3D asset generation and single-image 3D reconstruction.

Noteworthy papers include:

  • DoubleDiffusion: Introduces a novel framework combining heat and denoising diffusion for generative learning on 3D mesh surfaces, enabling geometry-aware signal diffusion.
  • Chirpy3D: Advances fine-grained 3D generation by enabling the creation of novel, plausible parts through continuous part latents and multi-view diffusion.
  • SPAR3D: Proposes a two-stage approach for single-image 3D object reconstruction, combining the efficiency of regression methods with the probabilistic modeling of generative methods.
  • Zero-1-to-G: Enables direct 3D generation from single-view images using pretrained 2D diffusion models, leveraging Gaussian splats for 3D representation.
  • Consistent Flow Distillation: Improves text-to-3D generation by ensuring multi-view consistency in the generation process, addressing limitations of previous distillation methods.
  • MEt3R: Introduces a metric for evaluating multi-view consistency in generated images, independent of the sampling procedure.
  • F3D-Gaus: Tackles generalizable 3D-aware generation from monocular datasets, improving the quality and consistency of 3D renderings.
  • SuperNeRF-GAN: Offers a universal framework for 3D-consistent super-resolution, enhancing the resolution of generated images while preserving 3D consistency.
  • C2PD: Proposes a novel approach for guided depth super-resolution, focusing on the continuity inherent in depth maps.
  • CaPa: Introduces a carve-and-paint framework for efficient 4K textured mesh generation, decoupling geometry generation from texture synthesis.

Sources

DoubleDiffusion: Combining Heat Diffusion with Denoising Diffusion for Generative Learning on 3D Meshes

Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation

SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images

Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation

Consistent Flow Distillation for Text-to-3D Generation

MEt3R: Measuring Multi-View Consistency in Generated Images

F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting

SuperNeRF-GAN: A Universal 3D-Consistent Super-Resolution Framework for Efficient and Enhanced 3D-Aware Image Synthesis

C2PD: Continuity-Constrained Pixelwise Deformation for Guided Depth Super-Resolution

CaPa: Carve-n-Paint Synthesis for Efficient 4K Textured Mesh Generation

Built with on top of