Controllable Text-to-3D Generation via Surface-Aligned Gaussian
  Splatting
While text-to-3D and image-to-3D generation tasks have received considerable
attention, one important but under-explored field between them is controllable
text-to-3D generation, which we mainly focus on in this work. To address this
task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network
architecture designed to enhance existing pre-trained multi-view diffusion
models by integrating additional input conditions, such as edge, depth, normal,
and scribble maps. Our innovation lies in the introduction of a conditioning
module that controls the base diffusion model using both local and global
embeddings, which are computed from the input condition images and camera
poses. Once trained, MVControl is able to offer 3D diffusion guidance for
optimization-based 3D generation. And, 2) we propose an efficient multi-stage
3D generation pipeline that leverages the benefits of recent large
reconstruction models and score distillation algorithm. Building upon our
MVControl architecture, we employ a unique hybrid diffusion guidance method to
direct the optimization process. In pursuit of efficiency, we adopt 3D
Gaussians as our representation instead of the commonly used implicit
representations. We also pioneer the use of SuGaR, a hybrid representation that
binds Gaussians to mesh triangle faces. This approach alleviates the issue of
poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained
geometry on the mesh. Extensive experiments demonstrate that our method
achieves robust generalization and enables the controllable generation of
high-quality 3D content.
65f79df901e0f4a95d662065
Zhiqi Li
admin_assigned
2024-03-18T09:16:41.827Z
6353b63b06d707b3324279e3
/avatars/d5f2de814a5f7570ad1710b28c22cf88.svg
Zhiqi Li
user
lzq49
65f79df901e0f4a95d662066
Yiming Chen
65f79df901e0f4a95d662067
Lingzhe Zhao
65f79df901e0f4a95d662068
Peidong Liu
Controllable Text-to-3D Generation via Surface-Aligned Gaussian
  Splatting
While text-to-3D and image-to-3D generation tasks have received considerable
attention, one important but under-explored field between them is controllable
text-to-3D generation, which we mainly focus on in this work. To address this
task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network
architecture designed to enhance existing pre-trained multi-view diffusion
models by integrating additional input conditions, such as edge, depth, normal,
and scribble maps. Our innovation lies in the introduction of a conditioning
module that controls the base diffusion model using both local and global
embeddings, which are computed from the input condition images and camera
poses. Once trained, MVControl is able to offer 3D diffusion guidance for
optimization-based 3D generation. And, 2) we propose an efficient multi-stage
3D generation pipeline that leverages the benefits of recent large
reconstruction models and score distillation algorithm. Building upon our
MVControl architecture, we employ a unique hybrid diffusion guidance method to
direct the optimization process. In pursuit of efficiency, we adopt 3D
Gaussians as our representation instead of the commonly used implicit
representations. We also pioneer the use of SuGaR, a hybrid representation that
binds Gaussians to mesh triangle faces. This approach alleviates the issue of
poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained
geometry on the mesh. Extensive experiments demonstrate that our method
achieves robust generalization and enables the controllable generation of
high-quality 3D content.
A novel architecture called Multi-view ControlNet and an efficient pipeline are introduced to achieve controllable text-to-3D generation by enhancing multi-view diffusion models with additional condition inputs and using hybrid diffusion guidance with 3D Gaussians and SuGaR representation.
Multi-view ControlNet diffusion models conditioning module local embeddings global embeddings camera poses optimization-based 3D generation multi-stage 3D generation pipeline score distillation algorithm hybrid diffusion guidance 3D Gaussians implicit representations SuGaR geometry representation mesh triangle faces