title: Controllable Text-to-3D Generation via Surface-Aligned Gaussian
    Splatting
summary: While text-to-3D and image-to-3D generation tasks have received considerable
  attention, one important but under-explored field between them is controllable
  text-to-3D generation, which we mainly focus on in this work. To address this
  task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network
  architecture designed to enhance existing pre-trained multi-view diffusion
  models by integrating additional input conditions, such as edge, depth, normal,
  and scribble maps. Our innovation lies in the introduction of a conditioning
  module that controls the base diffusion model using both local and global
  embeddings, which are computed from the input condition images and camera
  poses. Once trained, MVControl is able to offer 3D diffusion guidance for
  optimization-based 3D generation. And, 2) we propose an efficient multi-stage
  3D generation pipeline that leverages the benefits of recent large
  reconstruction models and score distillation algorithm. Building upon our
  MVControl architecture, we employ a unique hybrid diffusion guidance method to
  direct the optimization process. In pursuit of efficiency, we adopt 3D
  Gaussians as our representation instead of the commonly used implicit
  representations. We also pioneer the use of SuGaR, a hybrid representation that
  binds Gaussians to mesh triangle faces. This approach alleviates the issue of
  poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained
  geometry on the mesh. Extensive experiments demonstrate that our method
  achieves robust generalization and enables the controllable generation of
  high-quality 3D content.
paper_authors._id: 65f79df901e0f4a95d662065
paper_authors.name: Zhiqi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:16:41.827Z
paper_authors._id: 6353b63b06d707b3324279e3
paper_authors.avatarUrl: /avatars/d5f2de814a5f7570ad1710b28c22cf88.svg
paper_authors.fullname: Zhiqi Li
paper_authors.type: user
paper_authors.user: lzq49
paper_authors._id: 65f79df901e0f4a95d662066
paper_authors.name: Yiming Chen
paper_authors._id: 65f79df901e0f4a95d662067
paper_authors.name: Lingzhe Zhao
paper_authors._id: 65f79df901e0f4a95d662068
paper_authors.name: Peidong Liu
paper_title: Controllable Text-to-3D Generation via Surface-Aligned Gaussian
    Splatting
paper_summary: While text-to-3D and image-to-3D generation tasks have received considerable
  attention, one important but under-explored field between them is controllable
  text-to-3D generation, which we mainly focus on in this work. To address this
  task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network
  architecture designed to enhance existing pre-trained multi-view diffusion
  models by integrating additional input conditions, such as edge, depth, normal,
  and scribble maps. Our innovation lies in the introduction of a conditioning
  module that controls the base diffusion model using both local and global
  embeddings, which are computed from the input condition images and camera
  poses. Once trained, MVControl is able to offer 3D diffusion guidance for
  optimization-based 3D generation. And, 2) we propose an efficient multi-stage
  3D generation pipeline that leverages the benefits of recent large
  reconstruction models and score distillation algorithm. Building upon our
  MVControl architecture, we employ a unique hybrid diffusion guidance method to
  direct the optimization process. In pursuit of efficiency, we adopt 3D
  Gaussians as our representation instead of the commonly used implicit
  representations. We also pioneer the use of SuGaR, a hybrid representation that
  binds Gaussians to mesh triangle faces. This approach alleviates the issue of
  poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained
  geometry on the mesh. Extensive experiments demonstrate that our method
  achieves robust generalization and enables the controllable generation of
  high-quality 3D content.
paper_ai_keywords: Multi-view ControlNet (MVControl) diffusion models conditioning module local and global embeddings optimization-based 3D generation score distillation algorithm hybrid diffusion guidance method 3D Gaussians implicit representations SuGaR mesh triangle faces geometry in 3D Gaussians