title: StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
summary: In the realm of text-to-3D generation, utilizing 2D diffusion models through
  score distillation sampling (SDS) frequently leads to issues such as blurred
  appearances and multi-faced geometry, primarily due to the intrinsically noisy
  nature of the SDS loss. Our analysis identifies the core of these challenges as
  the interaction among noise levels in the 2D diffusion process, the
  architecture of the diffusion network, and the 3D model representation. To
  overcome these limitations, we present StableDreamer, a methodology
  incorporating three advances. First, inspired by InstructNeRF2NeRF, we
  formalize the equivalence of the SDS generative prior and a simple supervised
  L2 reconstruction loss. This finding provides a novel tool to debug SDS, which
  we use to show the impact of time-annealing noise levels on reducing
  multi-faced geometries. Second, our analysis shows that while image-space
  diffusion contributes to geometric precision, latent-space diffusion is crucial
  for vivid color rendition. Based on this observation, StableDreamer introduces
  a two-stage training strategy that effectively combines these aspects,
  resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D
  Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance
  the overall quality, reduce memory usage during training, and accelerate
  rendering speeds, and better capture semi-transparent objects. StableDreamer
  reduces multi-face geometries, generates fine details, and converges stably.
paper_authors._id: 656fdc8234440bdf2ed850d9
paper_authors.name: Pengsheng Guo
paper_authors._id: 656fdc8234440bdf2ed850da
paper_authors.name: Hans Hao
paper_authors._id: 656fdc8234440bdf2ed850db
paper_authors.name: Adam Caccavale
paper_authors._id: 656fdc8234440bdf2ed850dc
paper_authors.name: Zhongzheng Ren
paper_authors._id: 656fdc8234440bdf2ed850dd
paper_authors.name: Edward Zhang
paper_authors._id: 656fdc8234440bdf2ed850de
paper_authors.name: Qi Shan
paper_authors._id: 656fdc8234440bdf2ed850df
paper_authors.name: Aditya Sankar
paper_authors._id: 656fdc8234440bdf2ed850e0
paper_authors.name: Alexander G. Schwing
paper_authors._id: 656fdc8234440bdf2ed850e1
paper_authors.name: Alex Colburn
paper_authors._id: 656fdc8234440bdf2ed850e2
paper_authors.name: Fangchang Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-06T10:25:02.754Z
paper_authors._id: 656ea9c84b5b695353709f0d
paper_authors.avatarUrl: /avatars/0be91edfbfe9ab9a83325e2d844801f0.svg
paper_authors.fullname: Fangchang Ma
paper_authors.type: user
paper_authors.user: fangchangma
paper_title: StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
paper_summary: In the realm of text-to-3D generation, utilizing 2D diffusion models through
  score distillation sampling (SDS) frequently leads to issues such as blurred
  appearances and multi-faced geometry, primarily due to the intrinsically noisy
  nature of the SDS loss. Our analysis identifies the core of these challenges as
  the interaction among noise levels in the 2D diffusion process, the
  architecture of the diffusion network, and the 3D model representation. To
  overcome these limitations, we present StableDreamer, a methodology
  incorporating three advances. First, inspired by InstructNeRF2NeRF, we
  formalize the equivalence of the SDS generative prior and a simple supervised
  L2 reconstruction loss. This finding provides a novel tool to debug SDS, which
  we use to show the impact of time-annealing noise levels on reducing
  multi-faced geometries. Second, our analysis shows that while image-space
  diffusion contributes to geometric precision, latent-space diffusion is crucial
  for vivid color rendition. Based on this observation, StableDreamer introduces
  a two-stage training strategy that effectively combines these aspects,
  resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D
  Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance
  the overall quality, reduce memory usage during training, and accelerate
  rendering speeds, and better capture semi-transparent objects. StableDreamer
  reduces multi-face geometries, generates fine details, and converges stably.
paper_ai_keywords: 2D diffusion models score distillation sampling (SDS) noise levels diffusion network architecture 3D model representation StableDreamer InstructNeRF2NeRF supervised L2 reconstruction loss time-annealing image-space diffusion latent-space diffusion two-stage training strategy anisotropic 3D Gaussians Neural Radiance Fields (NeRFs) multi-faced geometries high-fidelity 3D models semi-transparent objects