title: ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with
    Variational Score Distillation
summary: Score distillation sampling (SDS) has shown great promise in text-to-3D
  generation by distilling pretrained large-scale text-to-image diffusion models,
  but suffers from over-saturation, over-smoothing, and low-diversity problems.
  In this work, we propose to model the 3D parameter as a random variable instead
  of a constant as in SDS and present variational score distillation (VSD), a
  principled particle-based variational framework to explain and address the
  aforementioned issues in text-to-3D generation. We show that SDS is a special
  case of VSD and leads to poor samples with both small and large CFG weights. In
  comparison, VSD works well with various CFG weights as ancestral sampling from
  diffusion models and simultaneously improves the diversity and sample quality
  with a common CFG weight (i.e., 7.5). We further present various improvements
  in the design space for text-to-3D such as distillation time schedule and
  density initialization, which are orthogonal to the distillation algorithm yet
  not well explored. Our overall approach, dubbed ProlificDreamer, can generate
  high rendering resolution (i.e., 512times512) and high-fidelity NeRF with
  rich structure and complex effects (e.g., smoke and drops). Further,
  initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and
  photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/
paper_authors._id: 64701c0a850a938d6c568531
paper_authors.name: Zhengyi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T13:58:00.955Z
paper_authors._id: 634e15aec1ce28f1de91c470
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
paper_authors.fullname: Zhengyi Wang
paper_authors.type: user
paper_authors.user: Zhengyi
paper_authors._id: 64701c0a850a938d6c568532
paper_authors.name: Cheng Lu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-26T03:39:57.857Z
paper_authors._id: 6308ab64e3246e2be8884f4f
paper_authors.avatarUrl: /avatars/7d8002caeacf27258996f31e4a2e186f.svg
paper_authors.fullname: LuChengTHU
paper_authors.type: user
paper_authors.user: LuChengTHU
paper_authors._id: 64701c0a850a938d6c568533
paper_authors.name: Yikai Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-26T02:42:51.456Z
paper_authors._id: 64206b35e40f66bcd1e3c985
paper_authors.avatarUrl: /avatars/a8f830d1c47702f642045399e59c9fe8.svg
paper_authors.fullname: Yikai Wang
paper_authors.type: user
paper_authors.user: yikaiw
paper_authors._id: 64701c0a850a938d6c568534
paper_authors.name: Fan Bao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T08:07:15.886Z
paper_authors._id: 633131798ef21f47308ce49b
paper_authors.avatarUrl: /avatars/1a3b5b032cbbfc1499ec96aba5260902.svg
paper_authors.fullname: fan bao
paper_authors.type: user
paper_authors.user: baofff
paper_authors._id: 64701c0a850a938d6c568535
paper_authors.name: Chongxuan Li
paper_authors._id: 64701c0a850a938d6c568536
paper_authors.name: Hang Su
paper_authors._id: 64701c0a850a938d6c568537
paper_authors.name: Jun Zhu
paper_title: ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with
    Variational Score Distillation
paper_summary: Score distillation sampling (SDS) has shown great promise in text-to-3D
  generation by distilling pretrained large-scale text-to-image diffusion models,
  but suffers from over-saturation, over-smoothing, and low-diversity problems.
  In this work, we propose to model the 3D parameter as a random variable instead
  of a constant as in SDS and present variational score distillation (VSD), a
  principled particle-based variational framework to explain and address the
  aforementioned issues in text-to-3D generation. We show that SDS is a special
  case of VSD and leads to poor samples with both small and large CFG weights. In
  comparison, VSD works well with various CFG weights as ancestral sampling from
  diffusion models and simultaneously improves the diversity and sample quality
  with a common CFG weight (i.e., 7.5). We further present various improvements
  in the design space for text-to-3D such as distillation time schedule and
  density initialization, which are orthogonal to the distillation algorithm yet
  not well explored. Our overall approach, dubbed ProlificDreamer, can generate
  high rendering resolution (i.e., 512times512) and high-fidelity NeRF with
  rich structure and complex effects (e.g., smoke and drops). Further,
  initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and
  photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/
paper_ai_summary: Variational score distillation addresses issues in text-to-3D generation by modeling 3D parameters as random variables, improving sample quality and diversity.
paper_ai_keywords: score distillation sampling variational score distillation particle-based variational framework ancestral sampling diffusion models text-to-3D generation NeRF rendering resolution high-fidelity fine-tuning ProlificDreamer