Progressive Rendering Distillation: Adapting Stable Diffusion for
  Instant Text-to-Mesh Generation without 3D Data
It is highly desirable to obtain a model that can generate high-quality 3D
meshes from text prompts in just seconds. While recent attempts have adapted
pre-trained text-to-image diffusion models, such as Stable Diffusion (SD), into
generators of 3D representations (e.g., Triplane), they often suffer from poor
quality due to the lack of sufficient high-quality 3D training data. Aiming at
overcoming the data shortage, we propose a novel training scheme, termed as
Progressive Rendering Distillation (PRD), eliminating the need for 3D
ground-truths by distilling multi-view diffusion models and adapting SD into a
native 3D generator. In each iteration of training, PRD uses the U-Net to
progressively denoise the latent from random noise for a few steps, and in each
step it decodes the denoised latent into 3D output. Multi-view diffusion
models, including MVDream and RichDreamer, are used in joint with SD to distill
text-consistent textures and geometries into the 3D outputs through score
distillation. Since PRD supports training without 3D ground-truths, we can
easily scale up the training data and improve generation quality for
challenging text prompts with creative concepts. Meanwhile, PRD can accelerate
the inference speed of the generation model in just a few steps. With PRD, we
train a Triplane generator, namely TriplaneTurbo, which adds only 2.5%
trainable parameters to adapt SD for Triplane generation. TriplaneTurbo
outperforms previous text-to-3D generators in both efficiency and quality.
Specifically, it can produce high-quality 3D meshes in 1.2 seconds and
generalize well for challenging text input. The code is available at
https://github.com/theEricMa/TriplaneTurbo.
67eb92defa85fe030e2db9e2
Zhiyuan Ma
admin_assigned
2025-04-01T08:14:51.486Z
64295d1f4e073875f6a605ac
https://cdn-avatars.huggingface.co/v1/production/uploads/64295d1f4e073875f6a605ac/HEK8dluqPUhiARW-0eBZG.jpeg
Zhiyuan Ma
user
ZhiyuanthePony
67eb92defa85fe030e2db9e3
Xinyue Liang
admin_assigned
2025-04-01T08:14:58.992Z
672111333ced358bdac2925d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/qHNDI3zYRkJZmhCHkSZwK.png
Xinyue Liang
user
DarklordLeto
67eb92defa85fe030e2db9e4
Rongyuan Wu
67eb92defa85fe030e2db9e5
Xiangyu Zhu
67eb92defa85fe030e2db9e6
Zhen Lei
67eb92defa85fe030e2db9e7
Lei Zhang
Progressive Rendering Distillation: Adapting Stable Diffusion for
  Instant Text-to-Mesh Generation without 3D Data
It is highly desirable to obtain a model that can generate high-quality 3D
meshes from text prompts in just seconds. While recent attempts have adapted
pre-trained text-to-image diffusion models, such as Stable Diffusion (SD), into
generators of 3D representations (e.g., Triplane), they often suffer from poor
quality due to the lack of sufficient high-quality 3D training data. Aiming at
overcoming the data shortage, we propose a novel training scheme, termed as
Progressive Rendering Distillation (PRD), eliminating the need for 3D
ground-truths by distilling multi-view diffusion models and adapting SD into a
native 3D generator. In each iteration of training, PRD uses the U-Net to
progressively denoise the latent from random noise for a few steps, and in each
step it decodes the denoised latent into 3D output. Multi-view diffusion
models, including MVDream and RichDreamer, are used in joint with SD to distill
text-consistent textures and geometries into the 3D outputs through score
distillation. Since PRD supports training without 3D ground-truths, we can
easily scale up the training data and improve generation quality for
challenging text prompts with creative concepts. Meanwhile, PRD can accelerate
the inference speed of the generation model in just a few steps. With PRD, we
train a Triplane generator, namely TriplaneTurbo, which adds only 2.5%
trainable parameters to adapt SD for Triplane generation. TriplaneTurbo
outperforms previous text-to-3D generators in both efficiency and quality.
Specifically, it can produce high-quality 3D meshes in 1.2 seconds and
generalize well for challenging text input. The code is available at
https://github.com/theEricMa/TriplaneTurbo.
A novel training scheme, Progressive Rendering Distillation (PRD), enhances text-to-3D generation by distilling multi-view diffusion models into a native 3D generator, overcoming data shortages and improving efficiency.
diffusion models Triplane Progressive Rendering Distillation PRD U-Net multi-view diffusion models MVDream RichDreamer score distillation TriplaneTurbo text-to-3D generators