VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
Recent innovations on text-to-3D generation have featured Score Distillation
Sampling (SDS), which enables the zero-shot learning of implicit 3D models
(NeRF) by directly distilling prior knowledge from 2D diffusion models.
However, current SDS-based models still struggle with intricate text prompts
and commonly result in distorted 3D models with unrealistic textures or
cross-view inconsistency issues. In this work, we introduce a novel Visual
Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the
visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
Instead of solely supervising SDS with text prompt, VP3D first capitalizes on
2D diffusion model to generate a high-quality image from input text, which
subsequently acts as visual prompt to strengthen SDS optimization with explicit
visual appearance. Meanwhile, we couple the SDS optimization with additional
differentiable reward function that encourages rendering images of 3D models to
better visually align with 2D visual prompt and semantically match with text
prompt. Through extensive experiments, we show that the 2D Visual Prompt in our
VP3D significantly eases the learning of visual appearance of 3D models and
thus leads to higher visual fidelity with more detailed textures. It is also
appealing in view that when replacing the self-generating visual prompt with a
given reference image, VP3D is able to trigger a new task of stylized
text-to-3D generation. Our project page is available at
https://vp3d-cvpr24.github.io.
660249a9c5665728f7c729b4
Yang Chen
660249a9c5665728f7c729b5
Yingwei Pan
660249a9c5665728f7c729b6
Haibo Yang
660249a9c5665728f7c729b7
Ting Yao
660249a9c5665728f7c729b8
Tao Mei
VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
Recent innovations on text-to-3D generation have featured Score Distillation
Sampling (SDS), which enables the zero-shot learning of implicit 3D models
(NeRF) by directly distilling prior knowledge from 2D diffusion models.
However, current SDS-based models still struggle with intricate text prompts
and commonly result in distorted 3D models with unrealistic textures or
cross-view inconsistency issues. In this work, we introduce a novel Visual
Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the
visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
Instead of solely supervising SDS with text prompt, VP3D first capitalizes on
2D diffusion model to generate a high-quality image from input text, which
subsequently acts as visual prompt to strengthen SDS optimization with explicit
visual appearance. Meanwhile, we couple the SDS optimization with additional
differentiable reward function that encourages rendering images of 3D models to
better visually align with 2D visual prompt and semantically match with text
prompt. Through extensive experiments, we show that the 2D Visual Prompt in our
VP3D significantly eases the learning of visual appearance of 3D models and
thus leads to higher visual fidelity with more detailed textures. It is also
appealing in view that when replacing the self-generating visual prompt with a
given reference image, VP3D is able to trigger a new task of stylized
text-to-3D generation. Our project page is available at
https://vp3d-cvpr24.github.io.
A novel Visual Prompt-guided text-to-3D diffusion model (VP3D) boosts text-to-3D generation by integrating visual appearance knowledge from 2D images.
Score Distillation Sampling (SDS) implicit 3D models (NeRF) 2D diffusion models visual prompts differentiable reward function stylized text-to-3D generation