title: VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
summary: Recent innovations on text-to-3D generation have featured Score Distillation
  Sampling (SDS), which enables the zero-shot learning of implicit 3D models
  (NeRF) by directly distilling prior knowledge from 2D diffusion models.
  However, current SDS-based models still struggle with intricate text prompts
  and commonly result in distorted 3D models with unrealistic textures or
  cross-view inconsistency issues. In this work, we introduce a novel Visual
  Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the
  visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
  Instead of solely supervising SDS with text prompt, VP3D first capitalizes on
  2D diffusion model to generate a high-quality image from input text, which
  subsequently acts as visual prompt to strengthen SDS optimization with explicit
  visual appearance. Meanwhile, we couple the SDS optimization with additional
  differentiable reward function that encourages rendering images of 3D models to
  better visually align with 2D visual prompt and semantically match with text
  prompt. Through extensive experiments, we show that the 2D Visual Prompt in our
  VP3D significantly eases the learning of visual appearance of 3D models and
  thus leads to higher visual fidelity with more detailed textures. It is also
  appealing in view that when replacing the self-generating visual prompt with a
  given reference image, VP3D is able to trigger a new task of stylized
  text-to-3D generation. Our project page is available at
  https://vp3d-cvpr24.github.io.
paper_authors._id: 660249a9c5665728f7c729b4
paper_authors.name: Yang Chen
paper_authors._id: 660249a9c5665728f7c729b5
paper_authors.name: Yingwei Pan
paper_authors._id: 660249a9c5665728f7c729b6
paper_authors.name: Haibo Yang
paper_authors._id: 660249a9c5665728f7c729b7
paper_authors.name: Ting Yao
paper_authors._id: 660249a9c5665728f7c729b8
paper_authors.name: Tao Mei
paper_title: VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
paper_summary: Recent innovations on text-to-3D generation have featured Score Distillation
  Sampling (SDS), which enables the zero-shot learning of implicit 3D models
  (NeRF) by directly distilling prior knowledge from 2D diffusion models.
  However, current SDS-based models still struggle with intricate text prompts
  and commonly result in distorted 3D models with unrealistic textures or
  cross-view inconsistency issues. In this work, we introduce a novel Visual
  Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the
  visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
  Instead of solely supervising SDS with text prompt, VP3D first capitalizes on
  2D diffusion model to generate a high-quality image from input text, which
  subsequently acts as visual prompt to strengthen SDS optimization with explicit
  visual appearance. Meanwhile, we couple the SDS optimization with additional
  differentiable reward function that encourages rendering images of 3D models to
  better visually align with 2D visual prompt and semantically match with text
  prompt. Through extensive experiments, we show that the 2D Visual Prompt in our
  VP3D significantly eases the learning of visual appearance of 3D models and
  thus leads to higher visual fidelity with more detailed textures. It is also
  appealing in view that when replacing the self-generating visual prompt with a
  given reference image, VP3D is able to trigger a new task of stylized
  text-to-3D generation. Our project page is available at
  https://vp3d-cvpr24.github.io.
paper_ai_keywords: SDS (Score Distillation Sampling) implicit 3D models (NeRF) 2D diffusion models zero-shot learning intricate text prompts distorted 3D models unrealistic textures cross-view inconsistency Visual Prompt-guided text-to-3D diffusion model (VP3D) high-quality image input text visual appearance knowledge 2D visual prompt SDS optimization differentiable reward function visually align with 2D visual prompt semantically match with text prompt visual fidelity detailed textures stylized text-to-3D generation