title: Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
summary: Text-driven 3D scene generation is widely applicable to video gaming, film
  industry, and metaverse applications that have a large demand for 3D scenes.
  However, existing text-to-3D generation methods are limited to producing 3D
  objects with simple geometries and dreamlike styles that lack realism. In this
  work, we present Text2NeRF, which is able to generate a wide range of 3D scenes
  with complicated geometric structures and high-fidelity textures purely from a
  text prompt. To this end, we adopt NeRF as the 3D representation and leverage a
  pre-trained text-to-image diffusion model to constrain the 3D reconstruction of
  the NeRF to reflect the scene description. Specifically, we employ the
  diffusion model to infer the text-related image as the content prior and use a
  monocular depth estimation method to offer the geometric prior. Both content
  and geometric priors are utilized to update the NeRF model. To guarantee
  textured and geometric consistency between different views, we introduce a
  progressive scene inpainting and updating strategy for novel view synthesis of
  the scene. Our method requires no additional training data but only a natural
  language description of the scene as the input. Extensive experiments
  demonstrate that our Text2NeRF outperforms existing methods in producing
  photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of
  natural language prompts.
paper_authors._id: 646abeff1556443f24b8d097
paper_authors.name: Jingbo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T11:57:40.766Z
paper_authors._id: 631ff2c8124782a19eff5e2f
paper_authors.avatarUrl: /avatars/76e9f74ecf6ed56e4abb89a25387b052.svg
paper_authors.fullname: Jingbo Zhang
paper_authors.type: user
paper_authors.user: Eckert
paper_authors._id: 646abeff1556443f24b8d098
paper_authors.name: Xiaoyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:03:01.826Z
paper_authors._id: 637360960938c0754236974e
paper_authors.avatarUrl: /avatars/4f47c06ca2f6bf5b6f33c0fba4f54892.svg
paper_authors.fullname: Xiaoyu Li
paper_authors.type: user
paper_authors.user: Axidy
paper_authors._id: 646abeff1556443f24b8d099
paper_authors.name: Ziyu Wan
paper_authors._id: 646abeff1556443f24b8d09a
paper_authors.name: Can Wang
paper_authors._id: 646abeff1556443f24b8d09b
paper_authors.name: Jing Liao
paper_title: Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
paper_summary: Text-driven 3D scene generation is widely applicable to video gaming, film
  industry, and metaverse applications that have a large demand for 3D scenes.
  However, existing text-to-3D generation methods are limited to producing 3D
  objects with simple geometries and dreamlike styles that lack realism. In this
  work, we present Text2NeRF, which is able to generate a wide range of 3D scenes
  with complicated geometric structures and high-fidelity textures purely from a
  text prompt. To this end, we adopt NeRF as the 3D representation and leverage a
  pre-trained text-to-image diffusion model to constrain the 3D reconstruction of
  the NeRF to reflect the scene description. Specifically, we employ the
  diffusion model to infer the text-related image as the content prior and use a
  monocular depth estimation method to offer the geometric prior. Both content
  and geometric priors are utilized to update the NeRF model. To guarantee
  textured and geometric consistency between different views, we introduce a
  progressive scene inpainting and updating strategy for novel view synthesis of
  the scene. Our method requires no additional training data but only a natural
  language description of the scene as the input. Extensive experiments
  demonstrate that our Text2NeRF outperforms existing methods in producing
  photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of
  natural language prompts.
paper_ai_summary: Text2NeRF generates high-fidelity, photo-realistic 3D scenes from text prompts by integrating NeRF with a text-to-image diffusion model and monocular depth estimation.
paper_ai_keywords: NeRF text-to-image diffusion model monocular depth estimation progressive scene inpainting novel view synthesis photo-realistic multi-view consistent