title: Viewpoint Textual Inversion: Unleashing Novel View Synthesis with
    Pretrained 2D Diffusion Models
summary: Text-to-image diffusion models understand spatial relationship between
  objects, but do they represent the true 3D structure of the world from only 2D
  supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
  diffusion models like Stable Diffusion, and we show that this structure can be
  exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
  (ViewNeTI), controls the 3D viewpoint of objects in generated images from
  frozen diffusion models. We train a small neural mapper to take camera
  viewpoint parameters and predict text encoder latents; the latents then
  condition the diffusion generation process to produce images with the desired
  camera viewpoint.
    ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
  frozen diffusion model as a prior, we can solve NVS with very few input views;
  we can even do single-view novel view synthesis. Our single-view NVS
  predictions have good semantic details and photorealism compared to prior
  methods. Our approach is well suited for modeling the uncertainty inherent in
  sparse 3D vision problems because it can efficiently generate diverse samples.
  Our view-control mechanism is general, and can even change the camera view in
  images generated by user-defined prompts.
paper_authors._id: 6507e8ff04d04d653d088f05
paper_authors.name: James Burgess
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-18T17:14:33.118Z
paper_authors._id: 650871aeb44445e9b3625c7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
paper_authors.fullname: James Burgess
paper_authors.type: user
paper_authors.user: jmhb
paper_authors._id: 6507e8ff04d04d653d088f06
paper_authors.name: Kuan-Chieh Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T11:04:08.815Z
paper_authors._id: 648ca58a39d2584ee47efef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
paper_authors.fullname: Kuan-Chieh Wang
paper_authors.type: user
paper_authors.user: wangkua1
paper_authors._id: 6507e8ff04d04d653d088f07
paper_authors.name: Serena Yeung
paper_title: Viewpoint Textual Inversion: Unleashing Novel View Synthesis with
    Pretrained 2D Diffusion Models
paper_summary: Text-to-image diffusion models understand spatial relationship between
  objects, but do they represent the true 3D structure of the world from only 2D
  supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
  diffusion models like Stable Diffusion, and we show that this structure can be
  exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
  (ViewNeTI), controls the 3D viewpoint of objects in generated images from
  frozen diffusion models. We train a small neural mapper to take camera
  viewpoint parameters and predict text encoder latents; the latents then
  condition the diffusion generation process to produce images with the desired
  camera viewpoint.
    ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
  frozen diffusion model as a prior, we can solve NVS with very few input views;
  we can even do single-view novel view synthesis. Our single-view NVS
  predictions have good semantic details and photorealism compared to prior
  methods. Our approach is well suited for modeling the uncertainty inherent in
  sparse 3D vision problems because it can efficiently generate diverse samples.
  Our view-control mechanism is general, and can even change the camera view in
  images generated by user-defined prompts.
paper_ai_summary: Viewpoint Neural Textual Inversion (ViewNeTI) leverages 3D information encoded in text-to-image diffusion models to control camera viewpoints and perform novel view synthesis with high photorealism and semantic detail.
paper_ai_keywords: diffusion models text-to-image 3D knowledge viewpoint control Viewpoint Neural Textual Inversion ViewNeTI text encoder latents novel view synthesis NVS single-view NVS semantic details photorealism sparse 3D vision diversity generation camera view control