title: ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
summary: 3D asset generation is getting massive amounts of attention, inspired by the
  recent success of text-guided 2D content creation. Existing text-to-3D methods
  use pretrained text-to-image diffusion models in an optimization problem or
  fine-tune them on synthetic data, which often results in non-photorealistic 3D
  objects without backgrounds. In this paper, we present a method that leverages
  pretrained text-to-image models as a prior, and learn to generate multi-view
  images in a single denoising process from real-world data. Concretely, we
  propose to integrate 3D volume-rendering and cross-frame-attention layers into
  each block of the existing U-Net network of the text-to-image model. Moreover,
  we design an autoregressive generation that renders more 3D-consistent images
  at any viewpoint. We train our model on real-world datasets of objects and
  showcase its capabilities to generate instances with a variety of high-quality
  shapes and textures in authentic surroundings. Compared to the existing
  methods, the results generated by our method are consistent, and have favorable
  visual quality (-30% FID, -37% KID).
paper_authors._id: 65e738d4f73c7bbb01500a14
paper_authors.name: Lukas Höllein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T12:43:28.426Z
paper_authors._id: 638711fd20244d72a73eb27b
paper_authors.avatarUrl: /avatars/162a798948bfe3df8e788aba5d024ffb.svg
paper_authors.fullname: Lukas Hoellein
paper_authors.type: user
paper_authors.user: lukasHoel
paper_authors._id: 65e738d4f73c7bbb01500a15
paper_authors.name: Aljaž Božič
paper_authors._id: 65e738d4f73c7bbb01500a16
paper_authors.name: Norman Müller
paper_authors._id: 65e738d4f73c7bbb01500a17
paper_authors.name: David Novotny
paper_authors._id: 65e738d4f73c7bbb01500a18
paper_authors.name: Hung-Yu Tseng
paper_authors._id: 65e738d4f73c7bbb01500a19
paper_authors.name: Christian Richardt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:39.201Z
paper_authors._id: 67adc0cfe1f8b9375a6c7a63
paper_authors.avatarUrl: /avatars/942d92ff9bd9e7a0f8f4687babbb8153.svg
paper_authors.fullname: Christian Richardt
paper_authors.type: user
paper_authors.user: cr333
paper_authors._id: 65e738d4f73c7bbb01500a1a
paper_authors.name: Michael Zollhöfer
paper_authors._id: 65e738d4f73c7bbb01500a1b
paper_authors.name: Matthias Nießner
paper_title: ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
paper_summary: 3D asset generation is getting massive amounts of attention, inspired by the
  recent success of text-guided 2D content creation. Existing text-to-3D methods
  use pretrained text-to-image diffusion models in an optimization problem or
  fine-tune them on synthetic data, which often results in non-photorealistic 3D
  objects without backgrounds. In this paper, we present a method that leverages
  pretrained text-to-image models as a prior, and learn to generate multi-view
  images in a single denoising process from real-world data. Concretely, we
  propose to integrate 3D volume-rendering and cross-frame-attention layers into
  each block of the existing U-Net network of the text-to-image model. Moreover,
  we design an autoregressive generation that renders more 3D-consistent images
  at any viewpoint. We train our model on real-world datasets of objects and
  showcase its capabilities to generate instances with a variety of high-quality
  shapes and textures in authentic surroundings. Compared to the existing
  methods, the results generated by our method are consistent, and have favorable
  visual quality (-30% FID, -37% KID).
paper_ai_summary: The paper introduces a method for generating photorealistic 3D assets using pretrained text-to-image models integrated with 3D volume-rendering and cross-frame-attention layers, achieving high-quality results in real-world settings.
paper_ai_keywords: text-to-3D text-to-image diffusion models optimization problem fine-tuning synthetic data pretrained text-to-image models denoising process 3D volume-rendering cross-frame-attention layers U-Net network autoregressive generation real-world datasets FID KID