ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
3D asset generation is getting massive amounts of attention, inspired by the
recent success of text-guided 2D content creation. Existing text-to-3D methods
use pretrained text-to-image diffusion models in an optimization problem or
fine-tune them on synthetic data, which often results in non-photorealistic 3D
objects without backgrounds. In this paper, we present a method that leverages
pretrained text-to-image models as a prior, and learn to generate multi-view
images in a single denoising process from real-world data. Concretely, we
propose to integrate 3D volume-rendering and cross-frame-attention layers into
each block of the existing U-Net network of the text-to-image model. Moreover,
we design an autoregressive generation that renders more 3D-consistent images
at any viewpoint. We train our model on real-world datasets of objects and
showcase its capabilities to generate instances with a variety of high-quality
shapes and textures in authentic surroundings. Compared to the existing
methods, the results generated by our method are consistent, and have favorable
visual quality (-30% FID, -37% KID).
65e738d4f73c7bbb01500a14
Lukas Höllein
claimed_verified
2024-09-20T12:43:28.426Z
638711fd20244d72a73eb27b
/avatars/162a798948bfe3df8e788aba5d024ffb.svg
Lukas Hoellein
user
lukasHoel
65e738d4f73c7bbb01500a15
Aljaž Božič
65e738d4f73c7bbb01500a16
Norman Müller
65e738d4f73c7bbb01500a17
David Novotny
65e738d4f73c7bbb01500a18
Hung-Yu Tseng
65e738d4f73c7bbb01500a19
Christian Richardt
claimed_verified
2025-02-13T20:36:39.201Z
67adc0cfe1f8b9375a6c7a63
/avatars/942d92ff9bd9e7a0f8f4687babbb8153.svg
Christian Richardt
user
cr333
65e738d4f73c7bbb01500a1a
Michael Zollhöfer
65e738d4f73c7bbb01500a1b
Matthias Nießner
ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
3D asset generation is getting massive amounts of attention, inspired by the
recent success of text-guided 2D content creation. Existing text-to-3D methods
use pretrained text-to-image diffusion models in an optimization problem or
fine-tune them on synthetic data, which often results in non-photorealistic 3D
objects without backgrounds. In this paper, we present a method that leverages
pretrained text-to-image models as a prior, and learn to generate multi-view
images in a single denoising process from real-world data. Concretely, we
propose to integrate 3D volume-rendering and cross-frame-attention layers into
each block of the existing U-Net network of the text-to-image model. Moreover,
we design an autoregressive generation that renders more 3D-consistent images
at any viewpoint. We train our model on real-world datasets of objects and
showcase its capabilities to generate instances with a variety of high-quality
shapes and textures in authentic surroundings. Compared to the existing
methods, the results generated by our method are consistent, and have favorable
visual quality (-30% FID, -37% KID).
The paper introduces a method for generating photorealistic 3D assets using pretrained text-to-image models integrated with 3D volume-rendering and cross-frame-attention layers, achieving high-quality results in real-world settings.
text-to-3D text-to-image diffusion models optimization problem fine-tuning synthetic data pretrained text-to-image models denoising process 3D volume-rendering cross-frame-attention layers U-Net network autoregressive generation real-world datasets FID KID