IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality
  3D Generation
Most text-to-3D generators build upon off-the-shelf text-to-image models
trained on billions of images. They use variants of Score Distillation Sampling
(SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation
is to fine-tune the 2D generator to be multi-view aware, which can help
distillation or can be combined with reconstruction networks to output 3D
objects directly. In this paper, we further explore the design space of
text-to-3D models. We significantly improve multi-view generation by
considering video instead of image generators. Combined with a 3D
reconstruction algorithm which, by using Gaussian splatting, can optimize a
robust image-based loss, we directly produce high-quality 3D outputs from the
generated views. Our new method, IM-3D, reduces the number of evaluations of
the 2D generator network 10-100x, resulting in a much more efficient pipeline,
better quality, fewer geometric inconsistencies, and higher yield of usable 3D
assets.
65cc30057643d0e69bb18894
Luke Melas-Kyriazi
admin_assigned
2024-02-14T11:03:40.117Z
6244593ec661f366527f6a78
/avatars/4e2cf8894c3268c824160249cdfaf3ab.svg
Luke Melas-Kyriazi
user
lukemelas
65cc30057643d0e69bb18895
Iro Laina
65cc30057643d0e69bb18896
Christian Rupprecht
65cc30057643d0e69bb18897
Natalia Neverova
65cc30057643d0e69bb18898
Andrea Vedaldi
65cc30057643d0e69bb18899
Oran Gafni
admin_assigned
2024-02-14T11:04:09.310Z
6345a97d547c70e4b7ce9169
/avatars/5d233e65eadfa7d9ef0390a584df5451.svg
Oran Gafni
user
Oran22
65cc30057643d0e69bb1889a
Filippos Kokkinos
extracted_pending
2024-02-14T03:14:14.497Z
64918e6bd0aa14078dee0929
/avatars/407d6996d5d297ad3492a9a2382205cd.svg
Filippos
user
fkokkinos
IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality
  3D Generation
Most text-to-3D generators build upon off-the-shelf text-to-image models
trained on billions of images. They use variants of Score Distillation Sampling
(SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation
is to fine-tune the 2D generator to be multi-view aware, which can help
distillation or can be combined with reconstruction networks to output 3D
objects directly. In this paper, we further explore the design space of
text-to-3D models. We significantly improve multi-view generation by
considering video instead of image generators. Combined with a 3D
reconstruction algorithm which, by using Gaussian splatting, can optimize a
robust image-based loss, we directly produce high-quality 3D outputs from the
generated views. Our new method, IM-3D, reduces the number of evaluations of
the 2D generator network 10-100x, resulting in a much more efficient pipeline,
better quality, fewer geometric inconsistencies, and higher yield of usable 3D
assets.
IM-3D improves text-to-3D generation by using video generators and Gaussian splatting to produce high-quality 3D outputs efficiently with minimized geometric inconsistencies.
Score Distillation Sampling multi-view aware 2D generator video generators 3D reconstruction Gaussian splatting image-based loss IM-3D