IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation
<|reference_start|>IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation: Most text-to-3D generators build upon off-the-shelf text-to-image models trained on billions of images. They use variants of Score Distillation Sampling (SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation is to fine-tune the 2D generator to be multi-view aware, which can help distillation or can be combined with reconstruction networks to output 3D objects directly. In this paper, we further explore the design space of text-to-3D models. We significantly improve multi-view generation by considering video instead of image generators. Combined with a 3D reconstruction algorithm which, by using Gaussian splatting, can optimize a robust image-based loss, we directly produce high-quality 3D outputs from the generated views. Our new method, IM-3D, reduces the number of evaluations of the 2D generator network 10-100x, resulting in a much more efficient pipeline, better quality, fewer geometric inconsistencies, and higher yield of usable 3D assets.<|reference_end|>
@article{melas-kyriazi2024im-3d:,
    title={IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality
  3D Generation},
    author={Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova,
  Andrea Vedaldi, Oran Gafni, Filippos Kokkinos},
    journal={arXiv preprint arXiv:2402.08682},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.08682},
    primaryClass={cs.CV cs.AI cs.LG}
}