SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse
  Views
Open-world 3D generation has recently attracted considerable attention. While
many single-image-to-3D methods have yielded visually appealing outcomes, they
often lack sufficient controllability and tend to produce hallucinated regions
that may not align with users' expectations. In this paper, we explore an
important scenario in which the input consists of one or a few unposed 2D
images of a single object, with little or no overlap. We propose a novel
method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative
camera poses for these sparse-view images. SpaRP distills knowledge from 2D
diffusion models and finetunes them to implicitly deduce the 3D spatial
relationships between the sparse views. The diffusion model is trained to
jointly predict surrogate representations for camera poses and multi-view
images of the object under known poses, integrating all information from the
input sparse views. These predictions are then leveraged to accomplish 3D
reconstruction and pose estimation, and the reconstructed 3D model can be used
to further refine the camera poses of input views. Through extensive
experiments on three datasets, we demonstrate that our method not only
significantly outperforms baseline methods in terms of 3D reconstruction
quality and pose prediction accuracy but also exhibits strong efficiency. It
requires only about 20 seconds to produce a textured mesh and camera poses for
the input views. Project page: https://chaoxu.xyz/sparp.
66c41d58ac74db25de785db9
Chao Xu
claimed_verified
2024-08-22T19:51:41.497Z
6468a48cff18750165a5aad2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
Chao Xu
user
chaoxu
66c41d58ac74db25de785dba
Ang Li
66c41d58ac74db25de785dbb
Linghao Chen
66c41d58ac74db25de785dbc
Yulin Liu
claimed_verified
2024-08-29T07:24:08.732Z
6587beaace38d143c423c1ae
/avatars/ce2c00385ac47a0b4f4698039befbf6f.svg
Yulin Liu
user
liuyulinn
66c41d58ac74db25de785dbd
Ruoxi Shi
66c41d58ac74db25de785dbe
Hao Su
66c41d58ac74db25de785dbf
Minghua Liu
SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse
  Views
Open-world 3D generation has recently attracted considerable attention. While
many single-image-to-3D methods have yielded visually appealing outcomes, they
often lack sufficient controllability and tend to produce hallucinated regions
that may not align with users' expectations. In this paper, we explore an
important scenario in which the input consists of one or a few unposed 2D
images of a single object, with little or no overlap. We propose a novel
method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative
camera poses for these sparse-view images. SpaRP distills knowledge from 2D
diffusion models and finetunes them to implicitly deduce the 3D spatial
relationships between the sparse views. The diffusion model is trained to
jointly predict surrogate representations for camera poses and multi-view
images of the object under known poses, integrating all information from the
input sparse views. These predictions are then leveraged to accomplish 3D
reconstruction and pose estimation, and the reconstructed 3D model can be used
to further refine the camera poses of input views. Through extensive
experiments on three datasets, we demonstrate that our method not only
significantly outperforms baseline methods in terms of 3D reconstruction
quality and pose prediction accuracy but also exhibits strong efficiency. It
requires only about 20 seconds to produce a textured mesh and camera poses for
the input views. Project page: https://chaoxu.xyz/sparp.
A novel method, SpaRP, reconstructs 3D textured meshes and estimates camera poses from sparse-view images by leveraging and finetuning 2D diffusion models.
diffusion models 3D reconstruction camera pose estimation sparse-view images surrogate representations multi-view images 3D textured mesh