title: SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse
    Views
summary: Open-world 3D generation has recently attracted considerable attention. While
  many single-image-to-3D methods have yielded visually appealing outcomes, they
  often lack sufficient controllability and tend to produce hallucinated regions
  that may not align with users' expectations. In this paper, we explore an
  important scenario in which the input consists of one or a few unposed 2D
  images of a single object, with little or no overlap. We propose a novel
  method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative
  camera poses for these sparse-view images. SpaRP distills knowledge from 2D
  diffusion models and finetunes them to implicitly deduce the 3D spatial
  relationships between the sparse views. The diffusion model is trained to
  jointly predict surrogate representations for camera poses and multi-view
  images of the object under known poses, integrating all information from the
  input sparse views. These predictions are then leveraged to accomplish 3D
  reconstruction and pose estimation, and the reconstructed 3D model can be used
  to further refine the camera poses of input views. Through extensive
  experiments on three datasets, we demonstrate that our method not only
  significantly outperforms baseline methods in terms of 3D reconstruction
  quality and pose prediction accuracy but also exhibits strong efficiency. It
  requires only about 20 seconds to produce a textured mesh and camera poses for
  the input views. Project page: https://chaoxu.xyz/sparp.
paper_authors._id: 66c41d58ac74db25de785db9
paper_authors.name: Chao Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:51:41.497Z
paper_authors._id: 6468a48cff18750165a5aad2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
paper_authors.fullname: Chao Xu
paper_authors.type: user
paper_authors.user: chaoxu
paper_authors._id: 66c41d58ac74db25de785dba
paper_authors.name: Ang Li
paper_authors._id: 66c41d58ac74db25de785dbb
paper_authors.name: Linghao Chen
paper_authors._id: 66c41d58ac74db25de785dbc
paper_authors.name: Yulin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:24:08.732Z
paper_authors._id: 6587beaace38d143c423c1ae
paper_authors.avatarUrl: /avatars/ce2c00385ac47a0b4f4698039befbf6f.svg
paper_authors.fullname: Yulin Liu
paper_authors.type: user
paper_authors.user: liuyulinn
paper_authors._id: 66c41d58ac74db25de785dbd
paper_authors.name: Ruoxi Shi
paper_authors._id: 66c41d58ac74db25de785dbe
paper_authors.name: Hao Su
paper_authors._id: 66c41d58ac74db25de785dbf
paper_authors.name: Minghua Liu
paper_title: SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse
    Views
paper_summary: Open-world 3D generation has recently attracted considerable attention. While
  many single-image-to-3D methods have yielded visually appealing outcomes, they
  often lack sufficient controllability and tend to produce hallucinated regions
  that may not align with users' expectations. In this paper, we explore an
  important scenario in which the input consists of one or a few unposed 2D
  images of a single object, with little or no overlap. We propose a novel
  method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative
  camera poses for these sparse-view images. SpaRP distills knowledge from 2D
  diffusion models and finetunes them to implicitly deduce the 3D spatial
  relationships between the sparse views. The diffusion model is trained to
  jointly predict surrogate representations for camera poses and multi-view
  images of the object under known poses, integrating all information from the
  input sparse views. These predictions are then leveraged to accomplish 3D
  reconstruction and pose estimation, and the reconstructed 3D model can be used
  to further refine the camera poses of input views. Through extensive
  experiments on three datasets, we demonstrate that our method not only
  significantly outperforms baseline methods in terms of 3D reconstruction
  quality and pose prediction accuracy but also exhibits strong efficiency. It
  requires only about 20 seconds to produce a textured mesh and camera poses for
  the input views. Project page: https://chaoxu.xyz/sparp.
paper_ai_summary: A novel method, SpaRP, reconstructs 3D textured meshes and estimates camera poses from sparse-view images by leveraging and finetuning 2D diffusion models.
paper_ai_keywords: diffusion models 3D reconstruction camera pose estimation sparse-view images surrogate representations multi-view images 3D textured mesh