title: ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion
    Model
summary: Advancements in 3D scene reconstruction have transformed 2D images from the
  real world into 3D models, producing realistic 3D results from hundreds of
  input photos. Despite great success in dense-view reconstruction scenarios,
  rendering a detailed scene from insufficient captured views is still an
  ill-posed optimization problem, often resulting in artifacts and distortions in
  unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction
  paradigm that reframes the ambiguous reconstruction challenge as a temporal
  generation task. The key insight is to unleash the strong generative prior of
  large pre-trained video diffusion models for sparse-view reconstruction.
  However, 3D view consistency struggles to be accurately preserved in directly
  generated video frames from pre-trained models. To address this, given limited
  input views, the proposed ReconX first constructs a global point cloud and
  encodes it into a contextual space as the 3D structure condition. Guided by the
  condition, the video diffusion model then synthesizes video frames that are
  both detail-preserved and exhibit a high degree of 3D consistency, ensuring the
  coherence of the scene from various perspectives. Finally, we recover the 3D
  scene from the generated video through a confidence-aware 3D Gaussian Splatting
  optimization scheme. Extensive experiments on various real-world datasets show
  the superiority of our ReconX over state-of-the-art methods in terms of quality
  and generalizability.
paper_authors._id: 66d126dd5ab9ab8cb4d8974d
paper_authors.name: Fangfu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:46:29.326Z
paper_authors._id: 6505a02f9310ce8c400edc63
paper_authors.avatarUrl: /avatars/bbf781594fc8c812316711aa8e2797aa.svg
paper_authors.fullname: Fangfu Liu
paper_authors.type: user
paper_authors.user: Liuff23
paper_authors._id: 66d126dd5ab9ab8cb4d8974e
paper_authors.name: Wenqiang Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T13:08:54.418Z
paper_authors._id: 64897b1f0ec897cfe579a399
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64897b1f0ec897cfe579a399/ICR_75b877BaSE94gjBuj.jpeg
paper_authors.fullname: wenq
paper_authors.type: user
paper_authors.user: wenqsun
paper_authors._id: 66d126dd5ab9ab8cb4d8974f
paper_authors.name: Hanyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:30:09.588Z
paper_authors._id: 65c38f6c137aba2aee524989
paper_authors.avatarUrl: /avatars/a93e29f55876df3e65e2532972e057e4.svg
paper_authors.fullname: Hanyang Wang
paper_authors.type: user
paper_authors.user: hanyang-21
paper_authors._id: 66d126dd5ab9ab8cb4d89750
paper_authors.name: Yikai Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-30T01:57:21.093Z
paper_authors._id: 64206b35e40f66bcd1e3c985
paper_authors.avatarUrl: /avatars/a8f830d1c47702f642045399e59c9fe8.svg
paper_authors.fullname: Yikai Wang
paper_authors.type: user
paper_authors.user: yikaiw
paper_authors._id: 66d126dd5ab9ab8cb4d89751
paper_authors.name: Haowen Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:17:49.797Z
paper_authors._id: 6577fd0c2c8d6e12c4df5865
paper_authors.avatarUrl: /avatars/83d264d01bc21765952225ddd1c1a389.svg
paper_authors.fullname: Haowen Sun
paper_authors.type: user
paper_authors.user: Sunboy0617
paper_authors._id: 66d126dd5ab9ab8cb4d89752
paper_authors.name: Junliang Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T07:17:57.430Z
paper_authors._id: 65a420cd90e65dc39a6abe9e
paper_authors.avatarUrl: /avatars/81ac5b749043e899f5017782409f9e28.svg
paper_authors.fullname: yejunliang
paper_authors.type: user
paper_authors.user: yejunliang23
paper_authors._id: 66d126dd5ab9ab8cb4d89753
paper_authors.name: Jun Zhang
paper_authors._id: 66d126dd5ab9ab8cb4d89754
paper_authors.name: Yueqi Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:56:21.321Z
paper_authors._id: 66c8131afafc0fc87ca99650
paper_authors.avatarUrl: /avatars/a6eeba2ccf011d5c9964fd38f85bd671.svg
paper_authors.fullname: Yueqi Duan
paper_authors.type: user
paper_authors.user: duanyueqi
paper_title: ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion
    Model
paper_summary: Advancements in 3D scene reconstruction have transformed 2D images from the
  real world into 3D models, producing realistic 3D results from hundreds of
  input photos. Despite great success in dense-view reconstruction scenarios,
  rendering a detailed scene from insufficient captured views is still an
  ill-posed optimization problem, often resulting in artifacts and distortions in
  unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction
  paradigm that reframes the ambiguous reconstruction challenge as a temporal
  generation task. The key insight is to unleash the strong generative prior of
  large pre-trained video diffusion models for sparse-view reconstruction.
  However, 3D view consistency struggles to be accurately preserved in directly
  generated video frames from pre-trained models. To address this, given limited
  input views, the proposed ReconX first constructs a global point cloud and
  encodes it into a contextual space as the 3D structure condition. Guided by the
  condition, the video diffusion model then synthesizes video frames that are
  both detail-preserved and exhibit a high degree of 3D consistency, ensuring the
  coherence of the scene from various perspectives. Finally, we recover the 3D
  scene from the generated video through a confidence-aware 3D Gaussian Splatting
  optimization scheme. Extensive experiments on various real-world datasets show
  the superiority of our ReconX over state-of-the-art methods in terms of quality
  and generalizability.
paper_ai_summary: ReconX uses video diffusion models guided by a 3D point cloud to achieve high-quality 3D scene reconstruction from sparse views while maintaining temporal consistency and coherence.
paper_ai_keywords: 3D scene reconstruction video diffusion models sparse-view reconstruction point cloud contextual space 3D structure condition 3D consistency confidence-aware 3D Gaussian Splatting 3D Gaussian Splatting