ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion
  Model
Advancements in 3D scene reconstruction have transformed 2D images from the
real world into 3D models, producing realistic 3D results from hundreds of
input photos. Despite great success in dense-view reconstruction scenarios,
rendering a detailed scene from insufficient captured views is still an
ill-posed optimization problem, often resulting in artifacts and distortions in
unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction
paradigm that reframes the ambiguous reconstruction challenge as a temporal
generation task. The key insight is to unleash the strong generative prior of
large pre-trained video diffusion models for sparse-view reconstruction.
However, 3D view consistency struggles to be accurately preserved in directly
generated video frames from pre-trained models. To address this, given limited
input views, the proposed ReconX first constructs a global point cloud and
encodes it into a contextual space as the 3D structure condition. Guided by the
condition, the video diffusion model then synthesizes video frames that are
both detail-preserved and exhibit a high degree of 3D consistency, ensuring the
coherence of the scene from various perspectives. Finally, we recover the 3D
scene from the generated video through a confidence-aware 3D Gaussian Splatting
optimization scheme. Extensive experiments on various real-world datasets show
the superiority of our ReconX over state-of-the-art methods in terms of quality
and generalizability.
66d126dd5ab9ab8cb4d8974d
Fangfu Liu
admin_assigned
2024-08-30T08:46:29.326Z
6505a02f9310ce8c400edc63
/avatars/bbf781594fc8c812316711aa8e2797aa.svg
Fangfu Liu
user
Liuff23
66d126dd5ab9ab8cb4d8974e
Wenqiang Sun
claimed_verified
2024-08-30T13:08:54.418Z
64897b1f0ec897cfe579a399
https://cdn-avatars.huggingface.co/v1/production/uploads/64897b1f0ec897cfe579a399/ICR_75b877BaSE94gjBuj.jpeg
wenq
user
wenqsun
66d126dd5ab9ab8cb4d8974f
Hanyang Wang
claimed_verified
2025-04-03T08:30:09.588Z
65c38f6c137aba2aee524989
/avatars/a93e29f55876df3e65e2532972e057e4.svg
Hanyang Wang
user
hanyang-21
66d126dd5ab9ab8cb4d89750
Yikai Wang
extracted_confirmed
2024-08-30T01:57:21.093Z
64206b35e40f66bcd1e3c985
/avatars/a8f830d1c47702f642045399e59c9fe8.svg
Yikai Wang
user
yikaiw
66d126dd5ab9ab8cb4d89751
Haowen Sun
claimed_verified
2024-09-02T07:17:49.797Z
6577fd0c2c8d6e12c4df5865
/avatars/83d264d01bc21765952225ddd1c1a389.svg
Haowen Sun
user
Sunboy0617
66d126dd5ab9ab8cb4d89752
Junliang Ye
claimed_verified
2024-08-30T07:17:57.430Z
65a420cd90e65dc39a6abe9e
/avatars/81ac5b749043e899f5017782409f9e28.svg
yejunliang
user
yejunliang23
66d126dd5ab9ab8cb4d89753
Jun Zhang
66d126dd5ab9ab8cb4d89754
Yueqi Duan
admin_assigned
2024-08-30T08:56:21.321Z
66c8131afafc0fc87ca99650
/avatars/a6eeba2ccf011d5c9964fd38f85bd671.svg
Yueqi Duan
user
duanyueqi
ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion
  Model
Advancements in 3D scene reconstruction have transformed 2D images from the
real world into 3D models, producing realistic 3D results from hundreds of
input photos. Despite great success in dense-view reconstruction scenarios,
rendering a detailed scene from insufficient captured views is still an
ill-posed optimization problem, often resulting in artifacts and distortions in
unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction
paradigm that reframes the ambiguous reconstruction challenge as a temporal
generation task. The key insight is to unleash the strong generative prior of
large pre-trained video diffusion models for sparse-view reconstruction.
However, 3D view consistency struggles to be accurately preserved in directly
generated video frames from pre-trained models. To address this, given limited
input views, the proposed ReconX first constructs a global point cloud and
encodes it into a contextual space as the 3D structure condition. Guided by the
condition, the video diffusion model then synthesizes video frames that are
both detail-preserved and exhibit a high degree of 3D consistency, ensuring the
coherence of the scene from various perspectives. Finally, we recover the 3D
scene from the generated video through a confidence-aware 3D Gaussian Splatting
optimization scheme. Extensive experiments on various real-world datasets show
the superiority of our ReconX over state-of-the-art methods in terms of quality
and generalizability.
ReconX uses video diffusion models guided by a 3D point cloud to achieve high-quality 3D scene reconstruction from sparse views while maintaining temporal consistency and coherence.
3D scene reconstruction video diffusion models sparse-view reconstruction point cloud contextual space 3D structure condition 3D consistency confidence-aware 3D Gaussian Splatting 3D Gaussian Splatting