GeometryCrafter: Consistent Geometry Estimation for Open-world Videos
  with Diffusion Priors
Despite remarkable advancements in video depth estimation, existing methods
exhibit inherent limitations in achieving geometric fidelity through the
affine-invariant predictions, limiting their applicability in reconstruction
and other metrically grounded downstream tasks. We propose GeometryCrafter, a
novel framework that recovers high-fidelity point map sequences with temporal
coherence from open-world videos, enabling accurate 3D/4D reconstruction,
camera parameter estimation, and other depth-based applications. At the core of
our approach lies a point map Variational Autoencoder (VAE) that learns a
latent space agnostic to video latent distributions for effective point map
encoding and decoding. Leveraging the VAE, we train a video diffusion model to
model the distribution of point map sequences conditioned on the input videos.
Extensive evaluations on diverse datasets demonstrate that GeometryCrafter
achieves state-of-the-art 3D accuracy, temporal consistency, and generalization
capability.
67ec958ebb1d6dd924f94a31
Tian-Xing Xu
admin_assigned
2025-04-02T09:51:06.226Z
65f8e4778dc7bb5b4db97f92
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/gBmQdovmANmjV72k3gaW8.png
Tian-Xing Xu
user
slothfulxtx
67ec958ebb1d6dd924f94a32
Xiangjun Gao
admin_assigned
2025-04-02T09:51:14.978Z
64c0953a8137192a1e2474dc
/avatars/546405a7eaf2f60ad108ceaa0dda7d08.svg
xiangjun gao
user
xiangjun0211
67ec958ebb1d6dd924f94a33
Wenbo Hu
claimed_verified
2025-04-02T08:22:52.313Z
657a7458afbb0117ba15c59f
https://cdn-avatars.huggingface.co/v1/production/uploads/657a7458afbb0117ba15c59f/8_iwTS1UG_mKnfylFbLsY.jpeg
Wenbo Hu
user
wbhu-tc
67ec958ebb1d6dd924f94a34
Xiaoyu Li
67ec958ebb1d6dd924f94a35
Song-Hai Zhang
67ec958ebb1d6dd924f94a36
Ying Shan
admin_assigned
2025-04-02T09:51:45.869Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
GeometryCrafter: Consistent Geometry Estimation for Open-world Videos
  with Diffusion Priors
Despite remarkable advancements in video depth estimation, existing methods
exhibit inherent limitations in achieving geometric fidelity through the
affine-invariant predictions, limiting their applicability in reconstruction
and other metrically grounded downstream tasks. We propose GeometryCrafter, a
novel framework that recovers high-fidelity point map sequences with temporal
coherence from open-world videos, enabling accurate 3D/4D reconstruction,
camera parameter estimation, and other depth-based applications. At the core of
our approach lies a point map Variational Autoencoder (VAE) that learns a
latent space agnostic to video latent distributions for effective point map
encoding and decoding. Leveraging the VAE, we train a video diffusion model to
model the distribution of point map sequences conditioned on the input videos.
Extensive evaluations on diverse datasets demonstrate that GeometryCrafter
achieves state-of-the-art 3D accuracy, temporal consistency, and generalization
capability.
GeometryCrafter uses a point map VAE and video diffusion model to estimate high-fidelity, temporally coherent depth maps from open-world videos, enhancing 3D reconstruction and camera parameter estimation.
point map VAE video diffusion model high-fidelity point maps temporal coherence 3D/4D reconstruction camera parameter estimation latent space distribution modeling