Wonderland: Navigating 3D Scenes from a Single Image
This paper addresses a challenging question: How can we efficiently create
high-quality, wide-scope 3D scenes from a single arbitrary image? Existing
methods face several constraints, such as requiring multi-view data,
time-consuming per-scene optimization, low visual quality in backgrounds, and
distorted reconstructions in unseen areas. We propose a novel pipeline to
overcome these limitations. Specifically, we introduce a large-scale
reconstruction model that uses latents from a video diffusion model to predict
3D Gaussian Splattings for the scenes in a feed-forward manner. The video
diffusion model is designed to create videos precisely following specified
camera trajectories, allowing it to generate compressed video latents that
contain multi-view information while maintaining 3D consistency. We train the
3D reconstruction model to operate on the video latent space with a progressive
training strategy, enabling the efficient generation of high-quality,
wide-scope, and generic 3D scenes. Extensive evaluations across various
datasets demonstrate that our model significantly outperforms existing methods
for single-view 3D scene generation, particularly with out-of-domain images.
For the first time, we demonstrate that a 3D reconstruction model can be
effectively built upon the latent space of a diffusion model to realize
efficient 3D scene generation.
67610a192504d8a0fbf6a83a
Hanwen Liang
claimed_verified
2024-12-18T09:53:00.045Z
64cb2227e984d09bed90fdc5
/avatars/15b04cfbd935bd01c84020c8a8e6c8c3.svg
Hanwen Liang
user
hw-liang
67610a192504d8a0fbf6a83b
Junli Cao
admin_assigned
2024-12-17T11:19:05.096Z
63f54aa73aa49d8cb97b84bc
/avatars/c73c5870039611ab9162daad46a1ba20.svg
junli cao
user
jlcao2
67610a192504d8a0fbf6a83c
Vidit Goel
admin_assigned
2024-12-17T11:19:00.007Z
636c0c1a15cd58e915bb8139
/avatars/7c675ac6a7d303d3425e498c4e939eb0.svg
Vidit Goel
user
vidit98
67610a192504d8a0fbf6a83d
Guocheng Qian
admin_assigned
2024-12-17T11:18:51.075Z
645fed74335c21d19f3bf76c
https://cdn-avatars.huggingface.co/v1/production/uploads/645fed74335c21d19f3bf76c/gwVsllRWtSHbg4a1erkdF.jpeg
Guocheng Qian
user
guochengqian
67610a192504d8a0fbf6a83e
Sergei Korolev
admin_assigned
2024-12-17T11:18:45.439Z
6338047fe1b78345008dcd22
/avatars/7306138e83ca642f1638a39efa169c7f.svg
Sergei Korolev
user
ZanyRumata
67610a192504d8a0fbf6a83f
Demetri Terzopoulos
67610a192504d8a0fbf6a840
Konstantinos N. Plataniotis
67610a192504d8a0fbf6a841
Sergey Tulyakov
67610a192504d8a0fbf6a842
Jian Ren
claimed_verified
2025-01-15T08:49:35.742Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
Wonderland: Navigating 3D Scenes from a Single Image
This paper addresses a challenging question: How can we efficiently create
high-quality, wide-scope 3D scenes from a single arbitrary image? Existing
methods face several constraints, such as requiring multi-view data,
time-consuming per-scene optimization, low visual quality in backgrounds, and
distorted reconstructions in unseen areas. We propose a novel pipeline to
overcome these limitations. Specifically, we introduce a large-scale
reconstruction model that uses latents from a video diffusion model to predict
3D Gaussian Splattings for the scenes in a feed-forward manner. The video
diffusion model is designed to create videos precisely following specified
camera trajectories, allowing it to generate compressed video latents that
contain multi-view information while maintaining 3D consistency. We train the
3D reconstruction model to operate on the video latent space with a progressive
training strategy, enabling the efficient generation of high-quality,
wide-scope, and generic 3D scenes. Extensive evaluations across various
datasets demonstrate that our model significantly outperforms existing methods
for single-view 3D scene generation, particularly with out-of-domain images.
For the first time, we demonstrate that a 3D reconstruction model can be
effectively built upon the latent space of a diffusion model to realize
efficient 3D scene generation.
A novel pipeline using latents from a video diffusion model to predict 3D scenes from single images achieves high quality and efficiency, outperforming existing methods.
video diffusion model 3D Gaussian Splattings feed-forward 3D reconstruction model video latent space progressive training single-view 3D scene generation out-of-domain images