Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal
  Consistency
We present Free4D, a novel tuning-free framework for 4D scene generation from
a single image. Existing methods either focus on object-level generation,
making scene-level generation infeasible, or rely on large-scale multi-view
video datasets for expensive training, with limited generalization ability due
to the scarcity of 4D scene data. In contrast, our key insight is to distill
pre-trained foundation models for consistent 4D scene representation, which
offers promising advantages such as efficiency and generalizability. 1) To
achieve this, we first animate the input image using image-to-video diffusion
models followed by 4D geometric structure initialization. 2) To turn this
coarse structure into spatial-temporal consistent multiview videos, we design
an adaptive guidance mechanism with a point-guided denoising strategy for
spatial consistency and a novel latent replacement strategy for temporal
coherence. 3) To lift these generated observations into consistent 4D
representation, we propose a modulation-based refinement to mitigate
inconsistencies while fully leveraging the generated information. The resulting
4D representation enables real-time, controllable rendering, marking a
significant advancement in single-image-based 4D scene generation.
67e7b4ba05d7355e476f4a10
Tianqi Liu
claimed_verified
2025-03-31T08:12:50.975Z
66d347eebb76fb26eedb256e
https://cdn-avatars.huggingface.co/v1/production/uploads/66d347eebb76fb26eedb256e/iCPF7GkmZu--XCsWzoucl.jpeg
tianqi liu
user
tqliu
67e7b4ba05d7355e476f4a11
Zihao Huang
admin_assigned
2025-03-31T08:33:22.318Z
631b24f2f6bc4be4a64c4d43
https://cdn-avatars.huggingface.co/v1/production/uploads/631b24f2f6bc4be4a64c4d43/P9_tVF7SESmVxxGKVCgCk.jpeg
Zihao Huang
user
Inso
67e7b4ba05d7355e476f4a12
Zhaoxi Chen
admin_assigned
2025-03-31T08:33:43.896Z
62fc8cf7ee999004b5a8b982
/avatars/6c5dda9e58747054a989f077a078f3dc.svg
Zhaoxi Chen
user
FrozenBurning
67e7b4ba05d7355e476f4a13
Guangcong Wang
admin_assigned
2025-03-31T08:33:54.167Z
62e893da40bd989bb71b8f89
/avatars/34e755d1124303a498429a3c4d01367b.svg
Guangcong Wang
user
GuangcongWang
67e7b4ba05d7355e476f4a14
Shoukang Hu
admin_assigned
2025-03-31T08:34:21.036Z
6503be91a450492f84314af8
/avatars/ef94efdad0bc8a423262d25a8cf77e41.svg
Shoukang Hu
user
skhu101
67e7b4ba05d7355e476f4a15
Liao Shen
67e7b4ba05d7355e476f4a16
Huiqiang Sun
67e7b4ba05d7355e476f4a17
Zhiguo Cao
67e7b4ba05d7355e476f4a18
Wei Li
67e7b4ba05d7355e476f4a19
Ziwei Liu
admin_assigned
2025-03-31T08:34:47.540Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal
  Consistency
We present Free4D, a novel tuning-free framework for 4D scene generation from
a single image. Existing methods either focus on object-level generation,
making scene-level generation infeasible, or rely on large-scale multi-view
video datasets for expensive training, with limited generalization ability due
to the scarcity of 4D scene data. In contrast, our key insight is to distill
pre-trained foundation models for consistent 4D scene representation, which
offers promising advantages such as efficiency and generalizability. 1) To
achieve this, we first animate the input image using image-to-video diffusion
models followed by 4D geometric structure initialization. 2) To turn this
coarse structure into spatial-temporal consistent multiview videos, we design
an adaptive guidance mechanism with a point-guided denoising strategy for
spatial consistency and a novel latent replacement strategy for temporal
coherence. 3) To lift these generated observations into consistent 4D
representation, we propose a modulation-based refinement to mitigate
inconsistencies while fully leveraging the generated information. The resulting
4D representation enables real-time, controllable rendering, marking a
significant advancement in single-image-based 4D scene generation.
A novel tuning-free framework Free4D generates 4D scenes from a single image using image-to-video diffusion models, adaptive guidance, and a modulation-based refinement to ensure spatial and temporal consistency.
Free4D image-to-video diffusion models adaptive guidance point-guided denoising latent replacement modulation-based refinement 4D scene generation spatial consistency temporal coherence