Human4DiT: Free-view Human Video Generation with 4D Diffusion
  Transformer
We present a novel approach for generating high-quality, spatio-temporally
coherent human videos from a single image under arbitrary viewpoints. Our
framework combines the strengths of U-Nets for accurate condition injection and
diffusion transformers for capturing global correlations across viewpoints and
time. The core is a cascaded 4D transformer architecture that factorizes
attention across views, time, and spatial dimensions, enabling efficient
modeling of the 4D space. Precise conditioning is achieved by injecting human
identity, camera parameters, and temporal signals into the respective
transformers. To train this model, we curate a multi-dimensional dataset
spanning images, videos, multi-view data and 3D/4D scans, along with a
multi-dimensional training strategy. Our approach overcomes the limitations of
previous methods based on GAN or UNet-based diffusion models, which struggle
with complex motions and viewpoint changes. Through extensive experiments, we
demonstrate our method's ability to synthesize realistic, coherent and
free-view human videos, paving the way for advanced multimedia applications in
areas such as virtual reality and animation. Our project website is
https://human4dit.github.io.
665549f95ca1e0473bfd8707
Ruizhi Shao
claimed_verified
2024-05-28T07:18:57.267Z
64785ad575a9a0cfabcf5083
/avatars/5b8526bf2d8389310ca1799655cbbcd8.svg
Ruizhi Shao
user
DSaurus
665549f95ca1e0473bfd8708
Youxin Pang
665549f95ca1e0473bfd8709
Zerong Zheng
665549f95ca1e0473bfd870a
Jingxiang Sun
admin_assigned
2024-05-28T10:10:19.626Z
6456018a95082f722d136946
/avatars/c9f825b9ac0af1e6f615ddcdd9a771ac.svg
jingxiang sun
user
starksun
665549f95ca1e0473bfd870b
Yebin Liu
admin_assigned
2024-05-28T10:10:25.792Z
62e14dbe4db2175cd2735a80
/avatars/e6385dcedcb97e1b36281b49210321aa.svg
Yebin Liu
user
YebinLiu
Human4DiT: Free-view Human Video Generation with 4D Diffusion
  Transformer
We present a novel approach for generating high-quality, spatio-temporally
coherent human videos from a single image under arbitrary viewpoints. Our
framework combines the strengths of U-Nets for accurate condition injection and
diffusion transformers for capturing global correlations across viewpoints and
time. The core is a cascaded 4D transformer architecture that factorizes
attention across views, time, and spatial dimensions, enabling efficient
modeling of the 4D space. Precise conditioning is achieved by injecting human
identity, camera parameters, and temporal signals into the respective
transformers. To train this model, we curate a multi-dimensional dataset
spanning images, videos, multi-view data and 3D/4D scans, along with a
multi-dimensional training strategy. Our approach overcomes the limitations of
previous methods based on GAN or UNet-based diffusion models, which struggle
with complex motions and viewpoint changes. Through extensive experiments, we
demonstrate our method's ability to synthesize realistic, coherent and
free-view human videos, paving the way for advanced multimedia applications in
areas such as virtual reality and animation. Our project website is
https://human4dit.github.io.
U-Nets diffusion transformers cascaded 4D transformer architecture attention human identity camera parameters temporal signals multi-dimensional dataset multi-dimensional training strategy GAN UNet-based diffusion models realistic coherent free-view human videos virtual reality animation