title: Human4DiT: Free-view Human Video Generation with 4D Diffusion
    Transformer
summary: We present a novel approach for generating high-quality, spatio-temporally
  coherent human videos from a single image under arbitrary viewpoints. Our
  framework combines the strengths of U-Nets for accurate condition injection and
  diffusion transformers for capturing global correlations across viewpoints and
  time. The core is a cascaded 4D transformer architecture that factorizes
  attention across views, time, and spatial dimensions, enabling efficient
  modeling of the 4D space. Precise conditioning is achieved by injecting human
  identity, camera parameters, and temporal signals into the respective
  transformers. To train this model, we curate a multi-dimensional dataset
  spanning images, videos, multi-view data and 3D/4D scans, along with a
  multi-dimensional training strategy. Our approach overcomes the limitations of
  previous methods based on GAN or UNet-based diffusion models, which struggle
  with complex motions and viewpoint changes. Through extensive experiments, we
  demonstrate our method's ability to synthesize realistic, coherent and
  free-view human videos, paving the way for advanced multimedia applications in
  areas such as virtual reality and animation. Our project website is
  https://human4dit.github.io.
paper_authors._id: 665549f95ca1e0473bfd8707
paper_authors.name: Ruizhi Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:18:57.267Z
paper_authors._id: 64785ad575a9a0cfabcf5083
paper_authors.avatarUrl: /avatars/5b8526bf2d8389310ca1799655cbbcd8.svg
paper_authors.fullname: Ruizhi Shao
paper_authors.type: user
paper_authors.user: DSaurus
paper_authors._id: 665549f95ca1e0473bfd8708
paper_authors.name: Youxin Pang
paper_authors._id: 665549f95ca1e0473bfd8709
paper_authors.name: Zerong Zheng
paper_authors._id: 665549f95ca1e0473bfd870a
paper_authors.name: Jingxiang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:10:19.626Z
paper_authors._id: 6456018a95082f722d136946
paper_authors.avatarUrl: /avatars/c9f825b9ac0af1e6f615ddcdd9a771ac.svg
paper_authors.fullname: jingxiang sun
paper_authors.type: user
paper_authors.user: starksun
paper_authors._id: 665549f95ca1e0473bfd870b
paper_authors.name: Yebin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:10:25.792Z
paper_authors._id: 62e14dbe4db2175cd2735a80
paper_authors.avatarUrl: /avatars/e6385dcedcb97e1b36281b49210321aa.svg
paper_authors.fullname: Yebin Liu
paper_authors.type: user
paper_authors.user: YebinLiu
paper_title: Human4DiT: Free-view Human Video Generation with 4D Diffusion
    Transformer
paper_summary: We present a novel approach for generating high-quality, spatio-temporally
  coherent human videos from a single image under arbitrary viewpoints. Our
  framework combines the strengths of U-Nets for accurate condition injection and
  diffusion transformers for capturing global correlations across viewpoints and
  time. The core is a cascaded 4D transformer architecture that factorizes
  attention across views, time, and spatial dimensions, enabling efficient
  modeling of the 4D space. Precise conditioning is achieved by injecting human
  identity, camera parameters, and temporal signals into the respective
  transformers. To train this model, we curate a multi-dimensional dataset
  spanning images, videos, multi-view data and 3D/4D scans, along with a
  multi-dimensional training strategy. Our approach overcomes the limitations of
  previous methods based on GAN or UNet-based diffusion models, which struggle
  with complex motions and viewpoint changes. Through extensive experiments, we
  demonstrate our method's ability to synthesize realistic, coherent and
  free-view human videos, paving the way for advanced multimedia applications in
  areas such as virtual reality and animation. Our project website is
  https://human4dit.github.io.
paper_ai_keywords: U-Nets diffusion transformers cascaded 4D transformer architecture attention human identity camera parameters temporal signals multi-dimensional dataset multi-dimensional training strategy GAN UNet-based diffusion models realistic coherent free-view human videos virtual reality animation