title: OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human
    Animation Models
summary: End-to-end human animation, such as audio-driven talking human generation,
  has undergone notable advancements in the recent few years. However, existing
  methods still struggle to scale up as large general video generation models,
  limiting their potential in real applications. In this paper, we propose
  OmniHuman, a Diffusion Transformer-based framework that scales up data by
  mixing motion-related conditions into the training phase. To this end, we
  introduce two training principles for these mixed conditions, along with the
  corresponding model architecture and inference strategy. These designs enable
  OmniHuman to fully leverage data-driven motion generation, ultimately achieving
  highly realistic human video generation. More importantly, OmniHuman supports
  various portrait contents (face close-up, portrait, half-body, full-body),
  supports both talking and singing, handles human-object interactions and
  challenging body poses, and accommodates different image styles. Compared to
  existing end-to-end audio-driven methods, OmniHuman not only produces more
  realistic videos, but also offers greater flexibility in inputs. It also
  supports multiple driving modalities (audio-driven, video-driven and combined
  driving signals). Video samples are provided on the ttfamily project page
  (https://omnihuman-lab.github.io)
paper_authors._id: 67a1a7a166a8a88726963ef4
paper_authors.name: Gaojie Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:49:22.604Z
paper_authors._id: 64802fcdcc9e514b3b031244
paper_authors.avatarUrl: /avatars/cc5979008bdb21a2be9575865dce909b.svg
paper_authors.fullname: Gaojie Lin
paper_authors.type: user
paper_authors.user: lingaojie
paper_authors._id: 67a1a7a166a8a88726963ef5
paper_authors.name: Jianwen Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:15:17.797Z
paper_authors._id: 643410509bd5a84b5dca1345
paper_authors.avatarUrl: /avatars/597823ce61c7b1a5da77e178820824f6.svg
paper_authors.fullname: Jianwen Jiang
paper_authors.type: user
paper_authors.user: JianwenJ
paper_authors._id: 67a1a7a166a8a88726963ef6
paper_authors.name: Jiaqi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:01:37.757Z
paper_authors._id: 6527603743db8c626c467726
paper_authors.avatarUrl: /avatars/216df0374b37355d57d2c76e6f08e4d6.svg
paper_authors.fullname: yang
paper_authors.type: user
paper_authors.user: jiaqi78
paper_authors._id: 67a1a7a166a8a88726963ef7
paper_authors.name: Zerong Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:50:10.474Z
paper_authors._id: 65b0b6d00648e8d10b609066
paper_authors.avatarUrl: /avatars/071d2a99f6d7a4e37d338e58d46c4bc2.svg
paper_authors.fullname: ZZerong
paper_authors.type: user
paper_authors.user: zerong2
paper_authors._id: 67a1a7a166a8a88726963ef8
paper_authors.name: Chao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:15:22.815Z
paper_authors._id: 66d7ca4c0429a62c38c46bf5
paper_authors.avatarUrl: /avatars/45c7859627ac4be70c40f5b1fd02b18a.svg
paper_authors.fullname: liangdebugger
paper_authors.type: user
paper_authors.user: chao0412
paper_title: OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human
    Animation Models
paper_summary: End-to-end human animation, such as audio-driven talking human generation,
  has undergone notable advancements in the recent few years. However, existing
  methods still struggle to scale up as large general video generation models,
  limiting their potential in real applications. In this paper, we propose
  OmniHuman, a Diffusion Transformer-based framework that scales up data by
  mixing motion-related conditions into the training phase. To this end, we
  introduce two training principles for these mixed conditions, along with the
  corresponding model architecture and inference strategy. These designs enable
  OmniHuman to fully leverage data-driven motion generation, ultimately achieving
  highly realistic human video generation. More importantly, OmniHuman supports
  various portrait contents (face close-up, portrait, half-body, full-body),
  supports both talking and singing, handles human-object interactions and
  challenging body poses, and accommodates different image styles. Compared to
  existing end-to-end audio-driven methods, OmniHuman not only produces more
  realistic videos, but also offers greater flexibility in inputs. It also
  supports multiple driving modalities (audio-driven, video-driven and combined
  driving signals). Video samples are provided on the ttfamily project page
  (https://omnihuman-lab.github.io)
paper_ai_summary: OmniHuman, a Diffusion Transformer-based framework, achieves realistic human video generation by integrating motion-related conditions into its training process, supporting various inputs and driving modalities.
paper_ai_keywords: Diffusion Transformer motion-related conditions model architecture inference strategy data-driven motion generation human video generation portrait contents human-object interactions body poses image styles audio-driven video-driven combined driving signals