title: Identity-Preserving Text-to-Video Generation by Frequency Decomposition
summary: Identity-preserving text-to-video (IPT2V) generation aims to create
  high-fidelity videos with consistent human identity. It is an important task in
  video generation but remains an open problem for generative models. This paper
  pushes the technical frontier of IPT2V in two directions that have not been
  resolved in literature: (1) A tuning-free pipeline without tedious case-by-case
  finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based
  control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V
  model to keep human identity consistent in the generated video. Inspired by
  prior findings in frequency analysis of diffusion transformers, it employs
  identity-control signals in the frequency domain, where facial features can be
  decomposed into low-frequency global features and high-frequency intrinsic
  features. First, from a low-frequency perspective, we introduce a global facial
  extractor, which encodes reference images and facial key points into a latent
  space, generating features enriched with low-frequency information. These
  features are then integrated into shallow layers of the network to alleviate
  training challenges associated with DiT. Second, from a high-frequency
  perspective, we design a local facial extractor to capture high-frequency
  details and inject them into transformer blocks, enhancing the model's ability
  to preserve fine-grained features. We propose a hierarchical training strategy
  to leverage frequency information for identity preservation, transforming a
  vanilla pre-trained video generation model into an IPT2V model. Extensive
  experiments demonstrate that our frequency-aware heuristic scheme provides an
  optimal control solution for DiT-based models. Thanks to this scheme, our
  ConsisID generates high-quality, identity-preserving videos, making strides
  towards more effective IPT2V.
paper_authors._id: 674698d29c25e8cefac0250f
paper_authors.name: Shenghai Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:50.317Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 674698d29c25e8cefac02510
paper_authors.name: Jinfa Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T11:36:16.287Z
paper_authors._id: 63f37af60be81bdc5d92eebb
paper_authors.avatarUrl: /avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Jinfa
paper_authors._id: 674698d29c25e8cefac02511
paper_authors.name: Xianyi He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-29T09:17:05.382Z
paper_authors._id: 6688f94517f8d8f81c2def5f
paper_authors.avatarUrl: /avatars/1064ccd62539a8ca2ec86fc87c818296.svg
paper_authors.fullname: hexianyi
paper_authors.type: user
paper_authors.user: pkuhexianyi
paper_authors._id: 674698d29c25e8cefac02512
paper_authors.name: Yunyuan Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-29T10:08:15.927Z
paper_authors._id: 646de6402fd5a8eb8c518aa6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646de6402fd5a8eb8c518aa6/HYWb8-fT1kTm-ROBr1-0X.jpeg
paper_authors.fullname: yunyangge
paper_authors.type: user
paper_authors.user: yunyangge
paper_authors._id: 674698d29c25e8cefac02513
paper_authors.name: Yujun Shi
paper_authors._id: 674698d29c25e8cefac02514
paper_authors.name: Liuhan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T11:35:39.975Z
paper_authors._id: 65c0588569429d85dc594a85
paper_authors.avatarUrl: /avatars/822203d982519b999247a53235e30c39.svg
paper_authors.fullname: liuhan
paper_authors.type: user
paper_authors.user: LiuhanChen
paper_authors._id: 674698d29c25e8cefac02515
paper_authors.name: Jiebo Luo
paper_authors._id: 674698d29c25e8cefac02516
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:11:13.193Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_title: Identity-Preserving Text-to-Video Generation by Frequency Decomposition
paper_summary: Identity-preserving text-to-video (IPT2V) generation aims to create
  high-fidelity videos with consistent human identity. It is an important task in
  video generation but remains an open problem for generative models. This paper
  pushes the technical frontier of IPT2V in two directions that have not been
  resolved in literature: (1) A tuning-free pipeline without tedious case-by-case
  finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based
  control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V
  model to keep human identity consistent in the generated video. Inspired by
  prior findings in frequency analysis of diffusion transformers, it employs
  identity-control signals in the frequency domain, where facial features can be
  decomposed into low-frequency global features and high-frequency intrinsic
  features. First, from a low-frequency perspective, we introduce a global facial
  extractor, which encodes reference images and facial key points into a latent
  space, generating features enriched with low-frequency information. These
  features are then integrated into shallow layers of the network to alleviate
  training challenges associated with DiT. Second, from a high-frequency
  perspective, we design a local facial extractor to capture high-frequency
  details and inject them into transformer blocks, enhancing the model's ability
  to preserve fine-grained features. We propose a hierarchical training strategy
  to leverage frequency information for identity preservation, transforming a
  vanilla pre-trained video generation model into an IPT2V model. Extensive
  experiments demonstrate that our frequency-aware heuristic scheme provides an
  optimal control solution for DiT-based models. Thanks to this scheme, our
  ConsisID generates high-quality, identity-preserving videos, making strides
  towards more effective IPT2V.
paper_ai_summary: ConsisID is a tuning-free, DiT-based IPT2V model that uses frequency-aware identity control to generate high-quality, identity-preserving videos by decomposing facial features into low and high-frequency elements.
paper_ai_keywords: DiT-based IPT2V identity-preserving frequency-aware heuristic control global facial extractor low-frequency latent space local facial extractor high-frequency hierarchical training strategy video generation