title: DisPose: Disentangling Pose Guidance for Controllable Human Image
    Animation
summary: Controllable human image animation aims to generate videos from reference
  images using driving videos. Due to the limited control signals provided by
  sparse guidance (e.g., skeleton pose), recent works have attempted to introduce
  additional dense conditions (e.g., depth map) to ensure motion alignment.
  However, such strict dense guidance impairs the quality of the generated video
  when the body shape of the reference character differs significantly from that
  of the driving video. In this paper, we present DisPose to mine more
  generalizable and effective control signals without additional dense input,
  which disentangles the sparse skeleton pose in human image animation into
  motion field guidance and keypoint correspondence. Specifically, we generate a
  dense motion field from a sparse motion field and the reference image, which
  provides region-level dense guidance while maintaining the generalization of
  the sparse pose control. We also extract diffusion features corresponding to
  pose keypoints from the reference image, and then these point features are
  transferred to the target pose to provide distinct identity information. To
  seamlessly integrate into existing models, we propose a plug-and-play hybrid
  ControlNet that improves the quality and consistency of generated videos while
  freezing the existing model parameters. Extensive qualitative and quantitative
  experiments demonstrate the superiority of DisPose compared to current methods.
  Code:
  https://github.com/lihxxx/DisPose{https://github.com/lihxxx/DisPose}.
paper_authors._id: 675ba7567a3e15dbc1834a8a
paper_authors.name: Hongxiang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:59:26.419Z
paper_authors._id: 653eb3bd4a52f10eaf72fbaf
paper_authors.avatarUrl: /avatars/b525482b61c6f6054bf44bbc3113c29f.svg
paper_authors.fullname: Hongxiang Li
paper_authors.type: user
paper_authors.user: HongxiangLi
paper_authors._id: 675ba7567a3e15dbc1834a8b
paper_authors.name: Yaowei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:59:32.938Z
paper_authors._id: 6362801380c1a705a6ea54ac
paper_authors.avatarUrl: /avatars/041ad5abf9be42e336938f51ebb8746c.svg
paper_authors.fullname: Yaowei Li
paper_authors.type: user
paper_authors.user: Yw22
paper_authors._id: 675ba7567a3e15dbc1834a8c
paper_authors.name: Yuhang Yang
paper_authors._id: 675ba7567a3e15dbc1834a8d
paper_authors.name: Junjie Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:00:01.835Z
paper_authors._id: 659a39b95f7a6d40f75404ec
paper_authors.avatarUrl: /avatars/791a6e8f31f0d0eead06374898a0ded7.svg
paper_authors.fullname: Junjie Cao
paper_authors.type: user
paper_authors.user: flyingtom
paper_authors._id: 675ba7567a3e15dbc1834a8e
paper_authors.name: Zhihong Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:00:07.949Z
paper_authors._id: 64b78e70da8017900e7d1972
paper_authors.avatarUrl: /avatars/f862e7f98cfc3ffb820309ffe7bc73d3.svg
paper_authors.fullname: zhihong zhu
paper_authors.type: user
paper_authors.user: PKUGGBond
paper_authors._id: 675ba7567a3e15dbc1834a8f
paper_authors.name: Xuxin Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:00:36.457Z
paper_authors._id: 65016826cc88b84c06e2582a
paper_authors.avatarUrl: /avatars/d3dba1cb0d2e4386f0a63c16f9cc0a13.svg
paper_authors.fullname: Xuxin Cheng
paper_authors.type: user
paper_authors.user: chengxx-pku
paper_authors._id: 675ba7567a3e15dbc1834a90
paper_authors.name: Long Chen
paper_title: DisPose: Disentangling Pose Guidance for Controllable Human Image
    Animation
paper_summary: Controllable human image animation aims to generate videos from reference
  images using driving videos. Due to the limited control signals provided by
  sparse guidance (e.g., skeleton pose), recent works have attempted to introduce
  additional dense conditions (e.g., depth map) to ensure motion alignment.
  However, such strict dense guidance impairs the quality of the generated video
  when the body shape of the reference character differs significantly from that
  of the driving video. In this paper, we present DisPose to mine more
  generalizable and effective control signals without additional dense input,
  which disentangles the sparse skeleton pose in human image animation into
  motion field guidance and keypoint correspondence. Specifically, we generate a
  dense motion field from a sparse motion field and the reference image, which
  provides region-level dense guidance while maintaining the generalization of
  the sparse pose control. We also extract diffusion features corresponding to
  pose keypoints from the reference image, and then these point features are
  transferred to the target pose to provide distinct identity information. To
  seamlessly integrate into existing models, we propose a plug-and-play hybrid
  ControlNet that improves the quality and consistency of generated videos while
  freezing the existing model parameters. Extensive qualitative and quantitative
  experiments demonstrate the superiority of DisPose compared to current methods.
  Code:
  https://github.com/lihxxx/DisPose{https://github.com/lihxxx/DisPose}.
paper_ai_keywords: human image animation driving videos sparse guidance skeleton pose dense conditions depth map motion alignment DisPose motion field guidance keypoint features diffusion features ControlNet qualitative experiments quantitative experiments