DisPose: Disentangling Pose Guidance for Controllable Human Image
  Animation
Controllable human image animation aims to generate videos from reference
images using driving videos. Due to the limited control signals provided by
sparse guidance (e.g., skeleton pose), recent works have attempted to introduce
additional dense conditions (e.g., depth map) to ensure motion alignment.
However, such strict dense guidance impairs the quality of the generated video
when the body shape of the reference character differs significantly from that
of the driving video. In this paper, we present DisPose to mine more
generalizable and effective control signals without additional dense input,
which disentangles the sparse skeleton pose in human image animation into
motion field guidance and keypoint correspondence. Specifically, we generate a
dense motion field from a sparse motion field and the reference image, which
provides region-level dense guidance while maintaining the generalization of
the sparse pose control. We also extract diffusion features corresponding to
pose keypoints from the reference image, and then these point features are
transferred to the target pose to provide distinct identity information. To
seamlessly integrate into existing models, we propose a plug-and-play hybrid
ControlNet that improves the quality and consistency of generated videos while
freezing the existing model parameters. Extensive qualitative and quantitative
experiments demonstrate the superiority of DisPose compared to current methods.
Code:
https://github.com/lihxxx/DisPose{https://github.com/lihxxx/DisPose}.
675ba7567a3e15dbc1834a8a
Hongxiang Li
admin_assigned
2024-12-16T10:59:26.419Z
653eb3bd4a52f10eaf72fbaf
/avatars/b525482b61c6f6054bf44bbc3113c29f.svg
Hongxiang Li
user
HongxiangLi
675ba7567a3e15dbc1834a8b
Yaowei Li
admin_assigned
2024-12-16T10:59:32.938Z
6362801380c1a705a6ea54ac
/avatars/041ad5abf9be42e336938f51ebb8746c.svg
Yaowei Li
user
Yw22
675ba7567a3e15dbc1834a8c
Yuhang Yang
675ba7567a3e15dbc1834a8d
Junjie Cao
admin_assigned
2024-12-16T11:00:01.835Z
659a39b95f7a6d40f75404ec
/avatars/791a6e8f31f0d0eead06374898a0ded7.svg
Junjie Cao
user
flyingtom
675ba7567a3e15dbc1834a8e
Zhihong Zhu
admin_assigned
2024-12-16T11:00:07.949Z
64b78e70da8017900e7d1972
/avatars/f862e7f98cfc3ffb820309ffe7bc73d3.svg
GGBond
user
PKUGGBond
675ba7567a3e15dbc1834a8f
Xuxin Cheng
admin_assigned
2024-12-16T11:00:36.457Z
65016826cc88b84c06e2582a
/avatars/d3dba1cb0d2e4386f0a63c16f9cc0a13.svg
Xuxin Cheng
user
chengxx-pku
675ba7567a3e15dbc1834a90
Long Chen
DisPose: Disentangling Pose Guidance for Controllable Human Image
  Animation
Controllable human image animation aims to generate videos from reference
images using driving videos. Due to the limited control signals provided by
sparse guidance (e.g., skeleton pose), recent works have attempted to introduce
additional dense conditions (e.g., depth map) to ensure motion alignment.
However, such strict dense guidance impairs the quality of the generated video
when the body shape of the reference character differs significantly from that
of the driving video. In this paper, we present DisPose to mine more
generalizable and effective control signals without additional dense input,
which disentangles the sparse skeleton pose in human image animation into
motion field guidance and keypoint correspondence. Specifically, we generate a
dense motion field from a sparse motion field and the reference image, which
provides region-level dense guidance while maintaining the generalization of
the sparse pose control. We also extract diffusion features corresponding to
pose keypoints from the reference image, and then these point features are
transferred to the target pose to provide distinct identity information. To
seamlessly integrate into existing models, we propose a plug-and-play hybrid
ControlNet that improves the quality and consistency of generated videos while
freezing the existing model parameters. Extensive qualitative and quantitative
experiments demonstrate the superiority of DisPose compared to current methods.
Code:
https://github.com/lihxxx/DisPose{https://github.com/lihxxx/DisPose}.
DisPose enhances human image animation by extracting dense motion field and diffusion features from sparse pose and reference images, improving video quality and consistency.
sparse skeleton pose dense conditions depth map motion field guidance keypoint correspondence dense motion field diffusion features ControlNet plug-and-play hybrid ControlNet