TCAN: Animating Human Images with Temporally Consistent Pose Guidance
  using Diffusion Models
Pose-driven human-image animation diffusion models have shown remarkable
capabilities in realistic human video synthesis. Despite the promising results
achieved by previous approaches, challenges persist in achieving temporally
consistent animation and ensuring robustness with off-the-shelf pose detectors.
In this paper, we present TCAN, a pose-driven human image animation method that
is robust to erroneous poses and consistent over time. In contrast to previous
methods, we utilize the pre-trained ControlNet without fine-tuning to leverage
its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To
keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the
network to align the latent space between the pose and appearance features.
Additionally, by introducing an additional temporal layer to the ControlNet, we
enhance robustness against outliers of the pose detector. Through the analysis
of attention maps over the temporal axis, we also designed a novel temperature
map leveraging pose information, allowing for a more static background.
Extensive experiments demonstrate that the proposed method can achieve
promising results in video synthesis tasks encompassing various poses, like
chibi. Project Page: https://eccv2024tcan.github.io/
669485a33ceee2f1203f6bef
Jeongho Kim
claimed_verified
2024-07-18T09:08:08.802Z
6100cc3189e20ab255c4a243
/avatars/36a30c0f9e03488b24313be4c1a6ed59.svg
Jeongho Kim
user
rlawjdghek
669485a33ceee2f1203f6bf0
Min-Jung Kim
669485a33ceee2f1203f6bf1
Junsoo Lee
669485a33ceee2f1203f6bf2
Jaegul Choo
TCAN: Animating Human Images with Temporally Consistent Pose Guidance
  using Diffusion Models
Pose-driven human-image animation diffusion models have shown remarkable
capabilities in realistic human video synthesis. Despite the promising results
achieved by previous approaches, challenges persist in achieving temporally
consistent animation and ensuring robustness with off-the-shelf pose detectors.
In this paper, we present TCAN, a pose-driven human image animation method that
is robust to erroneous poses and consistent over time. In contrast to previous
methods, we utilize the pre-trained ControlNet without fine-tuning to leverage
its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To
keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the
network to align the latent space between the pose and appearance features.
Additionally, by introducing an additional temporal layer to the ControlNet, we
enhance robustness against outliers of the pose detector. Through the analysis
of attention maps over the temporal axis, we also designed a novel temperature
map leveraging pose information, allowing for a more static background.
Extensive experiments demonstrate that the proposed method can achieve
promising results in video synthesis tasks encompassing various poses, like
chibi. Project Page: https://eccv2024tcan.github.io/
TCAN, a pose-driven video synthesis method using ControlNet and LoRA, enhances temporal consistency and robustness to erroneous poses.
Pose-driven human-image animation diffusion models video synthesis temporally consistent animation robustness off-the-shelf pose detectors TCAN ControlNet LoRA UNet layers latent space temporal layer attention maps temperature map chibi