title: TCAN: Animating Human Images with Temporally Consistent Pose Guidance
    using Diffusion Models
summary: Pose-driven human-image animation diffusion models have shown remarkable
  capabilities in realistic human video synthesis. Despite the promising results
  achieved by previous approaches, challenges persist in achieving temporally
  consistent animation and ensuring robustness with off-the-shelf pose detectors.
  In this paper, we present TCAN, a pose-driven human image animation method that
  is robust to erroneous poses and consistent over time. In contrast to previous
  methods, we utilize the pre-trained ControlNet without fine-tuning to leverage
  its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To
  keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the
  network to align the latent space between the pose and appearance features.
  Additionally, by introducing an additional temporal layer to the ControlNet, we
  enhance robustness against outliers of the pose detector. Through the analysis
  of attention maps over the temporal axis, we also designed a novel temperature
  map leveraging pose information, allowing for a more static background.
  Extensive experiments demonstrate that the proposed method can achieve
  promising results in video synthesis tasks encompassing various poses, like
  chibi. Project Page: https://eccv2024tcan.github.io/
paper_authors._id: 669485a33ceee2f1203f6bef
paper_authors.name: Jeongho Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:08.802Z
paper_authors._id: 6100cc3189e20ab255c4a243
paper_authors.avatarUrl: /avatars/36a30c0f9e03488b24313be4c1a6ed59.svg
paper_authors.fullname: Jeongho Kim
paper_authors.type: user
paper_authors.user: rlawjdghek
paper_authors._id: 669485a33ceee2f1203f6bf0
paper_authors.name: Min-Jung Kim
paper_authors._id: 669485a33ceee2f1203f6bf1
paper_authors.name: Junsoo Lee
paper_authors._id: 669485a33ceee2f1203f6bf2
paper_authors.name: Jaegul Choo
paper_title: TCAN: Animating Human Images with Temporally Consistent Pose Guidance
    using Diffusion Models
paper_summary: Pose-driven human-image animation diffusion models have shown remarkable
  capabilities in realistic human video synthesis. Despite the promising results
  achieved by previous approaches, challenges persist in achieving temporally
  consistent animation and ensuring robustness with off-the-shelf pose detectors.
  In this paper, we present TCAN, a pose-driven human image animation method that
  is robust to erroneous poses and consistent over time. In contrast to previous
  methods, we utilize the pre-trained ControlNet without fine-tuning to leverage
  its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To
  keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the
  network to align the latent space between the pose and appearance features.
  Additionally, by introducing an additional temporal layer to the ControlNet, we
  enhance robustness against outliers of the pose detector. Through the analysis
  of attention maps over the temporal axis, we also designed a novel temperature
  map leveraging pose information, allowing for a more static background.
  Extensive experiments demonstrate that the proposed method can achieve
  promising results in video synthesis tasks encompassing various poses, like
  chibi. Project Page: https://eccv2024tcan.github.io/
paper_ai_keywords: Pose-driven human-image animation diffusion models Realistic human video synthesis Temporally consistent animation Off-the-shelf pose detectors TCAN ControlNet LoRA UNet layers Latent space Appearance features Temporal layer Outliers of the pose detector Attention maps Temperature map Chibi