title: AnimateLCM: Accelerating the Animation of Personalized Diffusion Models
    and Adapters with Decoupled Consistency Learning
summary: Video diffusion models has been gaining increasing attention for its ability
  to produce videos that are both coherent and of high fidelity. However, the
  iterative denoising process makes it computationally intensive and
  time-consuming, thus limiting its applications. Inspired by the Consistency
  Model (CM) that distills pretrained image diffusion models to accelerate the
  sampling with minimal steps and its successful extension Latent Consistency
  Model (LCM) on conditional image generation, we propose AnimateLCM, allowing
  for high-fidelity video generation within minimal steps. Instead of directly
  conducting consistency learning on the raw video dataset, we propose a
  decoupled consistency learning strategy that decouples the distillation of
  image generation priors and motion generation priors, which improves the
  training efficiency and enhance the generation visual quality. Additionally, to
  enable the combination of plug-and-play adapters in stable diffusion community
  to achieve various functions (e.g., ControlNet for controllable generation). we
  propose an efficient strategy to adapt existing adapters to our distilled
  text-conditioned video consistency model or train adapters from scratch without
  harming the sampling speed. We validate the proposed strategy in
  image-conditioned video generation and layout-conditioned video generation, all
  achieving top-performing results. Experimental results validate the
  effectiveness of our proposed method. Code and weights will be made public.
  More details are available at https://github.com/G-U-N/AnimateLCM.
paper_authors._id: 65bc4ffbb54ab5b37d692f8b
paper_authors.name: Fu-Yun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T02:44:09.816Z
paper_authors._id: 63e9e92f20c109718713f5eb
paper_authors.avatarUrl: /avatars/9ff312e854d803e1a2e9e685a21d12f8.svg
paper_authors.fullname: Fu-Yun Wang
paper_authors.type: user
paper_authors.user: wangfuyun
paper_authors._id: 65bc4ffbb54ab5b37d692f8c
paper_authors.name: Zhaoyang Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:25:35.678Z
paper_authors._id: 64c26e0e400274d129a10d5d
paper_authors.avatarUrl: /avatars/c1debf3d7f61c2ea01d9a25d7e332712.svg
paper_authors.fullname: Zhaoyang Huang
paper_authors.type: user
paper_authors.user: DrinkingCoder
paper_authors._id: 65bc4ffbb54ab5b37d692f8d
paper_authors.name: Xiaoyu Shi
paper_authors._id: 65bc4ffbb54ab5b37d692f8e
paper_authors.name: Weikang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:26:10.126Z
paper_authors._id: 65b8b840a81930c593073370
paper_authors.avatarUrl: /avatars/31b399d3da33e8c02f9e06a16dd85779.svg
paper_authors.fullname: Weikang BIAN
paper_authors.type: user
paper_authors.user: wkbian
paper_authors._id: 65bc4ffbb54ab5b37d692f8f
paper_authors.name: Guanglu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:26:20.839Z
paper_authors._id: 636a2efffbc31ee68fee5c1c
paper_authors.avatarUrl: /avatars/21e119f23e1e6fe926567409e74d1a89.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Guanglu
paper_authors._id: 65bc4ffbb54ab5b37d692f90
paper_authors.name: Yu Liu
paper_authors._id: 65bc4ffbb54ab5b37d692f91
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:27:53.119Z
paper_authors._id: 64d97a3a67c967b01512dd1c
paper_authors.avatarUrl: /avatars/8a885f9290d4f42de4c481bcfcae8d85.svg
paper_authors.fullname: Hongsheng Liu
paper_authors.type: user
paper_authors.user: hsliu
paper_title: AnimateLCM: Accelerating the Animation of Personalized Diffusion Models
    and Adapters with Decoupled Consistency Learning
paper_summary: Video diffusion models has been gaining increasing attention for its ability
  to produce videos that are both coherent and of high fidelity. However, the
  iterative denoising process makes it computationally intensive and
  time-consuming, thus limiting its applications. Inspired by the Consistency
  Model (CM) that distills pretrained image diffusion models to accelerate the
  sampling with minimal steps and its successful extension Latent Consistency
  Model (LCM) on conditional image generation, we propose AnimateLCM, allowing
  for high-fidelity video generation within minimal steps. Instead of directly
  conducting consistency learning on the raw video dataset, we propose a
  decoupled consistency learning strategy that decouples the distillation of
  image generation priors and motion generation priors, which improves the
  training efficiency and enhance the generation visual quality. Additionally, to
  enable the combination of plug-and-play adapters in stable diffusion community
  to achieve various functions (e.g., ControlNet for controllable generation). we
  propose an efficient strategy to adapt existing adapters to our distilled
  text-conditioned video consistency model or train adapters from scratch without
  harming the sampling speed. We validate the proposed strategy in
  image-conditioned video generation and layout-conditioned video generation, all
  achieving top-performing results. Experimental results validate the
  effectiveness of our proposed method. Code and weights will be made public.
  More details are available at https://github.com/G-U-N/AnimateLCM.
paper_ai_keywords: Video diffusion models Coherent High fidelity Iterative denoising process Consistency Model (CM) Latent Consistency Model (LCM) Conditional image generation AnimateLCM Decoupled consistency learning strategy Image generation priors Motion generation priors Consistency learning Raw video dataset Plug-and-play adapters Stable diffusion community ControlNet Controllable generation Text-conditioned video consistency model Distilled video consistency model Image-conditioned video generation Layout-conditioned video generation