AnimateLCM: Accelerating the Animation of Personalized Diffusion Models
  and Adapters with Decoupled Consistency Learning
Video diffusion models has been gaining increasing attention for its ability
to produce videos that are both coherent and of high fidelity. However, the
iterative denoising process makes it computationally intensive and
time-consuming, thus limiting its applications. Inspired by the Consistency
Model (CM) that distills pretrained image diffusion models to accelerate the
sampling with minimal steps and its successful extension Latent Consistency
Model (LCM) on conditional image generation, we propose AnimateLCM, allowing
for high-fidelity video generation within minimal steps. Instead of directly
conducting consistency learning on the raw video dataset, we propose a
decoupled consistency learning strategy that decouples the distillation of
image generation priors and motion generation priors, which improves the
training efficiency and enhance the generation visual quality. Additionally, to
enable the combination of plug-and-play adapters in stable diffusion community
to achieve various functions (e.g., ControlNet for controllable generation). we
propose an efficient strategy to adapt existing adapters to our distilled
text-conditioned video consistency model or train adapters from scratch without
harming the sampling speed. We validate the proposed strategy in
image-conditioned video generation and layout-conditioned video generation, all
achieving top-performing results. Experimental results validate the
effectiveness of our proposed method. Code and weights will be made public.
More details are available at https://github.com/G-U-N/AnimateLCM.
65bc4ffbb54ab5b37d692f8b
Fu-Yun Wang
claimed_verified
2024-02-02T02:44:09.816Z
63e9e92f20c109718713f5eb
/avatars/9ff312e854d803e1a2e9e685a21d12f8.svg
Fu-Yun Wang
user
wangfuyun
65bc4ffbb54ab5b37d692f8c
Zhaoyang Huang
admin_assigned
2024-02-02T11:25:35.678Z
64c26e0e400274d129a10d5d
/avatars/c1debf3d7f61c2ea01d9a25d7e332712.svg
Zhaoyang Huang
user
DrinkingCoder
65bc4ffbb54ab5b37d692f8d
Xiaoyu Shi
65bc4ffbb54ab5b37d692f8e
Weikang Bian
admin_assigned
2024-02-02T11:26:10.126Z
65b8b840a81930c593073370
/avatars/31b399d3da33e8c02f9e06a16dd85779.svg
Weikang BIAN
user
wkbian
65bc4ffbb54ab5b37d692f8f
Guanglu Song
admin_assigned
2024-02-02T11:26:20.839Z
636a2efffbc31ee68fee5c1c
/avatars/21e119f23e1e6fe926567409e74d1a89.svg
Song
user
Guanglu
65bc4ffbb54ab5b37d692f90
Yu Liu
65bc4ffbb54ab5b37d692f91
Hongsheng Li
admin_assigned
2024-02-02T11:27:53.119Z
64d97a3a67c967b01512dd1c
/avatars/8a885f9290d4f42de4c481bcfcae8d85.svg
Hongsheng Liu
user
hsliu
AnimateLCM: Accelerating the Animation of Personalized Diffusion Models
  and Adapters with Decoupled Consistency Learning
Video diffusion models has been gaining increasing attention for its ability
to produce videos that are both coherent and of high fidelity. However, the
iterative denoising process makes it computationally intensive and
time-consuming, thus limiting its applications. Inspired by the Consistency
Model (CM) that distills pretrained image diffusion models to accelerate the
sampling with minimal steps and its successful extension Latent Consistency
Model (LCM) on conditional image generation, we propose AnimateLCM, allowing
for high-fidelity video generation within minimal steps. Instead of directly
conducting consistency learning on the raw video dataset, we propose a
decoupled consistency learning strategy that decouples the distillation of
image generation priors and motion generation priors, which improves the
training efficiency and enhance the generation visual quality. Additionally, to
enable the combination of plug-and-play adapters in stable diffusion community
to achieve various functions (e.g., ControlNet for controllable generation). we
propose an efficient strategy to adapt existing adapters to our distilled
text-conditioned video consistency model or train adapters from scratch without
harming the sampling speed. We validate the proposed strategy in
image-conditioned video generation and layout-conditioned video generation, all
achieving top-performing results. Experimental results validate the
effectiveness of our proposed method. Code and weights will be made public.
More details are available at https://github.com/G-U-N/AnimateLCM.
AnimateLCM, an extension of Latent Consistency Model, achieves high-fidelity video generation efficiently with decoupled consistency learning and adaptability to existing adapters, outperforming in image-conditioned and layout-conditioned video generation.
video diffusion models consistency model latent consistency model AnimateLCM consistency learning image generation priors motion generation priors plug-and-play adapters ControlNet text-conditioned video consistency model image-conditioned video generation layout-conditioned video generation