Adaptive Caching for Faster Video Generation with Diffusion Transformers
Generating temporally-consistent high-fidelity videos can be computationally
expensive, especially over longer temporal spans. More-recent Diffusion
Transformers (DiTs) -- despite making significant headway in this context --
have only heightened such challenges as they rely on larger models and heavier
attention mechanisms, resulting in slower inference speeds. In this paper, we
introduce a training-free method to accelerate video DiTs, termed Adaptive
Caching (AdaCache), which is motivated by the fact that "not all videos are
created equal": meaning, some videos require fewer denoising steps to attain a
reasonable quality than others. Building on this, we not only cache
computations through the diffusion process, but also devise a caching schedule
tailored to each video generation, maximizing the quality-latency trade-off. We
further introduce a Motion Regularization (MoReg) scheme to utilize video
information within AdaCache, essentially controlling the compute allocation
based on motion content. Altogether, our plug-and-play contributions grant
significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video
generation) without sacrificing the generation quality, across multiple video
DiT baselines.
672992dd829cb1cce1201665
Kumara Kahatapitiya
claimed_verified
2024-11-05T07:57:47.053Z
65a12ec4c9873890d15c4ab9
/avatars/ce4d46f575ba757f78eabdb25b394171.svg
Kumara Kahatapitiya
user
kumarak
672992dd829cb1cce1201666
Haozhe Liu
672992dd829cb1cce1201667
Sen He
672992dd829cb1cce1201668
Ding Liu
672992dd829cb1cce1201669
Menglin Jia
672992dd829cb1cce120166a
Michael S. Ryoo
672992dd829cb1cce120166b
Tian Xie
Adaptive Caching for Faster Video Generation with Diffusion Transformers
Generating temporally-consistent high-fidelity videos can be computationally
expensive, especially over longer temporal spans. More-recent Diffusion
Transformers (DiTs) -- despite making significant headway in this context --
have only heightened such challenges as they rely on larger models and heavier
attention mechanisms, resulting in slower inference speeds. In this paper, we
introduce a training-free method to accelerate video DiTs, termed Adaptive
Caching (AdaCache), which is motivated by the fact that "not all videos are
created equal": meaning, some videos require fewer denoising steps to attain a
reasonable quality than others. Building on this, we not only cache
computations through the diffusion process, but also devise a caching schedule
tailored to each video generation, maximizing the quality-latency trade-off. We
further introduce a Motion Regularization (MoReg) scheme to utilize video
information within AdaCache, essentially controlling the compute allocation
based on motion content. Altogether, our plug-and-play contributions grant
significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video
generation) without sacrificing the generation quality, across multiple video
DiT baselines.
Adaptive Caching and Motion Regularization accelerate video diffusion transformers by optimizing computational resources based on video content and denoising needs.
diffusion Transformers Adaptive Caching caching schedule Motion Regularization video generation inference speedups denoising steps