title: Adaptive Caching for Faster Video Generation with Diffusion Transformers
summary: Generating temporally-consistent high-fidelity videos can be computationally
  expensive, especially over longer temporal spans. More-recent Diffusion
  Transformers (DiTs) -- despite making significant headway in this context --
  have only heightened such challenges as they rely on larger models and heavier
  attention mechanisms, resulting in slower inference speeds. In this paper, we
  introduce a training-free method to accelerate video DiTs, termed Adaptive
  Caching (AdaCache), which is motivated by the fact that "not all videos are
  created equal": meaning, some videos require fewer denoising steps to attain a
  reasonable quality than others. Building on this, we not only cache
  computations through the diffusion process, but also devise a caching schedule
  tailored to each video generation, maximizing the quality-latency trade-off. We
  further introduce a Motion Regularization (MoReg) scheme to utilize video
  information within AdaCache, essentially controlling the compute allocation
  based on motion content. Altogether, our plug-and-play contributions grant
  significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video
  generation) without sacrificing the generation quality, across multiple video
  DiT baselines.
paper_authors._id: 672992dd829cb1cce1201665
paper_authors.name: Kumara Kahatapitiya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:57:47.053Z
paper_authors._id: 65a12ec4c9873890d15c4ab9
paper_authors.avatarUrl: /avatars/ce4d46f575ba757f78eabdb25b394171.svg
paper_authors.fullname: Kumara Kahatapitiya
paper_authors.type: user
paper_authors.user: kumarak
paper_authors._id: 672992dd829cb1cce1201666
paper_authors.name: Haozhe Liu
paper_authors._id: 672992dd829cb1cce1201667
paper_authors.name: Sen He
paper_authors._id: 672992dd829cb1cce1201668
paper_authors.name: Ding Liu
paper_authors._id: 672992dd829cb1cce1201669
paper_authors.name: Menglin Jia
paper_authors._id: 672992dd829cb1cce120166a
paper_authors.name: Michael S. Ryoo
paper_authors._id: 672992dd829cb1cce120166b
paper_authors.name: Tian Xie
paper_title: Adaptive Caching for Faster Video Generation with Diffusion Transformers
paper_summary: Generating temporally-consistent high-fidelity videos can be computationally
  expensive, especially over longer temporal spans. More-recent Diffusion
  Transformers (DiTs) -- despite making significant headway in this context --
  have only heightened such challenges as they rely on larger models and heavier
  attention mechanisms, resulting in slower inference speeds. In this paper, we
  introduce a training-free method to accelerate video DiTs, termed Adaptive
  Caching (AdaCache), which is motivated by the fact that "not all videos are
  created equal": meaning, some videos require fewer denoising steps to attain a
  reasonable quality than others. Building on this, we not only cache
  computations through the diffusion process, but also devise a caching schedule
  tailored to each video generation, maximizing the quality-latency trade-off. We
  further introduce a Motion Regularization (MoReg) scheme to utilize video
  information within AdaCache, essentially controlling the compute allocation
  based on motion content. Altogether, our plug-and-play contributions grant
  significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video
  generation) without sacrificing the generation quality, across multiple video
  DiT baselines.
paper_ai_summary: Adaptive Caching and Motion Regularization accelerate video diffusion transformers by optimizing computational resources based on video content and denoising needs.
paper_ai_keywords: diffusion Transformers Adaptive Caching caching schedule Motion Regularization video generation inference speedups denoising steps