SmoothCache: A Universal Inference Acceleration Technique for Diffusion
  Transformers
Diffusion Transformers (DiT) have emerged as powerful generative models for
various tasks, including image, video, and speech synthesis. However, their
inference process remains computationally expensive due to the repeated
evaluation of resource-intensive attention and feed-forward modules. To address
this, we introduce SmoothCache, a model-agnostic inference acceleration
technique for DiT architectures. SmoothCache leverages the observed high
similarity between layer outputs across adjacent diffusion timesteps. By
analyzing layer-wise representation errors from a small calibration set,
SmoothCache adaptively caches and reuses key features during inference. Our
experiments demonstrate that SmoothCache achieves 8% to 71% speed up while
maintaining or even improving generation quality across diverse modalities. We
showcase its effectiveness on DiT-XL for image generation, Open-Sora for
text-to-video, and Stable Audio Open for text-to-audio, highlighting its
potential to enable real-time applications and broaden the accessibility of
powerful DiT models.
673c2472d1794741928fa132
Joseph Liu
extracted_pending
2024-11-19T05:39:04.787Z
65330e31d690f3012e0557a2
/avatars/c0969c9643a6dc70ff7b3e65c9412e54.svg
Eng Ann Joseph Liu
user
josephliu-roblox
673c2472d1794741928fa133
Joshua Geddes
673c2472d1794741928fa134
Ziyu Guo
extracted_pending
2024-11-19T05:39:04.787Z
648cce407f7821f063c140be
/avatars/6a458c35f34bfbceba6be8e5b6fb1437.svg
Ziyu Guo
user
helloterran
673c2472d1794741928fa135
Haomiao Jiang
673c2472d1794741928fa136
Mahesh Kumar Nandwana
extracted_pending
2024-11-19T05:39:04.787Z
667f437aa94430cd2e723346
/avatars/c7d71f6789756082e129b0b9cf9dcfda.svg
Mahesh Nandwana
user
mnandwana
SmoothCache: A Universal Inference Acceleration Technique for Diffusion
  Transformers
Diffusion Transformers (DiT) have emerged as powerful generative models for
various tasks, including image, video, and speech synthesis. However, their
inference process remains computationally expensive due to the repeated
evaluation of resource-intensive attention and feed-forward modules. To address
this, we introduce SmoothCache, a model-agnostic inference acceleration
technique for DiT architectures. SmoothCache leverages the observed high
similarity between layer outputs across adjacent diffusion timesteps. By
analyzing layer-wise representation errors from a small calibration set,
SmoothCache adaptively caches and reuses key features during inference. Our
experiments demonstrate that SmoothCache achieves 8% to 71% speed up while
maintaining or even improving generation quality across diverse modalities. We
showcase its effectiveness on DiT-XL for image generation, Open-Sora for
text-to-video, and Stable Audio Open for text-to-audio, highlighting its
potential to enable real-time applications and broaden the accessibility of
powerful DiT models.
SmoothCache accelerates DiT inference by caching key features, achieving significant speed-ups while maintaining quality across image, video, and audio tasks.
Diffusion Transformers DiT SmoothCache inference acceleration attention modules feed-forward modules layer-wise representation errors real-time applications