Region-Adaptive Sampling for Diffusion Transformers
Diffusion models (DMs) have become the leading choice for generative tasks
across diverse domains. However, their reliance on multiple sequential forward
passes significantly limits real-time performance. Previous acceleration
methods have primarily focused on reducing the number of sampling steps or
reusing intermediate results, failing to leverage variations across spatial
regions within the image due to the constraints of convolutional U-Net
structures. By harnessing the flexibility of Diffusion Transformers (DiTs) in
handling variable number of tokens, we introduce RAS, a novel, training-free
sampling strategy that dynamically assigns different sampling ratios to regions
within an image based on the focus of the DiT model. Our key observation is
that during each sampling step, the model concentrates on semantically
meaningful regions, and these areas of focus exhibit strong continuity across
consecutive steps. Leveraging this insight, RAS updates only the regions
currently in focus, while other regions are updated using cached noise from the
previous step. The model's focus is determined based on the output from the
preceding step, capitalizing on the temporal consistency we observed. We
evaluate RAS on Stable Diffusion 3 and Lumina-Next-T2I, achieving speedups up
to 2.36x and 2.51x, respectively, with minimal degradation in generation
quality. Additionally, a user study reveals that RAS delivers comparable
qualities under human evaluation while achieving a 1.6x speedup. Our approach
makes a significant step towards more efficient diffusion transformers,
enhancing their potential for real-time applications.
67b2a89ebe31bfaa7cd2bff1
Ziming Liu
claimed_verified
2025-02-18T09:33:14.074Z
632924029c3f42ca7149f305
/avatars/080bc7da4ad2875bdfa359213c88feb7.svg
Liu Ziming
user
MaruyamaAya
67b2a89ebe31bfaa7cd2bff2
Yifan Yang
claimed_verified
2025-02-18T09:33:16.182Z
62d18eb81e36881a57f29bf4
/avatars/104851421b4ee9641daaf15942fa7ea1.svg
Yif Yang
user
Yif29
67b2a89ebe31bfaa7cd2bff3
Chengruidong Zhang
67b2a89ebe31bfaa7cd2bff4
Yiqi Zhang
claimed_verified
2025-02-18T09:33:07.641Z
6356bf52e983d3c51d212205
https://cdn-avatars.huggingface.co/v1/production/uploads/1666629451430-noauth.jpeg
Yiqi Zhang
user
Viscent
67b2a89ebe31bfaa7cd2bff5
Lili Qiu
67b2a89ebe31bfaa7cd2bff6
Yang You
67b2a89ebe31bfaa7cd2bff7
Yuqing Yang
Region-Adaptive Sampling for Diffusion Transformers
Diffusion models (DMs) have become the leading choice for generative tasks
across diverse domains. However, their reliance on multiple sequential forward
passes significantly limits real-time performance. Previous acceleration
methods have primarily focused on reducing the number of sampling steps or
reusing intermediate results, failing to leverage variations across spatial
regions within the image due to the constraints of convolutional U-Net
structures. By harnessing the flexibility of Diffusion Transformers (DiTs) in
handling variable number of tokens, we introduce RAS, a novel, training-free
sampling strategy that dynamically assigns different sampling ratios to regions
within an image based on the focus of the DiT model. Our key observation is
that during each sampling step, the model concentrates on semantically
meaningful regions, and these areas of focus exhibit strong continuity across
consecutive steps. Leveraging this insight, RAS updates only the regions
currently in focus, while other regions are updated using cached noise from the
previous step. The model's focus is determined based on the output from the
preceding step, capitalizing on the temporal consistency we observed. We
evaluate RAS on Stable Diffusion 3 and Lumina-Next-T2I, achieving speedups up
to 2.36x and 2.51x, respectively, with minimal degradation in generation
quality. Additionally, a user study reveals that RAS delivers comparable
qualities under human evaluation while achieving a 1.6x speedup. Our approach
makes a significant step towards more efficient diffusion transformers,
enhancing their potential for real-time applications.
RAS, a novel sampling strategy for diffusion transformers, dynamically adjusts sampling ratios based on regions of focus, achieving speedups in diffusion models with minimal quality loss.
diffusion models sampling strategy Diffusion Transformers (DiTs) tokens sampling ratios spatial regions U-Net structures consecutive steps temporal consistency Stable Diffusion 3 Lumina-Next-T2I generation quality user study