ToDo: Token Downsampling for Efficient Generation of High-Resolution
  Images
Attention mechanism has been crucial for image diffusion models, however,
their quadratic computational complexity limits the sizes of images we can
process within reasonable time and memory constraints. This paper investigates
the importance of dense attention in generative image models, which often
contain redundant features, making them suitable for sparser attention
mechanisms. We propose a novel training-free method ToDo that relies on token
downsampling of key and value tokens to accelerate Stable Diffusion inference
by up to 2x for common sizes and up to 4.5x or more for high resolutions like
2048x2048. We demonstrate that our approach outperforms previous methods in
balancing efficient throughput and fidelity.
65d758719e01a52135cc2d88
Ethan Smith
65d758719e01a52135cc2d89
Nayan Saxena
claimed_verified
2024-02-23T12:25:33.938Z
63b6de91b004c84aac8b970b
/avatars/fa27d141497093fe0f2d91bf537fad33.svg
Nayan Saxena
user
saxenan3
65d758719e01a52135cc2d8a
Aninda Saha
ToDo: Token Downsampling for Efficient Generation of High-Resolution
  Images
Attention mechanism has been crucial for image diffusion models, however,
their quadratic computational complexity limits the sizes of images we can
process within reasonable time and memory constraints. This paper investigates
the importance of dense attention in generative image models, which often
contain redundant features, making them suitable for sparser attention
mechanisms. We propose a novel training-free method ToDo that relies on token
downsampling of key and value tokens to accelerate Stable Diffusion inference
by up to 2x for common sizes and up to 4.5x or more for high resolutions like
2048x2048. We demonstrate that our approach outperforms previous methods in
balancing efficient throughput and fidelity.
A novel training-free method accelerates Stable Diffusion inference by downsampling key and value tokens, enhancing throughput and fidelity without additional training.
attention mechanism quadratic computational complexity dense attention generative image models token downsampling Stable Diffusion efficient throughput fidelity