Recent work on optimizing bandwidth in the attention block has generated huge speed-ups and reductions in GPU memory usage. The most recent type of memory-efficient attention is [Flash Attention](https://arxiv.org/abs/2205.14135) (you can check out the original code at [HazyResearch/flash-attention](https://github.com/HazyResearch/flash-attention)).  
<Tip>  
If you have PyTorch >= 2.0 installed, you should not expect a speed-up for inference when enabling `xformers`.  
</Tip>  
To use Flash Attention, install the following:  
- PyTorch > 1.12
- CUDA available
- [xFormers](xformers)  
Then call [`~ModelMixin.enable_xformers_memory_efficient_attention`] on the pipeline:  
```python
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
torch_dtype=torch.float16,
use_safetensors=True,
).to("cuda")

pipe.enable_xformers_memory_efficient_attention()

with torch.inference_mode():
sample = pipe("a small cat")

# optional: You can disable it via
# pipe.disable_xformers_memory_efficient_attention()
```  
The iteration speed when using `xformers` should match the iteration speed of PyTorch 2.0 as described [here](torch2.0).
/Users/nielsrogge/Documents/python_projecten/diffusers/docs/source/en/optimization/memory.md