Recent work on optimizing bandwidth in the attention block has generated huge speed-ups and reductions in GPU memory usage. The most recent type of memory-efficient attention is [Flash Attention](https://arxiv.org/abs/2205.14135) (you can check out the original code at [HazyResearch/flash-attention](https://github.com/HazyResearch/flash-attention)).  
<Tip>  
If you have PyTorch >= 2.0 installed, you should not expect a speed-up for inference when enabling `xformers`.  
</Tip>