Flash Attention
[flash-attention](https://github.com/HazyResearch/flash-attention)


**Tiling**과 **Recomputation**을 사용하여 Attention을 가속화
softmax를 block단위를 쪼개서 계산
CUDA kernel를 사용




> [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Wonbeom Jang](https://www.wonbeomjang.kr/blog/2023/fastattention/)

> [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Wonbeom Jang](https://www.wonbeomjang.kr/blog/2023/fastattention/)





