Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
heads reduces the memory overhead at inference by reducing the size of the KV cache. Linformer 📎 Paper 👩‍💻 Code A linear time self-attention is achieved by breaking down the scaled dot-product attention into multiple smaller attentions using linear projections. Together, these operations create a