Selective Attention Improves Transformer
Paper
•
2410.02703
•
Published
•
24
Attention
-
-
Differential Transformer
Paper • 2410.05258 • Published • 175 -
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention
Paper • 2410.05076 • Published • 8 -
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
Paper • 2410.13276 • Published • 28 -
Star Attention: Efficient LLM Inference over Long Sequences
Paper • 2411.17116 • Published • 54 -
KV Shifting Attention Enhances Language Modeling
Paper • 2411.19574 • Published • 9 -
Entropy-Guided Attention for Private LLMs
Paper • 2501.03489 • Published • 14 -
Not All Language Model Features Are Linear
Paper • 2405.14860 • Published • 41 -
Your Transformer is Secretly Linear
Paper • 2405.12250 • Published • 158 -
MiniMax-01: Scaling Foundation Models with Lightning Attention
Paper • 2501.08313 • Published • 282 -
Tensor Product Attention Is All You Need
Paper • 2501.06425 • Published • 87 -
Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
Paper • 2501.13629 • Published • 47 -
TransMLA: Multi-head Latent Attention Is All You Need
Paper • 2502.07864 • Published • 48 -
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Paper • 2502.11089 • Published • 150 -
Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information
Paper • 2502.14258 • Published • 26