Beyond Homogeneous Attention: Memory-Efficient LLMs via
  Fourier-Approximated KV Cache
Large Language Models struggle with memory demands from the growing Key-Value
(KV) cache as context lengths increase. Existing compression methods homogenize
head dimensions or rely on attention-guided token pruning, often sacrificing
accuracy or introducing computational overhead. We propose FourierAttention, a
training-free framework that exploits the heterogeneous roles of transformer
head dimensions: lower dimensions prioritize local context, while upper ones
capture long-range dependencies. By projecting the long-context-insensitive
dimensions onto orthogonal Fourier bases, FourierAttention approximates their
temporal evolution with fixed-length spectral coefficients. Evaluations on
LLaMA models show that FourierAttention achieves the best long-context accuracy
on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel,
FlashFourierAttention, is designed to optimize memory via streamlined
read-write operations, enabling efficient deployment without performance
compromise.
684fedfa1d9b438aa3957a6d
Xiaoran Liu
claimed_verified
2025-06-16T12:56:14.162Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Liu Xiaoran
user
LiuXR
684fedfa1d9b438aa3957a6e
Siyang He
684fedfa1d9b438aa3957a6f
Qiqi Wang
684fedfa1d9b438aa3957a70
Ruixiao Li
684fedfa1d9b438aa3957a71
Yuerong Song
684fedfa1d9b438aa3957a72
Zhigeng Liu
684fedfa1d9b438aa3957a73
Linlin Li
684fedfa1d9b438aa3957a74
Qun Liu
684fedfa1d9b438aa3957a75
Zengfeng Huang
684fedfa1d9b438aa3957a76
Qipeng Guo
684fedfa1d9b438aa3957a77
Ziwei He
684fedfa1d9b438aa3957a78
Xipeng Qiu
Beyond Homogeneous Attention: Memory-Efficient LLMs via
  Fourier-Approximated KV Cache
Large Language Models struggle with memory demands from the growing Key-Value
(KV) cache as context lengths increase. Existing compression methods homogenize
head dimensions or rely on attention-guided token pruning, often sacrificing
accuracy or introducing computational overhead. We propose FourierAttention, a
training-free framework that exploits the heterogeneous roles of transformer
head dimensions: lower dimensions prioritize local context, while upper ones
capture long-range dependencies. By projecting the long-context-insensitive
dimensions onto orthogonal Fourier bases, FourierAttention approximates their
temporal evolution with fixed-length spectral coefficients. Evaluations on
LLaMA models show that FourierAttention achieves the best long-context accuracy
on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel,
FlashFourierAttention, is designed to optimize memory via streamlined
read-write operations, enabling efficient deployment without performance
compromise.
FourierAttention is a training-free framework that enhances memory efficiency in Large Language Models by compressing long-context-insensitive transformer head dimensions using orthogonal Fourier bases, while maintaining accuracy.
FourierAttention transformer head dimensions LongBench Needle-In-A-Haystack FlashFourierAttention orthogonal Fourier bases long-context accuracy