Tensor Product Attention Is All You Need
Scaling language models to handle longer input sequences typically
necessitates large key-value (KV) caches, resulting in substantial memory
overhead during inference. In this paper, we propose Tensor Product Attention
(TPA), a novel attention mechanism that uses tensor decompositions to represent
queries, keys, and values compactly, significantly shrinking KV cache size at
inference time. By factorizing these representations into contextual low-rank
components (contextual factorization) and seamlessly integrating with RoPE, TPA
achieves improved model quality alongside memory efficiency. Based on TPA, we
introduce the Tensor ProducT ATTenTion Transformer (T6), a new model
architecture for sequence modeling. Through extensive empirical evaluation of
language modeling tasks, we demonstrate that T6 exceeds the performance of
standard Transformer baselines including MHA, MQA, GQA, and MLA across various
metrics, including perplexity and a range of renowned evaluation benchmarks.
Notably, TPAs memory efficiency enables the processing of significantly longer
sequences under fixed resource constraints, addressing a critical scalability
challenge in modern language models. The code is available at
https://github.com/tensorgi/T6.
6785ce62cb1fc2728334a5e2
Yifan Zhang
claimed_verified
2025-01-14T08:28:51.600Z
647bf082aba7062fe5c51ca9
https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
Yifan Zhang
user
yifAI
6785ce62cb1fc2728334a5e3
Yifeng Liu
claimed_verified
2025-01-16T08:31:50.343Z
653d276681f52ceb4d12bd85
/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
Yifeng Liu
user
Lewis-Lau
6785ce62cb1fc2728334a5e4
Huizhuo Yuan
claimed_verified
2025-02-11T10:03:54.519Z
657be24e8d360b690d5b665f
/avatars/423185c3f386e3fec2e5708c1030b013.svg
Angela Yuan
user
angelahzyuan
6785ce62cb1fc2728334a5e5
Zhen Qin
admin_assigned
2025-01-14T08:51:32.418Z
649014b91d71e55664838d2d
/avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
Zhen Qin
user
zhenqincn
6785ce62cb1fc2728334a5e6
Yang Yuan
6785ce62cb1fc2728334a5e7
Quanquan Gu
admin_assigned
2025-01-14T08:52:03.941Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
6785ce62cb1fc2728334a5e8
Andrew Chi-Chih Yao
Tensor Product Attention Is All You Need
Scaling language models to handle longer input sequences typically
necessitates large key-value (KV) caches, resulting in substantial memory
overhead during inference. In this paper, we propose Tensor Product Attention
(TPA), a novel attention mechanism that uses tensor decompositions to represent
queries, keys, and values compactly, significantly shrinking KV cache size at
inference time. By factorizing these representations into contextual low-rank
components (contextual factorization) and seamlessly integrating with RoPE, TPA
achieves improved model quality alongside memory efficiency. Based on TPA, we
introduce the Tensor ProducT ATTenTion Transformer (T6), a new model
architecture for sequence modeling. Through extensive empirical evaluation of
language modeling tasks, we demonstrate that T6 exceeds the performance of
standard Transformer baselines including MHA, MQA, GQA, and MLA across various
metrics, including perplexity and a range of renowned evaluation benchmarks.
Notably, TPAs memory efficiency enables the processing of significantly longer
sequences under fixed resource constraints, addressing a critical scalability
challenge in modern language models. The code is available at
https://github.com/tensorgi/T6.
T6, a new Transformer architecture using Tensor Product Attention, improves language model performance and memory efficiency, allowing for longer sequence processing.
Tensor Product Attention TPA tensor decompositions contextual factorization RoPE memory efficiency sequence modeling T6 MHA MQA GQA MLA perplexity evaluation benchmarks