title: Tensor Product Attention Is All You Need
summary: Scaling language models to handle longer input sequences typically
  necessitates large key-value (KV) caches, resulting in substantial memory
  overhead during inference. In this paper, we propose Tensor Product Attention
  (TPA), a novel attention mechanism that uses tensor decompositions to represent
  queries, keys, and values compactly, significantly shrinking KV cache size at
  inference time. By factorizing these representations into contextual low-rank
  components (contextual factorization) and seamlessly integrating with RoPE, TPA
  achieves improved model quality alongside memory efficiency. Based on TPA, we
  introduce the Tensor ProducT ATTenTion Transformer (T6), a new model
  architecture for sequence modeling. Through extensive empirical evaluation of
  language modeling tasks, we demonstrate that T6 exceeds the performance of
  standard Transformer baselines including MHA, MQA, GQA, and MLA across various
  metrics, including perplexity and a range of renowned evaluation benchmarks.
  Notably, TPAs memory efficiency enables the processing of significantly longer
  sequences under fixed resource constraints, addressing a critical scalability
  challenge in modern language models. The code is available at
  https://github.com/tensorgi/T6.
paper_authors._id: 6785ce62cb1fc2728334a5e2
paper_authors.name: Yifan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:28:51.600Z
paper_authors._id: 647bf082aba7062fe5c51ca9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
paper_authors.fullname: Yifan Zhang
paper_authors.type: user
paper_authors.user: yifAI
paper_authors._id: 6785ce62cb1fc2728334a5e3
paper_authors.name: Yifeng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T08:31:50.343Z
paper_authors._id: 653d276681f52ceb4d12bd85
paper_authors.avatarUrl: /avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
paper_authors.fullname: Yifeng Liu
paper_authors.type: user
paper_authors.user: Lewis-Lau
paper_authors._id: 6785ce62cb1fc2728334a5e4
paper_authors.name: Huizhuo Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:03:54.519Z
paper_authors._id: 657be24e8d360b690d5b665f
paper_authors.avatarUrl: /avatars/423185c3f386e3fec2e5708c1030b013.svg
paper_authors.fullname: Angela Yuan
paper_authors.type: user
paper_authors.user: angelahzyuan
paper_authors._id: 6785ce62cb1fc2728334a5e5
paper_authors.name: Zhen Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:51:32.418Z
paper_authors._id: 649014b91d71e55664838d2d
paper_authors.avatarUrl: /avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
paper_authors.fullname: Zhen Qin
paper_authors.type: user
paper_authors.user: zhenqincn
paper_authors._id: 6785ce62cb1fc2728334a5e6
paper_authors.name: Yang Yuan
paper_authors._id: 6785ce62cb1fc2728334a5e7
paper_authors.name: Quanquan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T08:52:03.941Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_authors._id: 6785ce62cb1fc2728334a5e8
paper_authors.name: Andrew Chi-Chih Yao
paper_title: Tensor Product Attention Is All You Need
paper_summary: Scaling language models to handle longer input sequences typically
  necessitates large key-value (KV) caches, resulting in substantial memory
  overhead during inference. In this paper, we propose Tensor Product Attention
  (TPA), a novel attention mechanism that uses tensor decompositions to represent
  queries, keys, and values compactly, significantly shrinking KV cache size at
  inference time. By factorizing these representations into contextual low-rank
  components (contextual factorization) and seamlessly integrating with RoPE, TPA
  achieves improved model quality alongside memory efficiency. Based on TPA, we
  introduce the Tensor ProducT ATTenTion Transformer (T6), a new model
  architecture for sequence modeling. Through extensive empirical evaluation of
  language modeling tasks, we demonstrate that T6 exceeds the performance of
  standard Transformer baselines including MHA, MQA, GQA, and MLA across various
  metrics, including perplexity and a range of renowned evaluation benchmarks.
  Notably, TPAs memory efficiency enables the processing of significantly longer
  sequences under fixed resource constraints, addressing a critical scalability
  challenge in modern language models. The code is available at
  https://github.com/tensorgi/T6.
paper_ai_summary: T6, a new Transformer architecture using Tensor Product Attention, improves language model performance and memory efficiency, allowing for longer sequence processing.
paper_ai_keywords: Tensor Product Attention TPA tensor decompositions contextual factorization RoPE memory efficiency sequence modeling T6 MHA MQA GQA MLA perplexity evaluation benchmarks