title: KV Shifting Attention Enhances Language Modeling
summary: The current large language models are mainly based on decode-only structure
  transformers, which have great in-context learning (ICL) capabilities. It is
  generally believed that the important foundation of its ICL capability is the
  induction heads mechanism, which requires at least two layers attention. In
  order to more efficiently implement the ability of the model's induction, we
  revisit the induction heads mechanism and proposed a KV shifting attention. We
  theoretically prove that the KV shifting attention reducing the model's
  requirements for the depth and width of the induction heads mechanism. Our
  experimental results demonstrate that KV shifting attention is beneficial to
  learning induction heads and language modeling, which lead to better
  performance or faster convergence from toy models to the pre-training models
  with more than 10 B parameters.
paper_authors._id: 675269205281c3cae493244d
paper_authors.name: Mingyu Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:12.672Z
paper_authors._id: 662f3c239e6d371ab7903b01
paper_authors.avatarUrl: /avatars/e36d774f4371a05f27c0950c624f8d3e.svg
paper_authors.fullname: xumingyu
paper_authors.type: user
paper_authors.user: xumingyu16
paper_authors._id: 675269205281c3cae493244e
paper_authors.name: Wei Cheng
paper_authors._id: 675269205281c3cae493244f
paper_authors.name: Bingning Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T15:15:23.438Z
paper_authors._id: 640dd3e0c364a086c6322ad2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dd3e0c364a086c6322ad2/z11CvZcbWGSRtWXzvem6I.png
paper_authors.fullname: Daniel Wang
paper_authors.type: user
paper_authors.user: DanielWang
paper_authors._id: 675269205281c3cae4932450
paper_authors.name: Weipeng Chen
paper_title: KV Shifting Attention Enhances Language Modeling
paper_summary: The current large language models are mainly based on decode-only structure
  transformers, which have great in-context learning (ICL) capabilities. It is
  generally believed that the important foundation of its ICL capability is the
  induction heads mechanism, which requires at least two layers attention. In
  order to more efficiently implement the ability of the model's induction, we
  revisit the induction heads mechanism and proposed a KV shifting attention. We
  theoretically prove that the KV shifting attention reducing the model's
  requirements for the depth and width of the induction heads mechanism. Our
  experimental results demonstrate that KV shifting attention is beneficial to
  learning induction heads and language modeling, which lead to better
  performance or faster convergence from toy models to the pre-training models
  with more than 10 B parameters.
paper_ai_summary: KV shifting attention improves the efficiency of in-context learning in large language models by reducing the depth and width requirements of induction heads.
paper_ai_keywords: decode-only structure transformers in-context learning induction heads mechanism KV shifting attention language modeling