title: Linear Transformers with Learnable Kernel Functions are Better
    In-Context Models
summary: Advancing the frontier of subquadratic architectures for Language Models
  (LMs) is crucial in the rapidly evolving field of natural language processing.
  Current innovations, including State Space Models, were initially celebrated
  for surpassing Transformer performance on language modeling tasks. However,
  these models have revealed deficiencies in essential In-Context Learning
  capabilities - a domain where the Transformer traditionally shines. The Based
  model emerged as a hybrid solution, blending a Linear Transformer with a kernel
  inspired by the Taylor expansion of exponential functions, augmented by
  convolutional networks. Mirroring the Transformer's in-context adeptness, it
  became a strong contender in the field. In our work, we present a singular,
  elegant alteration to the Based kernel that amplifies its In-Context Learning
  abilities evaluated with the Multi-Query Associative Recall task and overall
  language modeling process, as demonstrated on the Pile dataset.
paper_authors._id: 65d2e27c8120c6edb8230737
paper_authors.name: Yaroslav Aksenov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T17:05:22.159Z
paper_authors._id: 63ed5676684767daecac6f8a
paper_authors.avatarUrl: /avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
paper_authors.fullname: Yaroslav Aksenov
paper_authors.type: user
paper_authors.user: yaraksen
paper_authors._id: 65d2e27c8120c6edb8230738
paper_authors.name: Nikita Balagansky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T15:13:57.032Z
paper_authors._id: 60b364e7f88532cd79eaff7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
paper_authors.fullname: Nikita Balagansky
paper_authors.type: user
paper_authors.user: elephantmipt
paper_authors._id: 65d2e27c8120c6edb8230739
paper_authors.name: Sofia Maria Lo Cicero Vaina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T08:39:04.278Z
paper_authors._id: 65d384f2d717f5ae95dd9099
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KPyo3z_hjEs6qlSVRnNH9.jpeg
paper_authors.fullname: Sofia Lo Cicero
paper_authors.type: user
paper_authors.user: 2son1a
paper_authors._id: 65d2e27c8120c6edb823073a
paper_authors.name: Boris Shaposhnikov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T09:04:49.315Z
paper_authors._id: 637dd11dcbad6e62a5e39743
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637dd11dcbad6e62a5e39743/DG3rM8cy8inqbCoG4qizO.jpeg
paper_authors.fullname: Boris Shaposhnikov
paper_authors.type: user
paper_authors.user: borisshapa
paper_authors._id: 65d2e27c8120c6edb823073b
paper_authors.name: Alexey Gorbatovski
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T08:59:48.728Z
paper_authors._id: 62897fce5d9e25c10e4f319d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62897fce5d9e25c10e4f319d/bMlfAyzkNNZlkQ5mCW6Vc.jpeg
paper_authors.fullname: Alexey Gorbatovski
paper_authors.type: user
paper_authors.user: Myashka
paper_authors._id: 65d2e27c8120c6edb823073c
paper_authors.name: Daniil Gavrilov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T15:15:23.473Z
paper_authors._id: 62a9c8edc19f92ae443ab37f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
paper_authors.fullname: Daniil Gavrilov
paper_authors.type: user
paper_authors.user: kefirski
paper_title: Linear Transformers with Learnable Kernel Functions are Better
    In-Context Models
paper_summary: Advancing the frontier of subquadratic architectures for Language Models
  (LMs) is crucial in the rapidly evolving field of natural language processing.
  Current innovations, including State Space Models, were initially celebrated
  for surpassing Transformer performance on language modeling tasks. However,
  these models have revealed deficiencies in essential In-Context Learning
  capabilities - a domain where the Transformer traditionally shines. The Based
  model emerged as a hybrid solution, blending a Linear Transformer with a kernel
  inspired by the Taylor expansion of exponential functions, augmented by
  convolutional networks. Mirroring the Transformer's in-context adeptness, it
  became a strong contender in the field. In our work, we present a singular,
  elegant alteration to the Based kernel that amplifies its In-Context Learning
  abilities evaluated with the Multi-Query Associative Recall task and overall
  language modeling process, as demonstrated on the Pile dataset.
paper_ai_keywords: State Space Models Transformer In-Context Learning Based model Linear Transformer Taylor expansion exponential functions convolutional networks Multi-Query Associative Recall Pile dataset