title: Linear Transformers are Versatile In-Context Learners
summary: Recent research has demonstrated that transformers, particularly linear
  attention models, implicitly execute gradient-descent-like algorithms on data
  provided in-context during their forward inference step. However, their
  capability in handling more complex problems remains unexplored. In this paper,
  we prove that any linear transformer maintains an implicit linear model and can
  be interpreted as performing a variant of preconditioned gradient descent. We
  also investigate the use of linear transformers in a challenging scenario where
  the training data is corrupted with different levels of noise. Remarkably, we
  demonstrate that for this problem linear transformers discover an intricate and
  highly effective optimization algorithm, surpassing or matching in performance
  many reasonable baselines. We reverse-engineer this algorithm and show that it
  is a novel approach incorporating momentum and adaptive rescaling based on
  noise levels. Our findings show that even linear transformers possess the
  surprising ability to discover sophisticated optimization strategies.
paper_authors._id: 65d821412c9f76a077ef2b23
paper_authors.name: Max Vladymyrov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:18:02.726Z
paper_authors._id: 6502fe0bb1792803da806d42
paper_authors.avatarUrl: /avatars/60280ce59f1f0d67e4210b0453039282.svg
paper_authors.fullname: Max Vladymyrov
paper_authors.type: user
paper_authors.user: gozzo87
paper_authors._id: 65d821412c9f76a077ef2b24
paper_authors.name: Johannes von Oswald
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:18:09.984Z
paper_authors._id: 6502de229421d71875f52170
paper_authors.avatarUrl: /avatars/cff4c35bb8f5704d47bd8a74e7f605e6.svg
paper_authors.fullname: Johannes von Oswald
paper_authors.type: user
paper_authors.user: voswaldj
paper_authors._id: 65d821412c9f76a077ef2b25
paper_authors.name: Mark Sandler
paper_authors._id: 65d821412c9f76a077ef2b26
paper_authors.name: Rong Ge
paper_title: Linear Transformers are Versatile In-Context Learners
paper_summary: Recent research has demonstrated that transformers, particularly linear
  attention models, implicitly execute gradient-descent-like algorithms on data
  provided in-context during their forward inference step. However, their
  capability in handling more complex problems remains unexplored. In this paper,
  we prove that any linear transformer maintains an implicit linear model and can
  be interpreted as performing a variant of preconditioned gradient descent. We
  also investigate the use of linear transformers in a challenging scenario where
  the training data is corrupted with different levels of noise. Remarkably, we
  demonstrate that for this problem linear transformers discover an intricate and
  highly effective optimization algorithm, surpassing or matching in performance
  many reasonable baselines. We reverse-engineer this algorithm and show that it
  is a novel approach incorporating momentum and adaptive rescaling based on
  noise levels. Our findings show that even linear transformers possess the
  surprising ability to discover sophisticated optimization strategies.
paper_ai_summary: Linear transformers are shown to perform a variant of preconditioned gradient descent and discover noisy data optimization strategies that outperform baselines.
paper_ai_keywords: linear transformers gradient descent preconditioned gradient descent momentum adaptive rescaling