title: Repeat After Me: Transformers are Better than State Space Models at
    Copying
summary: Transformers are the dominant architecture for sequence modeling, but there
  is growing interest in models that use a fixed-size latent state that does not
  depend on the sequence length, which we refer to as "generalized state space
  models" (GSSMs). In this paper we show that while GSSMs are promising in terms
  of inference-time efficiency, they are limited compared to transformer models
  on tasks that require copying from the input context. We start with a
  theoretical analysis of the simple task of string copying and prove that a two
  layer transformer can copy strings of exponential length while GSSMs are
  fundamentally limited by their fixed-size latent state. Empirically, we find
  that transformers outperform GSSMs in terms of efficiency and generalization on
  synthetic tasks that require copying the context. Finally, we evaluate
  pretrained large language models and find that transformer models dramatically
  outperform state space models at copying and retrieving information from
  context. Taken together, these results suggest a fundamental gap between
  transformers and GSSMs on tasks of practical interest.
paper_authors._id: 65c04af9744da3e6394c9c05
paper_authors.name: Samy Jelassi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:06:58.589Z
paper_authors._id: 643740cab8ea86e503291f18
paper_authors.avatarUrl: /avatars/46a7e6ba97c89215aa89ba67e61e17e7.svg
paper_authors.fullname: Samy Jelassi
paper_authors.type: user
paper_authors.user: sjelassi
paper_authors._id: 65c04af9744da3e6394c9c06
paper_authors.name: David Brandfonbrener
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:07:05.329Z
paper_authors._id: 6314bdae4f283a6af50711da
paper_authors.avatarUrl: /avatars/ca3fcc8609b492124b4f97fab29a63f1.svg
paper_authors.fullname: David Brandfonbrener
paper_authors.type: user
paper_authors.user: davidbrandfonbrener
paper_authors._id: 65c04af9744da3e6394c9c07
paper_authors.name: Sham M. Kakade
paper_authors._id: 65c04af9744da3e6394c9c08
paper_authors.name: Eran Malach
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:07:17.958Z
paper_authors._id: 637b41b756db0404b7c615a5
paper_authors.avatarUrl: /avatars/ce33c956900cc711a237b532c12bdc66.svg
paper_authors.fullname: Eran Malach
paper_authors.type: user
paper_authors.user: emalach
paper_title: Repeat After Me: Transformers are Better than State Space Models at
    Copying
paper_summary: Transformers are the dominant architecture for sequence modeling, but there
  is growing interest in models that use a fixed-size latent state that does not
  depend on the sequence length, which we refer to as "generalized state space
  models" (GSSMs). In this paper we show that while GSSMs are promising in terms
  of inference-time efficiency, they are limited compared to transformer models
  on tasks that require copying from the input context. We start with a
  theoretical analysis of the simple task of string copying and prove that a two
  layer transformer can copy strings of exponential length while GSSMs are
  fundamentally limited by their fixed-size latent state. Empirically, we find
  that transformers outperform GSSMs in terms of efficiency and generalization on
  synthetic tasks that require copying the context. Finally, we evaluate
  pretrained large language models and find that transformer models dramatically
  outperform state space models at copying and retrieving information from
  context. Taken together, these results suggest a fundamental gap between
  transformers and GSSMs on tasks of practical interest.
paper_ai_summary: Transformers outperform generalized state space models (GSSMs) in copying and retrieving information from context due to their ability to handle variable-length sequences without a fixed-size latent state.
paper_ai_keywords: transformers generalized state space models GSSMs string copying fixed-size latent state exponential length