Repeat After Me: Transformers are Better than State Space Models at
  Copying
Transformers are the dominant architecture for sequence modeling, but there
is growing interest in models that use a fixed-size latent state that does not
depend on the sequence length, which we refer to as "generalized state space
models" (GSSMs). In this paper we show that while GSSMs are promising in terms
of inference-time efficiency, they are limited compared to transformer models
on tasks that require copying from the input context. We start with a
theoretical analysis of the simple task of string copying and prove that a two
layer transformer can copy strings of exponential length while GSSMs are
fundamentally limited by their fixed-size latent state. Empirically, we find
that transformers outperform GSSMs in terms of efficiency and generalization on
synthetic tasks that require copying the context. Finally, we evaluate
pretrained large language models and find that transformer models dramatically
outperform state space models at copying and retrieving information from
context. Taken together, these results suggest a fundamental gap between
transformers and GSSMs on tasks of practical interest.
65c04af9744da3e6394c9c05
Samy Jelassi
admin_assigned
2024-02-05T12:06:58.589Z
643740cab8ea86e503291f18
/avatars/46a7e6ba97c89215aa89ba67e61e17e7.svg
Samy Jelassi
user
sjelassi
65c04af9744da3e6394c9c06
David Brandfonbrener
admin_assigned
2024-02-05T12:07:05.329Z
6314bdae4f283a6af50711da
/avatars/ca3fcc8609b492124b4f97fab29a63f1.svg
David Brandfonbrener
user
davidbrandfonbrener
65c04af9744da3e6394c9c07
Sham M. Kakade
65c04af9744da3e6394c9c08
Eran Malach
admin_assigned
2024-02-05T12:07:17.958Z
637b41b756db0404b7c615a5
/avatars/ce33c956900cc711a237b532c12bdc66.svg
Eran Malach
user
emalach
Repeat After Me: Transformers are Better than State Space Models at
  Copying
Transformers are the dominant architecture for sequence modeling, but there
is growing interest in models that use a fixed-size latent state that does not
depend on the sequence length, which we refer to as "generalized state space
models" (GSSMs). In this paper we show that while GSSMs are promising in terms
of inference-time efficiency, they are limited compared to transformer models
on tasks that require copying from the input context. We start with a
theoretical analysis of the simple task of string copying and prove that a two
layer transformer can copy strings of exponential length while GSSMs are
fundamentally limited by their fixed-size latent state. Empirically, we find
that transformers outperform GSSMs in terms of efficiency and generalization on
synthetic tasks that require copying the context. Finally, we evaluate
pretrained large language models and find that transformer models dramatically
outperform state space models at copying and retrieving information from
context. Taken together, these results suggest a fundamental gap between
transformers and GSSMs on tasks of practical interest.
Transformers outperform generalized state space models (GSSMs) in copying and retrieving information from context due to their ability to handle variable-length sequences without a fixed-size latent state.
transformers generalized state space models GSSMs string copying fixed-size latent state exponential length