title: Transformers are Multi-State RNNs
summary: Transformers are considered conceptually different compared to the previous
  generation of state-of-the-art NLP models - recurrent neural networks (RNNs).
  In this work, we demonstrate that decoder-only transformers can in fact be
  conceptualized as infinite multi-state RNNs - an RNN variant with unlimited
  hidden state size. We further show that pretrained transformers can be
  converted into finite multi-state RNNs by fixing the size of their
  hidden state. We observe that several existing transformers cache compression
  techniques can be framed as such conversion policies, and introduce a novel
  policy, TOVA, which is simpler compared to these policies. Our experiments with
  several long range tasks indicate that TOVA outperforms all other baseline
  policies, while being nearly on par with the full (infinite) model, and using
  in some cases only 1{8} of the original cache size. Our results
  indicate that transformer decoder LLMs often behave in practice as RNNs. They
  also lay out the option of mitigating one of their most painful computational
  bottlenecks - the size of their cache memory. We publicly release our code at
  https://github.com/schwartz-lab-NLP/TOVA.
paper_authors._id: 65a0a3dadb5d37ad5e8b016a
paper_authors.name: Matanel Oren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:33:09.910Z
paper_authors._id: 6502b3f0cb1cf43486e1f333
paper_authors.avatarUrl: /avatars/ff4462062beb17b9ad427ae730e7974d.svg
paper_authors.fullname: Matanel Oren
paper_authors.type: user
paper_authors.user: matanelo
paper_authors._id: 65a0a3dadb5d37ad5e8b016b
paper_authors.name: Michael Hassid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:33:16.772Z
paper_authors._id: 6547411a9295970f878aa52e
paper_authors.avatarUrl: /avatars/6e240f0add27bf1a6c04a9618eccdf83.svg
paper_authors.fullname: Michael Hassid
paper_authors.type: user
paper_authors.user: hassid
paper_authors._id: 65a0a3dadb5d37ad5e8b016c
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:33:23.532Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_authors._id: 65a0a3dadb5d37ad5e8b016d
paper_authors.name: Roy Schwartz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:34:04.289Z
paper_authors._id: 641bef9da5f876fe30c94e3f
paper_authors.avatarUrl: /avatars/42b1ad679e84b3212e3770ca1e6d64f2.svg
paper_authors.fullname: Roy Schwartz
paper_authors.type: user
paper_authors.user: royschwartz
paper_title: Transformers are Multi-State RNNs
paper_summary: Transformers are considered conceptually different compared to the previous
  generation of state-of-the-art NLP models - recurrent neural networks (RNNs).
  In this work, we demonstrate that decoder-only transformers can in fact be
  conceptualized as infinite multi-state RNNs - an RNN variant with unlimited
  hidden state size. We further show that pretrained transformers can be
  converted into finite multi-state RNNs by fixing the size of their
  hidden state. We observe that several existing transformers cache compression
  techniques can be framed as such conversion policies, and introduce a novel
  policy, TOVA, which is simpler compared to these policies. Our experiments with
  several long range tasks indicate that TOVA outperforms all other baseline
  policies, while being nearly on par with the full (infinite) model, and using
  in some cases only 1{8} of the original cache size. Our results
  indicate that transformer decoder LLMs often behave in practice as RNNs. They
  also lay out the option of mitigating one of their most painful computational
  bottlenecks - the size of their cache memory. We publicly release our code at
  https://github.com/schwartz-lab-NLP/TOVA.
paper_ai_summary: Decoder-only transformers can be conceptualized as finite multi-state RNNs, and a new cache compression technique, TOVA, significantly reduces their computational cost while maintaining high performance.
paper_ai_keywords: transformers recurrent neural networks RNNs infinite multi-state RNNs finite multi-state RNNs cache compression TOVA long range tasks decoder LLMs