Transformers are Multi-State RNNs
Transformers are considered conceptually different compared to the previous
generation of state-of-the-art NLP models - recurrent neural networks (RNNs).
In this work, we demonstrate that decoder-only transformers can in fact be
conceptualized as infinite multi-state RNNs - an RNN variant with unlimited
hidden state size. We further show that pretrained transformers can be
converted into finite multi-state RNNs by fixing the size of their
hidden state. We observe that several existing transformers cache compression
techniques can be framed as such conversion policies, and introduce a novel
policy, TOVA, which is simpler compared to these policies. Our experiments with
several long range tasks indicate that TOVA outperforms all other baseline
policies, while being nearly on par with the full (infinite) model, and using
in some cases only 1{8} of the original cache size. Our results
indicate that transformer decoder LLMs often behave in practice as RNNs. They
also lay out the option of mitigating one of their most painful computational
bottlenecks - the size of their cache memory. We publicly release our code at
https://github.com/schwartz-lab-NLP/TOVA.
65a0a3dadb5d37ad5e8b016a
Matanel Oren
admin_assigned
2024-01-12T09:33:09.910Z
6502b3f0cb1cf43486e1f333
/avatars/ff4462062beb17b9ad427ae730e7974d.svg
Matanel Oren
user
matanelo
65a0a3dadb5d37ad5e8b016b
Michael Hassid
admin_assigned
2024-01-12T09:33:16.772Z
6547411a9295970f878aa52e
/avatars/6e240f0add27bf1a6c04a9618eccdf83.svg
Michael Hassid
user
hassid
65a0a3dadb5d37ad5e8b016c
Yossi Adi
admin_assigned
2024-01-12T09:33:23.532Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
65a0a3dadb5d37ad5e8b016d
Roy Schwartz
admin_assigned
2024-01-12T09:34:04.289Z
641bef9da5f876fe30c94e3f
/avatars/42b1ad679e84b3212e3770ca1e6d64f2.svg
Roy Schwartz
user
royschwartz
Transformers are Multi-State RNNs
Transformers are considered conceptually different compared to the previous
generation of state-of-the-art NLP models - recurrent neural networks (RNNs).
In this work, we demonstrate that decoder-only transformers can in fact be
conceptualized as infinite multi-state RNNs - an RNN variant with unlimited
hidden state size. We further show that pretrained transformers can be
converted into finite multi-state RNNs by fixing the size of their
hidden state. We observe that several existing transformers cache compression
techniques can be framed as such conversion policies, and introduce a novel
policy, TOVA, which is simpler compared to these policies. Our experiments with
several long range tasks indicate that TOVA outperforms all other baseline
policies, while being nearly on par with the full (infinite) model, and using
in some cases only 1{8} of the original cache size. Our results
indicate that transformer decoder LLMs often behave in practice as RNNs. They
also lay out the option of mitigating one of their most painful computational
bottlenecks - the size of their cache memory. We publicly release our code at
https://github.com/schwartz-lab-NLP/TOVA.
Decoder-only transformers can be conceptualized as finite multi-state RNNs, and a new cache compression technique, TOVA, significantly reduces their computational cost while maintaining high performance.
transformers recurrent neural networks RNNs infinite multi-state RNNs finite multi-state RNNs cache compression TOVA long range tasks decoder LLMs