Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Self-attention performs well in long context but has quadratic complexity.
Existing RNN layers have linear complexity, but their performance in long
context is limited by the expressive power of their hidden state. We propose a
new class of sequence modeling layers with linear complexity and an expressive
hidden state. The key idea is to make the hidden state a machine learning model
itself, and the update rule a step of self-supervised learning. Since the
hidden state is updated by training even on test sequences, our layers are
called Test-Time Training (TTT) layers. We consider two instantiations:
TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer
MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B
parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both
TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer,
they can keep reducing perplexity by conditioning on more tokens, while Mamba
cannot after 16k context. With preliminary systems optimization, TTT-Linear is
already faster than Transformer at 8k context and matches Mamba in wall-clock
time. TTT-MLP still faces challenges in memory I/O, but shows larger potential
in long context, pointing to a promising direction for future research.
668b5e98d358e8fd17762b9a
Yu Sun
668b5e98d358e8fd17762b9b
Xinhao Li
668b5e98d358e8fd17762b9c
Karan Dalal
extracted_confirmed
2024-07-08T04:11:37.802Z
646ebf2298e8f749fc60ce38
/avatars/5a76c9343451c24e9697d3165cdc0af6.svg
Karan Dalal
user
karansdalal
668b5e98d358e8fd17762b9d
Jiarui Xu
admin_assigned
2024-07-08T21:11:59.315Z
64f55c47708362ac9b5f5c45
/avatars/bd0ba54d6e75dc3662a3cbdec0091770.svg
Jiarui Xu
user
jxu605
668b5e98d358e8fd17762b9e
Arjun Vikram
admin_assigned
2024-07-08T21:11:33.053Z
63c89f809cf77a4393d2f1ce
https://cdn-avatars.huggingface.co/v1/production/uploads/1674092377599-noauth.jpeg
Arjun Vikram
user
arjvik
668b5e98d358e8fd17762b9f
Genghan Zhang
admin_assigned
2024-07-08T21:11:27.105Z
65a76ff1e504d9738d636217
/avatars/26bf5e3f19057835ee95d72c24904d77.svg
Genghan Zhang
user
Genghan
668b5e98d358e8fd17762ba0
Yann Dubois
admin_assigned
2024-07-08T21:11:21.048Z
641362754e5305c14f233f08
/avatars/bce974ef60c507d22702cc7662033b28.svg
Yann Dubois
user
YannDubs
668b5e98d358e8fd17762ba1
Xinlei Chen
admin_assigned
2024-07-08T21:11:00.420Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
668b5e98d358e8fd17762ba2
Xiaolong Wang
668b5e98d358e8fd17762ba3
Sanmi Koyejo
admin_assigned
2024-07-08T21:10:33.322Z
64931e7e2da595588288f161
https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
Sanmi Koyejo
user
sanmikoyejo
668b5e98d358e8fd17762ba4
Tatsunori Hashimoto
668b5e98d358e8fd17762ba5
Carlos Guestrin
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Self-attention performs well in long context but has quadratic complexity.
Existing RNN layers have linear complexity, but their performance in long
context is limited by the expressive power of their hidden state. We propose a
new class of sequence modeling layers with linear complexity and an expressive
hidden state. The key idea is to make the hidden state a machine learning model
itself, and the update rule a step of self-supervised learning. Since the
hidden state is updated by training even on test sequences, our layers are
called Test-Time Training (TTT) layers. We consider two instantiations:
TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer
MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B
parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both
TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer,
they can keep reducing perplexity by conditioning on more tokens, while Mamba
cannot after 16k context. With preliminary systems optimization, TTT-Linear is
already faster than Transformer at 8k context and matches Mamba in wall-clock
time. TTT-MLP still faces challenges in memory I/O, but shows larger potential
in long context, pointing to a promising direction for future research.
Test-Time Training (TTT) layers achieve linear complexity and competitive performance on long context tasks, surpassing both Transformers and RNNs in certain scenarios.
self-attention quadratic complexity RNN layers linear complexity expressive hidden state machine learning model self-supervised learning Test-Time Training (TTT) layers TTT-Linear TTT-MLP Transformer Mamba perplexity memory I/O