Gated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks
(RNNs) to reach and even surpass the performance of Transformers on certain
sequence modeling tasks. These modern RNNs feature a prominent design pattern:
linear recurrent layers interconnected by feedforward paths with multiplicative
gating. Here, we show how RNNs equipped with these two design elements can
exactly implement (linear) self-attention, the main building block of
Transformers. By reverse-engineering a set of trained RNNs, we find that
gradient descent in practice discovers our construction. In particular, we
examine RNNs trained to solve simple in-context learning tasks on which
Transformers are known to excel and find that gradient descent instills in our
RNNs the same attention-based in-context learning algorithm used by
Transformers. Our findings highlight the importance of multiplicative
interactions in neural networks and suggest that certain RNNs might be
unexpectedly implementing attention under the hood.
64f8321ee7584abc624dba4d
Nicolas Zucchet
claimed_verified
2025-04-24T11:25:36.724Z
680a081bc9c06490bc2a3f9c
/avatars/2f0a8abb30d28f9f0563d4968668f7aa.svg
Nicolas Zucchet
user
nzucchet
64f8321ee7584abc624dba4e
Seijin Kobayashi
64f8321ee7584abc624dba4f
Yassir Akram
64f8321ee7584abc624dba50
Johannes von Oswald
64f8321ee7584abc624dba51
Maxime Larcher
64f8321ee7584abc624dba52
Angelika Steger
64f8321ee7584abc624dba53
João Sacramento
Gated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks
(RNNs) to reach and even surpass the performance of Transformers on certain
sequence modeling tasks. These modern RNNs feature a prominent design pattern:
linear recurrent layers interconnected by feedforward paths with multiplicative
gating. Here, we show how RNNs equipped with these two design elements can
exactly implement (linear) self-attention, the main building block of
Transformers. By reverse-engineering a set of trained RNNs, we find that
gradient descent in practice discovers our construction. In particular, we
examine RNNs trained to solve simple in-context learning tasks on which
Transformers are known to excel and find that gradient descent instills in our
RNNs the same attention-based in-context learning algorithm used by
Transformers. Our findings highlight the importance of multiplicative
interactions in neural networks and suggest that certain RNNs might be
unexpectedly implementing attention under the hood.
Recent RNNs with linear recurrent layers and multiplicative gating can implement linear self-attention, similar to Transformers, as discovered through reverse-engineering of trained RNNs on in-context learning tasks.
RNNs recurrent neural networks linear self-attention Transformers multiplicative gating in-context learning