title: Gated recurrent neural networks discover attention
summary: Recent architectural developments have enabled recurrent neural networks
  (RNNs) to reach and even surpass the performance of Transformers on certain
  sequence modeling tasks. These modern RNNs feature a prominent design pattern:
  linear recurrent layers interconnected by feedforward paths with multiplicative
  gating. Here, we show how RNNs equipped with these two design elements can
  exactly implement (linear) self-attention, the main building block of
  Transformers. By reverse-engineering a set of trained RNNs, we find that
  gradient descent in practice discovers our construction. In particular, we
  examine RNNs trained to solve simple in-context learning tasks on which
  Transformers are known to excel and find that gradient descent instills in our
  RNNs the same attention-based in-context learning algorithm used by
  Transformers. Our findings highlight the importance of multiplicative
  interactions in neural networks and suggest that certain RNNs might be
  unexpectedly implementing attention under the hood.
paper_authors._id: 64f8321ee7584abc624dba4d
paper_authors.name: Nicolas Zucchet
paper_authors._id: 64f8321ee7584abc624dba4e
paper_authors.name: Seijin Kobayashi
paper_authors._id: 64f8321ee7584abc624dba4f
paper_authors.name: Yassir Akram
paper_authors._id: 64f8321ee7584abc624dba50
paper_authors.name: Johannes von Oswald
paper_authors._id: 64f8321ee7584abc624dba51
paper_authors.name: Maxime Larcher
paper_authors._id: 64f8321ee7584abc624dba52
paper_authors.name: Angelika Steger
paper_authors._id: 64f8321ee7584abc624dba53
paper_authors.name: João Sacramento
paper_title: Gated recurrent neural networks discover attention
paper_summary: Recent architectural developments have enabled recurrent neural networks
  (RNNs) to reach and even surpass the performance of Transformers on certain
  sequence modeling tasks. These modern RNNs feature a prominent design pattern:
  linear recurrent layers interconnected by feedforward paths with multiplicative
  gating. Here, we show how RNNs equipped with these two design elements can
  exactly implement (linear) self-attention, the main building block of
  Transformers. By reverse-engineering a set of trained RNNs, we find that
  gradient descent in practice discovers our construction. In particular, we
  examine RNNs trained to solve simple in-context learning tasks on which
  Transformers are known to excel and find that gradient descent instills in our
  RNNs the same attention-based in-context learning algorithm used by
  Transformers. Our findings highlight the importance of multiplicative
  interactions in neural networks and suggest that certain RNNs might be
  unexpectedly implementing attention under the hood.
paper_ai_keywords: recurrent neural networks (RNNs) Transformers sequence modeling tasks linear recurrent layers feedforward paths multiplicative gating (linear) self-attention reverse-engineering gradient descent in-context learning tasks multiplicative interactions