title: Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
summary: The role of hidden units in recurrent neural networks is typically seen as
  modeling memory, with research focusing on enhancing information retention
  through gating mechanisms. A less explored perspective views hidden units as
  active participants in the computation performed by the network, rather than
  passive memory stores. In this work, we revisit bi-linear operations, which
  involve multiplicative interactions between hidden units and input embeddings.
  We demonstrate theoretically and empirically that they constitute a natural
  inductive bias for representing the evolution of hidden states in state
  tracking tasks. These are the simplest type of task that require hidden units
  to actively contribute to the behavior of the network. We also show that
  bi-linear state updates form a natural hierarchy corresponding to state
  tracking tasks of increasing complexity, with popular linear recurrent networks
  such as Mamba residing at the lowest-complexity center of that hierarchy.
paper_authors._id: 683db6ff97f03081fefa4e9f
paper_authors.name: M. Reza Ebrahimi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:49:11.196Z
paper_authors._id: 648a1075e8bee5332919e0ab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a1075e8bee5332919e0ab/KDzi904zGPuy_PL3gbUYE.jpeg
paper_authors.fullname: M.Reza Ebrahimi
paper_authors.type: user
paper_authors.user: mamaj92
paper_authors._id: 683db6ff97f03081fefa4ea0
paper_authors.name: Roland Memisevic
paper_title: Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
paper_summary: The role of hidden units in recurrent neural networks is typically seen as
  modeling memory, with research focusing on enhancing information retention
  through gating mechanisms. A less explored perspective views hidden units as
  active participants in the computation performed by the network, rather than
  passive memory stores. In this work, we revisit bi-linear operations, which
  involve multiplicative interactions between hidden units and input embeddings.
  We demonstrate theoretically and empirically that they constitute a natural
  inductive bias for representing the evolution of hidden states in state
  tracking tasks. These are the simplest type of task that require hidden units
  to actively contribute to the behavior of the network. We also show that
  bi-linear state updates form a natural hierarchy corresponding to state
  tracking tasks of increasing complexity, with popular linear recurrent networks
  such as Mamba residing at the lowest-complexity center of that hierarchy.
paper_ai_summary: Bilinear operations in recurrent neural networks are shown to be a natural bias for state tracking tasks, forming a hierarchical structure where linear recurrent networks are the simplest form.
paper_ai_keywords: hidden units recurrent neural networks gating mechanisms bi-linear operations input embeddings state tracking tasks bi-linear state updates Mamba