Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) for processing sequential data, and also in neuroscience, to understand the emergent properties of networks of real neurons. Prior theoretical work in understanding the properties of RNNs has focused on models with additive interactions. However, real neurons can have gating -- i.e. multiplicative -- interactions  , and gating is also a central feature of the best performing RNNs in machine learning . Here, we develop a dynamical mean-field theory (DMFT) to study the consequences of gating in RNNs. We use random matrix theory to show how gating robustly produces marginal stability and line attractors -- important mechanisms for biologically-relevant computations requiring long memory. The long-time behavior of the gated network is studied using its Lyapunov spectrum, and the DMFT is used to provide a novel analytical expression for the maximum Lyapunov exponent demonstrating its close relation to relaxation time of the dynamics. Gating is also shown to give rise to a novel, discontinuous transition to chaos, where the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity), contrary to a seminal result for additive RNNs. Critical surfaces and regions of marginal stability in the parameter space are indicated in phase diagrams, thus providing a map for principled parameter choices for ML practitioners. Finally, we develop a field-theory for gradients that arise in training, by incorporating the adjoint sensitivity framework from control theory in the DMFT . This paves the way for the use of powerful field-theoretic techniques to study training / gradients in large RNNs.
Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) for processing sequential data, and in neuroscience, to understand the emergent properties of networks of real neurons. Prior theoretical work in understanding the properties of RNNs has focused on networks with additive interactions. However, gating -- i.e. multiplicative -- interactions are ubiquitous in real neurons , and gating is also the central feature of the best-performing RNNs in ML . Here, we study the consequences of gating for the dynamical behavior of RNNs. We show that gating leads to slow modes and a novel, marginally-stable state. The network in this marginally-stable state can function as a robust integrator, and unlike previous approaches, gating permits this function without parameter fine-tuning or special symmetries. We study the long-time behavior of the gated network using its Lyapunov spectrum, and provide a novel relation between the maximum Lyapunov exponent and the relaxation time of the dynamics. Gating is also shown to give rise to a novel, discontinuous transition to chaos, where the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity), in contrast to a seminal result for additive RNNs. The rich dynamical behavior is summarized in a phase diagram indicating critical surfaces and regions of marginal stability -- thus, providing a map for principled parameter choices to ML practitioners. Finally, we develop a field theory for gradients that arise in training, by combining the adjoint formalism from control theory with the dynamical mean-field theory . This paves the way for the use of powerful field theoretic techniques to study training and gradients in large RNNs.
D
also
R
models
networks
D
real neurons can have
A
are ubiquitous in real neurons
R
a
the
R
best performing RNNs in machine learning
best-performing RNNs in ML
D
develop a dynamical mean-field theory (DMFT) to
R
in
for the dynamical behavior of
R
use random matrix theory to show how gating robustly produces marginal stability and line attractors -- important mechanisms for biologically-relevant computations requiring long memory. The
show that gating leads to slow modes and a novel, marginally-stable state. The network in this marginally-stable state can function as a robust integrator, and unlike previous approaches, gating permits this function without parameter fine-tuning or special symmetries. We study the
D
is studied
D
the DMFT is used to
R
analytical expression for
relation between
R
demonstrating its close relation to
and the
R
contrary
in contrast
R
Critical
The rich dynamical behavior is summarized in a phase diagram indicating critical
R
in the parameter space are indicated in phase diagrams, thus
-- thus,
R
for
to
R
field-theory
field theory
R
incorporating the adjoint sensitivity framework
combining the adjoint formalism
R
in the DMFT
with the dynamical mean-field theory
R
field-theoretic
field theoretic
R
/
and