RNNs are popular dynamical models, used for processing sequential data  . Prior theoretical work in understanding the properties of RNNs has focused on models with additive interactions , where the input to a unit is a weighted sum of the output of the remaining units in network . However, there is ample evidence that neurons can have gating - i.e. multiplicative - interactions. Such gating interactions have significant effects on the collective dynamics of the network. Furthermore, the best performing RNNs in machine learning have gating interactions. Thus, gating interactions are beneficial for information processing and learning tasks. We develop a dynamical mean-field theory (DMFT) of gating to understand the dynamical regimes produced by gating . Our gated RNN reduces to the classical RNNs in certain limits and is closely related to popular gated models in machine learning . We use random matrix theory (RMT) to analytically characterize the spectrum of the Jacobian and show how gating produces slow modes and marginal stability . Thus, gating is a potential mechanism to implement computations involving line attractor dynamics . The long-time behavior of the gated network is studied using its Lyapunov spectrum, and the DMFT is used to provide an analytical prediction for the maximum Lyapunov exponent . We also show that gating gives rise to a novel, discontinuous transition to chaos, where the proliferation of critical points is decoupled with the appearance of chaotic dynamics ; the nature of this chaotic state is characterized in detail. Using the DMFT and RMT, we produce phase diagramsfor gated RNN . Finally, we address the gradients by leveraging the adjoint sensitivity framework to develop a DMFT for the gradients. The theory developed here sheds light on the rich dynamical behaviour produced by gating interactions and has implications for architectural choices and learning dynamics .
Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) for processing sequential data , and also in neuroscience, to understand the emergent properties of networks of real neurons . Prior theoretical work in understanding the properties of RNNs has focused on models with additive interactions . However, real neurons can have gating -- i.e. multiplicative -- interactions, and gating is also a central feature of the best performing RNNs in machine learning . Here, we develop a dynamical mean-field theory (DMFT) to study the consequences of gating in RNNs . We use random matrix theory to show how gating robustly produces marginal stability and line attractors -- important mechanisms for biologically-relevant computations requiring long memory . The long-time behavior of the gated network is studied using its Lyapunov spectrum, and the DMFT is used to provide a novel analytical expression for the maximum Lyapunov exponent demonstrating its close relation to relaxation time of the dynamics. Gating is also shown to give rise to a novel, discontinuous transition to chaos, where the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity), contrary to a seminal result for additive RNNs. Critical surfaces and regions of marginal stability in the parameter space are indicated in phase diagrams, thus providing a map for principled parameter choices for ML practitioners . Finally, we develop a field-theory for gradients that arise in training, by incorporating the adjoint sensitivity framework from control theory in the DMFT. This paves the way for the use of powerful field-theoretic techniques to study training/gradients in large RNNs .
R
RNNs are popular
Recurrent neural networks (RNNs) are powerful
R
used
widely used in machine learning (ML)
A
, and also in neuroscience, to understand the emergent properties of networks of real neurons
D
, where the input to a unit is a weighted sum of the output of the remaining units in network
R
there is ample evidence that
real
R
-
--
R
- interactions. Such gating interactions have significant effects on the collective dynamics of the network. Furthermore, the
-- interactions, and gating is also a central feature of the
R
have gating interactions. Thus, gating interactions are beneficial for information processing and learning tasks. We
. Here, we
R
of gating to understand the dynamical regimes produced by gating . Our gated RNN reduces to the classical RNNs in certain limits and is closely related to popular gated models in machine learning
to study the consequences of gating in RNNs
R
(RMT) to analytically characterize the spectrum of the Jacobian and
to
R
produces slow modes and marginal stability . Thus, gating is a potential mechanism to implement computations involving line attractor dynamics
robustly produces marginal stability and line attractors -- important mechanisms for biologically-relevant computations requiring long memory
R
an analytical prediction
a novel analytical expression
R
. We also show that gating gives
demonstrating its close relation to relaxation time of the dynamics. Gating is also shown to give
R
is decoupled with
(topological complexity) is decoupled from
R
; the nature of this chaotic state is characterized in detail. Using the DMFT and RMT, we produce phase diagramsfor gated RNN
(dynamical complexity), contrary to a seminal result for additive RNNs. Critical surfaces and regions of marginal stability in the parameter space are indicated in phase diagrams, thus providing a map for principled parameter choices for ML practitioners
R
address the gradients by leveraging
develop a field-theory for gradients that arise in training, by incorporating
R
to develop a DMFT for the gradients. The theory developed here sheds light on the rich dynamical behaviour produced by gating interactions and has implications for architectural choices and learning dynamics
from control theory in the DMFT. This paves the way for the use of powerful field-theoretic techniques to study training/gradients in large RNNs