summaries: This paper proposes a biologically motivated mechanism that could allow a network of slow-responding neurons (primarily due to the response lag) to perform computation at a much faster timescale. The key modeling idea is to assume that the neurons compute a "prospective" latent state, by linearly extrapolating from the current position in the phase space, by the current momentum and a specific timescale. The authors then assume that neural dynamics is along the extrema of an energy function in the latent state space; hence called the "Latent Equilibrium" framework. Using synthetic data, the authors demonstrate that the proposed model can indeed lead to fast computation and learning; they also apply the model to real data from cortical microcircuits. The authors also go further and discuss how the model can be modified to self-correct timescale mismatches. This paper asks an important question, proposes novel ideas, and presents compelling contributions. The problem being addressed is fundamental and yet has received not much attention,  with no existing good solution. The problem is that physical and biological elements in the brain are slow. Computations propagating through a neural network hierarchy with such elements will introduce time lag.  The mismatch between teaching signals and the current state of the system,  when the input is constantly changing, presents a problem for learning.  The two existing solutions involve (1) separating inference and learning into the "relaxation" phase for the computation to complete and the "learning" phase for weight update, or (2) use a very slow weight update rate. Both of these are suboptimal.  They proposed a prospective energy function in which the membrane potential of a neuron is trained to map onto a latent equilibrium space where during the relaxation state, the instantaneous outputs of the neurons are the prospective target membrane potentials of the neurons with those recurrent computations. This will allow weight update to be done concurrently with relaxation and solving a problem of achieving arbitrary fast computation with arbitrarily slow neurons.  The resulting model can be interpreted as a biologically plausible approximation of error backpropagation in deep cortical networks with continuous-time leaky neuronal dynamics and continuous active local plasticity. They showed this method work in MINST and CIFAR100, and in fact, the method works better than BP during the early epochs of training. The authors propose a framework from which a neuron model with (slow) neural dynamics close to physical substrates can be derived  along with a plasticity rule for learning that is similar to backprop. The neuron model allows fast propagation of signals through the network in spite of the slow dynamics of the components, which avoids having to control the timing of learning to turn it on only after the signal has propagated. The authors perform empirical evaluations of networks based on this model on various tasks. The authors introduce a novel framework called Latent Equilibrium that enables quasi-instantaneous inference regardless of network depth and learning without phased plasticity in networks with slow neuronal elements.  Notably, the framework offers a biologically plausible model of back-propagation. Authors show that the models generated according to the framework have reasonable prediction quality on standard benchmark datasets. The authors also show that the framework can generate good models of cortical networks. Finally, authors explore heterogeneous substrates and conclude that framework is robust. The paper introduces Latent Equilibrium, a new framework for inference and learning in networks of slow components.   Authors use the leaky integrator dynamics that characterize biological neurons to jointly derive disentangled neuron and synapse dynamics. The neuronal dynamics evolve along a constant-energy manifold, called the Latent Equilibrium.   They show that LE allows for performance gains after the beginning of training, when the disruptive effects of relaxation begin to dominate, as well as robustness of LE on a MNIST classification task. 
abstractText: The response time of physical computational elements is finite, and neurons are no exception. In hierarchical models of cortical networks each layer thus introduces a response lag. This inherent property of physical dynamical systems results in delayed processing of stimuli and causes a timing mismatch between network output and instructive signals, thus afflicting not only inference, but also learning. We introduce Latent Equilibrium, a new framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential. This principle enables quasi-instantaneous inference independent of network depth and avoids the need for phased plasticity or computationally expensive network relaxation phases. We jointly derive disentangled neuron and synapse dynamics from a prospective energy function that depends on a network’s generalized position and momentum. The resulting model can be interpreted as a biologically plausible approximation of error backpropagation in deep cortical networks with continuous-time, leaky neuronal dynamics and continuously active, local plasticity. We demonstrate successful learning of standard benchmark datasets, achieving competitive performance using both fully-connected and convolutional architectures, and show how our principle can be applied to detailed models of cortical microcircuitry. Furthermore, we study the robustness of our model to spatio-temporal substrate imperfections to demonstrate its feasibility for physical realization, be it in vivo or in silico.§
authors.name: Paul Haider
authors.name: Benjamin Ellenberger
authors.name: Laura Kriener
authors.name: Jakob Jordan
authors.name: Walter Senn
authors.name: Mihai A. Petrovici
references.authors: D.E. Rumelhart G.E. Hinton R.J. Williams
references.title: Learning Representations by Backpropagating Errors
references.venue: Nature 323,
references.authors: J.C.R. Whittington R. Bogacz
references.title: An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity
references.venue: Neural Computation
references.authors: B. Scellier Bengio Y. Equilibrium propagation
references.title: Bridging the Gap between Energy-Based Models and Backpropagation
references.venue: Frontiers in Computational Neuroscience 11, 24
references.authors: J. Guerguiev T.P. Lillicrap B.A. Richards
references.title: Towards Deep Learning with Segregated Dendrites
references.venue: eLife 6,
references.authors: J. Sacramento R. Ponte Costa Y. Bengio W. Senn
references.title: Dendritic Cortical Microcircuits Approximate the Backpropagation Algorithm
references.venue: Advances in Neural Information Processing Systems
references.authors: Y. Song T. Lukasiewicz Z. Xu R. Bogacz
references.title: Can the Brain do Backpropagation? – Exact Implementation of Backpropagation in Predictive Coding Networks
references.venue: Advances in Neural Information Processing Systems
references.authors: B. Millidge A. Tschantz C.L. Buckley
references.title: Predictive Coding Approximates Backprop along Arbitrary Computation Graphs
references.authors: B. Millidge A. Tschantz C.L. Buckley Seth A. Activation Relaxation
references.title: A Local Dynamical Approximation to Backpropagation in the Brain
references.venue: arXiv: 2009.05359
references.authors: Choromanska A. et al. Beyond Backprop
references.title: Online Alternating Minimization with Auxiliary Variables
references.venue: International Conference on Machine Learning 97, 1193–1202
references.authors: S Bartunov
references.title: Assessing the Scalability of Biologically-motivated Deep Learning Algorithms and Architectures
references.authors: R.P.N. Rao Ballard D.H. Predictive Coding in the Visual Cortex
references.title: A Functional Interpretation of Some Extra-Classical Receptive-Field Effects
references.venue: Nature Neuroscience 2, 79–87
references.authors: R. Bogacz
references.title: A tutorial on the free-energy framework for modelling perception and learning
references.venue: Journal of mathematical psychology 76,
references.authors: J.J. Hopfield
references.title: Neurons with Graded Response Have Collective Computational Properties like Those of Two-State Neurons
references.venue: Proceedings of the National Academy of Sciences
references.authors: N. Minorsky
references.title: Directional stability of automatically steered bodies
references.venue: Journal of the American Society for Naval Engineers
references.authors: A.L. Hodgkin A.F. Huxley
references.title: A Quantitative Description of Membrane Current and Its Application to Conduction and Excitation in Nerve
references.venue: The Journal of Physiology 117,
references.authors: J. Platkiewicz R. Brette
references.title: Impact of Fast Sodium Channel Inactivation on Spike Threshold Dynamics and Synaptic Integration
references.venue: PLOS Computational Biology
references.authors: D Dold
references.title: Lagrangian dynamics of dendritic microcircuits enables real-time backpropagation of errors
references.venue: Computational and Systems Neuroscience (Cosyne)
references.authors: A.F. Kungl D. Dold O. Riedler M.A. Petrovici W. Senn
references.title: Deep reinforcement learning for time-continuous substrates
references.venue: Neuro Inspired Computational Elements Conference
references.authors: C. Van Vreeswijk H. Sompolinsky
references.title: Chaotic Balanced State in a Model of Cortical Circuits
references.venue: Neural Computation
references.authors: B.W. Knight
references.title: Dynamics of Encoding in a Population of Neurons
references.venue: Journal of General Physiology
references.authors: H.E. Plesser W. Gerstner
references.title: Escape Rate Models for Noisy Integrate-and-Free Neurons
references.venue: Neurocomputing 32-33,
references.authors: H Köndgen
references.title: The Dynamical Response Properties of Neocortical Neurons to Temporally Modulated Noisy Inputs In Vitro
references.venue: Cerebral Cortex
references.authors: J. Brea A.T. Gaál R. Urbanczik W. Senn
references.title: Prospective Coding by Spiking Neurons
references.venue: PLOS Computational Biology
references.authors: T.P. Lillicrap A. Santoro L. Marris C.J. Akerman G. Hinton
references.title: Backpropagation and the Brain
references.venue: Nature Reviews Neuroscience
references.authors: Y. LeCun L. Bottou Y. Bengio P. Haffner
references.title: Gradient-Based Learning Applied to Document Recognition
references.venue: Proceedings of the IEEE 86,
references.authors: P. Baldi P. Sadowski D. Whiteson
references.title: Searching for Exotic Particles in High-energy Physics with Deep Learning
references.venue: Nature Communications
references.authors: T.P. Lillicrap D. Cownden D.B. Tweed C.J. Akerman
references.title: Random Synaptic Feedback Weights Support Error Backpropagation for Deep Learning
references.venue: Nature Communications 7,
references.authors: Y LeCun
references.title: Backpropagation Applied to Handwritten Zip Code Recognition
references.venue: Neural Computation
references.authors: A. Krizhevsky
references.title: Learning Multiple Layers of Features from Tiny Images
references.authors: J. Sacramento R.P. Costa Y. Bengio W. Senn
references.title: Dendritic error backpropagation in deep cortical microcircuits
references.authors: R. Urbanczik W. Senn
references.title: Learning by the Dendritic Prediction of Somatic Spiking
references.venue: Neuron 81,
references.authors: Wunderlich T. et al. Demonstrating Advantages of Neuromorphic Computation
references.title: A Pilot Study
references.venue: Frontiers in Neuroscience 13
references.authors: S Billaudelle
references.title: Versatile Emulation of Spiking Neural Networks on an Accelerated Neuromorphic Substrate
references.venue: IEEE International Symposium on Circuits and Systems
references.authors: J Göltz
references.title: Fast and Energy-Efficient Neuromorphic Deep Learning with First-Spike Times
references.venue: Nature Machine Intelligence 3,
references.authors: Thakur C.S. et al. Large-scale Neuromorphic Spiking Array Processors
references.title: A Quest to Mimic the Brain
references.venue: Frontiers in Neuroscience 12, 891
references.authors: E. Marder Goaillard J.-M
references.title: Variability, compensation and homeostasis in neuron and network function
references.venue: Nature Reviews Neuroscience
references.authors: N Spruston P Jonas B. Sakmann
references.title: Dendritic Glutamate Receptor Channels in Rat Hippocampal CA3 and CA1 Pyramidal Neurons
references.venue: The Journal of Physiology 482,
references.authors: M.C. Bellingham R. Lim B. Walmsley
references.title: Developmental Changes in EPSC Quantal Size and Quantal Content at a Central Glutamatergic Synapse in Rat
references.venue: The Journal of Physiology 511,
references.authors: J.R.P. Geiger J. Lübke A. Roth M. Frotscher P. Jonas
references.title: Submillisecond AMPA ReceptorMediated Signaling at a Principal NeuronInterneuron Synapse
references.venue: Neuron 18,
references.authors: S.B. Duss S. Oggier T.P. Reber K. Henke
references.title: Formation of semantic associations between subliminally presented face-word pairs
references.venue: Consciousness and cognition 20,
references.authors: C. Clopath L. Büsing E. Vasilaki Gerstner W. Connectivity reflects coding
references.title: a model of voltage-based STDP with homeostasis
references.venue: Nature neuroscience 13, 344–352
references.authors: M. Akrout C. Wilson P.C. Humphreys T. Lillicrap D. Tweed
references.title: Deep learning without weight transport
references.authors: J. Guerguiev K.P. Kording B.A. Richards
references.title: Spike-based causal inference for weight alignment
references.authors: Y. Amit
references.title: Deep Learning with Asymmetric Connections and Hebbian Updates
references.venue: Frontiers in Computational Neuroscience 13,
references.authors: D Kunin
references.title: Two routes to scalable credit assignment without weight symmetry
references.venue: International Conference on Machine Learning,
references.authors: R. VanRullen R. Guyonneau S.J. Thorpe
references.title: Spike times make sense
references.venue: Trends in neurosciences 28,
references.authors: T Pfeil
references.title: Six networks on a universal neuromorphic computing substrate
references.venue: Frontiers in neuroscience 7,
references.authors: Petrovici M. A
references.title: Characterization and Compensation of Network-Level Anomalies in Mixed-Signal Neuromorphic Modeling Platforms
references.venue: PLOS ONE
sections.heading: 1 Introduction
sections.text: Physical systems composed of large collections of simple, but intricately connected elements can exhibit powerful collective computational properties. A prime example are animals’ nervous systems, and most prominently the human brain. Its computational prowess has motivated a large, crossdisciplinary and ongoing endeavor to emulate aspects of its structure and dynamics in artificial substrates, with the aim of being ultimately able to replicate its function. The speed of information processing in such a system depends on the response time of its components; for neurons, for example, it can be the integration time scale determined by their membrane time constant.
  *Senior authors. §Code is available at https://github.com/unibe-cns/le_NeurIPS_code.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  If we consider hierarchically organized neural networks composed of such elements, each layer in the hierarchy causes a response lag with respect to a changing stimulus. This lag introduces two related critical issues. For one, the inference speed in these systems decreases with their depth. In turn, this induces timing mismatches between instructive signals and neural activity, which disrupts learning. For example, recent proposals for bio-plausible implementations of error backpropagation (BP) [1–4] in the brain all require some form of relaxation, both for inference and during learning [5–11]. Notably, this also affects some purely algorithmic methods involving auxiliary variables [12]. To deal with this inherent property of physical dynamical systems, two approaches have been suggested: either phased plasticity that is active only following a certain relaxation period, or long stimulus presentation times with small learning rates. Both of these solutions entail significant drawbacks: the former is challenging to implement in asynchronous, distributed systems such as cortical networks or neuromorphic hardware, while the latter results, by construction, in slow learning. This has prompted the critique that any algorithm requiring such a settling process is too slow to describe complex brain function, particularly when involving real-time responses [13]. To the best of our knowledge this fundamental problem affects all modern models of approximate BP in biological substrates [5–11].
  To overcome these issues, we propose a novel framework for fast computation and learning in physical substrates with slow components. As we show below, this framework jointly addresses multiple aspects of neuronal computation, including neuron morphology, membrane dynamics, synaptic plasticity and network structure. In particular, it provides a biologically plausible approximation of BP in deep cortical networks with continuous-time, leaky neuronal dynamics and local, continuous plasticity. Moreover, our model is easy to implement in both software and hardware and is well-suited for distributed, asynchronous systems.
  In our framework, inference can be arbitrarily fast (up to finite simulation resolution or finite communication speed across physical distances) despite a finite response time of individual system components; downstream responses to input changes thus become effectively instantaneous. Conversely, responses to instructive top-down input that generate local error signals are also near-instantaneous, thus effectively removing the need for any relaxation phase. This allows truly phase-free learning from signals that change on much faster time scales than the response speed of individual network components.
  Similarly to other approaches [5, 6, 9, 14, 15], we derive neuron and synapse dynamics from a joint energy function. However, our energy function is designed to effectively disentangle these dynamics, thus removing the disruptive co-dependencies that otherwise arise during relaxation. This is achieved by introducing a simple, but crucial new ingredient: neuronal outputs that try to guess their future state based on their current information, a property we describe as “prospective” (which should not be confused with the “predictive” in predictive coding, as we also discuss below). Thereby, our framework also constructs an intimate relationship between such “slow” neuronal networks and artificial neural network (ANN)1, thus enabling the application of various auxiliary methods from deep learning.
sections.heading: 2 The problems of slow components
sections.text: To illustrate the issue with relaxation, we consider two neurons arranged in a chain (Fig. 1a). Biological neuronal membrane potentials u are conventionally modeled as leaky integrators of their input I: Cmu̇ = −glu + I , where the membrane capacitance Cm and leak conductance gl determine the membrane time constant τm := Cm/gl and thereby its response speed. These dynamics attenuate and delay input, resulting in significant differences between neuronal output rates and those expected in an instantaneous system (Fig. 1b). This mismatch increases with every additional layer: a feedforward network with n layers has an effective relaxation time constant of approximately nτm.
  Besides slow inference, this delayed response leads to critical issues during learning from downstream instructive signals. Consider the typical scenario where such a target signal is present in the output layer and plasticity is continuously active (and not phased according to some complicated schedule). If the system fulfills its task correctly, the target signal corresponds to the output of the relaxed system and no learning should take place. However, due to delays in neuronal responses, the output signal differs from the target during relaxation, which causes plasticity to adapt synaptic weights in an effort to better match the target. As a result, the system “overshoots” during early relaxation and has to
  1To differentiate between biologically plausible, leaky neurons and abstract neurons with instantaneous response, we respectively use the terms “neuronal” and “neural”.
  undo these synaptic changes through further learning in the late relaxation phase (Fig. 1c). Also, since output errors need inputs to propagate forward through the entire network before propagating backwards themselves, the input layer only observes correct errors after about 2nτm. We refer to this issue as the “relaxation problem”. In the following, we present a solution for this problem, combining prospective neuron dynamics with continuously active, local synaptic plasticity.
sections.heading: 3 Fast computations in slow substrates
sections.text: Inspired by the canonical coordinates from classical mechanics, we describe the state of a neuronal network by its position in phase space (u, u̇), with the generalized position u and the generalized momentum u̇. The relation between these two components describes the physics of the system. To obtain the leaky integrator dynamics that characterize biological neurons, we first define the abstract network state ŭm as
  ŭm := u+ τmu̇ . (1)
  We next define an energy function E that characterizes this state:
  E(ŭm) := 1
  2 ‖ŭm −Wϕ(ŭm)− b‖2 + βL(ŭm) . (2)
  Here, W and b represent the weight matrix and bias vector, ϕ is the neuronal activation function and the loss L is scaled by a constant β; we use bold font for matrices, vectors, and vector-valued functions. Intuitively speaking, E measures the difference between “what neurons guess that they will be doing in the future” (ŭm) and “what their biases and presynaptic afferents believe they should be doing” (b+Wϕ(ŭm)). Furthermore, for a subset of neurons, it adds a loss L that provides an external measure of the error for the states of these neurons, determined by instructive signals such as labels, reconstruction errors or rewards. This formulation of the energy function is very generic and can, by appropriate choice of parameters, apply to different network topologies, including multilayer perceptrons, convolutional architectures and recurrent networks. Note that this energy can be written as a neuron-wise sum over mismatch energies plus, for factorizing loss functions, a local loss term Ei(ŭ m i , ri,pre) = 1 2 (ŭ m i −Wiri,pre − bi) 2 + βLi (ŭmi ), where ri,pre := ϕ(ŭ m i,pre) is the presynaptic input vector for the ith neuron in the network.
  We can now derive neuronal dynamics as extrema of the energy function from Eqn. 2:
  ∇ŭmE = 0 =⇒ τmu̇ = −u+Wϕ(ŭm) + b+ e , (3)
  with presynaptic bottom-up input Wϕ(ŭm) and top-down error signals
  e = ϕ′(ŭm)W T [ŭm −Wϕ(ŭm)− b] (4)
  for hidden neurons and e = −β∇ŭmL for neurons which directly contribute to the loss. Plugging Eqn. 3 into Eqn. 4, it is easy to see that in hierarchical networks these errors can be expressed recursively over layers `, e` = ϕ′(ŭm)W T`+1e`+1, thus instantiating a variant of BP. Eqn. 3 can be interpreted as the dynamics of a structured pyramidal neuron receiving presynaptic, bottom-up input via its basal dendrites and top-down input via its apical tree (Fig. 1d). We provide a more detailed description of our model’s biophysical implementation in Section 5. We note that our approach is in contrast to previous work that introduced neuron dynamics via gradient descent on an energy function, such as [5, 6, 16], whereas we require a stationary energy function with respect to ŭm. Indeed, this difference is crucial for solving the relaxation problem, as discussed below. Since for a given input our network moves, by construction, within a constant-energy manifold, we refer to our model as Latent Equilibrium (LE).
  We can now revisit our choice of ŭm from a functional point of view. Instead of the classical output rate ϕ(u), our neurons fire with ϕ(ŭm), which depends on both u and u̇ (Eqn. 1). As neuron membranes are low-pass filters (Eqn. 3), ŭm can be viewed as a prospective version of u: when firing, the neuron uses its currently available information to forecast the state of its membrane potential after relaxation. The prospective nature of ŭm also holds in a strict mathematical sense: the breve operator ˘m := (1 + τmd/dt) is the exact inverse of an exponential low-pass filter (see SI). While neuronal membranes continue to relax slowly towards their steady states, neuronal outputs use membrane momenta to compute a correct instantaneous reaction to their inputs, even in the case of jumps (Fig. 1e). Thus, information can propagate instantaneously throughout the network, similarly to an ANN, counterintuitively even when membrane dynamics are never in equilibrium. The activity of the output layer hence reflects arbitrarily fast changes in the input – even on time scales smaller than the neuronal time constant τm – rather than responding with a significant time lag and attenuation as in previous, gradient-based models.
  The idea of incorporating derivatives into the input-output function of a system has a long history in control theory [17] and also represents a known, though often neglected feature of (single) biological neurons [18, 19]. A related, but different form of neuronal prospectivity has also been considered in other models of bio-plausible BP derived from a stationary action [20, 21]. At the level of neuronal populations with additive Gaussian noise, there exists a long tradition of studying faster-than-τm responses, both with [22] and without [23] recurrent connectivity. Similar observations also hold for single neurons in the presence of noise [24, 25]. Building on these insights and integrating them into a unified theory of neuronal dynamics and learning, our model proposes a specific form of prospective coding that can also be learned by local adaptation mechanisms, as we discuss in Section 6.
  We should also stress the difference between the terms “prospective” and “predictive”. Predictive coding, or more generally, predictive processing, is a theory of brain function which proposes that
  brains maintain internal models of their environment which they update and learn by trying to predict sensory input at the same moment in time. Originally based on a specific Bayesian model and error communication scheme [14], the notion of predictive coding can be generalized to layers in a hierarchical model predicting the activities of subsequent layers. This principle is instantiated in our networks as well, as will become clear in the following sections. In contrast, prospective coding in our framework refers to the ability of neurons to “look forward in time” using the current state of the membrane potential in phase space (position and velocity), as described above.2 Prospective and predictive coding are thus complementary: our specific prospective mechanism provides a cure for relaxation in predictive coding networks, thus significantly speeding up both learning and inference, as we describe below.
  Note that, even for functionally feedforward networks, our resulting network structure is recurrent, with backward coupling induced by error inputs to the apical tree. As a non-linear recurrent network, it cannot settle instantaneously into the correct state; rather, in numerical simulations, it jumps quickly towards an estimated stationary activity state and reaches equilibrium within several such jumps (of infinitesimal duration). In practice, saturating activation functions can help avoid pathological behavior under strong coupling. Moreover, we can introduce a very short exponential low-pass filter τ s on top-down signals, slightly larger than the temporal resolution of the simulation. Thus, in physical systems operating in continuous time, τ s can effectively become infinitesimal as well and does not affect the speed of information propagation through the network. In particular, as we discuss below, the perpetual consistency between input and output allows our model to continuously learn to reduce the loss, obviating the need for network relaxation phases and the associated global control of precisely timed plasticity mechanisms.
sections.heading: 4 Fast learning in slow substrates
sections.text: Based on our prospective energy function (Eqn. 2), we define synaptic weight dynamics, i.e., learning, as time-continuous stochastic gradient descent with learning rate ηW :
  Ẇ ∝ −∇WE =⇒ Ẇ = ηW [ŭm −Wr − b] rT . (5)
  Thus, weights evolve continuously in time driven by local error signals without requiring any particular schedule. Neuronal biases are adapted according to the same principle. Note that this rule only uses quantities that are available at the locus of the synapse (see also Section 5). Intuitively, this locality is enabled by the recurrent nature of the network: errors in the output units spread throughout the system, attributing credit locally through changes in neuronal membrane potentials. These changes are then used by synapses to update their weight in order to reduce the network loss. However, our learning rule is not an exact replica of BP, although it does approximate it in the limit of infinitely weak supervision β → 0 (often referred to as “nudging”); strictly speaking, it minimizes the energy function E, which implicitly minimizes the loss L. This form of credit assignment can be related to previous models which similarly avoid a separate artificial backward pass (as necessary in classical BP) by allowing errors to influence neuronal activity [27]. Plasticity in the weights projecting to output neurons depends on the choice of L; for example, for an L2 loss, plasticity in the output layer corresponds to the classical delta rule [28]: ẆN = ηWβ [r∗N − rN ] rTN−1. Despite similarities to previous work, learning in our framework does not suffer from many of the shortcomings that we have already noted. Since activity propagates quasi-instantaneously throughout the network, our plasticity can be continuously active without disrupting learning performance. This is true by construction and most easily visualized for a sequence of (piecewise constant) input patterns: following a change in the input, membrane dynamics take place in a constant-energy manifold (Eqn. 3) across which synaptic weight dynamics remain unchanged, i.e., they equally and simultaneously pull downward on all points of this manifold (Fig. 1f). This disentangling of membrane and synaptic weight dynamics constitutes the crucial difference to previous work, where the neuronal mismatch energies Ei change as dynamics evolve and thus can not represent the true errors in the network before reaching a fixed point. We further note that LE also alleviates the problem of unlearning in these other models: due to the misrepresentation of errors during relaxation, continuous learning
  2A similar concept has also been discussed in [26], but with the aim of implementing the future discounted states required for temporal difference learning; this form of prospectivity thus addresses a different problem and results in very different neuronal and synaptic dynamics.
  changes synaptic weights even in perfectly trained networks (Fig. 1c). This disruption of existing memories is related, but not identical to catastrophic forgetting.
  To illustrate time-continuous learning on continuous data streams, we first consider a simple Fouriersynthesis task in which a network driven by oscillating inputs learns to approximate a periodic (here: sawtooth) target signal. Despite the wavelengths of both input and target being much smaller than the time constant of the neuron, continuous plasticity in our model allows the output neuron to approximate the target well (Fig. 2a). The remaining differences merely reflect the fundamental limitation of a finite set of input frequencies and hidden units.
  We next turn the more traditional paradigm of classification tasks to demonstrate the scalability of our model. We train feedforward networks using LE on several standard benchmark datasets (see SI for details) using sample presentation times much shorter than the membrane time constant. We first learn MNIST [29] and HIGGS [30] with a fully connected architecture. After 100 epochs, our model reaches classification test errors (mean ± std) of (1.98 ± 0.11) % (MNIST) and (27.6 ± 0.4) % (HIGGS), on par with a standard ANN trained using stochastic gradient descent (SGD) and reaching (1.93 ± 0.14) % (MNIST) and (27.8 ± 0.4) % (HIGGS) with the same architecture, i.e., with the same number of learnable parameters (Fig. 2b,c). With feedback alignment (FA, [31]), we achieve test errors of (2.6 ± 0.1) % for MNIST. To illustrate the indifference of LE with respect to neuronal time constants, we repeated the MNIST experiments with significantly slower neuronal dynamics, obtaining the same results. In contrast, a network following the classical paradigm without prospective rates performs poorly: for membrane time constants of τm = 10ms and presentation times of T = 100ms per sample, the network does not exceed 90% accuracy on MNIST. For even shorter presentation times, the performance of such models quickly degrades further.
  Since our model does not assume any specific connectivity pattern, we can easily integrate different network topologies. Here we demonstrate this by introducing convolutional architectures on both MNIST and the more challenging CIFAR10 [33] datasets. On these datasets, our LE networks achieve test errors of (1.1 ± 0.1) % (MNIST) and (38.0 ± 1.3) % (CIFAR10), again on par with ANNs with identical structure at (1.08 ± 0.07) % (MNIST) and (39.4 ± 5.6) % (CIFAR10) (Fig. 2d,e). These results show that LE enables time-continuous learning using arbitrarily short presentation times in networks of leaky integrator neurons to achieve results competitive with standard BP in ANNs.
sections.heading: 5 Fast computation and learning in cortical microcircuits
sections.text: Due to the simplicity of their implementation, the principles of LE can be applied to models of approximate BP in the brain in order to alleviate the issues discussed above. Here we demonstrate how a network of hierarchically organized dendritic microcircuits [8, 34] can make use of our theoretical advances to significantly increase both inference and training speed, thus removing several critical shortcomings towards its viability as a scalable model of cortical processing. The resulting dynamical system represents a detailed and biologically plausible version of BP, with real-time dynamics, and phase-free, continual local learning able to operate on effectively arbitrary sensory timescales.
  The fundamental building block of this architecture is a cortical microcircuit model consisting of pyramidal cells and interneurons (Fig. 3a,b). Pyramidal cells have three compartments: a basal dendrite receiving bottom-up input from lower areas, an apical dendrite receiving top-down input from higher areas and lateral input from interneurons, and a somatic compartment that integrates dendritic information and generates the neuronal output. Interneurons consist of two compartments: a basal dendrite receiving input from pyramidal cells in the same layer, and a somatic compartment that receives input from pyramidal cells in higher layers. Pyramidal somatic compartments are leaky integrators of input from neighboring compartments (see SI for the full set of equations):
  Cmu̇ som i = gl (El − usomi ) + gbas ( vbasi − usomi ) + gapi ( vapii − u som i ) , (6)
  where i is the neuron index, El the leak potential, vbasi and v api i the basal and apical membrane potentials, respectively, and gbas and gapi the dendro-somatic couplings. Due to the conductancebased interaction between compartments, the effective time constant of the soma is τ eff := Cm/(gl + gbas + gapi). For somatic membrane potentials, assuming that apical dendrites encode errors (see below) and basal dendrites represent the input, this corresponds directly to Eqn. 3. Following [35], plasticity in basal synapses is driven by the local error signal given by the discrepancy between
  a b
  0 200 400 600 800 1000 epochs
  0
  20
  40
  60
  80
  100
  va li d
  at io
  n er
  ro r
  [% ]
  Tpres = 1 τeff Tpres = 10 τeff Tpres = 100 τeff
  d
  10−1 100 101 102
  Tpres[τeff ]
  0
  50
  100
  te st
  er ro
  r [%
  ]
  MC by Sacramento MC with LE
  horizontal vertical diagonal
  c
  𝑢𝑢𝑖𝑖som
  𝑢𝑢𝑗𝑗som
  𝑣𝑣𝑗𝑗bas
  𝑣𝑣𝑖𝑖bas
  𝑣𝑣𝑗𝑗 api
  𝑣𝑣𝑖𝑖 api
  𝑟𝑟𝑗𝑗
  𝑟𝑟𝑖𝑖
  𝑔𝑔bas
  𝑔𝑔bas
  𝑔𝑔api
  𝑔𝑔api
  output
  input
  output
  input 𝑢𝑢𝑖𝑖som
  𝑢𝑢𝑗𝑗som
  𝑣𝑣𝑗𝑗bas
  𝑣𝑣𝑖𝑖bas
  𝑣𝑣𝑗𝑗 api
  𝑣𝑣𝑖𝑖 api
  𝑟𝑟𝑗𝑗
  𝑟𝑟𝑖𝑖
  𝑔𝑔bas
  𝑔𝑔bas
  𝑔𝑔api
  𝑔𝑔api
  output
  input
  Figure 3: LE in cortical microcircuits. (a,b) Model architecture following [8]. Red: pyramidal cells, blue: interneurons; somatic compartments have darker colors. Each soma sends out a single axon that transmits either ϕ(u) (Sacramento et al.) or ϕ(ŭm) (LE). Except for top-down synapses, all synapses are plastic. (c) Synthetic dataset with 8 images grouped in 3 classes used to train a 3-layer network with 9-30-3 pyramidal cells. (d) Model performance during (top) and after (bottom) learning with (blue) and without (orange) LE. Top: Note the similarity in performance gains at the beginning of training, before the disruptive effects of relaxation begin to dominate. For better visualization, fluctuations are smoothed with a sliding window over 10 epochs. Bottom: Model performance (min, max and mean over 10 seeds) after 1000 epochs for different input presentation times.
  somatic and dendric membrane potentials usomi and v bas i : ẇij = ηW [ ϕ(usomi )− ϕ(αvbasi ) ] ϕ(usomj ) , (7)
  which is analogous to Eqn. 5 up to a monotonic transformation on the voltages (see SI).
  In this architecture, plasticity serves two purposes. For pyramidal-to-pyramidal feedforward synapses, it implements error-correcting learning as a time-continuous approximation of BP. For pyramidal-tointerneuron synapses, it drives interneurons to mimic their pyramidal partners in the layers above (see also SI). Thus, in a well-trained network, apical compartments of pyramidal cells are at rest, reflecting zero error, as top-down and lateral inputs cancel out. When an output error propagates through the network, these two inputs can no longer cancel out and their difference represents the local error ei. This architecture does not rely on the transpose of the forward weight matrix, improving viability for implementation in distributed asynchronous systems. Here, we keep feedback weights fixed, realizing a variant of feedback alignment. In principle, these weights could also be learned in order to further improve the local representation of errors Section 7.
  Incorporating the principles of LE is straightforward and requires only that neurons fire prospectively: ϕ(u) → ϕ(ŭeff). While we have already addressed the evidence for prospective neuronal activity, we note that our plasticity also uses these prospective signals, which constitutes an interesting prediction for future in-vivo experiments. We can now compare the behavior and performance of our LEaugmented model to its original archetype. Since large networks using the original model require prohibitively long training times when simulated with full dynamics rather than just their steady state [8], we use small networks and a small synthetic dataset as a benchmark (Fig. 3c).
  Our microcircuit model can learn perfect classification even for very short presentation times. In contrast, the original model without the prospective mechanism stagnates at high error rates even for this simple task. As discussed earlier, this can be traced back to the learning process being disrupted during relaxation. Without prospective dynamics, the model requires presentation times on the order of 100 τ eff to achieve perfect accuracy (Fig. 3d). In contrast, LE only degrades for presentation times below 0.1 τ eff, which is due to the limited resolution of our numerical integration method. Thus, incorporating LE into cortical microcircuits can bring the required presentation times into biologically plausible regimes, allowing networks to deal with rich sensory data.
sections.heading: 6 Robustness to substrate imperfections
sections.text: Computer simulations often assume perfectly homogeneous parameters across the network. Models can hence inadvertently rely on this homogeneity, resulting in unpredictable behavior and possibly fatal dysfunction when faced with the physics of analog substrates which are characterized by both heterogeneity in their components as well as temporal perturbation of their dynamics. Therefore, we consider robustness to spatio-temporal noise to represent a necessary prerequisite for any mathematical model aspiring to physical implementation, be it biological or artificial.
  Spatial noise reflects the individuality of cortical neurons or the heterogeneity arising from device mismatch in hardware. Here, we focus on the heterogeneity of time constants; in contrast to, for example, variability in synaptic parameters or activation functions, these variations can not be “trained away” by adapting synaptic weights [36–38]. The two time constants that govern neuron dynamics in our model, namely integration (Eqn. 3) and prospective coding (Eqn. 1), previously assumed to be identical, are affected independently by such variability. To differentiate between the two, we assign the prospective dynamics their own time constant: ŭr := u + τ ru̇. We can now model heterogeneity as independent, multiplicative Gaussian noise on all time constants: τm/r → (1+ξ)τm/r, with ξ ∼ N (0, σ2τ ); we use multiplicative noise to emphasize that our model is agnostic to absolute time scales, so only relative relationships between specific time constants matter.
  Due to the resulting mismatches between the timing of neuronal input and output, neuronal outputs suffer from exponential transients, leading to relaxation issues similar to the ones we already addressed in detail. However, depending on the transmission of top-down signals, the effects on learning performance can be very different. According to the formal theory, backward errors use the correct prospective voltages: e ∝ [ŭm −Wϕ(ŭr)] (Eqns. 4 and 5); this allows robust learning even for relatively large perturbations in the forward signals (Fig. 4a). In contrast, in biophysical implementations such as the microcircuits discussed above, neurons can only transmit a single output signal ϕ(ŭri), which consequently also affects the errors: e ∝ [ŭ r −Wϕ(ŭr)]. Since deviations due
  to a mismatch between integration and prospective time constants persist on the time scale of τm (see SI), even small amounts of mismatch can lead to a significant loss in performance (Fig. 4a).
  Here we address this issue by introducing a neuron-local adaptation mechanism that corrects the difference between prospective voltages ŭm and ŭr induced by mismatching time constants:
  τ̇m = ητ (ŭ r −Wr − b) u̇ . (8)
  In biological substrates, this could, for example, correspond to an adaptation of the transmembrane ion channel densities, whereas on many neuromorphic substrates, neuronal time constants can be adapted individually [39]. Before training the network we allow it to go through a “developmental phase” in which individual neurons are not affected by top-down errors and learn to match their prospective firing to their membrane dynamics (Fig. 4b), thus recovering the performance of networks with perfectly matched time constants (Fig. 4a). This developmental phase merely consists of presenting input samples to the network, for a duration that depends on the required matching precision. Here, we achieved mismatches below 1 ‰ within the equivalent of 20 training epochs. Note that neuronal time constants remain diverse after this phase, but are matched in a way that enables fast reaction of downstream areas to sensory stimuli – an ability that is certainly helpful for survival. This aligns well with in-vivo observations of parameters that are highly diverse between neurons and individuals, but are fine-tuned within each neuron in order to reliably produce a desired input-output relationship [40].
  We next model additive temporal noise on neuronal outputs as might be induced, for example, by noisy transmission channels: r → r+ξ, with ξ ∼ N (0, σ2r ). Formally, this turns membrane dynamics into a Langevin process. These perturbations add up over consecutive layers and can also accumulate over time due to recurrent interactions in the network. This introduces noise to the weight updates that can impair learning performance. Due to their slow integration of inputs, traditional neuron models filter out this noise, but our prospective mechanism effectively removes this filter. We thus need an additional denoising mechanism, for which we again turn to biology: by introducing synaptic filtering with a short time constant τ s, synaptic input is denoised before reaching the membrane; formally, r = ϕ(ŭm) is replaced by a low-pass-filtered rs in the energy function (Eqn. 2, see also SI). This narrow filter also mitigates possible numerical instabilities, as discussed in Section 3.
  Networks equipped with synaptic filters learn reliably even in the presence of significant noise levels (Fig. 4c). However, introducing this filter affects the quasi-instantaneity of computation in our networks, which then require longer input presentation times. Even so, these presentation times need only be “long” with respect to the characteristic time constant of relaxation mismatches – in this case, τ s. Thus, for the typical scenario of white noise described above, minuscule τ s on and even below biological time scales (see, e.g., [41–43]) can achieve effective denoising, without significantly affecting the advantages conferred by prospective coding. In conclusion, the demonstrated robustness of our model to spatial and temporal substrate imperfections introduced by simple, biologically inspired mechanisms, make it a promising candidate for implementation in analog physical systems.
sections.heading: 7 Implications and limitations
sections.text: In this section, we briefly address several interesting questions that arise from physical embeddings of LE, both in vivo and in silico.
  Within the context of LE, the microcircuit model used to implement error backpropagation carries several implications for cortical phenomenology beyond specific connectivity patterns. In particular, we expect that during active, attentive sensory processing, both during and after learning, cortical pyramidal cells and inhibitory interneurons will react simultaneously to changes in the sensory stimulus. Furthermore, we would also expect such synchronization at the level of cross-cortical networks, for example across the ventral visual stream. This should be observable in large-scale activity recordings with sufficient temporal resolution, for example in iEEG data.
  A further implication of our framework is that plasticity is, in principle, equanimous about the speed of sensory input changes. In particular, learning should be possible without neurons ever reaching a steady state. We would argue that the ability of mammals to learn from a continuously, and often quickly changing input stream already provides evidence for such quick processing and learning. In fact, stimulus presentation times for humans in a subliminal face-word association paradigm can be as short as 17 ms and still induce association learning [44]. Furthermore, we expect in-vivo plasticity in cortical subnetworks responsible for such forms of pattern recognition to explicitly depend on prospective neuronal states; this would contrast with other paradigms that only propose a dependence on instantaneous (e.g., [35]) or even low-pass-filtered (e.g., [45]) state variables.
  By explicitly approximating error backpropagation, our framework inherits its challenges with respect to biological plausibility, in particular the weight transport problem. In part, this is addressed by feedback alignment, as used in our cortical microcircuits. However, these weights can themselves be plastic in order to further boost learning performance [46–49]. The representation of errors by nudging neuronal activity also has the effect of diluting error signals (in addition to the vanishing gradient problem); this could be mitigated by adapting learning rates as a function of layer identity.
  One major aspect of biology that our framework does not explicitly address is the spike-based communication between neurons. In silico, this represents less of an obstacle, because pulsed communication packets can easily carry more information than just the pulse arrival times by including additional payload. In vivo, a similar role could be played by inter-spike-intervals within spike doublets or bursts, or by the precise spike timing used in, for example, spike latency codes [50].
  With respect to neuromorphic implementation, we should stress that our robustness analysis only represents a starting point and not a quantitatively faithful study of the expected effects in analog/mixedsignal hardware. While the investigated forms of spatiotemporal noise certainly are among the most salient, there are many other substrate-induced distortive effects to be considered, such as neuronal delays, limited bandwidth or limited synaptic real-estate [51, 52]. Ultimately, an actual demonstration in silico will have to be the definitive arbiter of the neuromorphic feasibility of LE.
sections.heading: 8 Conclusion
sections.text: We have introduced a new framework for inference and learning in physical systems composed of computational elements with finite response times. Our model rests on four simple axioms: prospective coding (Eqn. 1), neuronal mismatch energy (Eqn. 2), energy conservation under neuronal dynamics (Eqn. 3) and gradient descent on the energy under synaptic plasticity (Eqn. 5). In particular, incorporating the simple, biologically inspired mechanism of prospective coding allows us to avoid critical issues and scalability bottlenecks inherent to many current models of approximate BP in cortex. Furthermore, we have demonstrated robustness of the resulting implementations to substrate imperfections, a prerequisite for deployment in analog neuronal systems, both biological and artificial.
  Our framework carries implications both for neuroscience and for the design of neuromorphic hardware. The prospective mechanism described here would allow biological circuits to respond much faster than previously assumed. Furthermore, our framework suggests that both inference and learning take place on prospective, rather than instantaneous neuronal quantities. From a hardware perspective, this lifts the previously perceived limitations of slow analog components (as compared to digital ones) without relinquishing their power efficiency.
sections.heading: 9 Acknowledgements
sections.text: This work has received funding from the European Union 7th Framework Programme under grant agreement 604102 (HBP), the Horizon 2020 Framework Programme under grant agreements 720270, 785907 and 945539 (HBP), the Swiss National Science Foundation (SNSF, Sinergia grant CRSII5180316) and the Manfred Stärk Foundation. We acknowledge the use of Fenix Infrastructure resources, which are partially funded from the European Union’s Horizon2020 research and innovation programme through the ICEI project under the grant agreement No. 800858. Furthermore, we thank Mathieu Le Douairon, Reinhard Dietrich and the Insel Data Science Center for the usage and outstanding support of their Research HPC Cluster. A very special thank you goes to Ellis, for his uniquely wholesome influence during the final stretch of this work.
title: Latent Equilibrium