Analog Memories in a Balanced Rate-Based Network of E-I Neurons
The persistent and graded activity often observed in cortical circuits is sometimes seen as a signature of autoassociative retrieval of memories stored earlier in synaptic efficacies. However, despite decades of theoretical work on the subject, the mechanisms that support the storage and retrieval of memories remain unclear. Previous proposals concerning the dynamics of memory networks have fallen short of incorporating some key physiological constraints in a unified way. Specifically, some models violate Dale's law (i.e. allow neurons to be both excitatory and inhibitory), while some others restrict the representation of memories to a binary format, or induce recall states in which some neurons fire at rates close to saturation. We propose a novel control-theoretic framework to build functioning attractor networks that satisfy a set of relevant physiological constraints. We directly optimize networks of excitatory and inhibitory neurons to force sets of arbitrary analog patterns to become stable fixed points of the dynamics. The resulting networks operate in the balanced regime, are robust to corruptions of the memory cue as well as to ongoing noise, and incidentally explain the reduction of trial-to-trial variability following stimulus onset that is ubiquitously observed in sensory and motor cortices. Our results constitute a step forward in our understanding of the neural substrate of memory.
The paper returns to the venerable problem of associative memory in neural networks with a new fresh perspective, adding much new insight to an old problem. The authors observe that much previous work in the field ignored biological aspects such as Dale’s law, used binary memory representations in rate based models, and used non-sparse representations for memories. While all these issues have been dealt with individually in other work, the authors develop a strategy which allows them to address all issues. More concretely, they make use of a recent control-theoretic approach, based on spectral bounding techniques, to construct networks which deal with all these issues. In this approach the authors propose a cost function balancing three terms to achieve the desired goal through gradient descent. The first term in the cost contributes to the fixed point nature of the desired memories, the second term to the stability of the fixed points, and the third term encourages small weights which prevent saturation. By directly enforcing Dale’s principle in the weight parameterization, the authors are able to address all the above shortcomings of previous work, leading to a network where excitations and inhibition are naturally balanced. Interestingly, while they do not enforce saturation in the single neuron firing, this arises naturally through the solution of the optimization problem. Importantly, the authors only make use of the excitatory neurons to implement the memory (which is consistent with the fact it is usually the excitatory neurons that send output to other regions), and view the inhibitory neurons as variables to be optimized over.  The authors validate their results through numerical simulations, and demonstrate its noise robustness. In addition to demonstrating the good associative memory properties of the network, the authors are also able to explain experimental findings showing that the trial to trial variability is reduced following stimulus onset (although this seems to be a general feature of associative memories with spontaneous baseline firing). This is explained through convergence to the attractor closets to the initial condition. In summary, this is a clear and well written paper. It tackles a classic problem in neural computation using novel control-theoretic tools, and demonstrates how a biologically plausible associative memory may function. They do not propose a biologically plausible learning mechanism at this point, but hint at some recent proposals that may lead to such rules. I found the utilization of the inhibitory neurons as resources to be interesting and beneficial. Overall, the paper presents a clear, concise and important contribution to neural computation. A few minor issues:The authors use \gamma=0.04 in eq. (3). It would good to discuss the sensitivity of the results to this choice (although I expect it to be small)A log-normal distribution for memory states is proposed on line 115. Please motivate this. The cost function (4) may yield solutions which violate the exact fixed-point nature of the memories. How severe is this violation? This should be quantified and discussed. Line 254-256: The statements here were not clear to me. Please clarify.The authors do not mention capacity issues. It would be nice to make some statement, even preliminary, about this important computational aspect. It would be nice if the authors could discuss the computational benefits of Dale’s law. Namely, can this law be shown to enhance computational power under specific constraints? This is not necessarily the case, and may be a result of evolutionary and/or biophysical constraints, but it would be nice to relate to this.