SPARSE DICTIONARY LEARNING BY DYNAMICAL NEURAL NETWORKS
A dynamical neural network consists of a set of interconnected neurons that interact over time continuously. It can exhibit computational properties in the sense that the dynamical system's evolution and/or limit points in the associated state space can correspond to numerical solutions to certain mathematical optimization or learning problems. Such a computational system is particularly attractive in that it can be mapped to a massively parallel computer architecture for power and throughput efficiency, especially if each neuron can rely solely on local information (i.e., local memory). Deriving gradients from the dynamical network's various states while conforming to this last constraint, however, is challenging. We show that by combining ideas of top-down feedback and contrastive learning, a dynamical network for solving the 1 -minimizing dictionary learning problem can be constructed, and the true gradients for learning are provably computable by individual neurons. Using spiking neurons to construct our dynamical network, we present a learning process, its rigorous mathematical analysis, and numerical results on several dictionary learning problems.
The authors study sparse coding models in which unit activations minimize a cost that combines: 1) the error between a linear generative model and the input data; an 2) the L1 norm of unit activations themselves. They seek models in which both the inference procedure -- generating unit activations in response to each input data example -- and the learning procedure -- updating network connections so that the inferences minimize the cost function -- are local. By "local" they mean that the update to each unit's activation, and the updates to the connection weights, rely only on information about the inputs and outputs from that unit / connection. In a biological neural network, these are the variables represented by pre- and post-synaptic action potentials and voltages, and in hardware implementations, operations on these variables can be performed without substantially coordinating between different parts of the chip, providing strong motivation for the locality constraint(s). 
The authors achieve a local algorithm that approximately optimizes the sparse coding objective function by using feedback: they send the sparse coding units' activities "back" to the input layer through feedback connections. In the case where the feedback connection matrix is the transpose of the sparse coding dictionary matrix (D), the elementwise errors in the linear generative model (e.g., the non-local part of the sparse coding learning rule obtained by gradient descent) are represented by the difference between the inputs and this feedback to the input layer: that difference can be computed locally at the input units and then sent back to the coding layer to implement the updates. The feedback connections B are updated in another local process that keeps them symmetric with the feedforward weights: B= D= F^T throughout the learning process. 
The authors provide several theorems showing that this setup approximately solves the sparse coding problem (again, using local information), and show via simulation that their setup shows similar evolution of the loss function during training, as does SGD on the sparse coding cost function.
I think that the paper presents a neat idea --  feedback connections are too often ignored in computational models of the nervous system, and correspondingly in machine learning. At the same time, I have some concerns about the novelty and the presentation. Those are described below:
1. The paper is unnecessarily hard to read, at least in part due to a lack of notational consistency. As just one example, with B=D, why use two different symbols for this matrix? This just makes is so that your reader needs to keep track mentally of which variable is actually which other variable, and that quickly becomes confusing. I strongly recommend choosing the simplest and most consistent notation that you can throughout the paper.
2. Other recent studies also showed that feedback connections can lead to local updates successfully training neural networks: three such papers are cited below. The first two papers do supervised learning, while the third does unsupervised learning. It would be helpful for the authors to explain the key points of novelty of their paper: application of these feedback connection ideas to sparse coding. Otherwise, readers may mistakenly get the impression that this work is the first to use feedback connections in training neural networks.
Guerguiev, J., Lillicrap, T.P. and Richards, B.A., 2017. Towards deep learning with segregated dendrites. ELife, 6, p.e22901.
Sacramento, J., Costa, R.P., Bengio, Y. and Senn, W., 2018. Dendritic cortical microcircuits approximate the backpropagation algorithm. arXiv preprint arXiv:1810.11393.
Federer, C. and Zylberberg, J., 2018. A self-organizing short-term dynamical memory network. Neural Networks.
3. Given that the performance gains of the locality (vs something like SparseNet) are given such emphasis in the paper, those should be shown from the numerical experiments. This could be quantified by runtime, or some other measure.
4. The discussion of prior work is a little misleading -- although I'm sure this is unintentional. For example, at the top of p. 3, it is mentioned that the previous local sparse coding models do not have rigorous learning objectives. But then the appendix describes the learning objectives, and the approximations, made in the prior work. I think that the introduction should have a more transparent discussion of what was, and was not, in the prior papers, and how the current work advances the field.
5. The paper -- and especially appendix C2 -- makes strong emphasis of the importance finding local implementations of true gradient descent, as opposed to the approximations made by prior authors. I'm not sure that's such a big deal, given that Lillicrap et al. showed nicely in the paper cited below that any learning rule that is within 90 degrees of true gradient descent will still minimize the cost function: even if an algorithm doesn't move down the steepest path, it can still have updates that always move "downhill", and hence minimize the loss function. Consequently, I think that some justification is needed showing that the current model, being closer to true gradient descent, really outperforms the previous ones. 
Lillicrap, T.P., Cownden, D., Tweed, D.B. and Akerman, C.J., 2016. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7, p.13276.