Backpropagation is driving today's artificial neural networks (ANNs). However, despite extensive research, it remains unclear if the brain implements this algorithm. Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach. Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide. We provide proof that our approach converges to the true gradient for certain classes of networks. In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match or the performance of exact gradient-based learning. Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.
This paper proposes a method that addresses the "weight transport" problem [1] (not cited by the authors) emphasizing the biological infeasibility of artificial neural networks (ANNs) which are trained by gradients computed by the backpropagation algorithm [2a, 2b]. It is arguably the most eminent criticism (among many many others) when learning in the brain is modelled by such ANNs.
The authors cite only Rumelhart et al. (1986) for backpropagation (which is the same as the reverse mode of automatic differentiation), although Rumelhart cited neither Linnainmaa, the inventor of the method [2a], nor Werbos, who first applied it to ANNs [2b]. 
The authors propose to estimate the weights for the backward pass using a noise-based estimator and provide theoretical and empirical arguments. The proposed method is compared with backpropagation (the ground truth) and direct feedback alignment (DFA; which resorts to random matrices for the backward pass) on small fully-connected, convolutional, and fully-connected auto-encoder networks for MNIST, CIFAR10, and CIFAR100 and offers analysis based on insights from recent work.
Generally, this work is well-placed and the method straightforward and novel. That said, we found the paper to be difficult to parse with some questionable claims which we don't believe are sufficiently backed up by experiments. Due to the following issues, we think that the paper is not yet quite ready for acceptance.
1.) As the authors point out, the recent work by Akrout et al. [3] is very closely related but the paper is lacking a significant discussion let alone an empirical comparison. Extrapolating from the experiments in this paper we think that the presented method is likely going to underperform since Akrout et al. show on-par performance with backprop on imagenet.
2.) The paper appears to blend synthetic gradients [4a, 4b] and feedback alignment to a somewhat questionable degree. A connection that is usually not done in other recent papers about biologically feasible ANNs. And that for good reasons: synthetic gradients do not address the weight transport problem. They backprop the synthetic gradients in order to train the gradient estimators of the previous layer.
Also, the paper cites only the recent reference [4b] (2016) on synthetic gradients but not the original work [4a] (1990).
3.) The authors claim "Thus our method illustrates a biologically realistic way by which the brain could perform gradient descent learning". Even though their method is a more plausible way by which the brain could perform gradient descent learning, it is still far from realistic. The proposed method still implicitly "transports" a lot of information from the forward pass to the backward pass (such as a. topology or b. the derivative of the activation function or c. the activations of the forward pass (h^i) that are necessary to compute the gradient estimate in either case).
4.) The experimental results of the proposed method but also the baseline are simply too far from the state of the art. This is partially dismissed in section 5 due to regularization and data augmentation "tricks". We'd like to point out that feedback alignment has been reported to achieve <2% error on the MNIST dataset [6] which is significantly better than the results reported in this work without any tricks. Similar seems to be the case for CIFAR10 and CIFAR100. The reported results on CIFAR 10 and 100 are over 20% below the state of the art. Furthermore, many previous papers provide code which arguably makes such experiments particularly easy (see [7]). We do not expect the authors to provide state of the art results but they should be at least in the same ball-park.
5.) The authors claim "(...) this hybrid approach can solve large-scale problems." We don't think this is backed up by experiments as most models are rather small (all < 50k parameters except the CNN architecture) and evaluated on toy datasets (for vision standards; while underperforming as already mentioned).
6.) It is not obvious how the noise-based estimation will scale with width and depth. The authors make the observation that "The number of neurons has an effect". This can be seen in the appendix in Fig 4E where the relative error increases exponentially with the number of neurons in the first layer (note that the second layer doesn't increase since it remains unchanged). This seems to indicate that the method would not scale well.
7.) Why do the authors decide to train the MNIST networks with such a small learning rate (4e-4) for 2 million steps? Better results can be achieved with just a few thousand steps. Does the method work in the "general" setting?
8.) With regards to Figure 2C for layer 2: Given a random initialization of the backward pass for feedback alignment but also the presented method, we would expect that the sign congruence of the first few iterations to be the roughly the same on average. That doesn't seem to be the case according to the figure. Is there are an explanation for this observation?
9.) Why the sudden switch from feedback alignment to direct feedback alignment for the CNN experiment in section 4.3? Does the approach of section 4.2 not work with CNNs?
10) We hope the authors could clear up some confusion with regards to section 3 and the consistency proofs in the appendix.
10.a) Apparently, the notation slightly changes? In section 2, layers are indicated by 1 <= i <= N as the superscript. In section 3, i prevails but a new superscript 1 <= n <= N is used for the layers. Theorem 1 then states that the least-squares estimator for B^(i+1)\tilde{e}^{i+1} is (\hat{B}^{N+1})'. There seems to be a notation issue since it is our understanding that every layer should have its own estimated weights.
10.b) We believe A03 should state that E[\tilde{e}^i (\tilde{e}^i)'] is of full rank and not simply \tilde{e}^i (\tilde{e}^i)' assuming that \tilde{e} stands for the equivalent X of regular least-squares. From the main text (specifically the eq in section 2.1) one might assume that \tilde{e}^i are vectors but the proof seems to treat them like the design matrices X in the regular least-squares estimator.
10.c) It is awkward that in the proof of Theorem 1 and 2 the population size is suddenly T, the same symbol for the transpose.
10.d) The regular OLS estimator of the coefficients is b = (X'X)(X'y) where X is a full-rank design matrix and y are the targets. It is our understanding that \lambda, as defined by eq 12, is equivalent to the y of the regular OLS estimator. But in the following equation (the bottom of page 14), the W^{N+1} of \lambda is factored out, apparently assuming e^{N+1}e^{N+1}' = I. Maybe we made a mistake, we'd appreciate if the authors could clarify this step in the proof.
11.) Finally, we add some additional minor comments or sources of confusion:
- we'd appreciate if all equations were numerated
- In section 2, the definition of the loss is verbose and partially unnecessary. L(y,\hat{y}(x)) never appears in the text. Furthermore, \hat{y} is not introduced as a function so writing \hat{y}(x) is awkward.
- Similar for the equation in section 2.2.
- 4.1 first line: the tilde is misplaced
- Figure 3A, the black line has no label (assumed to be DAE which is missing)
- 4.2 first inline equation: the tilde is misplaced
- Table 1: no decimal values, not std. Not clear if the difference between DFA and note perturbation is significant.
- Figure 4A-E is missing the crucial comparison with feedback alignment
References:
[1] Grossberg, Stephen. "Competitive learning: From interactive activation to adaptive resonance." Cognitive science 11.1 (1987): 23-63.
[2a] Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis, Univ. Helsinki, 1970. FORTRAN code on pages 58-60. See also BIT 16, 146-160, 1976.
[2b] Werbos, Paul J. "Applications of advances in nonlinear sensitivity analysis." System modeling and optimization. Springer, Berlin, Heidelberg, 1982. 762-770.
[3] Akrout, Mohamed MA, et al. "Deep learning without weight transport." Advances in Neural Information Processing Systems. 2019.
[4a] J.  Schmidhuber. Networks adjusting networks. In J. Kindermann and A. Linden, editors, Proceedings of `Distributed Adaptive Neural Information Processing', St. Augustin, 24.-25.5. 1989, pages 197-208. Oldenbourg, 1990. Extended version: TR FKI-125-90 (revised), Institut für Informatik, TUM. http://people.idsia.ch/~juergen/FKI-125-90ocr.pdf See section "An Approach to Local Supervised Learning in Recurrent Networks."
[4b] Jaderberg, Max, et al. "Decoupled neural interfaces using synthetic gradients." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[5] Czarnecki, Wojciech Marian, et al. "Understanding synthetic gradients and decoupled neural interfaces." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[6] Nøkland, Arild. "Direct feedback alignment provides learning in deep neural networks." Advances in neural information processing systems. 2016.
[7] https://paperswithcode.com/sota/image-classification-on-cifar-10?p=maxout-networks
For now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way. Let us wait for the rebuttal.
Edit: After the rebuttal, we increased our score.