The family of feedback alignment (FA) algorithms aims to provide a more biologically motivated alternative to backpropagation (BP), by substituting the computations that are unrealistic to be implemented in physical brains. While FA algorithms have been shown to work well in practice, there is a lack of rigorous theory proofing their learning capabilities. Here we introduce the first feedback alignment algorithm with provable learning guarantees. In contrast to existing work, we do not require any assumption about the size or depth of the network except that it has a single output neuron, i.e., such as for binary classification tasks. We show that our FA algorithm can deliver its theoretical promises in practice, surpassing the learning performance of existing FA methods and matching backpropagation in binary classification tasks. Finally, we demonstrate the limits of our FA variant when the number of output neurons grows beyond a certain quantity.
This paper examines the question of learning in neural networks with random, fixed feedback weights, a technique known as “feedback alignment”. Feedback alignment was originally discovered by Lillicrap et al. (2016; Nature Communications, 7, 13276) when they were exploring potential means of solving the “weight transport problem” for neural networks. Essentially, the weight transport problem refers to the fact that the backpropagation-of-error algorithm requires feedback pathways for communicating errors that have synaptic weights that are symmetric to the feedforward pathway, which is biologically questionable. Feedback alignment is one approach to solving the weight transport problem, which as stated above, relies on the use of random, fixed weights for communicating the error backwards. It has been shown that in some cases, feedback alignment converges to weight updates that are reasonably well-aligned to the true gradient. Though initially considered a good potential solution for biologically realistic learning, feedback alignment both has not scaled up to difficult datasets and has no theoretical guarantees that it converges to the true gradient. This paper addresses both these issues.
To address these issues, the authors introduce two restrictions on the networks: (1) They enforce “monotone” networks, meaning that following the first layer, all synaptic weights are positive. This also holds for the feedback weights. (2) They require that the task in question be a binary classification task. The authors demonstrate analytically that with these restrictions, direct feedback alignment (where the errors are communicated directly to each hidden layer by separate feedback weights) is guaranteed to follow the sign of the gradient.  (Importantly, they also show that monotone nets are universal approximators.) Empirically, they back up their analysis by demonstrating that in fully connected networks that obey these two restrictions they can get nearly as good performance as back propagation on training sets, and even better performance on tests sets sometimes. However, they also demonstrate (empirically and analytically) that violating the second requirement (by introducing more classes) leads to divergence from the gradient and major impairments in performance relative to backpropagation.
Ultimately, I think this is a great paper, and I think it should be accepted at ICLR. It provides some of the first rigorous analysis of feedback alignment since the original paper came out, and unlike those original analyses, it is not restricted to linear networks (which are certainly not universal function approximators). I have looked over the proofs, and they seem to all be correct. As well, I found the paper easy to read, which was nice. However, there are a few things that could be done to clarify the contributions and situate the work within the field of biological learning algorithms better:
1) Though they do not include rigorous analyses, two previous papers have demonstrated empirically that feedback alignment works extremely well as long as the feedback weights share the same sign as the feedforward weights (see: Moskovitz, Theodore H., Ashok Litwin-Kumar, and L. F. Abbott. "Feedback alignment in deep convolutional networks." arXiv preprint arXiv:1812.06488 (2018) and Liao, Qianli, Joel Z. Leibo, and Tomaso Poggio. "How important is weight symmetry in backpropagation?." In Thirtieth AAAI Conference on Artificial Intelligence. 2016). Due to the requirement for monotone networks, this work is also providing a guarantee that the sign of feedforward and feedback weights are the same. That does not subtract substantially from the contributions of this paper, as the provision of the analytical guarantees is important. But, it is important for the authors to consider how their work relates to this past work. For example, could their analytical approach work equally well with nothing more than a sign symmetry guarantee? This should at least be discussed.
2) It should be admitted somewhere in the paper that the second requirement on the networks for binary tasks is deeply unbiological. As such, it should be recognized in discussion that this paper provides some important contributions to our understanding of feedback alignment, but does not ultimately move the question of biologically realistic learning forward all that much. Indeed, the discussion at the end about applications notably ignores biology. But, rather than just ignoring it, the biological mismatch should be openly admitted.
3) The results with the test sets are a little strange, at least for the tests with larger numbers of categories. In Bartunov et al. (2016), they reported not only better training set results with backprop, but also better test set results generally, than feedback alignment. Are the authors sure that their results, in say, Table 4, are not indicative of insufficient hyperparameter optimization?
Small notes:
- Lillicrap et al.’s paper was eventually published in Nature Communications (see citation above), and the reference should be changed to reflect this.
- The discussion on the impact of convolutions could be beefed up a little bit. In particular, it could be discussed relative to the results of Moskovitz et al. (above) who show that convnets work fine with nothing but guaranteed sign symmetry.