Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fullyconnected networks. Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on the MNIST, CIFAR-10, and ImageNet datasets, explore variants of target-propagation (TP) and feedback alignment (FA) algorithms, and examine performance in both fully-and locally-connected architectures. We also introduce weight-transport-free variants of difference target propagation (DTP) modified to remove backpropagation from the penultimate layer. Many of these algorithms perform well for MNIST, but for CIFAR and ImageNet we find that TP and FA variants perform significantly worse than BP, especially for networks composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.
Summary  Under the objective of finding biologically plausible alternatives to backpropagation that perform well on difficult tasks, variants of difference target propagation are compared to direct feedback alignment and backpropagation. All variants are found to perform significantly worse than backpropagation in digit classification (MNIST) and, in particular, in object classification (CIFAR-10 and ImageNet).   Quality  The study is mostly well done and the results seem plausible. But, to qualify as a solid and reproducible benchmark, I expect more details. Also, some parts feel a bit preliminary.  1. For how many epochs were the different methods trained? Do you report optimal values for a fixed number of training epochs or do you report the test error at the optimal validation value for each method? It seems, for example, that the test classification errors in Figure 2 (right) are still decreasing after epoch 500 for all variants of DTP. 2. With how many different initialisations where the results obtained in table 1 and 2? Did you repeat each experiment so many times that confidence interval are negligible? 3. Can you report the best hyperparameters in addition to the details of hyperparameter optimization? 4. AO-SDTP was tested only for 512 auxiliary outputs and the authors refer to future work (line 268). It would be interesting to know more about the performance of AO-SDTP for different dimensionality of z; potentially even for different ways to obtain the random features. 5. Since DTP alternating was always best on MNIST and CIFAR-10, I think it would be interesting to see it's performance on ImageNet, also to have an estimate of what ideally could be achieved with AO-SDTP.   Clarity  The paper is well written and easy to follow.  One minor point minor point where I stumbled, though, was in table 2, where top-1 and top-5 is not defined. For readers familiar with ILSVRC2012 - I guess this is what you are using - it is probably immediately clear what is meant. For everybody else, one sentence of explanation would help.  Another minor point that I believe could still be improved is the notation of the different losses. For example, why L(h_L) and not L(Theta)? What is L_y in section 5.5 of the appendix?   Originality  The work is original in the sense that the evaluation of DTP variants on MNIST, CIFAR-10 and Imagenet was never published before. The variants themselves are original as well, but minor and straightforward modifications of the original DTP.   Significance  It is important that people working on biologically plausible alternatives to backpropagation know about the limitations of DTP and variants. But since the paper does not present an advancement of state-of-the-art alternatives to backpropagation I think it would be better suited for a more specialized audience, maybe at a NIPS workshop.     Minor Points  - In Figure 2 & 4 the colours for parallel and alternating are hard to distinguish in print. - References 12 & 13 (identical up to title) could be replaced by https://elifesciences.org/articles/22901  - line 47: connections - line 68: to unveil - lines 77-78: I am not a native English speaker but "which eliminate significant lingering biologically implausible features" sounds strange to me.  - lines 109-110: I don't think that the backprop equations themselves imply a mode of information propagation that does not influence neural activity. The neural activity in different phases could corresponds to different quantities of the backprop algorithm. - I don't understand the sentence that starts on line 182. - line 259: perform better - I think the statement starting on line 294 is a bit strong. Reference 37 falls in this period and I am sure one can find further works in computational neuroscience on this topic between the mid 1990s and 2010.   Let me conclude with two questions that go beyond the discussion of this specific paper but address the attempt to find biologically plausible alternatives to backpropagation more broadly. I agree with the authors that biologically plausible methods should also be evaluated on behavioural realism. But what is a behaviourally realistic task? Is the repetitive presentation of static images in minibatches together with corresponding labels a behaviourally realistic task? Shouldn't we rather search for difficult and behaviourally realistic tasks that are less inspired by machine learning challenges but more by actual tasks humans solve?  ==== The author's response answered most questions and concerns. Suboptimal I find the decision to compare the methods on a fixed number of epochs. I can understand it because of limited computation resources. But since some of the curves did not seem to saturate (see point 1 above) I would be curious to know whether the bio-plausible alternatives would become significantly better if the number of epochs were a tunable hyper-parameter. Note that I think optimal performance after a fixed number of epochs is a perfect metric in machine learning but I am not convinced it is the right one to compare bio-plausible alternatives, since our random initializations of the network weights probably don't capture the strong inductive biases that evolutionary processes instantiate in biological neural networks.