Improved Expressivity Through Dendritic Neural Networks
A typical biological neuron, such as a pyramidal neuron of the neocortex, receives thousands of afferent synaptic inputs on its dendrite tree and sends the efferent axonal output downstream. In typical artificial neural networks, dendrite trees are modeled as linear structures that funnel weighted synaptic inputs to the cell bodies. However, numerous experimental and theoretical studies have shown that dendritic arbors are far more than simple linear accumulators. That is, synaptic inputs can actively modulate their neighboring synaptic activities; therefore, the dendritic structures are highly nonlinear. In this study, we model such local nonlinearity of dendritic trees with our dendritic neural network (DENN) structure and apply this structure to typical machine learning tasks. Equipped with localized nonlinearities, DENNs can attain greater model expressivity than regular neural networks while maintaining efficient network inference. Such strength is evidenced by the increased fitting power when we train DENNs with supervised machine learning tasks. We also empirically show that the locality structure of DENNs can improve the generalization performance, as exemplified by DENNs outranking naive deep neural network architectures when tested on classification tasks from the UCI machine learning repository.
UPDATED REVIEW: the authors responded to most comments. Although they provided details in the response, It's not clear to me if they will include discussions of (1) piecewise activation functions and in neuroscience e.g. two-compartment models in the related work (2) generalization (+learning curves) in the manuscript; I think both these things would improve the work. I still find it strange that no generalization results were presented in the original submission.  I agree with the other reviewers that the title sounds strange (typically you're advised not to use subjective descriptions like "great" or "amazing" in scientific work). I have updated my rating to 6.  ========================  PAPER SUMMARY:The authors propose a network architecture to model synaptic plasticity in dendritic arbours, which ends up looking to me like alternating layers of maxout and regular units, with some randomly-dropped connections. The proposed architecture is tested on MNIST, CIFAR, and a range of UCI classification tasks, and found to outperform alternatives. Insight about why this is the case is provided in the form of an analysis of expressivity, measured as number of linear regions in loss space.  STRENGTHS:   - quality: The paper is generally well-written and the structure and experiments are largely well-suited to evaluate the proposed architecture. The introduction and related work do a good job contextualizing the work. The experiments about expressivity are interesting and relevant.   - clarity: What is proposed and what is done are both clear. Details of datasets, experiments, and explanations of related work are generally very good.   - originality: To my knowledge, the proposed architecture is novel. Being somewhat familiar with the biology, I find piecewise linear/max operations a reasonable approximation intuitively.   WEAKNESSES:   - quality: It seems to me that the chosen "algorithm" for choosing dendrite synapses is very much like dropout with a fixed mask. Introducing this sparsity is a form of regularization, and a more fair comparison would be to do a similar regularization for the feed-forward nets (e.g. dropout, instead of bn/ln; for small networks like this as far as I know bn/ln are more helpful for optimization than regularization). It also seems to me that the proposed structure is very much like alternating layers of maxout and regular units, with this random-fixed dropout; I think this would be worth comparing to.  I think there are some references missing, in the area of similar/relevant neuroscience models and in the area of learned piecewise activation functions.  It would be reassuring to mention the computation time required and whether this differs from standard ff nets. Also, most notably, there are no accuracy results presented, no val/test results, and no mention is made of generalization performance for the MNIST/CIFAR experiments.    - clarity:  Some of the sentences are oddly constructed, long, or contain minor spelling and grammar errors .The manuscript should be further proofread for these issues. For readers not familiar with the biological language, it would be helpful to have a diagram of a neuron/dendritic arbour; in the appendix if necessary. It was not 100% clear to me from the explanations whether or not the networks compared have the same numbers of parameters; this seems like an important point to confirm.   - significance: I find it hard to assess the significance without generalizaion/accuracy results for the MNIST/CIFAR experiments.   REPRODUCABILITY: For the most part the experiments and hyperparameters are well-explained (some specific comments below), and I would hope the authors would make their code available.  SPECIFIC COMMENTS:   - in the abstract, I think it should say something like "...attain greater expressivity, as measured by the change in linear regions in output space after [citation]. " instead of just "attain greater expressivity"   - it would be nice to see learning curves for all experiments, at least in an appendix.    - in Figure 1, it would be very helpful to show a FNN and D-Net with the same number of parameters in each (unless I misunderstood, the FNN has 20 and the DNN has 16).   - There are some "For the rest part" -> for the rest of (or rephrase)   - missing references: instead of having a section just about Maxout networks, I think the related work should have a section called something like "learned piecewise-linear activation functions" which includes maxout and other works in this category, e.g. Noisy Activation Functions (Gulcehre 2016). Also, it's not really my field but I believe there is some work on two-compartment models in neuroscience and modeling these as deep nets which would be quite relevant for this work.   - It always bothers me somewhat when people refer to the brain as 'sparse' and use this as a justification for sparse neural networks. Yes, overall/considering all neurons the brain as one network it would be sparse, but building a 1000 unit network to do classification is much more analogous to a functional subunit of the brain (e.g. a subsection of the visual pathway), and connections in these networks are frequently quite dense. The authors are not the first to make this argument and I am certainly not blaming them for its origin, but I take this opportunity to point it out as (I believe) flawed. :)   - the definition of "neuron transition" is not clear to me - the sentence before Definition 2 suggests that it is a change in _classification_ (output space), which leads to a switch in the linear region of a piecewise linear function, but the Definition and the subsequent sentence seem to imply it is only the latter part (a change from one linear region of the activation function to another; nothing to do with the output space). If the latter, it is not clear to me how/whether or not these "transitions" say anything useful about learning. If it is the former (more similar to Raghu et al), I find the definition given unclear.    - I like the expressiveness experiments, but It would be nice to see some actual numbers instead of just descriptions.   - unless I missed it somehow, the "SNN" is never defined. and it is very unclear to me whether it refers to a self-organizing neural network cited in [12]. or a "sparse" neural network, and in any case what exactly this architecture is.   - also possible I missed it despite looking, but I could not find what non-linearity is used on D-Nets for the non-dendrite units   OVERALL ASSESSMENT: My biggest issue with the paper is the lack of mention/results about generalization on MNIST/CIFAR, and the ambiguity about fair comparison. If these issues are resolved I would be very willing to change my rating.  CONFIDENCE IN MY SCORE: This is the first time I've given a confidence of 5. With due credit to the authors, I believe I've understood most things about the paper, and I am familiar with the relevant work. Of course I'm not infallible and it's possible I've missed or misunderstood something, especially relating to the things I noted finding unclear.