Do Deep Nets Really Need to be Deep?
Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this paper we empirically demonstrate that shallow feed-forward nets can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow nets can learn these deep functions using the same number of parameters as the original deep models. On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional models.
This paper presents experiments with shallow network trained to mimicthe predictions of deeper, more complex networks trained on the samedata.  When trained to predict the pre-softmax outputs of the"teacher" networks, these mimic networks are able to approximate theperformance of the deeper networks, albeit with more parameters.  Theyconsistently outperform networks with the same architecture trainedthe cross-entropy objective function.The paper presents an interesting -- and perhaps useful -- set ofexperiments, but the theoretical analysis is extremely weak.  Thereare a number of bold claims that in my opinion are not demonstrated bythe experimental work, and many speculative statements, the truth ofwhich there has been little effort to investigate.  The paper isreasonably original, though methods similar to the mimic-network havebeen presented before.The paper is clearly written and reasonably well-organised, althoughthe colloquial style ("what is the source of this magic?") in theintroduction is not to my taste.  The strong part of the paper is thedemonstration that a shallower neural network can achieve goodperformance when trained to mimic a deeper "teacher" network, orensemble.  This is good news for software engineers tasked withbuilding low-footprint systems, although, as noted, related techniqueshave already been employed [20].  It is presented as a surprisingfinding, but I am not sure why this should be so: not dissimilarmethods have also had success, such as the KL-HMM.  As in that model,it may be seen that the teacher network is providing a large amount ofinformation to the learner in the vector z(t), as in the KL-HMM wherethe p(t) are used directly, so it is somewhat misleading to say thatthe learner is simply trained to mimic the (one-best) predictions ofthe teacher network - it's being given much more information thanthat.  Analagously, the purported benefit of the KL-HMM is that theposteriors generated by one NN are helpful targets for another (abeitmuch simpler) model.  (Note, I am aware that the KL-HMM differsmaterially from the mimic-model).Section 5.1 is particularly weak in my opinion.  If I were to write itwith my own theories as to why the mimic models work, they would bequite different to those being presented, but no more speculative.The first theory, that the teacher eliminates errors, could easily betested, but isn't.  The second, "the complexity in the data set hasbeen washed away" makes no sense at all -- what does this mean?  Thethird seems more plausible, but again, this could easily be tested."Model compression appears to be a form of regularization" needs muchmore support, as it underpins the core claim that if only shallowmodels could be regularized more successfully, then they could betrained to perform as well as the deep models without the help of ateacher. However, this is a diversion from my main point, which is that Ibelieve that the theoretical premise of the paper is flawed.  Aswidely known and cited [4], a neural network with a single hiddenlayer can approximate any decision boundary, with a large enoughnumber of hidden units.  This point, which seems to be the coreexperimental finding of the paper, is not novel.  However, the "largeenough" number of hidden units implies the requirement for anabritrarily large amount of training data.  Isn't one of the keystrengths of a deep network not the deep functional architecture perse, but rather its ability to make more efficient use of the availabledata when learning the weights, due to its hierarchical structure?This would certainly be the conclusion I would be tempted to draw fromTable 1, for example, where you demonstrate that a deep networkconsiderably outperforms a shallow network with the same number ofparameters, but that the deep structure isn't actually necessary toproduce equivalently-good decision boundaries.  So my answer to thequestion posed in the second paragraph of the introduction wouldprobably be (g).  There argument that the shallow network couldperform better, if only the regularization were more effective or thelearning algorithm improve, just isn't supported by the experiments.And sadly there is no theoretical justification either.This is a shame, because as I said, the experiements are nice, andthere are lots of future directions the work could go.  In speechrecognition, for example, it would be interesting to investigatewhether the learner/teacher methodology would work across differenttraining sets: for example, for speaker adaptation; or multi-lingualor cross-domain ASR.  There are other interesting questions that areleft unanswered: for example, what would happen if you had a muchlarger quantity of unlabelled data, and a teacher network trained on alabelled subset.  Would the mimic network assymptotically approach theperformance of the teacher, or given the extra data, could the methodexceed the performance of the teacher?  I suspect not, but it would begood to find out.In summary, this paper an interesting concept, but it needs to bepresented quite differently, with more theoretical analysis and lessspeculation.Some specific points:090 -> "training on these logit values makes learning easier for theshallow net by placing emphasis on confident predictions" - really?308 -> "shallow net achieves accuracy comparable to CNNs" - this is a bit misleading as the accuracy is some way off that of the teacher CNN here.