Constructing Deep Neural Networks by Bayesian Network Structure Learning
We introduce a principled approach for unsupervised structure learning of deep neural networks. We propose a new interpretation for depth and inter-layer connectivity where conditional independencies in the input distribution are encoded hierarchically in the network structure. Thus, the depth of the network is determined inherently. The proposed method casts the problem of neural network structure learning as a problem of Bayesian network structure learning. Then, instead of directly learning the discriminative structure, it learns a generative graph, constructs its stochastic inverse, and then constructs a discriminative graph. We prove that conditional-dependency relations among the latent variables in the generative graph are preserved in the class-conditional discriminative graph. We demonstrate on image classification benchmarks that the deepest layers (convolutional and dense) of common networks can be replaced by significantly smaller learned structures, while maintaining classification accuracy-state-of-the-art on tested benchmarks. Our structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.
The presented method learns a structure of a deep ANN by first learning a BN and then constructing the ANN from this BN.  The authors state that they "propose a new interpretation for depth and inter-layer connectivity in deep neural networks". Neurons in deep layers represent low-order conditional independencies (ie small conditioning set) and those in 'early' (non-deep) layers represent high-order CI relationships. These are all CI relations in the "X" ie the input vector of (observed) random variables.  Perhaps I am missing something here but I could not find an argument as to why this is a principled way to build deep ANNs with good performance. After all, when we actually use an ANN to make a prediction, all the X input variables will be observed, so it is not obvious to me why we should care about CI relations between the components of X. So we are using unsupervised learning to help supervised learning. Nothing wrong with that, but I think we need more about how the former helps the latter in this case.  The actual process of geneating the ANN structure is clearly described. Since the ultimate goal of the BN learning is somewhat unusual so is the BN learning algorithm - essentially we add latent variables to represent the (estimated) CI relations in the input.  Although I can't see why this is a good method for making ANN structure, perhaps it just is, so the empirical evaluation is very important. 5 image classification and 7 other benchmarks are used. Not too surprisingly the empirical results are positive: the learned (more sparse) structures do better than fully connected ones, particularly with smaller training sets. The authors also (in Section 4.2) compare to other methods: getting comparable accuracies to NAS but with smaller networks.  I think this is promising work and reasonable results have been produced but I judge that for publication in NIPS we need a fuller account of why/when this method should work.  SPECIFIC POINTS  Using a threshold for the CI tests set by a validation set is a good approach (certainly better than many constraint-based approaches where some arbitrary threshold is set and then CI test results on finite data are treated as 100% reliable). However, the authors do not address the issue of the inevitable errors in CI test results, despite flagging up this issue on lines 45-46.  line 255: smaller number of -> fewer  AFTER READING AUTHOR FEEDBACK  The authors have provided a better explanation of the rationale of their approach, which I hope will be included in any future version, and I have upped my rating appropriately.