We introduce an unsupervised structure learning algorithm for deep, feed-forward, neural networks. We propose a new interpretation for depth and inter-layer connectivity where a hierarchy of independencies in the input distribution is encoded in the network structure. This results in structures allowing neurons to connect to neurons in any deeper layer skipping intermediate layers. Moreover, neurons in deeper layers encode low-order (small condition sets) independencies and have a wide scope of the input, whereas neurons in the first layers encode higher-order (larger condition sets) independencies and have a narrower scope. Thus, the depth of the network is automatically determined-equal to the maximal order of independence in the input distribution, which is the recursion-depth of the algorithm. The proposed algorithm constructs two main graphical models: 1) a generative latent graph (a deep belief network) learned from data and 2) a deep discriminative graph constructed from the generative latent graph. We prove that conditional dependencies between the nodes in the learned generative latent graph are preserved in the class-conditional discriminative graph. Finally, a deep neural network structure is constructed based on the discriminative graph. We demonstrate on image classification benchmarks that the algorithm replaces the deepest layers (convolutional and dense layers) of common convolutional networks, achieving high classification accuracy, while constructing significantly smaller structures. The proposed structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.
The paper proposes an unsupervised structure learning method for deep neural networks. It first constructs a fully visible DAG by learning from data, and decomposes variables into autonomous sets. Then latent variables are introduced and stochastic inverse is generated. Later a deep neural network structure is constructed based on the discriminative graph. Both the problem considered in the paper and the proposed method look interesting. The resulting structure seems nice.
However, the reviewer indeed finds a major technical flaw in the paper. The foundation of the proposed method is on preserving the conditional dependencies in graph G. And each step mentioned in the paper, as it claims, can preserve all the conditional dependencies. However, in section 2.2, it seems that the stochastic inverse cannot. In Fig. 3(b), A and B are no longer dependent conditioned on {C,D,E} due to the v-structure induced in node H_A and H_B. Also in Fig. 3(c), if the reviewer understands correctly, the bidirectional edge between H_A and H_B is equivalent to H_A <- h -> H_B, which also induces a v-structure, blocking the dependency between A and B. Therefore, the very foundation of the proposed method is shattered. And the reviewer requests an explicit explanation of this issue.
Besides that, the reviewer also finds unfair comparisons in the experiments.
1. In section 5.1, although the authors show that the learned structure achieves 99.04%-99.07% compared with 98.4%-98.75% for fully connected layers, the comparisons are made by keeping the number of parameters similar in both cases. The comparisons are reasonable but not very convincing. Observing that the learned structures would be much sparser than the fully connected ones, it means that the number of neurons in the fully connected network is significantly smaller. Did the authors compare with fully connected network with similar number of neurons? In such case, which one is better? (Having fewer parameters is a plus, but in terms of accuracy the number of neurons really matters for fair comparison. In practice, we definitely would not use that small number of neurons in fully connected layers.)
2. In section 5.2, it is interesting to observe that using features from conv10 is better than that from last dense layer. But it is not a fair comparison with vanilla network. In vanilla VGG-16-D, there are 3 more conv layers and 3 more fully connected layers. If you find that taking features from conv10 is good for the learned structure, then maybe it will also be good by taking features from conv10 and then apply 2-3 fully-connected layers directly (The proposed structure learning is not comparable to convolutional layers, and what it should really compare to is fully-connected layers.) In such case, which one is better? 
Secondly, VGG-16 is a large network designed for ImageNet data. For small dataset such as CIFAR10 and CIFAR100, it is really overkilled. That's maybe the reason why taking the output of shallow layers could achieve pretty good results.
3. In Fig. 6, again, comparing the learned structure with fully-connected network by keeping parameters to be similar and resulting in large difference of the number of neurons is unfair from my point of view.
Furthermore, all the comparisons are made with respect to fully-connected network or vanilla CNNs. No other structure learning methods are compared with. Reasonable baseline methods should be included.
In conclusion, due to the above issues both in method and experiments, the reviewer thinks that this paper is not ready for publication.