In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) do not satisfy the i.i.d. prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs, thus the Gaussian process cannot correctly model the distribution of the hidden layers. Alternatively, we propose a novel probabilistic representation for the hidden layers of DNNs in two aspects: (i) a hidden layer formulates a Gibbs distribution, of which the energy function is determined by the activations, and (ii) the statistical connection between two adjacent layers can be formulated as a product of experts model. Based on the proposed probabilistic representation, we further confirm the equivalence between the variational inference and the stochastic gradient descent, and prove the later is also equivalent to the energy minimization optimization. As a result, the entire architecture of DNNs can be viewed as a Bayesian network, in which the hidden layers close to the input formulate the prior distributions of the training dataset. In addition, we propose a new regularization approach to improve the generalization performance of DNNs by pre-training the hidden layers corresponding to the prior distributions. Simulation results validate the proposed theories.
Main contribution of the paper
- The paper argues that the base assumption, the i.i.d. of the activated elements (activations) in the hidden layers, the existing methods (lee.et.al 2018) hold is not convincible.
- Instead, the author proposes a new way to probabilistically model the hidden layers, activations, and layer/layer connections.
- Based on the probabilistic model, the paper proposes a new regularizer.
Methods
- The author argues that the activation is not iid by empirically showing that the trained MLP (in most cases) does not un-correlated.
- The author proposes a new probabilistic model for MLP, and CNN assuming the Gibbs distribution to each activation and also assuming the product of expert (poE) model to explain the layer/layer relationship.
- And according to their model, CNN will be explained by the MRF model.
- The author proposes a regularization term regarding layer/layer connection.
- They argue that the SGD training can be seen as a first-order approximation of the inference of the hidden activations in MLP.
Questions
- See the Concerns
Strongpoints
- The probabilistic explanation of the MLP and the CNN seems novel and was interesting to the reviewer
- The proposed explanation assumes a weaker condition compared to the existing methods.
Concerns
- The main concern is that the reviewer cannot fully convince that i.i.d. assumption is wrong. 
Even though the trained MLP does not support the i.i.d. condition, one can suppose that the reason would be the typical training method (SGD), just finding the local minima in a deterministic way.
Maybe the proof in Appendix.G. supports the argument of the author, but the reviewer failed to clearly agree with the argument.
A clear explanation regarding the issue would be required.
- As far as the author understands, the paper proposes a probabilistic (Bayesian) model for explaining MLP, but it seems that they just used SGD for training the model. 
In that case, the reviewer is little suspicious of the role of the proposed regularization in that the regularization comes from Bayesian formulation, but the model was trained in a deterministic way.
The reviewer wants to ask the author that 
(1) is it possible to infer the model in a Bayesian manner such as sampling?
(2) Is there any justification for using SGD when conducting the experiments regarding the regularization? If it is related to Appendix.G, clearer explanation would be appreciated.
- As far as the reviewer understands, the regularization deals with the practical part of the paper. It would be better to see the effect of the regularization of widely used networks such as small-layered ResNet or others.
If the proposed formulation has other practical strongpoints, it would be nice to clarify them.
- The explanation using Gibbs distribution and PoE looks similar to RBM. The reviewer strongly wants a clear explanation of the difference and the strongpoints compared to RBM.
Conclusion
- The author proposed a new probabilistic explanation of the neural network, which seems novel and worth reporting.
- However, the reviewer failed to fully agree on some steps in the process of the paper.
Therefore, the reviewer temporary rates the paper as weak-reject, but this can be adjusted after seeing the answers of the author.
 
Inquiries
- See the concerns parts.