Deep learning achieves remarkable generalization capability with overwhelming number of model parameters. Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored. This paper attempts to provide an alternative understanding from the perspective of maximum entropy. We first derive two feature conditions that softmax regression strictly apply maximum entropy principle. DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle. The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development.
Summary:
This paper presents a derivation which links a DNN to recursive application of
maximum entropy model fitting. The mathematical notation is unclear, and in
one cases the lemmas are circular (i.e. two lemmas each assume the other is
correct for their proof). Additionally the main theorem requires complete
independence, but the second theorem provides pairwise independence, and the
two are not the same.
Major comments:
- The second condition of the maximum entropy equivalence theorem requires
  that all T are conditionally independent of Y. This statement is unclear, as
it could mean pairwise independence, or it could mean jointly independent
(i.e. for all pairs of non-overlapping subsets A & B of T I(T_A;T_B|Y) = 0).
This is the same as saying the mapping X->T is making each dimension of T
orthogonal, as otherwise it would introduce correlations. The proof of the
theorem assumes that pairwise independence induces joint independence and this
is not correct.
- Section 4.1 makes an analogy to EM, but gradient descent is not like this
  process as all the parameters are updated at once, and only optimised by a
single (noisy) step. The optimisation with respect to a single layer is
conditional on all the other layers remaining fixed, but the gradient
information is stale (as it knows about the previous step of the parameters in
the layer above). This means that gradient descent does all 1..L steps in
parallel, and this is different to the definition given.
- The proofs in Appendix C which are used for the statement I(T_i;T_j) >=
  I(T_i;T_j|Y) are incomplete, and in generate this statement is not true, so
requires proof.
- Lemma 1 appears to assume Lemma 2, and Lemma 2 appears to assume Lemma 1.
  Either these lemmas are circular or the derivations of both of them are
unclear.
- In Lemma 3 what is the minimum taken over for the left hand side? Elsewhere
  the minimum is taken over T, but T does not appear on the left hand side.
Explicit minimums help the reader to follow the logic, and implicit ones
should only be used when it is obvious what the minimum is over.
- In Lemma 5, what does "T is only related to X" mean? The proof states that
  Y -> T -> X forms a Markov chain, but this implies that T is a function of
Y, not X.
Minor comments:
- I assume that the E_{P(X,Y)} notation is the expectation of that probability
  distribution, but this notation is uncommon, and should be replaced with a
more explicit one.
- Markov is usually romanized with a "k" not a "c".
- The paper is missing numerous prepositions and articles, and contains
  multiple spelling mistakes & typos.