Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artificial Neural Networks (ANNs) has gained popularity in recent years. Nevertheless, these type of experiments require estimating mutual information and entropy which becomes intractable for moderately large problems. In this work we propose a framework for understanding SGD learning in the information plane which consists of observing entropy and conditional entropy of the output labels of ANNs. Through experimental results and theoretical justifications it is shown that, under some assumptions, the SGD learning trajectories appear to be similar for different ANN architectures. First, the SGD learning is modeled as a Hidden Markov Process (HMP) whose entropy tends to increase to the maximum. Then, it is shown that the SGD learning trajectory appears to move close to the shortest path between the initial and final joint distributions in the space of probability measures equipped with the total variation metric. Furthermore, it is shown that the trajectory of learning in the information plane can provide an alternative for observing the learning process, with potentially richer information about the learning than the trajectories in training and test error.
This paper study the trajectory of H(\hat{y}) versus H(\hat{y}|y) on the information plane for stochastic gradient descent methods for training neural networks. This paper was inspired by (Ziv and Tishby 17'), but instead of measuring the mutual information I(X;T) and I(Y:T), this paper proposed to measure H(\hat{y}) and H(\hat{y}|y), which are much easier to compute but carries similar meaning as I(Y;T) and I(X;T).
The interesting part of this paper appears in Section 4, where the author makes a connection between the SGD training process and \alpha-SMLC(strong Markov learning chain). SMLC is just simply linear combination of the initial distribution and the final stable distribution of the labels. The authors show that the trajectory of the real experiment is similar to that of SMLC.
Generally I think the paper is well-written and clearly present the ideas. Here are some pros and cons.
Pros 1: The trajectory presented in this paper is much more reliable than that in (Ziv and Tishby 17'), since measuring the entropy and conditional entropy of discrete random variables are much easier. Also it is easy for people to believe that the trajectory holds for various neural network structure and various activation functions.
Pros 2: The connection to SMLC is interesting and it may contain lot of insights.
Cons 1: One of my major concern is --- if you look at the trajectory of the experiment v.s. SMLC (Figure 3), they look similar at first glance. But if you look at it carefully, you will notice that the color of them are different! For SGD, the trajectory goes to the turning point very soon (usually no more than 10% of the training steps), whereas SMLC goes to the turning point much slower. How do the authors think about this phenomenon and what does this mean?
Cons 2: This paper is going to be more meaningful if the author can provide some discussions, especially about (1) what does the shape trajectory mean (2) what do the connection between the trajectory and Markov chain means (3) how can these connections be potentially useful to improve training algorithm? I understand that these questions may not be clearly answerable, but the authors should make this paper more inspiring such that other researchers can think deeper after reading this paper.
Cons 3: I suggest the authors using SGD instead of GD throughout the paper. Usually GD means true gradient descent, but the paper is talking about batched stochastic gradient descent. GD does not have Markovity.
Generally, I think the paper is on the borderline. I think the paper is acceptable if the author can provide more insights (against Cons 2).