Deep neural networks (DNNs) have achieved unprecedented practical success in many applications. However, how to interpret DNNs is still an open problem. In particular, what do hidden layers behave is not clearly understood. In this paper, relying on a teacher-student paradigm, we seek to understand the layer behaviors of DNNs by "monitoring" both across-layer and single-layer distribution evolution. Here, the "across-layer" and "single-layer" considers the layer behavior along the depth and a specific layer along training epochs, respectively. Relying on optimal transport theory, we employ the Wasserstein distance (W-distance) to measure the divergence between the layer distribution and the target distribution. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth. ii) For a specific layer, the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations. iii) However, a deep layer is not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs. Extensive experiments justify our theoretical findings.
This paper presents a method to compute the distance of distribution of two layers in neural networks by using the label distribution mapping (e.g., Frogner et al., 2015). With the tool, authors could see how individual layers could related each other across-layer (along the depth) and single layer (training epoch). 
I believe that the contributions of this paper are week in analyzing individual layers across-layer since there are many extensive studies are conducted on information bottleneck methods with mutual information. I believe that those methods are better to analyze the dynamics of learning even without the additional label distribution mapping.
However, authors of this paper presents a way to utilize the label distribution mapping to compare the distance of individual layers when an input image come as shown in Figure 5 which I believe the main contribution of this paper. 
The (somehow artificial and ambiguous) term, label distribution is used several places before it is defined. Even in Section 3, the label distribution mapping is not clearly explained except for the description of FC+softmax. Thus, it would be better to clarify the definition. Also, it is not clear that the label distribution reflect the actual distribution of (nodes or feature maps) in a specific layer. It would be good to spend more space and resources (e.g., image and/or running examples) to explain the definition of label distribution.