We propose a new notion of 'non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our results demonstrate that as an indicator for memorization, our technique can be used to perform early stopping.
This paper propose a new way of analyzing the robustness of neural network layers by measuring the level of "non-linearity" in the activation patterns on samples belonging to the same class, and correlate that to the level of "memorization" and generalization.
More specifically, the paper argues that a good representation cluster all the samples in a class together, therefore, in higher layers, the activation pattern of samples from the same class will be almost identical. In this case, the activation matrix will have a small non-negative rank. An approximation algorithm (via non-negative matrix factorization) is then used to compute the robustness and evaluate the robustness (by replacing the activation matrix with its low rank non-negative activation) is measured in a number of experiments with different amount of random label corruptions. The experiments show that networks trained on random labels are less robust than networks trained on true labels.
While the concept is interesting, I find the arguments in the paper a bit vague, and the usefulness of the algorithm might be hampered by its computation complexity, which is not discussed in the paper.
First of all, the paper lacks a clear notion of "memorization". While it is generally accepted that learns on random labels can be called "memorization", the paper seem to be defining it as how well is the network clustering points from the same class. Several questions need to be addressed in order for this notion to be justified:
1. Are (well generalized) networks really clustering samples of the same class to a centroid? It would be great if some empirical verifications are shown. Because the networks are using linear classifier in the last layer to classify the samples, it seems only linearly separability would be suffice for the work, which does not necessarily imply clustering.
2. Given two networks (of the same architecture), assume somehow network-1 decides to use the first 9 layers to compute a well clustered representation, while network-2 decides to use the first 5 layers to do the same thing. Do we say network-1 is (more) memorizing in this case?
3. The notion seems to be more about the underlying task than about the networks. Given the measurement, if a task is more complicated, meaning the input samples in the class have higher variance and requiring more efforts to cluster, then it seems the network will be doing more memorization. In other words, while networks will be doing more memorization when comparing a random label task to a true label task, it might also be "doing more memorization" when comparing learning on imagenet to learning on MNIST / CIFAR. One the one hand, this does not seem to fit our "intuition" about memorization; on the other hand, the heavy dependency on the underlying data distribution makes it difficult to compare results learned on different data -- especially since the measurements are based on per-class samples, "random labels" and "true labels" have very different class-conditional distributions.
I also have some questions about Figure 2(c). I will continue numbering the question for easier discussion.
4. Why for all cases, the lower layers all have higher AUC than the higher layers (except the last one)? The argument given in the paper is that the lower layers are the feature extraction phase while the upper layers are memorization phase. I think if clearly verified, this is a very interesting observation. But the paper currently do not have experiments to verify the hypothesis. Also more studies on this with different networks would be good. For example, with deeper networks, does the feature extraction phase include more layers?
5. The p=1 and p<1 curves seem to be very different. If one is to sample more densely between p=0.8 and p=1, would there still be a clear phase transition?
Some other questions:
6. Can you add discussions to the computation requirements for the proposed analysis? This is especially important for the cases where the analysis is used during training as tools to help deciding early stopping.
7. For the early stopping experiment, the main text says "These include the test error (in blue)" while in the figure the label axis is "Test loss". I'm assuming it is the cross entropy loss given the values are greater than 1. In this case, can you show in parallel the same plots in error rate, as the test error is more important than the test loss and the test loss could sometimes be artificially huge due to high confident mistakes on ambiguous test examples.
Some minor issues:
* Please proof read the paper for typos. E.g. on the 3rd paragraph of the 1st page: "that networks that networks that".
* The convention with subplots seem to be putting sub-captions under the figures, not above.