Workshop track -ICLR 2018 FEATURE INCAY FOR REPRESENTATION REGULARIZA- TION
Softmax-based loss is widely used in deep learning for multi-class classification, where each class is represented by a weight vector and each sample is represented as a feature vector. Inspired by that weight decay is a common practice to regularize the weight vectors, we investigate how to regularize the feature vectors since representation is also tunable in deep learning. One main observation is that elongating the feature norm of both correctlyclassified and mis-classified feature vectors improves learning: (1) increasing the feature norm of correctly-classified examples enlarges the probability margin among different classes and ensures better generalization. (2) increasing the feature norm of mis-classified examples can up-weight the contribution from hard examples. Accordingly, we propose feature incay to regularize feature vectors by encouraging larger feature norm. Extensive empirical results on MNIST, CI-FAR10, CIFAR100 and LFW demonstrate the effectiveness of feature incay.
The manuscript proposes to increase the norm of the last hidden layer to promote better classification accuracy. However, the motivation is a bit less convincing. Here are a few motivations that are mentioned.
(1) Increasing the feature norm of correctly classified examples helps cross entropy, which is of course correct. However, it only decreases the training loss. How do we know it will not lead to overfitting?
(2) Increasing the feature norm of mis-classified examples will make gradient larger for self-correction. And the manuscript proves it in property 2. However, the proof seems not complete. In Eq (7), increasing the feature norm would also affect the value of the term in parenthesis. As an example, if a negative example is already mis-classified as a positive, and its current probability is very close to 1, then further increasing feature norm would make the probability even closer to 1, leading to saturation and smaller gradient.
(3) Figure 1 shows that examples with larger feature norm tend to be predicted well. However, it is not very convincing since it is only a correlation rather than causality. Let's use simple linear softmax regression as a sanity check, where features to softmax are real features rather than hidden units. Increasing the feature norm seems to be against the best practice of feature normalization in which each feature after normalization is of variance 1.
The manuscript states that the feature norm won't be infinitely increased since there is an upper bound. However, the proof of property 3 seems to only apply to the certain cases where K<2D. In addition, alpha is in the formula of upper bound, but what is the upper bound of alpha?
The manuscript does comprehensive experiments to test the proposed method. The results are good, since the proposed method outperforms other baselines in most datasets. But the results are not impressively strong.
Minor issues:
(1) For proof of property 3, it seems that alpha and beta are used before defined. Are they the radius of two circles?