Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, clipping controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum. This intuition has been made precise in a line of recent works, which show that suitable clipping can yield significantly faster convergence than vanilla gradient descent. In this paper, we propose a new lens for studying gradient clipping, namely, robustness: informally, one expects clipping to mitigate the effects of noise, since one does not overly trust any single sample. Surprisingly, we prove that for the common problem of label noise in classification, standard gradient clipping does not in general provide robustness. On the other hand, we show that a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function. As a special case, this yields a simple, noise-robust modification of the standard cross-entropy loss which performs well empirically.
In this paper, we propose a new lens with which to study gradient clipping, namely, robustness: intuitively, clipping the gradient prevents over-confident descent steps, which is plausibly beneficial in the presence of noise. Given this intuition, our interest is whether gradient clipping can mitigate the problem of label noise in classification, which has received significant recent interest (Scott et al., 2013;Natarajan et al., 2013;Liu & Tao, 2016;Patrini et al., 2017;Ghosh et al., 2017;Han et al., 2018;Zhang & Sabuncu, 2018;Song et al., 2019;Thulasidasan et al., 2019;Charoenphakdee et al., 2019). We study this question, and provide three main contributions:
(a) we show that gradient clipping alone does not endow label noise robustness to even simple models. Specifically, we show that under stochastic gradient descent with linear models, gradient clipping is related to optimising a "Huberised" loss (Lemma 1, 2). While such Huberised losses preserve classification calibration (Lemma 3), they are not robust to label noise (Proposition 4). (b) we propose composite loss-based gradient clipping, a variant that does have label noise robustness.
Specifically, for losses comprising a base loss composed with a link function, (e.g., softmax crossentropy), we only clip the contribution of the base loss. The resulting partially Huberised loss preserves classification calibration (Lemma 6), while being robust to label noise (Proposition 7). (c) we empirically verify that on both synthetic and real-world datasets, partially Huberised versions of standard losses (e.g., softmax cross-entropy) perform well in the presence of label noise ( §5).
To illustrate the difference between standard and composite loss-based gradient clipping in (a) and (b), consider the pervasive softmax cross-entropy loss, viz. the logistic loss for binary classification. Recall that the logistic loss comprises the log-loss (or cross-entropy) composed with a sigmoid link. The Huberised loss arises from loss-based gradient clipping, and linearises the entire logistic loss beyond a threshold. The partially Huberised loss arises from composite loss-based gradient clipping, and linearises only the base loss (i.e., log-loss) beyond a threshold (right panel). Once combined with the sigmoid link, the overall partially Huberised loss asymptotically saturates. Our analysis in §3, §4 establishes that Huberised losses are not robust to label noise, but partially Huberised losses are (Proposition 4, 7).
In (a), we relate gradient clipping to loss "Huberisation", which linearises the logistic loss when it exceeds a fixed threshold. In (b), we introduce composite loss-based gradient clipping, which is equivalent to a "partial loss Huberisation" that linearises only the cross-entropy loss but leaves the sigmoid link untouched. Figure 1 illustrates the Huberised and partially Huberised logistic loss.
This paper studied a fundamental problem in robust machine learning, that is, can gradient clipping mitigate label noise? The paper is well written, clearly motivated, highly novel and significant not only in a theoretical sense but also in a practical sense.
As argued in the abstract, gradient clipping is a widely-used technique which is generally motivated from the OPTIMIZATION point of view. In this paper, the authors proposed an entirely new motivation of gradient clipping from the ROBUSTNESS point of view, since intuitively it should be able to mitigate label noise. Surprisingly, the authors proved that for some simple binary classification with label noise, standard gradient clipping does not provide robustness; on the other hand, a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function.
The major contributions were well summarized in the bottom of page 1, the related work was discussed in sec 4.3, and the caveats of this new methodology were also given in sec 4.4. Note that the proposed composite loss-based gradient clipping is applicable even on top of existing noise-robust losses, for example, the generalized cross-entropy loss, and this serves as a convincing demonstration of the great significance of the paper. Actually, the paper is full of insights, and I really enjoyed reviewing it.
I have a few questions on Tables 1 and 2. Table 1 said the standard gradient clipping is not robust according to Proposition 4. However, Proposition 4 is for the loss-based gradient clipping in (7) rather than the standard gradient clipping in (6). Perhaps I have missed something, but why the non-robustness of (7) can imply the non-robustness of (6)?
In Table 2, Linear loss on MNIST, the test accuracy was 9.6 when rho=0.6. Is this a typo? The accuracy was still 78.8 when rho=0.4. Moreover, the authors explained why the theoretically grounded linear loss performed so badly, that is, the optimization was so difficult. How about CE+clipping loss on CIFAR-100? The standard gradient clipping was also harmful here, even when there was no label noise at all. Was this due to bad optimization or robustness (or both)? 
Since distributionally robust supervised learning is a future direction to go, I think the authors may be interested in a thought-provoking paper:
W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? ICML 2018.