It is fundamental and challenging to train robust and accurate Deep Neural Networks (DNNs) when semantically abnormal examples exist. Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused on and how much more should they be emphasised to achieve robust learning? In this work, we study this question and propose gradient rescaling (GR) to solve it. GR modifies the magnitude of logit vector's gradient to emphasise on relatively easier training data points when noise becomes more severe, which functions as explicit emphasis regularisation to improve the generalisation performance of DNNs. Apart from regularisation, we connect GR to examples weighting and designing robust loss functions. We empirically demonstrate that GR is highly anomaly-robust and outperforms the state-of-the-art by a large margin, e.g., increasing 7% on CIFAR-100 with 40% noisy labels. It is also significantly superior to standard regularisers in both clean and abnormal settings. Furthermore, we present comprehensive ablation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios.
Summary
This paper presents Gradient Rescaling (GR) for robust learning to combat label noise. They propose to treat each data sample with different significance scores: some samples are important to learning, and some examples are insignificant (or even detrimental) to learning. So they desire to weight each samples according to their significance. They propose the notion of emphasis focus (When learning, whether we should put emphasis on learning “hard” examples or “easy” examples) and emphasis spread (the variance of these significance weights). The authors propose that this “difficulty” of samples are proportional to their network output logit values.
The authors examine the analytical forms of the gradients of popular loss functions such as Categorical Cross Entropy, Mean Absolute Error and Generalized Cross Entropy. They find that the formulas for the gradient are of similar family with varying hyperparameters. Authors claim that tweaking these hyperparameters result in tuning the emphasis focus and spread.
The authors conduct Experiments on CIFAR10, CIFAR100 with simulated symmetric noise. Also, they conduct experiments on real-world noisy datasets: Clothing 1M dataset and MARS video dataset. The authors claim that the performance of GR exceeds various baselines.
Significance/Novelty/Clarity
Significance: Low-Medium. The performance increase exhibited in the experiments are a bit underwhelming (when considering the fact that benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are missing).
Novelty: Medium. The paper is interesting in the sense that the authors integrated (and allegedly generalized) the gradient formulas for several losses into one family, and tried to integrate and tweak their postulation of  “Emphasis focus” and “emphasis spread” into the framework. However, the theoretical ground and convincing reasoning for their claim seems a bit lackluster.
Clarity: Low. The overall flow of the paper is a bit fuzzy - exhibiting a stream-of-consciousness style flow.
Pros and Cons in Detail
Pros:
1.The authors try to unify the analytical forms of the gradients of various loss functions into a single family equipped with hyperparameters that control emphasis focus and spread.
2.Conducted experiments show that GR achieved increased performance when compared to the baselines.
Cons:
My major concern is about tuning newly introduced hyperparameters in practical settings. How can we guarantee to have intact validation set?  Can we get any improvement via GR even with corrupted validation set for tuning hyperparameters? 
1. The arguments of the authors are grounded in the premise that “difficult” samples will exhibit small logit values, and “easy” samples high logit values.
2. No justifications (both theoretical and experimental) are provided on the claim that controlling emphasis focus/spread will result in more robust learning. 
3. This algorithm introduces 2 additional hyperparameters that are correlated with each other. This introduces additional labor.
4. By changing the loss function, the outputs of the network might lose its interpretation as a probability distribution.
5. No confidence intervals are shown except for the CIFAR-100 experiment.
6. Experiments are only conducted on vision tasks.
7. The baseline menagerie also changes when the authors change the target dataset.
8. Additional benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are required.
Questions
1. Is it always the case that “difficult” samples exhibit small logit values, and “easy” samples high logit values?
2. If not, GR’s emphasis manipulation might result in neglecting samples containing valuable information.
3. Can GR be used simultaneously with other noise-robust learning methods to further boost the performance?
4. Technically, GR aims to rescale the gradients of the logits. How will it interact with optimizers  other than SGD such as Adam?
5. Does GR still work well on small datasets(#points < 5000)?
Misc. Comments
Page 3-> inside L1 norm, no differentiation sign in the denominator.
Around eq 2 and 4: missing derivative symbol w.r.t. z