Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets. However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels. Are CNNs robust or fragile to label noise? Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic. In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k). Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise. We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations.
This paper demonstrates that CNNs are more robust to class-relevant label noise. They argue that real-world noise should be class-relevant.
Pros:
1. The authors find a new angle to exploit robust learning with noisy labels.
2. The authors perform numerical experiments to demonstrate the effectiveness of their proposal. And their experimental result support their previous claims.
Cons:
We have two questions in the following.
1. Basic definition: in learning with noisy labels, there are two basic models. First, most research focuses on class-conditional noise (CCN) model [1]. Second, recent research explore a bit on instance-dependent noise (IDN) model [2, 3]. As far as I know, there is no class-irrelevant label noise and class-relevevant label noise. In CCN mode, people would like to use symmetric noise and asymmetric noise as a basic benchmark to conduct experiments.
2. Motivation: The authors want to claim CNNs are more robust to such realistic label noise than class-irrelevant label noise. However, they make one mistake. They do not have a clear definition about realistic label noise. In my mind, I believe Clothing1M [4] should be realistic label noise dataset.
By the way, in learning with noisy labels, there are two kinds of research. First, people propose new robust methods for CCN model. Second, people propose new robust methods for IDN models. Proposing new setting should be encouraged. However, the setting and conclusion should be reasonable.
References:
[1] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning, 1988.
[2] A. Menon, B. Rooyen, and N. Natarajan. Learning from binary labels with instance-dependent corruption. Machine Learning, 2018.
[3] J. Cheng, T. Liu, K. Ramamohanarao, D. Tao. Learning with bounded instance-and label-dependent label noise. arxiv 1709.03768, 2017.
[4] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.