Investigating CNNs* Learning Representation under label noise
Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets. However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels. Are CNNs robust or fragile to label noise? Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic. In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k). Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise. We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations.
 The authors challenge the CNNs robustness to label noise, but when the label noise is class dependent, more realistic scenario than class independent noise. To analyse the CNNs behavior in such a scenario, they consider the ImageNet 1k dataset, and change some labels to labels that are close according to the ImageNet 1k tree of WordNet. The authors conduct multiple experiment to compare the effect of class dependent and class independent noise on: * the model accuracy * the robustness to adversarial perturbation * the learned representation The paper is generally well written and well structured. The analysis is sound and addresses interesting points, giving insightful results. Nevertheless, the overall conclusion is not very surprising. This work confirms the commonly admitted fact that CNNs learn features that are visually meaningful. Moreover, there is no significant novelty in the paper. The paper only analyses the CNNs behavior, without suggesting any new algorithm based on the observations. One specific point that seems under-investigated in my sense is the observation about the robustness to adversarial perturbations. The model with the class dependent noisy labels is in average less sensitive to the perturbations, even if this is not significant for the tested noise level. Did the authors test with different noise levels? This calls for a further analysis. It has the potential to give more insights, and probably inspire new methods to improve training robustness.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Investigating CNNs* Learning Representation under label noise
Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets. However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels. Are CNNs robust or fragile to label noise? Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic. In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k). Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise. We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors challenge the CNNs robustness to label noise, but when the label noise is class dependent, more realistic scenario than class independent noise. To analyse the CNNs behavior in such a scenario, they consider the ImageNet 1k dataset, and change some labels to labels that are close according to the ImageNet 1k tree of WordNet. The authors conduct multiple experiment to compare the effect of class dependent and class independent noise on: * the model accuracy * the robustness to adversarial perturbation * the learned representation The paper is generally well written and well structured. The analysis is sound and addresses interesting points, giving insightful results. Nevertheless, the overall conclusion is not very surprising. This work confirms the commonly admitted fact that CNNs learn features that are visually meaningful. Moreover, there is no significant novelty in the paper. The paper only analyses the CNNs behavior, without suggesting any new algorithm based on the observations. One specific point that seems under-investigated in my sense is the observation about the robustness to adversarial perturbations. The model with the class dependent noisy labels is in average less sensitive to the perturbations, even if this is not significant for the tested noise level. Did the authors test with different noise levels? This calls for a further analysis. It has the potential to give more insights, and probably inspire new methods to improve training robustness.
assistant
 The authors challenge the CNNs robustness to label noise, but when the label noise is class dependent, more realistic scenario than class independent noise. To analyse the CNNs behavior in such a scenario, they consider the ImageNet 1k dataset, and change some labels to labels that are close according to the ImageNet 1k tree of WordNet. The authors conduct multiple experiment to compare the effect of class dependent and class independent noise on: * the model accuracy * the robustness to adversarial perturbation * the learned representation The paper is generally well written and well structured. The analysis is sound and addresses interesting points, giving insightful results. Nevertheless, the overall conclusion is not very surprising. This work confirms the commonly admitted fact that CNNs learn features that are visually meaningful. Moreover, there is no significant novelty in the paper. The paper only analyses the CNNs behavior, without suggesting any new algorithm based on the observations. One specific point that seems under-investigated in my sense is the observation about the robustness to adversarial perturbations. The model with the class dependent noisy labels is in average less sensitive to the perturbations, even if this is not significant for the tested noise level. Did the authors test with different noise levels? This calls for a further analysis. It has the potential to give more insights, and probably inspire new methods to improve training robustness.