Knows When it Doesn’t Know: Deep Abstaining Classifiers
We introduce the deep abstaining classifier -- a deep neural network trained with a novel loss function that provides an abstention option during training. This allows the DNN to abstain on confusing or difficult-to-learn examples while improving performance on the non-abstained samples. We show that such deep abstaining classifiers can: (i) learn representations for structured noise -- where noisy training labels or confusing examples are correlated with underlying features -- and then learn to abstain based on such features; (ii) enable robust learning in the presence of arbitrary or unstructured noise by identifying noisy samples; and (iii) be used as an effective out-of-category detector that learns to reliably abstain when presented with samples from unknown classes. We provide analytical results on loss function behavior that enable automatic tuning of accuracy and coverage, and demonstrate the utility of the deep abstaining classifier using multiple image benchmarks, Results indicate significant improvement in learning in the presence of label noise.
 The paper introduces a new loss function for training a deep neural network which can abstain. The paper was easy to read, and they had thorough experiments and looked at their model performance in different angles (in existence of structured noise, in existence of unstructured noise and open world detection). However, I think this paper has some issues which are listed below: 1) Although there are very few works regarding abstaining in DNN, I would like to see what the paper offers that is not addressed by the existing literature. Right now, in the experiment, there is no comparison to the previous work, and in the introduction, the difference is not clear. I think having an extra related work section regarding comparison would be useful. 2) The experiment section was thorough, and the authors look at the performance of DAC at different angles; however, as far as I understand one of the significant contributions of the paper is to define abstain class during training instead of post-processing (e.g., abstaining on all examples where the network has low confidence). Therefore, I would like to see a better comparison to a network that has soft-max score cut-off rather than plain DNN. In figure 1-d the comparison is not clear since you did not report the coverage. I think it would be great if you can compare either with related work or tune a softmax-score on a validation set and then compare with your method. 3) There are some typos, misuse of .citet instead of .citep spacing between parenthesis; especially in figures, texts overlap, the spacing is not correct, some figures don’t have a caption, etc.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Knows When it Doesn’t Know: Deep Abstaining Classifiers
We introduce the deep abstaining classifier -- a deep neural network trained with a novel loss function that provides an abstention option during training. This allows the DNN to abstain on confusing or difficult-to-learn examples while improving performance on the non-abstained samples. We show that such deep abstaining classifiers can: (i) learn representations for structured noise -- where noisy training labels or confusing examples are correlated with underlying features -- and then learn to abstain based on such features; (ii) enable robust learning in the presence of arbitrary or unstructured noise by identifying noisy samples; and (iii) be used as an effective out-of-category detector that learns to reliably abstain when presented with samples from unknown classes. We provide analytical results on loss function behavior that enable automatic tuning of accuracy and coverage, and demonstrate the utility of the deep abstaining classifier using multiple image benchmarks, Results indicate significant improvement in learning in the presence of label noise.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper introduces a new loss function for training a deep neural network which can abstain. The paper was easy to read, and they had thorough experiments and looked at their model performance in different angles (in existence of structured noise, in existence of unstructured noise and open world detection). However, I think this paper has some issues which are listed below: 1) Although there are very few works regarding abstaining in DNN, I would like to see what the paper offers that is not addressed by the existing literature. Right now, in the experiment, there is no comparison to the previous work, and in the introduction, the difference is not clear. I think having an extra related work section regarding comparison would be useful. 2) The experiment section was thorough, and the authors look at the performance of DAC at different angles; however, as far as I understand one of the significant contributions of the paper is to define abstain class during training instead of post-processing (e.g., abstaining on all examples where the network has low confidence). Therefore, I would like to see a better comparison to a network that has soft-max score cut-off rather than plain DNN. In figure 1-d the comparison is not clear since you did not report the coverage. I think it would be great if you can compare either with related work or tune a softmax-score on a validation set and then compare with your method. 3) There are some typos, misuse of .citet instead of .citep spacing between parenthesis; especially in figures, texts overlap, the spacing is not correct, some figures don’t have a caption, etc.
assistant
 The paper introduces a new loss function for training a deep neural network which can abstain. The paper was easy to read, and they had thorough experiments and looked at their model performance in different angles (in existence of structured noise, in existence of unstructured noise and open world detection). However, I think this paper has some issues which are listed below: 1) Although there are very few works regarding abstaining in DNN, I would like to see what the paper offers that is not addressed by the existing literature. Right now, in the experiment, there is no comparison to the previous work, and in the introduction, the difference is not clear. I think having an extra related work section regarding comparison would be useful. 2) The experiment section was thorough, and the authors look at the performance of DAC at different angles; however, as far as I understand one of the significant contributions of the paper is to define abstain class during training instead of post-processing (e.g., abstaining on all examples where the network has low confidence). Therefore, I would like to see a better comparison to a network that has soft-max score cut-off rather than plain DNN. In figure 1-d the comparison is not clear since you did not report the coverage. I think it would be great if you can compare either with related work or tune a softmax-score on a validation set and then compare with your method. 3) There are some typos, misuse of .citet instead of .citep spacing between parenthesis; especially in figures, texts overlap, the spacing is not correct, some figures don’t have a caption, etc.