Cautious Deep Learning
Most classifiers operate by selecting the maximum of an estimate of the conditional distribution where stands for the features of the instance to be classified and denotes its label. This often results in a hubristic bias: overconfidence in the assignment of a definite label. Usually, the observations are concentrated on a small volume but the classifier provides definite predictions for the entire space. We propose constructing conformal prediction sets which contain a set of labels rather than a single label. These conformal prediction sets contain the true label with probability . Our construction is based on rather than which results in a classifier that is very cautious: it outputs the null set --- meaning ``I don*t know** --- when the object does not resemble the training examples. An important property of our approach is that classes can be added or removed without having to retrain the classifier. We demonstrate the performance on the ImageNet ILSVRC dataset and the CelebA and IMDB-Wiki facial datasets using high dimensional features obtained from state of the art convolutional neural networks.
 This paper applies Conformal Methods to multi-class classification. I am unfamiliar with the field, but the authors seem to be the first to attempt multiclass classification with Conformal Methods by estimating p(x|y) instead of the usual p(y|x). In doing so, they effectively build an independent classifier for each class that estimates whether an example comes from that class within a certain confidence which is set before training time. In this way, they create meaning NULL predictions for an unidentified example, instead of the usual low-probability of an erroneous class. The paper is well written, although it is difficult for me to work out which parts are the author*s contributions, and which parts are tutorial/introduction to known Conformal Methods techniques, again this might be because this is not my subject area. The ImageNet results look OK, not great, I would prefer to see a table in section 6. The transfer of features from CelebA to IMDB-wiki is good, but it is hard to tell how good. I feel there should be more comparisons to other methods, even, perhaps an ROC curve for a set of 1 vs all mlp binary/SVM classifiers along with the conformal results (mlp using different cutoff thresholds, conformal method being trained with different confidence levels). I feel like this paper may be important, but it is a little difficult for me (myself) to judge without clear empirical evidence of a strong method vs other methods.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Cautious Deep Learning
Most classifiers operate by selecting the maximum of an estimate of the conditional distribution where stands for the features of the instance to be classified and denotes its label. This often results in a hubristic bias: overconfidence in the assignment of a definite label. Usually, the observations are concentrated on a small volume but the classifier provides definite predictions for the entire space. We propose constructing conformal prediction sets which contain a set of labels rather than a single label. These conformal prediction sets contain the true label with probability . Our construction is based on rather than which results in a classifier that is very cautious: it outputs the null set --- meaning ``I don*t know** --- when the object does not resemble the training examples. An important property of our approach is that classes can be added or removed without having to retrain the classifier. We demonstrate the performance on the ImageNet ILSVRC dataset and the CelebA and IMDB-Wiki facial datasets using high dimensional features obtained from state of the art convolutional neural networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper applies Conformal Methods to multi-class classification. I am unfamiliar with the field, but the authors seem to be the first to attempt multiclass classification with Conformal Methods by estimating p(x|y) instead of the usual p(y|x). In doing so, they effectively build an independent classifier for each class that estimates whether an example comes from that class within a certain confidence which is set before training time. In this way, they create meaning NULL predictions for an unidentified example, instead of the usual low-probability of an erroneous class. The paper is well written, although it is difficult for me to work out which parts are the author*s contributions, and which parts are tutorial/introduction to known Conformal Methods techniques, again this might be because this is not my subject area. The ImageNet results look OK, not great, I would prefer to see a table in section 6. The transfer of features from CelebA to IMDB-wiki is good, but it is hard to tell how good. I feel there should be more comparisons to other methods, even, perhaps an ROC curve for a set of 1 vs all mlp binary/SVM classifiers along with the conformal results (mlp using different cutoff thresholds, conformal method being trained with different confidence levels). I feel like this paper may be important, but it is a little difficult for me (myself) to judge without clear empirical evidence of a strong method vs other methods.
assistant
 This paper applies Conformal Methods to multi-class classification. I am unfamiliar with the field, but the authors seem to be the first to attempt multiclass classification with Conformal Methods by estimating p(x|y) instead of the usual p(y|x). In doing so, they effectively build an independent classifier for each class that estimates whether an example comes from that class within a certain confidence which is set before training time. In this way, they create meaning NULL predictions for an unidentified example, instead of the usual low-probability of an erroneous class. The paper is well written, although it is difficult for me to work out which parts are the author*s contributions, and which parts are tutorial/introduction to known Conformal Methods techniques, again this might be because this is not my subject area. The ImageNet results look OK, not great, I would prefer to see a table in section 6. The transfer of features from CelebA to IMDB-wiki is good, but it is hard to tell how good. I feel there should be more comparisons to other methods, even, perhaps an ROC curve for a set of 1 vs all mlp binary/SVM classifiers along with the conformal results (mlp using different cutoff thresholds, conformal method being trained with different confidence levels). I feel like this paper may be important, but it is a little difficult for me (myself) to judge without clear empirical evidence of a strong method vs other methods.