Current machine learning algorithms can be easily fooled by adversarial examples. One possible solution path is to make models that use confidence thresholding to avoid making mistakes. Such models refuse to make a prediction when they are not confident of their answer. We propose to evaluate such models in terms of tradeoff curves with the goal of high success rate on clean examples and low failure rate on adversarial examples. Existing untargeted attacks developed for models that do not use confidence thresholding tend to underestimate such models' vulnerability. We propose the MaxConfidence family of attacks, which are optimal in a variety of theoretical settings, including one realistic setting: attacks against linear models. Experiments show the attack attains good results in practice. We show that simple defenses are able to perform well on MNIST but not on CIFAR, contributing further to previous calls that MNIST should be retired as a benchmarking dataset for adversarial robustness research. We release code for these evaluations as part of the cleverhans (Papernot et al., 2018) library 1 .
(Left) A success-fail curve. The horizontal axis shows the failure rate (fraction of examples for which the model returns a response and the response is incorrect) on adversarial examples. The vertical axis shows the success rate (fraction of examples for which the model returns a response and the response is correct) on clean examples. Each point on the curve corresponds to using a different confidence threshold. Dashed lines indicate estimates of upper and lower bounds (section 6.1) for thresholds t < 1 2 where our attack is not provably optimal. The upper and lower bounds are nearly equal so the two bounds are not separately visible at most points. (Right) We compare our new MaxConfidence attack to a traditional attack that maximizes the loss . The traditional attack is designed to cause a high loss, not a high failure rate in the presence of confidence thresholding. Our MaxConfidence attack performs better or equal at every point on the curve, with the most pronounced differences in the parts of the curve where confidence thresholding is the most effective. The staircase pattern for the undefended model results because few examples lie near the middle of the curve, and when drawing the plot we connect consecutive points in a staircase pattern in order to show the maximum possible failure rate. A curve formed with linear connections between points may false imply that some (success, failure) points are feasible even though they were not reached in the experiment. (Right) Success-fail curves for the same MNIST models on semantic adversarial examples. Surprisingly, adversarial training now performs worse than the undefended baseline. The regularized model performs slightly better than the undefended baselines. This illustrates some of the advantages of generic regularization combined with confidence thresholding: models that are generally skeptical of a wide variety of unusual data can avoid being fooled in settings that the designer did not need to explicitly specify ahead of time.
Adversarial examples "are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake" (Goodfellow et al., 2017). All currently known models are highly vulnerable to most known kinds of adversarial examples. See Goodfellow (2018) for one summary of important open problems and research directions.
One popular approach to defending against adversarial examples is to refuse to classify adversarial examples while still providing predictions for clean examples. The evaluation methodology for such defenses has been of limited quality. In this work, we present both metrics and attack methods for security evaluation of models that use confidence thresholding to refuse to classify some examples.
Models based on confidence thresholding have the potential to solve many open problems in adversarial robustness:
• The current state of the art in adversarial robustness , and with some caveats, Kannan et al. (2018)) is based on adversarial training (Szegedy et al., 2013;Goodfellow et al., 2014b;Warde-Farley & Goodfellow, 2016;Kurakin et al., 2016). So far adversarial training tends to result in robustness to the attack used during training but does not result in robustness to other kinds of attacks (e.g. Sharma & Chen (2017) showed that adversarially trained models designed to be robust to small perturbations as measured by the L ∞ norm do not also become robust to small perturbations as measured by the L 1 norm). It seems unlikely that adversarial training will scale to make the model give the right answer everywhere in the model's input domain, but maybe models based on confidence thresholding can learn to assign low confidence almost everywhere in the input domain, and thereby resist a very wide range of potential attacks. Confidence thresholding thus seems like a viable candidate to solve harder problems involving adversarial examples with fewer restrictions .
• Adversarial training has high computational cost, but maybe models with appropriate confidence can be trained at lower cost. We show that this is feasible on MNIST, and our evaluation methods pave the way to test future models on more difficult datasets.
• Confidence thresholding offers a simple, flexible way to trade performance on clean data for robustness to adversarial examples, simply by adjusting the threshold t. Our proposed evaluation methods can automatically produce entire tradeoff curves after running our proposed attack only once per example. Choosing the right point on the tradeoff curve does not require retraining the model or re-running the evaluation for each hypothetical threshold.
• Adversarial training often causes a reduction in accuracy on naturally occurring data (Su et al., 2018). Models that use confidence thresholding can actually achieve higher accuracy than undefended models, albeit with reduced coverage. When a model reduces coverage, it reduces the total volume of input space for which it can have an error. Other methods run the risk of moving errors into locations not measured by the current benchmark.
Many proposed defenses have been based on detecting adversarial examples and then refusing to classify them. Many of these proposed defenses have been broken (Carlini & Wagner, 2017). We observe that confidence thresholding is different from detection in a subtle but important way. In the case of confidence thresholding, refusing to return a response does not imply that the input is necessarily an "adversarial example". The classifier is free to shut off for a wide variety of inputs, so long as they do not appear often in the naturally occuring data. This makes the confidence thresholding defense much less likely to overfit to a specific conception of what an adversarial example looks like.
Despite this potential, it has not yet been clear how to evaluate models based on confidence thresholding. In this work, we advocate analyzing models in terms of a tradeoff curve: models should have high success rate on naturally occurring data and low failure rate on adversarial data, defined in section 3. Models that use confidence thresholding must also be benchmarked appropriately, against attacks that are aware the defender will use confidence thresholding. We propose a new family of attacks called MaxConfidence that is optimal in a variety of theoretical settings, and optimal in practice against linear models.
This paper introduces a family of attack on confidence thresholding algortihms. Such algorithms are allowed to refuse to make predictions when their confidence is below a certain threshold. 
There are certainly interesting links between such models and KWIK [1] algorithms (which are also supposed to be able to respond 'null' to queries), however they are not mentioned in this paper, which focuses mainly on evaluation methodologies.
The definition of the metric is certainly natural: you would expect some trade-off between performance in the normal versus the adversarial regime. I am not certain why the authors don't simply measure the success rate on both natural and adversarial conditions, so as to have the performance metric uniform. Unfortunately the paper's notationleaves something to be desired, as it fails to concretely define the metric.
Let me do so instead, and consider the classification accuracy of a classification rule $P_t$ using a threshold $t$ under a (possibly adaptive) distribution $Q$ to be $U(P,Q)$. Then, we can consider $Q_N, Q_A$ as the normal and adversarial distribution and measure the corresponding accuracies. 
Even if we do this, however, the authors do not clarify how they propose to select the classification rule. Should they employ something like a convex combination:
\[
V(P_t) := \alpha U(P_t, Q_N) + (1 - \alpha) U(P_t, Q_A) 
\]
or maybe take a nimimax approach
\[
V(P_t) := \min \{U(P_t, Q) | Q = Q_A, Q_N\}
\]
In addition, the authors simply plot curves for various choices of $t$, however it is necessary to take into account the fact that measuring performance in this way and selecting $t$ aftewards amounts to a hyperparameter selection [2]. Thus, the thresholding should be chosen on an independent validation set in order to optimise the chosen performance measure, and then the choice should evaluated on a new test set with respect to the same measure $V$
The MaxConfidence attack is not very well described, in my opinion. However, it seems it simply wishes to find to find a single point $x \in \mathbb{S}$ that maximises the probability of misclassification. It is not clear to me why performance against an attack of this type is interesting to measure.
The main contribution of the paper seems to be the generalisation of the attack by Goodfellow et al to softmax regression. The proof of this statement is in a rather obscure place in the paper. 
I am not sure I follow the idea for the proof, or what they are trying to prove. The authors should follow a standard Theorem/Proof organisation, clearing stating assumptions and what the theorem is showing us. It seems that they want to prove that if a solution to (1) exists, then MaxConfidence() finds it. But the only definition of MaxConfidence is (1). Hence I think that their theorem is vacuous. There are quite a few details that are also unclear such as what the authors mean by 'clean example' etc. 
However the authors do not explain their attack very well, their definition of the performance metric is not sufficiently formal, and their evaluation methodology is weak. Since evaluation methodology is the central point of the paper, this is a serious weaknes. Finally, there doesn't seem to be a lot of connection with the conference's topic.
[1] Li, Lihong, Michael L. Littman, and Thomas J. Walsh. "Knows what it knows: a framework for self-aware learning." Proceedings of the 25th international conference on Machine learning. ACM, 2008.
[2] Bengio, Samy, Johnny Mariéthoz, and Mikaela Keller. "The expected performance curve." International Conference on Machine Learning, ICML, Workshop on ROC Analysis in Machine Learning. No. EPFL-CONF-83266. 2005.