Adversarial training is the standard to train models robust against adversarial examples. However, especially for complex datasets, adversarial training incurs a significant loss in accuracy and is known to generalize poorly to stronger attacks, e.g., larger perturbations or other threat models. In this paper, we introduce confidence-calibrated adversarial training (CCAT) where the key idea is to enforce that the confidence on adversarial examples decays with their distance to the attacked examples. We show that CCAT preserves better the accuracy of normal training while robustness against adversarial examples is achieved via confidence thresholding, i.e., detecting adversarial examples based on their confidence. Most importantly, in strong contrast to adversarial training, the robustness of CCAT generalizes to larger perturbations and other threat models, not encountered during training. For evaluation, we extend the commonly used robust test error to our detection setting, present an adaptive attack with backtracking and allow the attacker to select, per test example, the worst-case adversarial example from multiple black-and white-box attacks. We present experimental results using L ∞ , L 2 , L 1 and L 0 attacks on MNIST, SVHN and Cifar10.
*Edit after rebuttal*
See the comment below for a response to the author's rebuttal. I have updated my score to a weak accept.
Summary
========
This paper proposes a more granular adversarial training scheme, where the model is trained to have decaying confidence as the size of the adversarial perturbations increase. 
The idea is quite natural and well motivated and presented. The paper suffers in its theoretical and evaluation sections though, which were both fairly confusing to me. I lean towards rejection for this paper but would consider changing my score if the presentation was improved.
Comments
=========
The observation that adversarial training enforces high confidence in the entirety of the epsilon ball is interesting, and the motivation for the proposed scheme is sound.
Yet, while the intuition of CCAT for addressing the accuracy-robustness trade-off makes sense to me, the purported beneficial effect for perturbations outside the given epsilon ball are not as clear.
The authors note in Section 3.1 that adversarial training gives no indications on how the model should behave outside the given epsilon ball, or on other threat models. While this is true, it is not clear how CCAT supposedly remedies this. By making the confidence decay within the epsilon ball, the training also doesn't explicitly enforce any behavior outside this ball or on other threat models. So if CCAT indeed provides benefits in those settings, it isn't clear why those gains should be expected.
The exposition of the technique in Section 3.2. is very clear, and Figure 1 nicely illustrates the effect of CCAT on the class confidences during an attack.
The theoretical toy example in Section 3.3. is a bit confusing, and I'm not convinced that the proof of Proposition 1 is correct. First, a classification problem parametrized by the class imbalance is a little weird in this setting. I have a few questions about the proof:
- You seem to be assuming that the model's confidence is calculated using a softmax. This assumption should be stated.
- The expressions for \hat{p}(y=2|x=x) and \hat{p}(y=1|x=x) are confusing to me. Why are you taking a log over the softmax? These probabilities are now negative... Also, in the second step, an equality of the form e^x / (e^y + e^x) = 1+e^(y-x) is used. This is incorrect, it should be 1/(1+e^(y-x)). In the end, the values log(1+e^a) and log(1+e^(-b)) are not in [0, 1] so they cannot be probabilities.
- From there on, it isn't clear what the rest of the proof is really saying
The experimental section is quite thorough and the results seem convincing. The considered adaptive attack, that optimizes (4) seems natural given that the defense thresholds on the model's confidence. I wonder how this is optimized though: Do you try a targeted attack for every possible target other than the true class?
I found the conflation of evaluation metrics quite confusing. It seems the authors know this and have done an effort to explain their choices, but I feel like this could still be improved further. For example, some of the metrics (e.g., AUC) should be maximized, while others (Err or RErr) should be minimized. This makes the reading of the result tables quite difficult.I would also suggest dropping one digit of precision in the Tables for better readability.
Overall, I have some mixed feelings about this paper. The idea is simple and well explained. The motivation, theoretical analysis, and reporting of results are somewhat confusing though. Some extra work on the presentation could definitely help this paper.
Some additional references
========================
- In the introduction, when mentioning the trade-off between robustness and accuracy, you should cite "Robustness May Be at Odds with Accuracy" by Tsipras et al. rather than Schmidt et al. (that paper shows that robustness may require more training data).
- Also in the introduction, you could consider referencing the following papers on robustness to different threat models:
    * Engstrom et al., "Exploring the Landscape of Spatial Robustness"
    * Tramer and Boneh, "Adversarial Training and Robustness for Multiple Perturbations"
- In the related work, you reference a number of works for adversarial training but not the original idea in the paper by Szegedy et al.
- In Section 3.1., the observation that the epsilon-ball around training examples can cross class boundaries is further analyzed in "Excessive Invariance Causes Adversarial Vulnerability" and "Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness" by Jacobsen et al.
Typos
=====
p.6 enumerator -> numerator