Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly. In this paper, we propose the MACER algorithm, which learns robust models without using adversarial training but performs better than all existing provable l 2 -defenses. Recent work (Cohen et al., 2019) shows that randomized smoothing can be used to provide certified l 2 radius to smoothed classifiers, and our algorithm trains provably robust smoothed classifiers via MAximizing the CErtified Radius (MACER). The attack-free characteristic makes MACER faster to train and easier to optimize. In our experiments, we show that our method can be applied to modern deep neural networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and SVHN. For all tasks, MACER spends less training time than state-of-the-art adversarial training algorithms, and the learned models achieve larger average certified radius. Our code is available at https://github.com/MacerAuthors/macer.
This paper improves the robustness of smoothed classifiers by maximizing the certified radius, which is more efficient than adversarially train the smoothed classifier and achieves higher average robust radius and better certified robustness when the radius is not much larger than the training sigma. It proposes a novel objective which is derived by decomposing the 0/1 certified loss into the sum of 0/1 classification error and 0/1 robustness error. Three conditions are identified to make the optimization doable. Two surrogate losses (CE and hinge loss on the certified radius) for the two 0/1 errors are proposed as upper bounds of the 0/1 loss. Certified radius is derived as a function of the logits of Soft-RS to make the hinge loss differentiable. Numerical stability of the proposed objective is also analyzed by showing its gradient is bounded.
In general, the paper is well-written and the proposed objective is novel to my knowledge. I tend to accept the paper. Still, I am not sure about how much MACER improves upon the baselines, and would like to ask some questions.
1. Cross entropy is used as a surrogate for the 0/1 classification error. This is true for all cases (including all experiments in this paper) except for binary classification, where the cross entropy is less than 1 when the score on the correct class is around 0.5. It is not important but would be better if you could mention this point.
2. Have you ever tried using a tighter upper bound for the 0/1 classification error, e.g., using cross entropy loss only for the wrongly classified samples? How does it affect the results?
3. Despite showing better results, MACER seems to be using much more epochs than the two baselines (but the total hrs is smaller than (Salman et al. 2019)). Also, MACER is using a much larger k than (Cohen et al., 2019). From Figure 3 (a) we can see a larger k improves the result a lot, and from Figure 3 (b) it seems that setting lambda to a non-zero value only improves the accuracy when the radius is large. For fair comparisons, could the authors give the ACR with different values of lambda while keeping other hyper parameters unchanged? Is Salman's method still not as good when using the same number of epochs?