In recent years several adversarial attacks and defenses have been proposed. Often seemingly robust models turn out to be non-robust when more sophisticated attacks are used. One way out of this dilemma are provable robustness guarantees. While provably robust models for specific l p -perturbation models have been developed, we show that they do not come with any guarantee against other l q -perturbations. We propose a new regularization scheme, MMR-Universal, for ReLU networks which enforces robustness wrt l 1 -and l ∞ -perturbations and show how that leads to the first provably robust models wrt any l p -norm for p ≥ 1.
Summary of the paper's contributions:
This paper proves a result on the l_p robustness (p \neq 1, \infty) of a piecewise affine classifier in terms of its l_1 and l_\infty robustness. This result is based on the insight that a guarantee for l_1 and l_\infty robustness of a piecewise affine classifier also guarantees robustness within the convex hull of the union of the l_1 and l_\infty balls. The paper then proposes a regularization scheme called MMR-Universal for a ReLU neural network that simultaneously trains for l_1 and l_\infty robustness. This scheme is based on maximizing the linear regions of the network as in Croce et al., AISTATS 2019. Using the main result of the paper, it is implied that the proposed regularization scheme also enforces l_p robustness for any p \geq 1.
Main comments: (1) The paper provides an interesting result that guarantees l_p robustness for a piecewise affine classifier based on only l_1 and l_\infty robustness. (2) The experiments show that the proposed regularization scheme is indeed effective in simultaneously guaranteeing robustness with respect to all l_p balls.
Detailed comments:
- The proposed regularization scheme does not come with any explicit robustness guarantee. Is it possible to show that a model that minimizes the regularized loss is guaranteed to be l_p robust for some radius? In Appendix C.3, it is mentioned that the best values for \gamma_p were empirically found to be 1-2 times the desired \epsilon_p robustness. Could this be formalized in theory?
- From Figure 2, it seems that there is an optimal ratio of l_1 to l_\infty robustness for which the l_2 guarantee is maximized. It would be interesting to see what this optimal ratio is for maximizing the l_p guarantee, as a function of dimension d and p.
- In the experiments, the MMR-Universal scheme is compared with MMR+AT-l_2 and MMR+AT-l_\infty. It would be interesting to compare it with MMR-l_1+AT-l_\infty or with MMR-l_\infty+AT-l_1.
- In eq(1), is c intended to be the true label at x?
- In Figure 2, the red curves are missing in the first and third plots.