Training certifiable neural networks enables one to obtain models with robustness guarantees against adversarial attacks. In this work, we use a linear approximation to bound model's output given an input adversarial budget. This allows us to bound the adversary-free region in the data neighborhood by a polyhedral envelope and yields finer-grained certified robustness than existing methods. We further exploit this certifier to introduce a framework called polyhedral envelope regularization (PER), which encourages larger polyhedral envelopes and thus improves the provable robustness of the models. We demonstrate the flexibility and effectiveness of our framework on standard benchmarks; it applies to networks with general activation functions and obtains comparable or better robustness guarantees than state-of-the-art methods, with very little cost in clean accuracy, i.e., without over-regularizing the model.
This paper proposes a certifiable NN training method, "polyhedral envelope
regularization" (PER) for defending against adversarial examples.  The defense
is based on the same linear relaxation based outer bounds of neural networks
(KW/CROWN) used in many previous works.  The paper makes a few new (but small)
technical contributions:
1. this paper uses a different loss function (7), which is essentially Hinge
loss on the lower bounds of distance to decision boundary. Previous works like
KW used cross-entropy loss on the lower bound of prediction margin instead,
which was based on minimax robust optimization theory. But I am not fully
convinced if the new loss function is better or not.
2. in (5), the authors solve the bounded input case more carefully than
previous works.  (5) is trivial to solve in the L infinity case and has been
used in previous works like (Wong & Kolter 2018, Gowal et al., 2018 and Zhang
et al., 2019); but solving it for other norms requires some efforts, and this
paper proposes a good solution for it (Algorithm 2);
3. In previous works like KW/CROWN, to find the largest certifiable radius, a
binary search is needed. The authors proposes a very small improvement to the
binary search process by setting the lower bound of search to the largest
epsilon that is certifiable using the current linear relaxations obtained from
a larger epsilon.
The authors does not improve any bounds proposed in KW/CROWN, and they reuse
the same bounds. I see the main contribution as the new hinge-like loss
function for training, and a more careful procedure to find the largest
certifiable radius in bounded input case.
Empirically, the improvement of the proposed algorithm is limited - based on
Table 1 it is hard to say if PER is better than KW or not. PER+at outperforms
KW sometimes, however it is not a completely fair comparison, as we can add a
PGD based adversarial training loss to KW as well, as done in DiffAI (Mirman et
al., https://github.com/eth-sri/diffai).
Questions:
1. In my personal experience I usually found Hinge loss not as effective as
cross-entropy loss in deep learning based tasks probably due to its
non-smoothness. The claim that (7) is better than cross-entropy loss is that it
does not overregularize the network. The authors should provide more evidence
to show if this argument holds, e.g., plotting the norm of weight matrices
during the training for the two losses to show that it can reduce
overregularization.
2. I think the metric ACB KW and ACB CRO (average certified radius of KW/CROWN)
in Table 1 and 2 are confusing and not fair. In KW and CROWN's evaluation,
giving an epsilon, if an example cannot be certified due to epsilon to large
(i.e., ||A|| \epsilon + b > 0), certifiable radius will be count as 0 (flat
line in Figure 1(a)). In this paper, the authors instead in this case use -b /
||A|| as the certifiable radius. This is merely a different way of evaluation,
and I don't see this as a contribution, as the "improvement" does not come from
a tighter bound.  In the same sense, I don't think Figure 1(a) and the
discussions on page 3 are appropriate characterization of KW/CROWN. PEC uses
exactly the same linear bounds as in KW/CROWN, and has the same certification
power.
3. For L2 based perturbations, in Table 1, the epsilon used for MNIST is too
small. It is better to use an epsilon that is aligned with previous works. For
example in Wong et al., 2018 (https://arxiv.org/pdf/1805.12514.pdf), page 22,
you will find the epsilon used for MNIST and CIFAR.
4. As discussed above, it is probably not fair to compare PER+at with KW. A new
baseline like KW+at should also be considered.
5. For norms other than L infinity norms, solving (5) for getting $d$ can be
time consuming (Algorithm 2). How much additional time does it comparing
to KW?
Overall, I cannot recommend accepting this paper due to its limited theoretical
contribution as well as unconvincing empirical results comparing to previous
methods. I suggest rephrasing some parts of the paper and providing more
experimental results as discussed above.