Training Provably Robust Models by Polyhedral Envelope Regularization
Training certifiable neural networks enables one to obtain models with robustness guarantees against adversarial attacks. In this work, we use a linear approximation to bound model’s output given an input adversarial budget. This allows us to bound the adversary-free region in the data neighborhood by a polyhedral envelope and yields finer-grained certified robustness than existing methods. We further exploit this certifier to introduce a framework called polyhedral envelope regular- ization (PER), which encourages larger polyhedral envelopes and thus improves the provable robustness of the models. We demonstrate the flexibility and effectiveness of our framework on standard benchmarks; it applies to networks with general activation functions and obtains comparable or better robustness guarantees than state-of-the-art methods, with very little cost in clean accuracy, i.e., without over-regularizing the model.
Summary: There exists several method (KW, Crown, zonotope transformers) which essentially propagate linear lower and upper bound through the network (bounds is a linear function of the input variables). So far, to check for robustness, these methods propagated the bounds until the last layer (+ margin computation) and then concretized them and compared the result to zero to see if the studied radius was safe or not. This paper highlights the fact that in the case where the studied radius is not safe, it is possible to extract a safe radius from the linear bound. The authors then show that: -> this can be used to make binary search to find the larger verifiable epsilon faster (by providing a lower bound on epsilon even when failing to verify, rather than simply when succeeding verification) -> this can be employed during the training process to improve regularization, similarly to another previously proposed method (Croce et al.). While the methods are similar, the bounds are much less conservative than Croce*s one and should therefore be more helpful The presentation of the content is quite clear, and Figure 1.a is extremely useful in conveying the benefits of the method. Experimental validation is thorough: -> Comparing the ACB KW and ACB PEC columns in Table 1 and 2, for all type of activation functions, shows that this is as expected a strict improvement in the generated bound. -> Comparing the various rows in Table 1 shows the improvement with regards to other methods, both in terms of clean accuracy (this produces less over-regularization), while maintaining similar or better robust accuracy. -> Table 6 shows the benefit in the application to binary search. Opinion: This paper is clearly written and I think that it*s an interesting insight and that the authors do a good job at conveying its usefulness. I*m happy for this paper to be accepted.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Training Provably Robust Models by Polyhedral Envelope Regularization
Training certifiable neural networks enables one to obtain models with robustness guarantees against adversarial attacks. In this work, we use a linear approximation to bound model’s output given an input adversarial budget. This allows us to bound the adversary-free region in the data neighborhood by a polyhedral envelope and yields finer-grained certified robustness than existing methods. We further exploit this certifier to introduce a framework called polyhedral envelope regular- ization (PER), which encourages larger polyhedral envelopes and thus improves the provable robustness of the models. We demonstrate the flexibility and effectiveness of our framework on standard benchmarks; it applies to networks with general activation functions and obtains comparable or better robustness guarantees than state-of-the-art methods, with very little cost in clean accuracy, i.e., without over-regularizing the model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: There exists several method (KW, Crown, zonotope transformers) which essentially propagate linear lower and upper bound through the network (bounds is a linear function of the input variables). So far, to check for robustness, these methods propagated the bounds until the last layer (+ margin computation) and then concretized them and compared the result to zero to see if the studied radius was safe or not. This paper highlights the fact that in the case where the studied radius is not safe, it is possible to extract a safe radius from the linear bound. The authors then show that: -> this can be used to make binary search to find the larger verifiable epsilon faster (by providing a lower bound on epsilon even when failing to verify, rather than simply when succeeding verification) -> this can be employed during the training process to improve regularization, similarly to another previously proposed method (Croce et al.). While the methods are similar, the bounds are much less conservative than Croce*s one and should therefore be more helpful The presentation of the content is quite clear, and Figure 1.a is extremely useful in conveying the benefits of the method. Experimental validation is thorough: -> Comparing the ACB KW and ACB PEC columns in Table 1 and 2, for all type of activation functions, shows that this is as expected a strict improvement in the generated bound. -> Comparing the various rows in Table 1 shows the improvement with regards to other methods, both in terms of clean accuracy (this produces less over-regularization), while maintaining similar or better robust accuracy. -> Table 6 shows the benefit in the application to binary search. Opinion: This paper is clearly written and I think that it*s an interesting insight and that the authors do a good job at conveying its usefulness. I*m happy for this paper to be accepted.
assistant
Summary: There exists several method (KW, Crown, zonotope transformers) which essentially propagate linear lower and upper bound through the network (bounds is a linear function of the input variables). So far, to check for robustness, these methods propagated the bounds until the last layer (+ margin computation) and then concretized them and compared the result to zero to see if the studied radius was safe or not. This paper highlights the fact that in the case where the studied radius is not safe, it is possible to extract a safe radius from the linear bound. The authors then show that: -> this can be used to make binary search to find the larger verifiable epsilon faster (by providing a lower bound on epsilon even when failing to verify, rather than simply when succeeding verification) -> this can be employed during the training process to improve regularization, similarly to another previously proposed method (Croce et al.). While the methods are similar, the bounds are much less conservative than Croce*s one and should therefore be more helpful The presentation of the content is quite clear, and Figure 1.a is extremely useful in conveying the benefits of the method. Experimental validation is thorough: -> Comparing the ACB KW and ACB PEC columns in Table 1 and 2, for all type of activation functions, shows that this is as expected a strict improvement in the generated bound. -> Comparing the various rows in Table 1 shows the improvement with regards to other methods, both in terms of clean accuracy (this produces less over-regularization), while maintaining similar or better robust accuracy. -> Table 6 shows the benefit in the application to binary search. Opinion: This paper is clearly written and I think that it*s an interesting insight and that the authors do a good job at conveying its usefulness. I*m happy for this paper to be accepted.