Training neural networks with verifiable robustness guarantees is challenging. Several existing approaches utilize linear relaxation based neural network output bounds under perturbation, but they can slow down training by a factor of hundreds depending on the underlying network architectures. Meanwhile, interval bound propagation (IBP) based training is efficient and significantly outperforms linear relaxation based methods on many tasks, yet it may suffer from stability issues since the bounds are much looser especially at the beginning of training. In this paper, we propose a new certified adversarial training method, CROWN-IBP, by combining the fast IBP bounds in a forward bounding pass and a tight linear relaxation based bound, CROWN, in a backward bounding pass. CROWN-IBP is computationally efficient and consistently outperforms IBP baselines on training verifiably robust neural networks. We conduct large scale experiments on MNIST and CIFAR datasets, and outperform all previous linear relaxation and bound propagation based certified defenses in ∞ robustness. Notably, we achieve 7.02% verified test error on MNIST at = 0.3, and 66.94% on CIFAR-10 with = 8/255.
This work proposes CROWN-IBP - novel and efficient certified defense method against adversarial attacks, by combining linear relaxation methods which tend to have tighter bounds with the more efficient interval-based methods. With an attempt to augment the IBP method with its lower computation complexity with the tight CROWN bounds, to get the best of both worlds. One of the primary contributions here is that reduction of computation complexity by an order of \Ln while maintaining similar or better bounds on error. The authors show compelling results with varied sized networks on both MNIST and CIFAR dataset, providing significant improvements over past baselines.
The paper itself is very well written, lucidly articulating the key contributions of the paper and highlighting the key results. The method and rationale behind it quite easy to follow.
Pros:
> Show significant benefits over previous baseline with 7.02% verified test error on MNIST at  \epsilon = 0.3, and 66.94% on CIFAR-10 with \epsilon = 8/255
> The proposed method is computationally viable, with up to 20X faster than linear relaxation methods with similar. better test errors and within 5-7X slower than the conventional IBP methods with worse errors
Cons:
> Extensive experiments with more advanced networks/datasets would have been more convincing, esp. given the computation efficiency that enables such experiments
> More elaborate insights into the choice of the training config/hyper-params esp. with the choice of \K_start, \K_end across the different datasets
Other comments:
> For the computational efficiency studies, it would be helpful to provide a breakdown of the costs between the different layers and operations, to better asses/confirm that benefits of CROWN-IBP method
> Impact of other complementary techniques such a lower precision/quantization? One fo the references compared against is the Gowal et al. 2018 for the as a baseline, however, it seems to be those results were obtained on a different HW platform (TPUs - motioned in Appendix-B), with potentially different computational accuracies (BFLOAT16 ?). So, this bears to question of the impact of precision on these methods and also the computation complexity.
> Since I'm not very well versed with the current baseline and state-of-art for variable robust training of DNN, it would be good to get an additional confirmation on the validity of the used baselines.