PEERNETS: EXPLOITING PEER WISDOM AGAINST ADVERSARIAL ATTACKS
Deep learning systems have become ubiquitous in many aspects of our lives. Unfortunately, it has been shown that such systems are vulnerable to adversarial attacks, making them prone to potential unlawful uses. Designing deep neural networks that are robust to adversarial attacks is a fundamental step in making such systems safer and deployable in a broader variety of applications (e.g. autonomous driving), but more importantly is a necessary step to design novel and more advanced architectures built on new computational paradigms rather than marginally building on the existing ones. In this paper we introduce PeerNets, a novel family of convolutional networks alternating classical Euclidean convolutions with graph convolutions to harness information from a graph of peer samples. This results in a form of non-local forward propagation in the model, where latent features are conditioned on the global structure induced by the graph, that is up to 3× more robust to a variety of white-and black-box adversarial attacks compared to conventional architectures with almost no drop in accuracy.
The paper presents a an interesting novel approach to train neural networks with so called peer regularization which aims to provide robustness to adversarial attacks. The idea is to add a graph neural network to a spatial CNN. A graph is defined over similar training samples which are found using a Monte Carlo approximation.
The regularization using graphs reminds me of recent work at ICML on semi-supervised learning (Kamnitsas et al. (2018) Semi-supervised learning via compact latent space clustering) which is using a graph to approximate cluster density which acts as a regularizer for training on labelled data.
The main problem I see with these approaches is that they rely on sufficiently large batch sizes which could be (currently) problematic for many real-world applications. Memory and computation limitations are mentioned, but not sufficently discussed. It would be good to add further details on practical limitations.
Experiments are limited to benchmark data using MNIST, CIFAR-10, CIFAR-100. Comprehensive evaluation has been carried out with insightful experiments and good comparison to state-of-the-art. Both white- and black-box adversarial attacks are explored with promising results for the proposed approach.
However, it is difficult to draw conclusions for real-world problems of larger scale. The authors state that proposed framework can be added to any baseline model, but miss to clearly mention the limitations. It is stated that future work will aim at scaling PeerNets to benchmarks like ImageNet, but it is unclear how this could be done. Is there any hope this could be applied to problems like 3D imaging data or videos?