CONTROLLING OVER-GENERALIZATION AND ITS EF- FECT ON ADVERSARIAL EXAMPLES GENERATION AND DETECTION
Convolutional Neural Networks (CNNs) significantly improve the state-of-the-art for many applications, especially in computer vision. However, CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes. Further, they are also vulnerable to adversarial examples. We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set. We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization. As an appropriate training set for the extra class, we introduce two resources that are computationally efficient to obtain: a representative natural out-distribution set and interpolated in-distribution samples. To help select a representative natural out-distribution set among available ones, we propose a simple measurement to assess an outdistribution set's fitness. We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. Finally, we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder, as the attack algorithms have to get around the rejection (dustbin) regions when generating actual adversaries.
The idea of having a separate class for out-distribution is a very interesting idea but unfortunately previously explored. In fact, in machine learning and NLP there is the OOV class which sometimes people in computer vision also use. Some of the claims in the paper can be further substantiated or explored. For example in abstract there is a simple claim that is presented too strong: We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. This claim is bigger than just CNNs and needs to be studied in a theoretical framework not an empirical one. Also, one simple way to stop these adversarial cases would be to explore using Sigmoid as opposed to softmax. In general it is very unlikely that you will be able to choose every variation of out-distribution cases. Much easier if you just try to solve the problem using a set of n Sigmoids (n total number of classes) and consider each output a probability distribution. 
However, the studies in this paper are still valuable and I strongly recommend continuing on the same direction.