Recent work has demonstrated the lack of robustness of well-trained deep neural networks (DNNs) to adversarial examples. For example, visually indistinguishable perturbations, when mixed with an original image, can easily lead deep learning models to misclassifications. In light of a recent study on the mutual influence between robustness and accuracy over 18 different ImageNet models, this paper investigates how training data affect the accuracy and robustness of deep neural networks. We conduct extensive experiments on four different datasets, including CIFAR-10, MNIST, STL-10, and Tiny ImageNet, with several representative neural networks. Our results reveal previously unknown phenomena that exist between the size of training data and characteristics of the resulting models. In particular, besides confirming that the model accuracy improves as the amount of training data increases, we also observe that the model robustness improves initially, but there exists a turning point after which robustness starts to decrease. How and when such turning points occur vary for different neural networks and different datasets.
This paper empirically evaluates the effect of the training dataset size on accuracy and robustness against adversarial attacks. The methodology of the paper is generally easy to assess and the overall idea well communicated.
For the motivation example I assume the following assessment holds true. Several linear functions are sampled and compose S_1, S_2, and T. A single linear regression model is used to fit all the data, either S_1 or (S_1 and S_2). If that is the case the experiment is not clear to me since the single linear model can only fit the data mean, mean slope (a) and constant (mu). Since the joint dataset better captures the mean of T the error for the joint training should be lower indeed. However, to actually compare both values the same threshold theta should be used for both and not a percentage of their performance. I would argue that this very simple model does not provide any valuable insight into the problem due to its construction.
The experimental setup presented in Section 4 only considers examples which are classified correctly by all data subsets. However, it is crucial to also consider the mistakes of these subsequent sets. For example, the learned model for the most restrictive dataset is most likely not exposed to a complex decision boundary, therefore it will exhibit a much smoother prediction at the cost that it will simply classify many more examples as the target class. In this case using data perturbations is not even the problem since completely different examples might be classified wrongly. Although not entirely clear, it would be very useful to consider the nearest negative neighbor in the dataset in the embedding space of the classifier to capture this problem at least partially. In general if the test accuracy is lower the learned classifier exhibits less performance, thus, adversary examples, distorted examples are not the main issue since it simply makes mistakes on visually different examples. Therefore, the overall analysis should be much more focused on models which achieve the same test performance but use require less data to achieve this performance.