Generalizing to Unseen Domains via Adversarial Data Augmentation
We are concerned with learning models that generalize well to different unseen domains. We consider a worst-case formulation over data distributions that are near the source domain in the feature space. Only using training data from a single source distribution, we propose an iterative procedure that augments the dataset with examples from a fictitious target domain that is "hard" under the current model. We show that our iterative scheme is an adaptive data augmentation method where we append adversarial examples at each iteration. For softmax losses, we show that our method is a data-dependent regularization scheme that behaves differently from classical regularizers that regularize towards zero (e.g., ridge or lasso). On digit recognition and semantic segmentation tasks, our method learns models improve performance across a range of a priori unknown target domains.
The paper attacks a novel problem: one-shot domain generalization where given samples from a single domain one requires robustness to unknown domain covariate shift. The problem is extremely hard and the related works claim that no other work exists attacking the same problem.  The paper incrementally builds up on a recently proposed method to defend against adversarial attacks [33]. In fact, the work uses the formulation of [33] and repurposes the procedure for one-shot domain generalization. The original additions are 3-fold: 1) the closeness constraint is changed from pixel-space to feature-space 2) an ensemble of models are trained with different neighborhood thresholds 3) a new theoretical motivation is provided.  It tests the method on the transfer between digit datasets as well as road datasets. The results show that the proposed training work better when tested on unseen domains compared to standard training.  The main issue is that, since no prior works exist to compare against, it needs more comprehensive experimental design to show the real benefits of the method. The experimental setup is currently very limited. For instance, I would like to see how are the benefits of using the new method related to the capacity of the model and how regularized it is (by design and by choice of hyperparameters). Would we see the same benefits if the model is highly regularized already?  What is the benefit of the constraint in the feature space instead of pixel space? A transition between using different layers would probably give some insight.  Some analysis on the samples which are only correctly classified using the proposed method will be helpful. At the same time, an analysis of the samples that are added to the dataset during the training procedure can be insightful. These two sets can be further studied together.  All in all, the originality of the paper is lacking, the experimental setup is not convincing, and there are not much insights given by the paper into the novel problem. So, I think, the paper is not ready for publication at NIPS.  ------------------------------------------ I read the authors' rebuttal and other reviewers comments. The rebuttal partially mitigates my general concerns with the paper: 1) regularization and 2) general insights. On the other hand, I can see the other reviewers' main positive point that from the theoretical point of view it is a simple and clear work on top of the existing ones. So, I increase my rating slightly but still lean towards the rejection of the paper. I think the paper will become significantly stronger if along the lines of the rebuttal further and more comprehensive studies are pursued.