Efficient Augmentation via Data Subsampling
Data augmentation is commonly used to encode invariances in learning methods. However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set. The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply. In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset. We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.
 Data augmentation is a useful technique, but can lead to undesirably large data sets. The authors propose to use influence or loss-based methods to select a small subset of points to use in augmenting data sets for training models where the loss is additive over data points, and investigate the performance of their schemes when logistic loss is used over CNN features. Specifically, they propose selecting which data points to augment by either choosing points where the training loss is high, or where the statistical influence score is high (as defined in Koh and Liang 2017). The cost of their method is that of fitting an initial model on the training set, then fitting the final model on the augmented data set. They compare to reasonable baselines: no augmentation, augmentation by transforming only a uniformly random chosen portion of the training data, and full training data augmentation; and show that augmenting even 10% of the data with their schemes can give loss competitive with full data augmentation, and lower than the loss achievable with no augmentation or augmentation of a uniformly random chosen portion of the data of similar size. Experiments were done on MNIST, CIFAR, and NORB. The paper is clearly written, the idea is intuitively attractive, and the experiments give convincing evidence that the method is practically useful. I believe it will be of interest to a large portion of the ICLR community, given the usefulness of data augmentation.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Efficient Augmentation via Data Subsampling
Data augmentation is commonly used to encode invariances in learning methods. However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set. The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply. In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset. We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Data augmentation is a useful technique, but can lead to undesirably large data sets. The authors propose to use influence or loss-based methods to select a small subset of points to use in augmenting data sets for training models where the loss is additive over data points, and investigate the performance of their schemes when logistic loss is used over CNN features. Specifically, they propose selecting which data points to augment by either choosing points where the training loss is high, or where the statistical influence score is high (as defined in Koh and Liang 2017). The cost of their method is that of fitting an initial model on the training set, then fitting the final model on the augmented data set. They compare to reasonable baselines: no augmentation, augmentation by transforming only a uniformly random chosen portion of the training data, and full training data augmentation; and show that augmenting even 10% of the data with their schemes can give loss competitive with full data augmentation, and lower than the loss achievable with no augmentation or augmentation of a uniformly random chosen portion of the data of similar size. Experiments were done on MNIST, CIFAR, and NORB. The paper is clearly written, the idea is intuitively attractive, and the experiments give convincing evidence that the method is practically useful. I believe it will be of interest to a large portion of the ICLR community, given the usefulness of data augmentation.
assistant
 Data augmentation is a useful technique, but can lead to undesirably large data sets. The authors propose to use influence or loss-based methods to select a small subset of points to use in augmenting data sets for training models where the loss is additive over data points, and investigate the performance of their schemes when logistic loss is used over CNN features. Specifically, they propose selecting which data points to augment by either choosing points where the training loss is high, or where the statistical influence score is high (as defined in Koh and Liang 2017). The cost of their method is that of fitting an initial model on the training set, then fitting the final model on the augmented data set. They compare to reasonable baselines: no augmentation, augmentation by transforming only a uniformly random chosen portion of the training data, and full training data augmentation; and show that augmenting even 10% of the data with their schemes can give loss competitive with full data augmentation, and lower than the loss achievable with no augmentation or augmentation of a uniformly random chosen portion of the data of similar size. Experiments were done on MNIST, CIFAR, and NORB. The paper is clearly written, the idea is intuitively attractive, and the experiments give convincing evidence that the method is practically useful. I believe it will be of interest to a large portion of the ICLR community, given the usefulness of data augmentation.