At internet scale, applications collect a tremendous amount of data by logging user events, analyzing text, and collecting images. This data powers a variety of machine learning models for tasks such as image classification, language modeling, content recommendation, and advertising. However, training large models over all available data can be computationally expensive, creating a bottleneck in the development of new machine learning models. In this work, we develop a novel approach to efficiently select a subset of training data to achieve faster training with no loss in model predictive performance. In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model. Extensive experiments show that our approach leads to a 1.6× and 1.8× speed-up on CIFAR10 and SVHN by selecting 60% and 50% subsets of the data, while maintaining the predictive performance of the model trained on the entire dataset.
# Summary
The paper presents a method for identifying and selecting the most informative subset of the training dataset in order to reduce training time while maintaining test accuracy. The method consists of training a proxy model that is smaller and has been trained for fewer epochs, and which can optionally be ensembled. Experiments show promising results, indicating that some datasets can be reduced to half the size without impacting model performance.
# Quality
The paper appears sound and of good quality. Background literature is cited and the proposed method is discussed in sufficient detail.
I would, however, like to see some additional comparative experiments. All experiments are constructed to show that the method can indeed achieve accuracy comparable to the full model but with a smaller training set. I would like to see how it compares to existing strategies -- are there any reason to pick this method over existing ones?
Since the last sentence in section 2 states that the proposed method is orthogonal to previous subsampling techniques, and therefore can be combined with any of them, it would be interesting to see how SVP compares to these and whether a combination of, say, SVP and importance sampling will in fact achieve better performance than the importance sampling on its own.
Additionally, given the model's high resemblance to active learning, it would be interesting to see it compared to some prominent active learning methods.
# Clarity
The paper reads quite well. I particularly like the paragraph headlines, which makes it easy to get an overview of the paper.
The figures are generally nice and readable, except for figure 3, which I don't understand. Maybe I am missing it, but I can't find an explanation for what the rows and columns indicate, and the labels themselves should also be increased in size.
# Originality
I do not find the paper particularly novel. To me, the proposed method seems to be a variant of active learning, not orthogonal to this as it is claimed in section 2. The choice of surrogate model and uncertainty metric might be new, but the method itself boils down to uncertainty sampling, a well-known strategy in active learning.
However, I am happy to change my mind if the authors can explain to me exactly how their method differs from active learning.
# Significance
While techniques for speeding up training without sacrificing performance are, of course, always interesting, I find the proposed method to be rather incremental and not significant enough for ICLR. It would be better suited as a workshop paper.
# Other notes
In the last paragraph of section 1, you write that "Our proposed framework is robust to the choice of proxy model architecture." I am not sure what you mean by this. Do you mean that one can choose any model as the proxy (which is clearly correct) or do you mean that the method is "proxy agnostic" in the sense that any proxy model will work better than no proxy? If the latter is the case, I would like some arguments for this. Also, if the method is indeed proxy agnostic, it should be possible to remove the proxy completely and select the data in some other way.