Long training times have increasingly become a burden for researchers by slowing down the pace of innovation, with some models taking days or weeks to train. In this paper, a new, general technique is presented that aims to speed up the training process by using a thinned-down training dataset. By leveraging autoencoders and the unique properties of embedding spaces, we are able to filter training datasets to include only those samples that matter the most. Through evaluation on a standard CIFAR-10 image classification task, this technique is shown to be effective. With this technique, training times can be reduced with a minimal loss in accuracy. Conversely, given a fixed training time budget, the technique was shown to improve accuracy by over 50%. This technique is a practical tool for achieving better results with large datasets and limited computational budgets.
This paper presents the idea of splitting the training process into two phases: fast training on a subset of the original dataset and finetining on the full dataset. To find a good subset of the training dataset it is proposed to train an autoencoder and use its embeddings to choose examples that have large values of the embedding features. The experiments show that on CIFAR-10 dataset this may speed up the convergence.
In general, I like the idea of being smart about which data and in which order to feed to the learner.
Nonetheless, I disagree with several premises of this paper. The paper claims that by making the dataset smaller one can speed up the training by the means of fitting the dataset into the accelerator memory and thus avoiding slow memory copies from CPU to accelerator memory. However, modern deep learning data pipelines are built in a way that has virtually zero overhead, since the data is loaded from disk and preprocessed on CPU and then copied on the accelerator asynchronously (i.e. the GPU doesn’t have to wait for the data, it can process the current batch and at the same time load the next one). Moreover, moving the data to GPU will introduce additional overheads in the case of random data augmentation, since this additional work would have to be done by GPU (while current deep learning frameworks asynchronously do this work on CPU). And finally, the authors claim that most modern datasets can fit to the accelerator memory if reduced 10x, but in my experience the network and it’s activations (which are stored during training) occupies most of the GPU memory even on high end accelerators, not leaving enough space to store a large dataset even after data reduction.
The paper cites Dunner  et al. (2017) as related work that focus on the similar problem: how to find a subset of the dataset to fit it into the GPU memory. However, I would argue that their setup is very different because they are using linear models (such as SVM): their learning steps are very fast compared to CNNs (which makes the memory bandwidth much slower in comparison), they don’t have to store activations of the layers (which allows them to fit much more training samples into GPU memory), and they don’t use data augmentation.
Also, I don’t think that the experimental comparison provides a strong enough evidence supporting the benefits of the proposed scheme. First, the experiments are only done on a small scale dataset (CIFAR-10), which is OK in general, but questionable when the proposed method explicitly targets big data regime and making the training faster. Second, the only baseline considered is choosing subset of data randomly. Third, the optimization method is plain SGD with momentum, while when presenting techniques for faster convergence it would make sense to compare on at least several standard optimization algorithms (e.g. Adam). Finally, the presented results are weak: in Fig 4 any improvements over random baseline are noticeable only after degrading the performance of the network by a large margin (to less than 67% accuracy on CIFAR-10);
Fig 5 looks like it has an error: training on the full dataset performs on the level of random guess after some training, which contradicts the fact that the same network converged to something reasonable in Fig. 6. Also, I believe that the training on the full dataset is strictly better than training on a random subset of data for a few epochs and then finetuning on the full dataset (with the same time budget). The latter sees the same number of updates but with less data, which should only decrease the test performance. If it’s correct, the results in Fig 5 for the full training should look similar (or better) than the results in Fig 4 for the random subset baseline, but it’s very far from being the case.
In Fig 6 I’m not sure what is being compared. Is a train or test loss? If it’s train, then it’s not a fair comparisons, since the two network are optimizing different train losses. I’m also very surprised not to see the moment of training mode transition on the plot (i.e. the moment when the model switched from restricted dataset to the full one), the lack of it can indicate an implementation error.
And finally, I would like to see the text being improved. Right now the language is confusing, for example: “this technique is shown to be effective” (what does it mean “effective” and compared to what?), “Unfortunately, while these techniques may be viable for smaller networks or datasets, large datasets have shown that they do not scale well.” (who have shown that the techniques doesn’t work on large dataset?), “the testing network is initialized using a weighted average of the final weights learned during training” (what does it mean?), “Qualitatively, the trained autoencoder succeeded in learning an adequate embedding.” (what does it mean?), etc.
Also, there is a typo in formulas 1, 2, and 3: it probably should sum up to n-1.
And I didn’t get what formula 4 means, what is “union of (for all i in n)”?
This bit I also didn’t get: “This simple loss function, in essence, forces the network to learn to extract the key features from the input, so that it can reproduce it using said features only. If desired, one could elect to use a more sophisticated loss, such as the Wasserstein distance metric (Gulrajani et al., 2017; Arjovsky et al., 2017), that takes more into account than raw pixel values.”. How can you substitute L2 loss in an autoencoder with Wasserstein metric (which is a metric between probability distributions, not images)?
There is also some missing related work, e.g. the idea mentioned in conclusion on augmenting the dataset in the latent space is presented in DeVries et al. “Dataset augmentation in feature space”.
It would be interesting to connect this work with importance sampling off-policy RL (see e.g. “Prioritized Experience Replay”) and look into sampling dataset points proportional to some importance probability with importance sampling correction.
On the positive side, I really enjoyed the look of the figures and diagrams.