Training Deep Models Faster with Robust, Approximate Importance Sampling
In theory, importance sampling speeds up stochastic gradient algorithms for supervised learning by prioritizing training examples. In practice, the cost of computing importances greatly limits the impact of importance sampling. We propose a robust, approximate importance sampling procedure (RAIS) for stochastic gradient descent. By approximating the ideal sampling distribution using robust optimization, RAIS provides much of the benefit of exact importance sampling with drastically reduced overhead. Empirically, we find RAIS-SGD and standard SGD follow similar learning curves, but RAIS moves faster through these paths, achieving speed-ups of at least 20% and sometimes much more.
The paper proposes a robust way to approximate optimal importance sampling weights to reduce the variance of stochastic gradient for training machine learning models. I think the idea (and implementation) is definitely valuable for the community and deserves a publication, but improving experiments can make the paper much more useful and insightful. In particular, I believe that 1) the exact O-SGD baseline (sampling without approximations) should be included in the plots at least for small-scale experiments; 2) there should be not only epoch plots, but the time plots as well to directly see the overhead of the sampling scheme; 3) there should be oblation studies for the factors you think make your method particularly effective (the ones listed in sec 6.2: large dataset size, lack of random data augmentation; etc). Also, you mention that you use only the loss layer of the networks to compute the gradient norms, which I see as a major approximation. Consider using the method from [1] at least as a baseline if the overhead is too large to use it on practice.  Again, the paper is really good, but I would love to see it getting even better before the publication.  Minor points: Would be nice to have pointer to appendix whenever you included the proof in where. E.g. now it looks like you consider proof of (4) too trivial to write down and I spent some time to rederive it only to find later that it’s actually proven in the appendix. The introduction of the dataset D on line 109 is confusing because it appears out of nowhere and when get explained slowly. Also, I read this part many times and still not sure how do you choose objects for D. I don’t get equation (6): how does it follow from (4) and what is g_{U1} and g_{R1}? Figure 6.2 referenced on line 228 doesn’t seem to exist Since you mentioned generalization as something you pay attention to (line 237), it would be nice to have test errors as well (in contrast to only having validation errors, since you used validation set for hyperparameters tuning). BTW, did you find optimal hyperparameters for SGD and for your method independently?  [1] Goodfellow, Ian. "Efficient per-example gradient computations." arXiv preprint arXiv:1510.01799 (2015).  UPD: thanks for addressing my concerns, I'm now even more willing to accept the paper.