Without-Replacement Sampling for Stochastic Gradient Methods
Stochastic gradient methods for machine learning and optimization problems are usually analyzed assuming data points are sampled with replacement. In contrast, sampling without replacement is far less understood, yet in practice it is very common, often easier to implement, and usually performs better. In this paper, we provide competitive convergence guarantees for without-replacement sampling under several scenarios, focusing on the natural regime of few passes over the data. Moreover, we describe a useful application of these results in the context of distributed optimization with randomly-partitioned data, yielding a nearly-optimal algorithm for regularized least squares (in terms of both communication complexity and runtime complexity) under broad parameter regimes. Our proof techniques combine ideas from stochastic optimization, adversarial online learning and transductive learning theory, and can potentially be applied to other stochastic optimization and learning problems.
The paper studies the problem of minimizing the average of a finite sum of convex functions over a convex domain using stochastic algorithms that, opposed to most popular methods, apply WITHOUT-replacement sampling to the data. More specifically, the paper considers methods that first randomly permute the functions, and then process the functions via incremental updates one at a time, making at most a single pass over the data (hence the data is only shuffled once). There are three main motivations for considering stochastic methods with without-replacement sampling: 1. it is observed many times empirically (and to the best of my knowledge this is what ML practitioners really do) that applying random shuffles to the data and then apply incremental updates works better than with-replacement SGD. 2. After the data is randomly permuted, these algorithms require only sequential access to the memory, which is much more efficient than standard with-replacement SGD methods that require random access to perform the updates. Since the main setting under consideration here is when making at most a single pass, it sounds plausible to assume that the data is already stored in some random permutation, and hence the algorithm is fully sequential, and there is no need to even artificially permute the data. 3. In a distributed setting in which data is partitioned to several machine (not assuming the data is sampled i.i.d from a distribution), it is more natural and efficient to analyze without-replacement algorithms.    While this setting is well motivated there has not been a large body of work with strong positive results on it which is understandable, since analyzing without-replacement sampling algorithms is much more difficult than the with-resampling variants. Once such previous recent result focused on the regime in which the algorithm makes a large number of passes over the data, re-permuting the data before each new pass. This current paper focuses on a regime which I find more interesting and relevant to ML applications, in which at most a single pass is made over the data. Indeed the celebrated with-replacement SGD method is most interesting exactly in this regime (though suffering from the short-coming of with-replacement sampling discussed above).  The main results are as follows: 1. the paper shows that any algorithm for which one can give a regret bound in the online learning adversarial model (which holds for SGD for instance) can be applied in the setting above with convergence rate resembling the average regret bounds. In particular, when all functions are convex Lipschitz losses for a linear predictor, wo-replacement SGD exhibits roughly the same convergence rate as standard SGD.   2. when the average over all functions is also strongly convex, wo-replacement SGD gives an improved rate, again similar to standard SGD up to log factors. 3. The authors gives a linearly converging wo-replacment variant for the celebrated SVRG algorithm in the specific yet important case in which the loss functions are the squared-loss. 4. The authors use the previous result to improve the application of SVRG for distributed optimization in the wo-replacement setting, improving the regime of parameters (i.e. the condition number) for which the method is highly efficient (in terms of communication).
As I detailed above, I believe the setting under consideration (without-replacment sampling for finite sum optimization using single pass) is very well motivated and widely applicable in ML problems. The results, to the best of my knowledge, are very novel and will be potentially interesting to many in the community, both theorists and practitioners. The paper is also very well written and results are carefully explained.  Some minor comments: -when defining F_{1:t-1}, I think it will also be beneficial for the reader to add a subscript of \sigma, reminding the reader this function is random. - in Lemma 1, please define s_{1:t-1} and s_{t:m}. - in Theorem 2, in the centered equation, it should be \mathcal{R}_{t-1,m-t+1}, i.e., comma instead of colon. - first eq. in proof of Theorem 3: on the RHS it should be the squared norm of w_{t+1}.