Variance Reduced Stochastic Gradient Descent with Neighbors
Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its slow convergence can be a computational bottleneck. Variance reduction techniques such as SAG, SVRG and SAGA have been proposed to overcome this weakness, achieving linear convergence. However, these methods are either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory. Therefore speed-ups relative to SGD may need a minimal number of epochs in order to materialize. This paper investigates algorithms that can exploit neighborhood structure in the training data to share and re-use information about past stochastic gradients across data points, which offers advantages in the transient optimization phase. As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms. We provide experimental results supporting our theory.
-- Comments after rebuttal -- The rebuttal from the authors was concise.However, I was not convinced about the assumption in Eq. 14 of the paper and how theauthors defended it. The authors say: "Many documents (text categorization),[..] or time series signals (speech recognition) in a training set are alike. This fact is not systematically exploited by any existing stochastic optimization method!" I don't think this is correct. In fact I think this is what statistical learning is all about,i.e., exploit similarities in data. Also in their point (B.2) the reviewers say "Eq. 14 should not be seen as an assumption but rather as one part of a trade-off. We can make \eta arbitrarily small by restricting the size of the neighborhood." These trade-offs are very hard to do in practice. This is in fact where the crux of their method lies. It would be very interesting if the authors had a concrete answer to that.Still, I slightly increased my rating for this paper because, judging by the rebuttal,it seems that the authors understand the shortcomings of this paper and how to resolve them. This should allow the paper to get published if other reviewers are convinced by the rebuttal. ---- The authors study a general formulation of stochastic variance reduction (SVRG)methods and derive a non-asymptotic analysis under strong convexity.This work is ambitious and creative. The idea of using neighborhoodstructure between data points to share information is very smart,and looks natural only in hindsight.However, by reading this paper I felt that it is work under progress with severalimportant details left unfinished. If I am mistaken I would be happy to recommend an accept.On a high-level the paper aims to do many things at the same time, e.g.,have a common analytic framework for SVRG methods, introduce and analyze theidea of data neighborhoods in stochastic approximations, apply SVRG in streaming settings, etc. As a reader I felt that none of these goals were fully achieved.   Regarding the common analytic framework of SVRG methodsI struggled with assumptions in Ineq. (14) and the uniformity assumptions for the memorizationalgorithms (e.g., Lines #213-215).Why are these assumptions reasonable? I am especially skeptical of Ineq. (14) because it uniformly bounds all past and present gradients and basically assumes everything is fine with the stochastic approximation. Although I could buy such an assumption for implicit procedures (e.g.,implicit SGD) which often have these uniform bounds,explicit procedures as in Eq. (2) generally do not.Regarding the streaming setting, I am confused how the authors define the streamingsetting and how they analyze and test in practice. In Line #411 the authors note: "We have found very similar results in a streaming setting, i.e., presenting data once in randomorder."I don't think this defines a streaming setting, and also seems to disagree with authors'discussion on the streaming setting in Abstract and intro sections. My own understanding is that in streaming settings one has access to infinitedata streamed one-by-one and then discarded. One key difficulty in the analysis of SVRG procedures in streaming settings is that one has no access to unbiased estimates of the full gradient.How do the authors cope with this difficulty? I suspect they use Lipschitz bounds as Ineq. (14)however this is not made explicit.Another issue here -- potentially more important -- is thatin streaming settings the estimation problem is different. In finite data sets SVRGmethods -- as well as other methods -- compute the empirical risk minimizer, e.g.,the MLE. However, in streaming settings one estimates the true model parameters. In the former case one approximates an estimator (e.g., MLE) and in the latter case one approximates the true model parameters. Thus, in the streaming case the "linear convergence rate" ofaveraging methods (mentioned by the authors in the abstract) cannot be surpassed. In the non-streaming case one can geometrically converge to the empirical minimizer, howeverthis minimizer is an estimator that is roughly O(1/n) away from the true model parameters. It is the nature of stochastic learning problems where the 1/n simply cannot be surpassed. This is a key issue that causes confusion in stochastic optimization from my own experience. Therefore, it would be important that the authors clarify the streaming setting theymention in the abstract, where it appears in the analysis, and how it is being tested in experiments. Under this light, it is unclear what they mean in Line #411.Finally, the promising idea of using neighbors of data remains underdeveloped. In Lines #138-144 the authors attempt to give practical advice on the matter but they are not specific. I am unsure how one canefficiently define these potentially O(n^2)neighborhood relations. The related analysis is solid using the Lipschitz conditions,but in the experiments it was unclear how the neighborhood SVRG variants were implemented.Other issues: - Figure 3 is not explained in main text. - Captions in figures are not informative. - Line #53, the Robbins-Monro paper is not about SGD. It is about the much more general idea ofstochastic approximations in the streaming setting.