Variance-Reduced Stochastic Gradient Descent on Streaming Data
We present an algorithm STRSAGA that can efficiently maintain a machine learning model over data points that arrive over time, and quickly update the model as new training data are observed. We present a competitive analysis that compares the suboptimality of the model maintained by STRSAGA with that of an offline algorithm that is given the entire data beforehand. Our theoretical and experimental results show that the risk of STRSAGA is comparable to that of an offline algorithm on a variety of input arrival patterns, and its experimental performance is significantly better than prior algorithms suited for streaming data, such as SGD and SSVRG.
This paper considers the problem of streaming stochastic optimization taking into account the arrival patterns of examples in time. Whereas the relevant previous work focuses on learning from a stream (i.e. in one pass over the data, with O(dimension) memory), this work attempts to utilize the time spent between the arrival of examples in order to revisit previously encountered examples.  The problem description is novel and appealing, both in its possible practical relevance and as an interesting new model for consideration in algorithm analysis. However, there are central issues with the comparisons that the paper draws between algorithms, both conceptually and experimentally.  Conceptually, it seems incorrect to say that STRSAGA is a streaming algorithm, and in turn in to repeatedly compare it to one (such as SSVRG), since STRSAGA's memory complexity actually grows linearly with the dataset size (in order to maintain its "effective sample set"). In all previous related work, a streaming algorithm essentially means one whose memory usage does not grow with the dataset size. This is exactly what distinguishes streaming algorithms from ERM-based algorithms. The latter maintain an empirical sample of points in memory, whereas the former cannot maintain more than a constant number of them.  Experimentally, the paper's main algorithm (STRSAGA) is missing some comparisons to natural baselines. Both DYNASAGA and SSVRG are not basic (nor widely used) algorithms. Meanwhile, a comparison to the simplest and most basic streaming algorithm, plain SGD, is missing.  Relating to baselines, the paper includes the introductory remark (line 74):  > We also show that STRSAGA significantly outperforms SSVRG [FGKS15], the state-of-the-art streaming data algorithm  The original SSVRG paper [FGKS15] offers no experiments, and I don't know of any published since. It's a theoretical result, being used here as an experimental baseline, as a means of demonstrating an empirical advantage. Because it isn't actually a commonly accepted state of the art in the practical sense, outperforming SSVRG in practice does not imply an improvement relative to the state of scientific knowledge. Again, the more obvious comparison here would be to plain one-pass SGD.  ----  Edited after author response:  The authors addressed the concern about experimental comparisons to SGD in their response. I suggest including this in the paper.  On the point of "streaming", here is a valid run of the algorithm:  Fix an integer k. Suppose there is a dataset of size 2k, and STRSAGA is run for steps 1, ..., k, where at each step it receives two subsequent data points from the dataset. After step k, STRSAGA has half of the original dataset stored in its effective sample set. Now suppose STRSAGA continues to run for T additional steps, receiving zero new points at each step. Then, during these T additional steps, STRSAGA processes that half of the original dataset that it had stored.  So STRSAGA's description includes, as a special case, an algorithm for minimizing empirical risk on half of a dataset, and this indicates that the paper needs work in its formal setup, not only in terminology. Maybe the right direction is to require a limit on memory, as reviewer 2 suggests as well.