Distributed Stochastic Optimization via Adaptive SGD
Stochastic convex optimization algorithms are the most popular way to train machine learning models on large-scale data. Scaling up the training process of these models is crucial, but the most popular algorithm, Stochastic Gradient Descent (SGD), is a serial method that is surprisingly hard to parallelize. In this paper, we propose an efficient distributed stochastic optimization method by combining adaptivity with variance reduction techniques. Our analysis yields a linear speedup in the number of machines, constant memory footprint, and only a logarithmic number of communication rounds. Critically, our approach is a black-box reduction that parallelizes any serial online learning algorithm, streamlining prior analysis and allowing us to leverage the significant progress that has been made in designing adaptive algorithms. In particular, we achieve optimal convergence rates without any prior knowledge of smoothness parameters, yielding a more robust algorithm that reduces the need for hyperparameter tuning. We implement our algorithm in the Spark distributed framework and exhibit dramatic performance gains on large-scale logistic regression problems.
=========== after rebuttal I carefully read the authors’ response. Some of my criticism has been addressed, yet I think that some critical flaws are still in the paper and I am not confident that these would be fixable in a final version within this submission round.  I believe that the authors do not make a strong case for their algorithm, since they do not appropriately compare with the state of the art. Eg, they do not compare to any of the algorithms mentioned below: https://arxiv.org/pdf/1511.01942.pdf and https://arxiv.org/pdf/1603.06861.pdf The authors mention in their experimental section: “Implementing our algorithm in the Spark environment is fairly straightforward. SVRG OL switches between two phases: a batch gradient computation phase and a serial SGD phase. “ As I mentioned in my review, this is very very similar -modulo the varying stepsize- to the algorithms above. At the very least the authors should have compared with the above.  Moreover, the authors *do not* support a key claim of the paper, ie that the proposed algorithm achieves a linear speedup? (this claim is made in the 4th sentence of their abstract, and repeated through out their text) There is no single experiment that compares the serial implementation of the algorithm with its distributed version, hence no scaling/speedup result can be inferred.   Since they addressed some of my criticisms, but not the major points above, I will update my score from 3 to 4.  ===============original review The authors present SVRG OL, an adaptive SVRG algorithm (not quite sure why it is refered as SGD), that is designed to get linear speedups on parallel architectures.  The theoretical contribution of the paper is that the algorithm requires log number of comm rounds to achieve an epsilon accurate result.  Although the problem setup is interesting, there are several issues with the paper that I list below:  - First of all, the presented algorithm is *identical* to  Algorithm 1 here:https://arxiv.org/pdf/1412.6606.pdf  Algorithm 1 here https://arxiv.org/pdf/1511.01942.pdf  and CheapSVRG here: https://arxiv.org/pdf/1603.06861.pdf It is unclear to me what is algorithmically novel between the above and the presented SVRG OL algorithm.  - This is certainly subjective, but I do not understand the usefulness of online optimization results in the context of distributed setups. In most dist. setups that are of interest, all data is available for batch processing. In this case convergence rates are indeed important, but online regret bounds don’t shed a particular insight on how to implement these algorithms.   - The authors mention “Unfortunately, minibatch-SGD obtains a communication complexity that scales as sqrt{N} (or N^1/4 for accelerated variants). In modern problems when N is extremely large, this overhead is prohibitively large.” This is absolutely not true, as every single large scale ML problem that trains DNNs uses backprop and minibatch SGD implementations. So although the # comm rounds is high, it is not prohibitive.   - The authors mention “prior analyses require specific settings for ⌘ that incorporate L and fail to converge with incorrect settings, requiring the user to manually tune ⌘ to obtain the desired performance. “  That is true, but in practice one almost never computes L or the str. cvx constant, as that can be as expensive as solving the problem. For non-convex problems that actually *never* happens (eg pre computing L).  - The authors implement the algorithm in spark. All the data sets tested fit in a single machine, and can most probably be run much faster with a simple python implementation of SVRG. I will not be convinced that the implementations presented are relevant, unless the authors have a comparison with a single machine, SVRG non-Spark baseline.  - Although the authors make a claim on linear speedups, no speedup curves are presented in the paper.  - minor: Although the authors mention they need log number of comm rounds, in Table 1 they present constants and omit the log factors. There is enough space to list the details in this table.   Overall, this paper deals with an interesting problem. however, the presented algorithm is not novel, the implementation and experiments might not be as relevant, and comparisons with serial SVRG are missing. I do not think the paper should be accepted to NIPS.