A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order
Asynchronous parallel optimization received substantial successes and extensive attention recently. One of core theoretical questions is how much speedup (or benefit) the asynchronous parallelization can bring to us. This paper provides a comprehensive and generic analysis to study the speedup property for a broad range of asynchronous parallel stochastic algorithms from the zeroth order to the first order methods. Our result recovers or improves existing analysis on special cases, provides more insights for understanding the asynchronous parallel behaviors, and suggests a novel asynchronous parallel zeroth order method for the first time. Our experiments provide novel applications of the proposed asynchronous parallel zeroth order method on hyper parameter tuning and model blending problems.
The paper provides a general analysis of a wide class of asynchronous stochastic algorithms, including stochastic gradient descent, and stochastic coordinate descent, all in the lock-free case. Serious numerical examples are provided to back up their claims of near linear speedup.
This is the best paper I have read for NIPS this year. I think it is a great contribution. Not only is a powerful framework and theorem provided, but the authors take the time to show how to apply it to other cases, and they discuss interesting features (for example, they make sure the results make sense in light of previous results, and discuss the effects of the variance in the gradient estimate). The authors instill confidence to the reader that the authors did a good job critically examining their own results.  The numerical examples are not the typical small-scale test that claim this is the best algorithm ever, but rather they are well-down tests with good implementations on real-world datasets, and they do not claim to be the “best” algorithm ever, but rather test the stated claim of the paper — namely, that the asynchronous nature leads to speedup benefits up to a certain point.  My main suggestion is that the authors re-proof the paper, as there are still many grammar mistakes (for example, many missing articles before nouns).  I also must mention that I did not have time to read the appendix, so I am taking the proofs at the authors’ word.   Other minor issues I noticed:  - “sever” is listed several times, and should be “server” as in the title of [17].  - You might want to put the equation on line 107 as its own numbered equation, and mention that this is actually a change in the algorithm when mu=0.   - Line 113, I don’t see where \hat{x}_k has been defined, so I am a bit confused about where this fits into the discussion.  - Equation (4), I don’t see any coordinate indices, just the iteration indices, so it seems like this is a consistent read, not an inconsistent read. Maybe this is a notational issue? I’m confused.  - Line 150, the choice of gamma for the Theorem to hold requires a lot of known parameters. Could you analyze the algorithm assuming that gamma is this quantity, up to a fixed constant?  Or at least run numerical examples showing the robustness of the algorithm to incorrect choices in gamma?  - Line 241 is not a complete sentence.