Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations and the asynchronous parallel mechanism. To fill the gaps in theory and provide theoretical supports, this paper studies two asynchronous parallel implementations of SG: one is over a computer network and the other is on a shared memory system. We establish an ergodic convergence rate O(1/ √ K) for both algorithms and prove that the linear speedup is achievable if the number of workers is bounded by √ K (K is the total number of iterations). Our results generalize and improve existing analysis for convex minimization.
The authors extended the stochastic gradient and coordinate descent to the asynchronous and nonconvex differentiable setting, assuming that the asynchronism is bounded. The authors showed that the average of the expected gradient converges to 0 at the usual sublinear rate O(1/sqrt{K}), where K is the number of iterations. The proofs appear to be solid and the authors did a fair job in comparing with relevant works in the literature. Very limited experiments were given in the appendix for a quick sanity check. Overall, the results in this work may be a good addition to the field.Quality: The proofs mostly appear to be correct, and the model, much built on previous work, is useful and relevant in practice. However, I am not quite convinced by the measure of convergence: the average of the expected gradients converge to 0 seems to be too weak. First, what can we conclude from the expectation here? If we run the algorithm for say 10 repetitions, we cannot really average the iterates to get a single iterate that has a strong guarantee. This is the cost of not having convexity. Second, the obtained results do not cover the known facts for convex functions. It would be much more convincing had one been able to include the special convex case. Some technical issues:1). In assumption 3, i_k is allowed to be dependent on xi_k. This seems to contradict Eq. (33) in line 937, i.e., conditioned on xi_k, i_k may not be uniformly distributed. This may invalidate the claim in Eq. (16) (line 406). 2). Line 814, here one should first condition on all xi_s, s < j'' (for fixed m), and then take expectation wrt xi_{j''} to zero out the term involving xi_{j''}. This is because x_{j''-tau_{j'',m}} may depend on xi_{j'} so we cannot first take expectation wrt xi_{j'}. (Also, here \nabla_{i_{j'}} should be \nabla, multiple times.) 3). Some calculations from line 1228 to 1236 seems to be messed up, but it does not seem to affect the claim. 4). The conditions in theorem 1 and 2 both involve f(x^*). This is undesirable. It would be great if one can come up with practical, usable bounds.Clarity: The paper is mostly easy to follow.Originality: The authors mostly follow the existing model in the literature but extend them to the nonconvex setting. The proofs do not involve any new techniques. So overall the work is incremental, but I think it is still of some interest.Significance: I feel that the authors oversell their work a bit, for the obtained results are far from "explaining deep learning or nonconvex stochastic optimization." The obtained sublinear rates also need more careful interpretation.Reference: The following work also seems to be relevant and perhaps should be cited: Ho et al., More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server. NIPS 2013.