  Stochastic gradient methods (SGMs) are the predominant approaches to train
deep learning models. The adaptive versions (e.g., Adam and AMSGrad) have been
extensively used in practice, partly because they achieve faster convergence
than the non-adaptive versions while incurring little overhead. On the other
hand, asynchronous (async) parallel computing has exhibited significantly
higher speed-up over its synchronous (sync) counterpart. Async-parallel
non-adaptive SGMs have been well studied in the literature from the
perspectives of both theory and practical performance. Adaptive SGMs can also
be implemented without much difficulty in an async-parallel way. However, to
the best of our knowledge, no theoretical result of async-parallel adaptive
SGMs has been established. The difficulty for analyzing adaptive SGMs with
async updates originates from the second moment term. In this paper, we propose
an async-parallel adaptive SGM based on AMSGrad. We show that the proposed
method inherits the convergence guarantee of AMSGrad for both convex and
non-convex problems, if the staleness (also called delay) caused by asynchrony
is bounded. Our convergence rate results indicate a nearly linear
parallelization speed-up if $\tau=o(K^{\frac{1}{4}})$, where $\tau$ is the
staleness and $K$ is the number of iterations. The proposed method is tested on
both convex and non-convex machine learning problems, and the numerical results
demonstrate its clear advantages over the sync counterpart and the
async-parallel nonadaptive SGM.

Optimization and Control;Distributed, Parallel, and Cluster Computing;Numerical Analysis;Numerical Analysis