  High-concurrency asynchronous training upon parameter server (PS)
architecture and high-performance synchronous training upon all-reduce (AR)
architecture are the most commonly deployed distributed training modes for
recommendation models. Although synchronous AR training is designed to have
higher training efficiency, asynchronous PS training would be a better choice
for training speed when there are stragglers (slow workers) in the shared
cluster, especially under limited computing resources. An ideal way to take
full advantage of these two training modes is to switch between them upon the
cluster status. However, switching training modes often requires tuning
hyper-parameters, which is extremely time- and resource-consuming. We find two
obstacles to a tuning-free approach: the different distribution of the gradient
values and the stale gradients from the stragglers. This paper proposes Global
Batch gradients Aggregation (GBA) over PS, which aggregates and applies
gradients with the same global batch size as the synchronous training. A
token-control process is implemented to assemble the gradients and decay the
gradients with severe staleness. We provide the convergence analysis to reveal
that GBA has comparable convergence properties with the synchronous training,
and demonstrate the robustness of GBA the recommendation models against the
gradient staleness. Experiments on three industrial-scale recommendation tasks
show that GBA is an effective tuning-free approach for switching. Compared to
the state-of-the-art derived asynchronous training, GBA achieves up to 0.2%
improvement on the AUC metric, which is significant for the recommendation
models. Meanwhile, under the strained hardware resource, GBA speeds up at least
2.4x compared to synchronous training.

Machine Learning;Distributed, Parallel, and Cluster Computing;Information Retrieval