Traditional synchronous distributed training is performed inside a cluster since it requires high bandwidth and low latency network (e.g. 25Gb Ethernet or Infiniband). However, in many application scenarios, training data are often distributed across many geographic locations, where physical distance is long and latency is high. Traditional synchronous distributed training cannot scale well under such limited network conditions. In this work, we aim to scale distributed learning under high-latency network. To achieve this, we propose Delayed and Temporally Sparse (DTS) update that enables synchronous training to tolerate extreme network conditions without compromising accuracy. We benchmark our algorithms on servers deployed across three continents in the world: London (Europe), Tokyo (Asia), Oregon (North America) and Ohio (North America). Under such challenging settings, DTS achieves 90× speedup over traditional methods without loss of accuracy on ImageNet.
This paper tackles the issue of scalability in distributed SGD over a high-latency network.
Along with experiments, this paper contributes two ideas: delayed updates and temporally sparse updates.
- strengths: 
· Clear-looking overall presentation.
  Good efforts to explain the network problems (latency, congestion) and how they are tackled by delayed updates and temporally sparse updates.
- weaknesses:
· While the overall presentation looks clean, the paper does not talk about the setting (one parameter server and many workers, only workers, etc).
  This is arguably basic information, and the readers is left to understand by themselves that the setting consists in many workers without a parameter server.
· There is no theoretical analysis of the soundness of the proposed algorithm.
  For instance in ASGD (that the authors cite), stale gradients are dampened before being used, which in turn is used to guarantee convergence.
  In the proposed algorithm, there is no dampening nor apparent limit of the maximum value of t; such a difference with prior art should entail a serious (theoretical) analysis.
· Finally, using SSGD for comparison is not very "fair", as communication-efficient algorithms have already been published for quite some time [1, 2, and follow-ups (e.g. searching for "local sgd")].
  At the very least a comparison with ASGD (cited) is necessary, as in a realistic setting latency is indeed a problem but arguably not bandwidth
  (plus, orthogonal gradient compression techniques do exist, e.g. as in "Federated Learning: Strategies for Improving Communication Efficiency").
questions to the authors: 
- Can you clarify the setting?
- Can you give at least an intuition why accepting stale gradient is correct (i.e. does not impede convergence)?
  There is no theoretical limit on the value of t; can workers take any arbitrary old gradient?
  So when the training is close to convergence (if it reaches it), i.e., when the norm of the gradients are close to 0,  the algorithm could then use very old gradients (which norms could be orders of magnitude larger) to "correct the mismatch" caused by the local training; is this correct?
  To solve that issue, ASGD introduces a simple dampening mechanism (which is necessary in the convergence proof), which your algorithm does not have.
The related work section focuses on gradient compression techniques (which tackle low bandwidth, not latency) and asynchronous SGD (which is more prone to congestion, with a single parameter server),
but seems to overlook that sparse communication techniques already exist (this fact should at least be mentioned).
The idea of sparse communication for SGD exists since at least 2012 [1, Algorithm 3].
A first mention of the use of such techniques for "communication efficiency" dates from (at least) 2015 [2].
[1] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao.
    Optimal distributed online prediction using mini-batches.
    J. Mach. Learn. Res., 13(1):165–202, January 2012.
[2] Sixin Zhang, Anna E. Choromanska, Yann LeCun.
    Deep learning with Elastic Averaging SGD.
    NeurIPS, 2015.
I do not see a clear novelty, nor a proof (or even intuitions) that the proposed algorithm is theoretically sound.
The comparison with SSGD is arguably unfair, since SSGD is arguably not at all the state-of-the-art in the proposed setting (hence the claimed speedup of x90 can be very misleading).