Large mini-batch parallel SGD is commonly used for distributed training of deep networks. Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication. In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training. SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneficial in high-latency or high-variability scenarios. The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies. We prove that SGP converges to a stationary point of smooth, non-convex objective functions. Furthermore, we validate empirically the potential of SGP. For example, using 32 nodes with 8 GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.6 hours while AllReduce SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.
# overview
This paper leverages a consensus based approach for computing and communicating approximate gradient averages to each node running a decentralized version of stochastic gradient descent.
Though the PushSum idea isn't new, its application to distributed SGD and corresponding convergence analysis represents a valuable contribution, and the experimental results indicate a potentially large speedup (in highly variable or latent networks) without substantially sacrificing model accuracy.
The paper itself is reasonably comprehensive but does miss out on comparisons with more recent but equally promising approaches, namely AD-PSGD. 
# pros
* Empirically shown to be significantly faster than SGD, D-PSGD in high-latency, communication bound configurations which is a fairly common real-world setup. There is an accuracy tradeoff at work here, but performance doesn't seem to suffer too much as the node count scales.
* introduces and proves theoretical guarantees for SGP approximate distributed average convergence for smooth, non-convex case, including upper bounded convergence rates.
# cons
* biggest criticism is that AD-PSGD from Lian et al 2018 is not included in experimental comparisons even though the paper is referenced. Authors state that asynchronous methods typically generalize worse than their synchonous counterparts but that isn't what Lian et al found in their comparison with D-PSGD (see table 2 and 3 from their paper). This comparison would be particularly interesting as AD-PSGD also performs well in the high network latency regime that SGP is touted for.
* would've liked to see comparison on other tasks beyond just image classification on ResNet.
# other comments
* Didn't see mention of specific iteration count value(s) K used in experiments or hyperparameters A.3. Since it bounds the convergence rate, this would be important to include.
* Found a few small typos:
  * pg. 5: Relatively -> Relative
  * pg. 7, fig. 2: part -> par
  * pg. 8, sec. 5.3. par. 2: achieves -> achieved
  * pg. 8, sec. 5.3, par. 2: "neighbors also to increases" (drop "to")
  * pg. 12, sec. A.2: "send-buffer to filled" -> "send-buffer to be filled"