TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {−1, 0, 1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet doesn't incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available 1 .
The authors tried to reduce communication cost in distributed deep learning. They proposed a new method called Ternary Gradients to reduce the overhead of gradient synchronization. The technique is correct and the authors made some experiments to evaluate their method. Overall, the method is novel and interesting and may have significant impact on large scale deep learning. Some comments below for the authors to improve their paper.
1. Some notations and words should be made clearer.  For example, s is defined as the max value of abs(g), but the domain of the max operation fails to show up in the equation. 
2. The author mentioned that they tried to narrow the gap between the two bounds in the paper for better convergence, but not mentioned for better convergence rate or better convergence probability. And it’s a pity that there’s no theoretical analysis of the convergence rate.
3.  One big drawback is that the author made too strong assumption, i.e. the cost function has only one single minimum, which means the objective function is convex. It’s rare in practice. To my best knowledge, there’re many efficient algorithms for convex optimization in distributed systems. Therefore, the proposed method will not have big impacts.
4.  The author should use more metrics in the experiments instead of just accuracy.