Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since the synchronization of the local models or gradients can be a bottleneck for large-scale distributed training, compressing communication traffic has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches-it can significantly compress the transmitting message size (0.1% of the gradient size) of each node and still preserve accuracy. However, the literature on compressing deep networks focuses almost exclusively on achieving good compression rate, while the efficiency of RGC in real implementation has been less investigated. In this paper, we develop an RGC method that achieves significant training time improvement in real-world multi-GPU systems. Our proposed RGC system design called RedSync, introduces a set of optimizations to reduce communication bandwidth while introducing limited overhead. We examine the performance of RedSync on two different multiple GPU platforms, including a supercomputer and a multi-card server. Our test cases include image classification on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which has long been considered with poor scalability, RedSync shows significant performance improvement.
Quality and clarity:
The paper proposes an approach to reduce the communication bandwidth and overhead in distributed deep learning. The approach leverages on previous work (mainly the residual gradient compression (RGC) algorithm), and proposes several implementation optimizations. From what I can read, it is the basic RGC algorithm that is used, but with some clever optimization to improve the performance of it. 
The quality of the paper is good, it is well-written and easy to read. The evaluation of the proposed approach is well done, using several diverse datasets and models, and executed on two different parallel systems. However, the reasons why RGC and qRGC sometimes have better accuracy than SGD needs to be analyzed and explained. 
Originality and significance:
The originality of the paper is relatively low (optimization of an existing algorithm) and the contributions are incremental. However, the paper addresses an important practical problem in distributed learning, and thus can have a significant practical impact on how distributed deep learning systems are implemented.
Pros:
* Addresses an important issue. 
* Good performance.
* Good evaluation on two different systems. 
Cons:
* Limited contribution. Although I like implementation papers (very important), I think the contribution is to low for ICLR.
Minor:
* In general, the figures are hard to read (the main problem is to small text)
* Compression in the title is slightly misleading, since it's mainly selection that is done (top-0.1% gradients). Although the values are packed in a data structure for transmission, it's not compression in a information theory perspective.