RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since the synchronization of the local models or gradients can be a bottleneck for large-scale distributed training, compressing communication traffic has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches---it can significantly compress the transmitting message size (0.1% of the gradient size) of each node and still preserve accuracy. However, the literature on compressing deep networks focuses almost exclusively on achieving good compression rate, while the efficiency of RGC in real implementation has been less investigated. In this paper, we develop an RGC method that achieves significant training time improvement in real-world multi-GPU systems. Our proposed RGC system design called RedSync, introduces a set of optimizations to reduce communication bandwidth while introducing limited overhead. We examine the performance of RedSync on two different multiple GPU platforms, including a supercomputer and a multi-card server. Our test cases include image classification on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which has long been considered with poor scalability, RedSync shows significant performance improvement.
 Paper focuses on Residual Gradient Compression (RGC) as a promising approach to reducing the synchronization cost of gradients in a distributed settings. Prior approaches focus on the theoretical value of good compression rates without looking into the overall cost of the changes. This paper introduces RedSync that builds on the existing approaches by picking the most appropriate ones that reduce the overall cost for gradient reduction without unduly focusing on the compression rate. The paper does this by providing an analysis of the cost of RGC and also the limitations in scaling as the bandwidth required grows with the number of nodes. It also highlights the value of applying different algorithms in this process for compression and the benefits and issues with each. Pros: - Useful analysis that will help direct research in this area - Shows that this approach works for models that have a high communication to computation ratio - Provides a useful approach that works for a number of models Cons: - Positive experimental results are on models that are typically not used in practice e.g. AlexNet and VGG16 - Speedups shown on LSTMs don*t see worthwhile to scale, and in practice a model-parallelism approach may scale better Corrections: - Typo in notes for Table 1 last sentence RCG => RGC - Typo in first sentence in section 3.2: RedSycn => RedSync - Section 3.3, #2 last sentence: maybe overdrafts => overshadows ?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since the synchronization of the local models or gradients can be a bottleneck for large-scale distributed training, compressing communication traffic has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches---it can significantly compress the transmitting message size (0.1% of the gradient size) of each node and still preserve accuracy. However, the literature on compressing deep networks focuses almost exclusively on achieving good compression rate, while the efficiency of RGC in real implementation has been less investigated. In this paper, we develop an RGC method that achieves significant training time improvement in real-world multi-GPU systems. Our proposed RGC system design called RedSync, introduces a set of optimizations to reduce communication bandwidth while introducing limited overhead. We examine the performance of RedSync on two different multiple GPU platforms, including a supercomputer and a multi-card server. Our test cases include image classification on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which has long been considered with poor scalability, RedSync shows significant performance improvement.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Paper focuses on Residual Gradient Compression (RGC) as a promising approach to reducing the synchronization cost of gradients in a distributed settings. Prior approaches focus on the theoretical value of good compression rates without looking into the overall cost of the changes. This paper introduces RedSync that builds on the existing approaches by picking the most appropriate ones that reduce the overall cost for gradient reduction without unduly focusing on the compression rate. The paper does this by providing an analysis of the cost of RGC and also the limitations in scaling as the bandwidth required grows with the number of nodes. It also highlights the value of applying different algorithms in this process for compression and the benefits and issues with each. Pros: - Useful analysis that will help direct research in this area - Shows that this approach works for models that have a high communication to computation ratio - Provides a useful approach that works for a number of models Cons: - Positive experimental results are on models that are typically not used in practice e.g. AlexNet and VGG16 - Speedups shown on LSTMs don*t see worthwhile to scale, and in practice a model-parallelism approach may scale better Corrections: - Typo in notes for Table 1 last sentence RCG => RGC - Typo in first sentence in section 3.2: RedSycn => RedSync - Section 3.3, #2 last sentence: maybe overdrafts => overshadows ?
assistant
 Paper focuses on Residual Gradient Compression (RGC) as a promising approach to reducing the synchronization cost of gradients in a distributed settings. Prior approaches focus on the theoretical value of good compression rates without looking into the overall cost of the changes. This paper introduces RedSync that builds on the existing approaches by picking the most appropriate ones that reduce the overall cost for gradient reduction without unduly focusing on the compression rate. The paper does this by providing an analysis of the cost of RGC and also the limitations in scaling as the bandwidth required grows with the number of nodes. It also highlights the value of applying different algorithms in this process for compression and the benefits and issues with each. Pros: - Useful analysis that will help direct research in this area - Shows that this approach works for models that have a high communication to computation ratio - Provides a useful approach that works for a number of models Cons: - Positive experimental results are on models that are typically not used in practice e.g. AlexNet and VGG16 - Speedups shown on LSTMs don*t see worthwhile to scale, and in practice a model-parallelism approach may scale better Corrections: - Typo in notes for Table 1 last sentence RCG => RGC - Typo in first sentence in section 3.2: RedSycn => RedSync - Section 3.3, #2 last sentence: maybe overdrafts => overshadows ?