GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training
Data parallelism can boost the training speed of convolutional neural networks (CNN), but could suffer from significant communication costs caused by gradient aggregation. To alleviate this problem, several scalar quantization techniques have been developed to compress the gradients. But these techniques could perform poorly when used together with decentralized aggregation protocols like ring all-reduce (RAR), mainly due to their inability to directly aggregate compressed gradients. In this paper, we empirically demonstrate the strong linear correlations between CNN gradients, and propose a gradient vector quantization technique, named GradiVeQ, to exploit these correlations through principal component analysis (PCA) for substantial gradient dimension reduction. GradiVeQ enables direct aggregation of compressed gradients, hence allows us to build a distributed learning system that parallelizes GradiVeQ gradient compression and RAR communications. Extensive experiments on popular CNNs demonstrate that applying GradiVeQ slashes the wall-clock gradient aggregation time of the original RAR by more than 5X without noticeable accuracy loss, and reduces the end-to-end training time by almost 50%. The results also show that GradiVeQ is compatible with scalar quantization techniques such as QSGD (Quantized SGD), and achieves a much higher speed-up gain under the same compression ratio.
This describes how a PCA during full-vector can be used to predict a good compression method. They do a good job arguing that PCA among gradient vectors should be a common case in machine learning. The results look good.  On the down side, it requires a fair amount of coding work to include it in what you do, because you still have a periodic "full gradient" phase. The PCA and how it is approximated are practical heuristics, so I don't expect a proof to be possible without a bit of fine print. I don't think there was a discussion of what happens with rare classes.  In this last case, some gradients might be really badly represented.  In such cases, I think you can detect the badly compressed gradient representations and "do something different".  But I think the paper is OK as is, because it proposes a clean approach that can hopefully be compared with the other "do something different" compression schemes that target more the communication than the aggregation cost of distributed SGD methods.  #### After authors' rebuttal and other reviews #### I thought the authors responded well to the reviewers' questions and propose some good changes to the paper.  The rebuttal did not really address my class imbalance concern, which I guess also bothered Reviewer #3, who commented about the time-invariance assumption.  In both cases a bothersome feature is that the applicability of a PCA assumption for gradient compression might break down, and you might not notice it.  So if "GradiVeQ is able to detect compression performance by measuring the training compression loss" [rebuttal] refers to a locally measured compression fidelity, this should be in the paper --- if you can formulate a loss that degrades somewhat gracefully over the time-invariance "L" period, then both Reviewer-3 and  I might be satisfied that a feature implementation has a decent chance of automagically detecting when the compressor's assumptions are beginning to malfunction.  I'm keeping my overall score the same.