Communication Compression for Decentralized Training
Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: communication compression for low bandwidth networks, and decentralization for high latency networks. In this paper, We explore a natural question: can the combination of both techniques lead to a system that is robust to both bandwidth and latency?
Although the system implication of such combination is trivial, the underlying theoretical principle and algorithm design is challenging: unlike centralized algorithms, simply compressing exchanged information, even in an unbiased stochastic way, within the decentralized network would accumulate the error and fail to converge. In this paper, we develop a framework of compressed, decentralized training and propose two different strategies, which we call extrapolation compression and difference compression. We analyze both algorithms and prove both converge at the rate of O(1/ √ nT ) where n is the number of workers and T is the number of iterations, matching the convergence rate for full precision, centralized training. We validate our algorithms and find that our proposed algorithm outperforms the best of merely decentralized and merely quantized algorithm significantly for networks with both high latency and low bandwidth.
The paper presents two approaches to combining decentralized learning with model compression. Both method use gossip communication over a given network of nodes and model averaging. Before sharing the models with other nodes, they are compressed by either by using an unbiased compression on the difference of the model to the last version (the algorithm is called DCD-PSGD), or by an unbiased compression of a linear extrapolation of the model using the last version (ECD-PSGD). For both approaches, their convergence rate is theoretically analyzed. The combination of decentralized learning and model compression allows to handle networks with both low bandwidth and high latency. This claim is substantiated by an empirical evaluation.   Quality: The idea of using compression in decentralized learning seems natural but interesting and the results appear to be useful. The technical content is well-presented and sound, the proofs appear to be correct. The claims of the paper are supported by both the theoretical analysis and the experiments. There is one minor issue I have. In Section 4.1 the paper claims a linear speedup of the DCD-PSGD algorithm (and the same for ECD-PSGD in Section 4.2). This definitely holds as long as the computational cost of calculating the average is in O(1), which is true as long as the network has an upper bounded node degree. However, for a fully connected network, the cost for calculating the average at each node would be in O(n), which is a constant overhead. This should reduce the speedup to sublinear. If this is correct, I would ask the authors to add an assumption, like an upper bounded node degree of the network, where the bound does not depend on the network size.   Clarity: The paper is very well-written and clearly structured. The theoretical parts are comprehensible. The paper is overall a pleasure to read. There are some minor issues with English style and grammar, so I would ask the authors to have the paper proof-read. There are also some minor points and typos: - In Section 4.1 "Consistence with D-PSGD": I guess, ECD-PSGD should be DCD-PSGD, right?  - In Section 4.2, Eq. (3) there is a t missing before x_t^(j) - In Section 5: it would be helpful to know which exact compression operator was used - In Section 5.4: blackucing should be reducing - In Appendix A.2, proof of Lemma 7: in the equation after "Therefore it yields", I don't see a difference between the lines 3 and 4. I guess one is a copy artifact.  I would also like to ask the authors to check the formatting of the paper, as it seems to be a bit non-standard: the abstract is in italic, the placement of Figure 1 and the Definitions and Notations subsection is unusual, and there are weird line spaces (e.g., 226 to 227).  Originality: Both decentralized learning by model averaging and communication compression have been studied before, as the authors admit, but their combination is relatively new (it was discussed in an arxiv paper of Konecny et al. "Federated Learning: Strategies for Improving Communication Efficiency" that is not yet published, but there it is only studied empirically). Moreover, they are to the best of my knowledge the first to theoretically analyse this approach. Thus, the main contribution of the paper are two novel and theoretically sound algorithms for decentralized learning with compression.   Significance: Even though not groundbreaking, I really liked the paper. The algorithms it proposes seem to be useful for decentralized machine learning in many real-world application. The algorithms are proven to be theoretically sound and the empirical evaluation indicates the usefulness of the approach.   The author's response has addressed my concern about the assumption of O(1) runtime complexity for calculating the average.