Federated Learning is a machine learning setting where the goal is to train a highquality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.
The authors examine several techniques that lead to low communication updates during distributed training in the context of Federated learning (FL). Under the setup of FL, it is assumed that training takes place over edge-device like compute nodes that have access to subsets of data (potentially of different size), and each node can potentially be of different computational power. Most importantly, in the FL setup, communication is the bottleneck. Eg a global model is to be trained by local updates that occur on mobile phones, and communication cost is high due to slow up-link.
The authors present techniques that are of similar flavor to quantized+sparsified updates. They distinguish theirs approaches into 1) structured updates and 2) sketched updates. For 1) they examine a low-rank version of distributed SGD where instead of communicating full-rank model updates, the updates are factored into two low rank components, and only one of them is optimized at each iteration, while the other can be randomly sampled.
They also examine random masking, eg a sparsification of the updates, that retains a random subset of the entries of the gradient update (eg by zero-ing out a random subset of elements). This latter technique is similar to randomized coordinate descent.
Under the theme of sketched updates, they examine quantized and sparsified updates with the property that in expectation they are identical to the true updates. The authors specifically examine random subsampling (which is the same as random masking, with different weights) and probabilistic quantization, where each element of a gradient update is randomly quantized to b bits. 
The major contribution of this paper is their experimental section, where the authors show the effects of training with structured, or sketched updates, in terms of reduced communication cost, and the effect on the training accuracy. They present experiments on several data sets, and observe that among all the techniques, random quantization can have a significant reduction of up to 32x in communication with minimal loss in accuracy.
My main concern about this paper is that although the presented techniques work well in practice, some of the algorithms tested are similar algorithms that have already been proven to work well in practice. For example, it is unclear how the performance of the presented quantization algorithms compares to say  QSGD [1] and Terngrad [2]. Although the authors cite QSGD, they do not directly compare against it in experiments.
As a matter of fact, one of the issues of the presented quantized techniques (the fact that random rotations might be needed when the dynamic range of elements is large, or when the updates are nearly sparse) is easily resolved by algorithms like QSGD and Terngrad that respect (and promote) sparsity in the updates. 
A more minor comment is that it is unclear that averaging is the right way to combine locally trained models for nonconvex problems. Recently, it has been shown that averaging can be suboptimal for nonconvex problems, eg a better averaging scheme can be used in place [3]. However, I would not worry too much about that issue, as the same techniques presented in this paper apply to any weighted linear averaging algorithm.
Another minor comment: The legends in the figures are tiny, and really hard to read.
Overall this paper examines interesting structured and randomized low communication updates for distributed FL, but lacks some important experimental comparisons.
[1] QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks https://arxiv.org/abs/1610.02132
[2] TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
https://arxiv.org/abs/1705.07878
[3] Parallel SGD: When does averaging help? 
https://arxiv.org/abs/1606.07365