Byzantine Stochastic Gradient Descent
This paper studies the problem of distributed stochastic optimization in an adversarial setting where, out of m machines which allegedly compute stochastic gradients every iteration, an ↵-fraction are Byzantine, and may behave adversarially. Our main result is a variant of stochastic gradient descent (SGD) which finds "-approximate minimizers of convex functions in
" 2 iterations. In contrast, traditional mini-batch SGD needs T = O 1 " 2 m iterations, but cannot tolerate Byzantine failures. Further, we provide a lower bound showing that, up to logarithmic factors, our algorithm is information-theoretically optimal both in terms of sample complexity and time complexity.
The paper studies stochastic convex optimization in a distributed master/workers framework, where on each round each machine out of m produces a stochastic gradient and sends it to the master, which aggregates these into a mini-batch. In this paper the authors allow a fraction of alpha of the machines to be Byzantine, i.e., they do not need to report valid stochastic gradients but may produce arbitrary vectors, even in an adversarial manner. The goal is to aggregate the reports of the machines and to converge to an optimal solution of the convex objective despite the malicious Byzantine machines.  The authors present a novel variant of minibatch-SGD which tackles the difficulty the dealing with Byzantine machines. They prove upper-bounds on the convergence and nearly optimal matching lower-bounds on any algorithm working in such framework, and in this sense the results are quite satisfactory. Importantly, the convergence is roughly the same as mini-batch SGD but includes an additional natural term that accounts for the Byzantine machines.   A somewhat similar result appeared in ICML18 [31]. [31] considers a very related problem, yet different, in which each machine is not represented by a stochastic first-order oracle, but is given a batch of m samples for the population distribution. The authors in [31] give a master/workers algorithm not based on SGD but on deterministic GD, where on each round, each machine reports the gradient w.r.t. its entire batch.  Both the current work and [31] can be compared w.r.t to sample complexity, i.e., number of samples each machine uses to guarantee a certain approximation error. While the two works use somewhat different assumptions, still the current work improves a key term in the convergence rate in terms of the dimension of the problem (linear dependence on d in [31] vs. no dependence in current paper). I find this improvement to be of great importance, especially for large-scale problems. Moreover, the algorithmic techniques used in this current paper seems to be quite different than those in [31], e.g., current paper tries to track the set of non-Byzntine machines throughout the run, while no such argument is present in [31]. Hence, also in terms of techniques the current paper seems sufficiently novel.