We propose Zeno++, a new robust asynchronous Stochastic Gradient Descent (SGD) procedure which tolerates Byzantine failures of the workers. In contrast to previous work, Zeno++ removes some unrealistic restrictions on workerserver communications, allowing for fully asynchronous updates from anonymous workers, arbitrarily stale worker updates, and the possibility of an unbounded number of Byzantine workers. The key idea is to estimate the descent of the loss value after the candidate gradient is applied, where large descent values indicate that the update results in optimization progress. We prove the convergence of Zeno++ for non-convex problems under Byzantine failures. Experimental results show that Zeno++ outperforms existing approaches.
This paper proposes an approach to Byzantine fault tolerance in asynchronous distributed SGD. The approach appears to be novel. Theoretical convergence guarantees are provided, and an empirical evaluation illustrates very promising results.
Overall, the paper is well-written and the results appear to be novel and interesting. Although I have a few questions, listed below, I generally lean towards accepting this paper.
The assumption of a lower bound on validation gradients is somewhat troubling, especially for over-parameterized problems where so-called "interpolation" may be possible. I realize that validation samples are never used explicitly for stochastic gradient updates, but the algorithm does ensure that the stochastic gradients used are similar to gradients of validation samples. If one is converging to a (local) minimizer, one wants the gradient to vanish. How do we reconcile these points? Also, to properly set $\rho$, for the theory to be valid, one needs to know this bound (or a lower bound on $V_2$). Is this practical?
The paper claims that the computational overhead of Zeno+ is too great to evaluate for comparison with Zeno++. From my reading of the two methods, it isn't immediately obvious to me why this is the case. Including experiments which at least compare the per-iteration runtime (even if not running Zeno+ for training to completion) would make the paper more compelling. After all, Zeno++ still involves periodically evaluating the gradient at a validation sample.
The paper makes the reasonable point that it is not reasonable to assume a bounded number of adversaries in the asynchronous setting, and the theorem statements make no assumption about the number of adversaries or rate at which received gradients are from a Byzantine worker. However, there are also no guarantees about whether the algorithm will ever make progress (i.e., will line 8 ever be reached?). This should be stated more transparently in the paper. Also, I was wondering, given that a gradient has been computed on the parameter server's validation set, which is assumed to be "clean", why not take a step using this gradient when the test in line 7 fails?
Finally, the paper titles includes SGD, but the description in Def 1 doesn't appear to involve stochastic gradients. Typical parameter server implementations have workers compute mini-batch stochastic gradients, not full gradients on their shard of the training set. Does Zeno++ need to be modified to run in this setting? Does the theory still hold?
Minor:
- Is there a typo in line 5 of Zeno++? Should this be $\nabla f_s$ instead of $f_s$? Otherwise, what does it mean to take the inner product of $v$ and $g$ in line 7?