Machine Learning (ML) solutions are nowadays distributed and are prone to various types of component failures, which can be encompassed in so-called Byzantine behavior. This paper introduces LIUBEI, a Byzantine-resilient ML algorithm that does not trust any individual component in the network (neither workers nor servers), nor does it induce additional communication rounds (on average), compared to standard non-Byzantine resilient algorithms. LIUBEI builds upon gradient aggregation rules (GARs) to tolerate a minority of Byzantine workers. Besides, LIUBEI replicates the parameter server on multiple machines instead of trusting it. We introduce a novel filtering mechanism that enables workers to filter out replies from Byzantine server replicas without requiring communication with all servers. Such a filtering mechanism is based on network synchrony, Lipschitz continuity of the loss function, and the GAR used to aggregate workers' gradients. We also introduce a protocol, scatter/gather, to bound drifts between models on correct servers with a small number of communication messages. We theoretically prove that LIUBEI achieves Byzantine resilience to both servers and workers and guarantees convergence. We build LIUBEI using TensorFlow, and we show that LIUBEI tolerates Byzantine behavior with an accuracy loss of around 5% and around 24% convergence overhead compared to vanilla TensorFlow. We moreover show that the throughput gain of LIUBEI compared to another state-of-the-art Byzantine-resilient ML algorithm (that assumes network asynchrony) is 70%.
The paper considers distributed stochastic gradient descent, where some (unknown) compute nodes may be unreliable. New heuristics for filtering out replies from unreliable servers are introduced alongside a new protocol that helps keeping nodes in sync.
In general, I miss a more clear indication of how the individual contributions are different from other methods. I am also missing more detailed ablation studies showing which of the new ideas contribute the most to efficient learning. As far as I can tell, the experiments do not really show an improvement over existing methods in this domain.
This is not my area of expertise, but I cannot recommend the paper for publication in its current form as
(a) it's not clear to me that the paper improves on existing methods, and
(b) it's not clear to me what the real novelty of the work is.
Post-rebuttal:
I acknowledge the response of the authors. They clarified some aspects for me, and the paper appears to have improved over the rebuttal period.
I did not change my rating, but I want to emphasize that this is only because my knowledge of this field is so limited. My rating is largely based on "gut feeling" rather than actual knowledge, and I won't argue against acceptance.