  Decentralized stochastic gradient descent (D-SGD) allows collaborative
learning on massive devices simultaneously without the control of a central
server. However, existing theories claim that decentralization invariably
undermines generalization. In this paper, we challenge the conventional belief
and present a completely new perspective for understanding decentralized
learning. We prove that D-SGD implicitly minimizes the loss function of an
average-direction Sharpness-aware minimization (SAM) algorithm under general
non-convex non-$\beta$-smooth settings. This surprising asymptotic equivalence
reveals an intrinsic regularization-optimization trade-off and three advantages
of decentralization: (1) there exists a free uncertainty evaluation mechanism
in D-SGD to improve posterior estimation; (2) D-SGD exhibits a gradient
smoothing effect; and (3) the sharpness regularization effect of D-SGD does not
decrease as total batch size increases, which justifies the potential
generalization benefit of D-SGD over centralized SGD (C-SGD) in large-batch
scenarios. The code is available at
https://github.com/Raiden-Zhu/ICML-2023-DSGD-and-SAM.

Machine Learning;Computers and Society;Distributed, Parallel, and Cluster Computing;Systems and Control;Systems and Control;Machine Learning