  This paper focuses on Bayesian inference in a federated learning context
(FL). While several distributed MCMC algorithms have been proposed, few
consider the specific limitations of FL such as communication bottlenecks and
statistical heterogeneity. Recently, Federated Averaging Langevin Dynamics
(FALD) was introduced, which extends the Federated Averaging algorithm to
Bayesian inference. We obtain a novel tight non-asymptotic upper bound on the
Wasserstein distance to the global posterior for FALD. This bound highlights
the effects of statistical heterogeneity, which causes a drift in the local
updates that negatively impacts convergence. We propose a new algorithm
VR-FALD* that uses control variates to correct the client drift. We establish
non-asymptotic bounds showing that VR-FALD* is not affected by statistical
heterogeneity. Finally, we illustrate our results on several FL benchmarks for
Bayesian inference.

Machine Learning;Machine Learning