Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
<|reference_start|>Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning: Virtually all federated learning (FL) methods, including FedAvg, operate in the following manner: i) an orchestrating server sends the current model parameters to a cohort of clients selected via certain rule, ii) these clients then independently perform a local training procedure (e.g., via SGD or Adam) using their own training data, and iii) the resulting models are shipped to the server for aggregation. This process is repeated until a model of suitable quality is found. A notable feature of these methods is that each cohort is involved in a single communication round with the server only. In this work we challenge this algorithmic design primitive and investigate whether it is possible to ``squeeze more juice" out of each cohort than what is possible in a single communication round. Surprisingly, we find that this is indeed the case, and our approach leads to up to 74% reduction in the total communication cost needed to train a FL model in the cross-device setting. Our method is based on a novel variant of the stochastic proximal point method (SPPM-AS) which supports a large collection of client sampling procedures some of which lead to further gains when compared to classical client selection approaches.<|reference_end|>
@article{yi2024cohort,
    title={Cohort Squeeze: Beyond a Single Communication Round per Cohort in
  Cross-Device Federated Learning},
    author={Kai Yi, Timur Kharisov, Igor Sokolov, Peter Richt'arik},
    journal={arXiv preprint arXiv:2406.01115},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01115},
    primaryClass={cs.LG}
}