Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization
<|reference_start|>Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization: Federated Learning (FL) faces significant challenges related to communication efficiency and heterogeneity. To address these issues, we explore the potential of using low-rank updates. Our theoretical analysis reveals that client's loss exhibits a higher rank structure (gradients span higher rank subspace of Hessian) compared to the server's loss. Based on this insight, we hypothesize that constraining client-side optimization to a low-rank subspace could provide an implicit regularization effect. Consequently, we propose FedLoRU, a general low-rank update framework for federated learning. Our framework enforces low-rank client-side updates and accumulates these updates to form a higher-rank model. Additionally, variants of FedLoRU can adapt to environments with statistical and model heterogeneity by employing multiple or hierarchical low-rank updates. Experimental results demonstrate that FedLoRU performs comparably to full-rank algorithms and exhibits robustness to heterogeneous and large numbers of clients.<|reference_end|>
@article{park2024communication-efficient,
    title={Communication-Efficient Federated Low-Rank Update Algorithm and its
  Connection to Implicit Regularization},
    author={Haemin Park, Diego Klabjan},
    journal={arXiv preprint arXiv:2409.12371},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.12371},
    primaryClass={cs.LG cs.AI stat.ML}
}