Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm
<|reference_start|>Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm: Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models. Our method, coined Fed-Sophia, combines a weighted moving average of the gradient with a clipping operation to find the descent direction. In addition to that, a lightweight estimation of the Hessian's diagonal is used to incorporate the curvature information. Numerical evaluation shows the superiority, robustness, and scalability of the proposed Fed-Sophia scheme compared to first and second-order baselines.<|reference_end|>
@article{elbakary2024fed-sophia:,
    title={Fed-Sophia: A Communication-Efficient Second-Order Federated Learning
  Algorithm},
    author={Ahmed Elbakary, Chaouki Ben Issaid, Mohammad Shehab, Karim Seddik,
  Tamer ElBatt, Mehdi Bennis},
    journal={arXiv preprint arXiv:2406.06655},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06655},
    primaryClass={cs.LG cs.AI cs.DC}
}