Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point
<|reference_start|>Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point: Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational overhead compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline.<|reference_end|>
@article{wang2024towards,
    title={Towards Federated Learning with On-device Training and Communication in
  8-bit Floating Point},
    author={Bokun Wang and Axel Berg and Durmus Alp Emre Acar and Chuteng Zhou},
    journal={arXiv preprint arXiv:2407.02610},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.02610},
    primaryClass={cs.LG cs.DC}
}