Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees
<|reference_start|>Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees: In this work, we propose a federated dynamical low-rank training (FeDLRT) scheme to reduce client compute and communication costs - two significant performance bottlenecks in horizontal federated learning. Our method builds upon dynamical low-rank splitting schemes for manifold-constrained optimization to create a global low-rank basis of network weights, which enables client training on a small coefficient matrix. A consistent global low-rank basis allows us to incorporate a variance correction scheme and prove global loss descent and convergence to a stationary point. Dynamic augmentation and truncation of the low-rank bases automatically optimizes computing and communication resource utilization. We demonstrate the efficiency of FeDLRT in an array of computer vision benchmarks and show a reduction of client compute and communication costs by up to an order of magnitude with minimal impacts on global accuracy.<|reference_end|>
@article{schotthöfer2024federated,
    title={Federated Dynamical Low-Rank Training with Global Loss Convergence
  Guarantees},
    author={Steffen Schotth"ofer and M. Paul Laiu},
    journal={arXiv preprint arXiv:2406.17887},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17887},
    primaryClass={cs.LG cs.AI math.OC}
}