Pre-Training and Personalized Fine-Tuning via Over-the-Air Federated Meta-Learning: Convergence-Generalization Trade-Offs
<|reference_start|>Pre-Training and Personalized Fine-Tuning via Over-the-Air Federated Meta-Learning: Convergence-Generalization Trade-Offs: For modern artificial intelligence (AI) applications such as large language models (LLMs), the training paradigm has recently shifted to pre-training followed by fine-tuning. Furthermore, owing to dwindling open repositories of data and thanks to efforts to democratize access to AI models, pre-training is expected to increasingly migrate from the current centralized deployments to federated learning (FL) implementations. Meta-learning provides a general framework in which pre-training and fine-tuning can be formalized. Meta-learning-based personalized FL (meta-pFL) moves beyond basic personalization by targeting generalization to new agents and tasks. This paper studies the generalization performance of meta-pFL for a wireless setting in which the agents participating in the pre-training phase, i.e., meta-learning, are connected via a shared wireless channel to the server. Adopting over-the-air computing, we study the trade-off between generalization to new agents and tasks, on the one hand, and convergence, on the other hand. The trade-off arises from the fact that channel impairments may enhance generalization, while degrading convergence. Extensive numerical results validate the theory.<|reference_end|>
@article{wen2024pre-training,
    title={Pre-Training and Personalized Fine-Tuning via Over-the-Air Federated
  Meta-Learning: Convergence-Generalization Trade-Offs},
    author={Haifeng Wen, Hong Xing, Osvaldo Simeone},
    journal={arXiv preprint arXiv:2406.11569},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11569},
    primaryClass={cs.LG cs.IT eess.SP math.IT}
}