FedPFT: Federated Proxy Fine-Tuning of Foundation Models
<|reference_start|>FedPFT: Federated Proxy Fine-Tuning of Foundation Models: Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, however, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. In this paper, we propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules. First, the sub-FM construction module employs a layer-wise compression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing those crucial neurons. Second, the sub-FM alignment module conducts a two-step distillations-layer-level and neuron-level-before and during FL fine-tuning respectively, to reduce error of gradient by accurately aligning sub-FM with FM under theoretical guarantees. Experimental results on seven commonly used datasets (i.e., four text and three vision) demonstrate the superiority of FedPFT.<|reference_end|>
@article{peng2024fedpft:,
    title={FedPFT: Federated Proxy Fine-Tuning of Foundation Models},
    author={Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan,
  Chenglu Wen, Ruisheng Zhang, Cheng Wang},
    journal={arXiv preprint arXiv:2404.11536},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.11536},
    primaryClass={cs.LG cs.AI}
}