HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning
<|reference_start|>HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning: Data heterogeneity among Federated Learning (FL) users poses a significant challenge, resulting in reduced global model performance. The community has designed various techniques to tackle this issue, among which Knowledge Distillation (KD)-based techniques are common.   While these techniques effectively improve performance under high heterogeneity, they inadvertently cause higher accuracy degradation under model poisoning attacks (known as attack amplification). This paper presents a case study to reveal this critical vulnerability in KD-based FL systems. We show why KD causes this issue through empirical evidence and use it as motivation to design a hybrid distillation technique. We introduce a novel algorithm, Hybrid Knowledge Distillation for Robust and Accurate FL (HYDRA-FL), which reduces the impact of attacks in attack scenarios by offloading some of the KD loss to a shallow layer via an auxiliary classifier. We model HYDRA-FL as a generic framework and adapt it to two KD-based FL algorithms, FedNTD and MOON. Using these two as case studies, we demonstrate that our technique outperforms baselines in attack settings while maintaining comparable performance in benign settings.<|reference_end|>
@article{khan2024hydra-fl:,
    title={HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate
  Federated Learning},
    author={Momin Ahmad Khan, Yasra Chandio, Fatima Muhammad Anwar},
    journal={arXiv preprint arXiv:2409.19912},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.19912},
    primaryClass={cs.LG cs.CR}
}