Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency
<|reference_start|>Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency: Fine-tuning pre-trained models is a widely employed technique in numerous real-world applications. However, fine-tuning these models on new tasks can lead to unfair outcomes. This is due to the absence of generalization guarantees for fairness properties, regardless of whether the original pre-trained model was developed with fairness considerations. To tackle this issue, we introduce an efficient and robust fine-tuning framework specifically designed to mitigate biases in new tasks. Our empirical analysis shows that the parameters in the pre-trained model that affect predictions for different demographic groups are different, so based on this observation, we employ a transfer learning strategy that neutralizes the importance of these influential weights, determined using Fisher information across demographic groups. Additionally, we integrate this weight importance neutralization strategy with a matrix factorization technique, which provides a low-rank approximation of the weight matrix using fewer parameters, reducing the computational demands. Experiments on multiple pre-trained models and new tasks demonstrate the effectiveness of our method.<|reference_end|>
@article{zhang2024bias,
    title={Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness
  and Efficiency},
    author={Yixuan Zhang and Feng Zhou},
    journal={arXiv preprint arXiv:2403.00625},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00625},
    primaryClass={cs.LG cs.CY}
}