BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization
<|reference_start|>BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization: Large Language Models (LLMs) have become pivotal in advancing natural language processing, yet their potential to perpetuate biases poses significant concerns. This paper introduces a new framework employing Direct Preference Optimization (DPO) to mitigate gender, racial, and religious biases in LLM-generated English text. By developing a loss function that favors less biased over biased completions, our approach cultivates a preference for respectful and non-discriminatory language in LLMs. We also contribute a manually designed dataset for training LLMs to recognize and correct biases. This dataset encompasses a diverse range of prompts paired with both biased and unbiased completions. Implementing this approach on the Microsoft Phi-2 model, we demonstrate substantial reductions in biased outputs as our model outperforms the baseline model on almost all bias benchmarks. Our model also achieves better performance compared to other open-source models on most benchmarks. By reducing biases in the language generated by the model, our study marks a significant step towards developing more ethical and socially responsible LLMs. We publicly release BiasDPO dataset on HuggingFace.<|reference_end|>
@article{allam2024biasdpo:,
    title={BiasDPO: Mitigating Bias in Language Models through Direct Preference
  Optimization},
    author={Ahmed Allam},
    journal={arXiv preprint arXiv:2407.13928},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.13928},
    primaryClass={cs.CL}
}