Social Debiasing for Fair Multi-modal LLMs
<|reference_start|>Social Debiasing for Fair Multi-modal LLMs: Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities. However, these models often inherit severe social biases from their training datasets, leading to unfair predictions based on attributes like race and gender. This paper addresses the issue of social biases in MLLMs by i) Introducing a comprehensive Counterfactual dataset with Multiple Social Concepts (CMSC), which provides a more diverse and extensive training set compared to existing datasets. ii) Proposing an Anti-Stereotype Debiasing strategy (ASD). Our method works by revisiting the MLLM training process, rescaling the autoregressive loss function, and improving data sampling methods to counteract biases. Through extensive experiments on various MLLMs, our CMSC dataset and ASD method demonstrate a significant reduction in social biases while maintaining the models' original performance.<|reference_end|>
@article{cheng2024social,
    title={Social Debiasing for Fair Multi-modal LLMs},
    author={Harry Cheng, Yangyang Guo, Qingpei Guo, Ming Yang, Tian Gan, Liqiang
  Nie},
    journal={arXiv preprint arXiv:2408.06569},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.06569},
    primaryClass={cs.CL cs.AI}
}