Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement
<|reference_start|>Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement: Recent studies show that large language models (LLMs) improve their performance through self-feedback on certain tasks while degrade on others. We discovered that such a contrary is due to LLM's bias in evaluating their own output. In this paper, we formally define LLM's self-bias - the tendency to favor its own generation - using two statistics. We analyze six LLMs (GPT-4, GPT-3.5, Gemini, LLaMA2, Mixtral and DeepSeek) on translation, constrained text generation, and mathematical reasoning tasks. We find that self-bias is prevalent in all examined LLMs across multiple languages and tasks. Our analysis reveals that while the self-refine pipeline improves the fluency and understandability of model outputs, it further amplifies self-bias. To mitigate such biases, we discover that larger model size and external feedback with accurate assessment can significantly reduce bias in the self-refine pipeline, leading to actual performance improvement in downstream tasks. The code and data are released at https://github.com/xu1998hz/llm_self_bias.<|reference_end|>
@article{xu2024pride,
    title={Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement},
    author={Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, William
  Yang Wang},
    journal={arXiv preprint arXiv:2402.11436},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11436},
    primaryClass={cs.CL cs.AI}
}