title: B-score: Detecting biases in large language models using response
    history
summary: Large language models (LLMs) often exhibit strong biases, e.g, against women
  or in favor of the number 7. We investigate whether LLMs would be able to
  output less biased answers when allowed to observe their prior answers to the
  same question in a multi-turn conversation. To understand which types of
  questions invite more biased answers, we test LLMs on our proposed set of
  questions that span 9 topics and belong to three types: (1) Subjective; (2)
  Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves
  in a multi-turn conversation in response to questions that seek an Random,
  unbiased answer. Furthermore, we propose B-score, a novel metric that is
  effective in detecting biases to Subjective, Random, Easy, and Hard questions.
  On MMLU, HLE, and CSQA, leveraging B-score substantially improves the
  verification accuracy of LLM answers (i.e, accepting LLM correct answers and
  rejecting incorrect ones) compared to using verbalized confidence scores or the
  frequency of single-turn answers alone. Code and data are available at:
  https://b-score.github.io.
paper_authors._id: 6835217ee759f596d018f72c
paper_authors.name: An Vo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:52:08.994Z
paper_authors._id: 6631fd5961a4305e5610d403
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6631fd5961a4305e5610d403/P1Dtxzn-KIbYDDsiw60nr.jpeg
paper_authors.fullname: An Vo
paper_authors.type: user
paper_authors.user: anvo25
paper_authors._id: 6835217ee759f596d018f72d
paper_authors.name: Mohammad Reza Taesiri
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-27T02:21:46.082Z
paper_authors._id: 6039478ab3ecf716b1a5fd4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg
paper_authors.fullname: taesiri
paper_authors.type: user
paper_authors.user: taesiri
paper_authors._id: 6835217ee759f596d018f72e
paper_authors.name: Daeyoung Kim
paper_authors._id: 6835217ee759f596d018f72f
paper_authors.name: Anh Totti Nguyen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-27T02:20:46.958Z
paper_authors._id: 60e85b3fcd1cf4e418fff651
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e85b3fcd1cf4e418fff651/TUaEA4SqB6xuG2JhwNETC.jpeg
paper_authors.fullname: Anh (Totti) Nguyen
paper_authors.type: user
paper_authors.user: anhng8
paper_title: B-score: Detecting biases in large language models using response
    history
paper_summary: Large language models (LLMs) often exhibit strong biases, e.g, against women
  or in favor of the number 7. We investigate whether LLMs would be able to
  output less biased answers when allowed to observe their prior answers to the
  same question in a multi-turn conversation. To understand which types of
  questions invite more biased answers, we test LLMs on our proposed set of
  questions that span 9 topics and belong to three types: (1) Subjective; (2)
  Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves
  in a multi-turn conversation in response to questions that seek an Random,
  unbiased answer. Furthermore, we propose B-score, a novel metric that is
  effective in detecting biases to Subjective, Random, Easy, and Hard questions.
  On MMLU, HLE, and CSQA, leveraging B-score substantially improves the
  verification accuracy of LLM answers (i.e, accepting LLM correct answers and
  rejecting incorrect ones) compared to using verbalized confidence scores or the
  frequency of single-turn answers alone. Code and data are available at:
  https://b-score.github.io.
paper_ai_summary: LLMs can reduce biases in multi-turn conversations for certain types of questions, and a novel B-score metric improves the accuracy of verifying LLM answers.
paper_ai_keywords: large language models biases multi-turn conversation B-score MMLU HLE CSQA verification accuracy verbalized confidence scores