title: Think Again! The Effect of Test-Time Compute on Preferences, Opinions,
    and Beliefs of Large Language Models
summary: As Large Language Models (LLMs) become deeply integrated into human life and
  increasingly influence decision-making, it's crucial to evaluate whether and to
  what extent they exhibit subjective preferences, opinions, and beliefs. These
  tendencies may stem from biases within the models, which may shape their
  behavior, influence the advice and recommendations they offer to users, and
  potentially reinforce certain viewpoints. This paper presents the Preference,
  Opinion, and Belief survey (POBs), a benchmark developed to assess LLMs'
  subjective inclinations across societal, cultural, ethical, and personal
  domains. We applied our benchmark to evaluate leading open- and closed-source
  LLMs, measuring desired properties such as reliability, neutrality, and
  consistency. In addition, we investigated the effect of increasing the
  test-time compute, through reasoning and self-reflection mechanisms, on those
  metrics. While effective in other tasks, our results show that these mechanisms
  offer only limited gains in our domain. Furthermore, we reveal that newer model
  versions are becoming less consistent and more biased toward specific
  viewpoints, highlighting a blind spot and a concerning trend. POBS:
  https://ibm.github.io/POBS
paper_authors._id: 683ea7297e58553a7f73c210
paper_authors.name: George Kour
paper_authors._id: 683ea7297e58553a7f73c211
paper_authors.name: Itay Nakash
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:44:49.716Z
paper_authors._id: 671f8106d677d3a764a6f9a5
paper_authors.avatarUrl: /avatars/90b4b00058aac30c060c5eac8debb1c7.svg
paper_authors.fullname: itay nakash
paper_authors.type: user
paper_authors.user: itaynakash
paper_authors._id: 683ea7297e58553a7f73c212
paper_authors.name: Ateret Anaby-Tavor
paper_authors._id: 683ea7297e58553a7f73c213
paper_authors.name: Michal Shmueli-Scheuer
paper_title: Think Again! The Effect of Test-Time Compute on Preferences, Opinions,
    and Beliefs of Large Language Models
paper_summary: As Large Language Models (LLMs) become deeply integrated into human life and
  increasingly influence decision-making, it's crucial to evaluate whether and to
  what extent they exhibit subjective preferences, opinions, and beliefs. These
  tendencies may stem from biases within the models, which may shape their
  behavior, influence the advice and recommendations they offer to users, and
  potentially reinforce certain viewpoints. This paper presents the Preference,
  Opinion, and Belief survey (POBs), a benchmark developed to assess LLMs'
  subjective inclinations across societal, cultural, ethical, and personal
  domains. We applied our benchmark to evaluate leading open- and closed-source
  LLMs, measuring desired properties such as reliability, neutrality, and
  consistency. In addition, we investigated the effect of increasing the
  test-time compute, through reasoning and self-reflection mechanisms, on those
  metrics. While effective in other tasks, our results show that these mechanisms
  offer only limited gains in our domain. Furthermore, we reveal that newer model
  versions are becoming less consistent and more biased toward specific
  viewpoints, highlighting a blind spot and a concerning trend. POBS:
  https://ibm.github.io/POBS
paper_ai_summary: The Preference, Opinion, and Belief survey assesses the subjective tendencies and biases of Large Language Models across various domains and highlights a trend of increased bias in newer model versions.
paper_ai_keywords: Large Language Models Preference Opinion and Belief survey reliability neutrality consistency reasoning mechanisms self-reflection mechanisms