Think Again! The Effect of Test-Time Compute on Preferences, Opinions,
  and Beliefs of Large Language Models
As Large Language Models (LLMs) become deeply integrated into human life and
increasingly influence decision-making, it's crucial to evaluate whether and to
what extent they exhibit subjective preferences, opinions, and beliefs. These
tendencies may stem from biases within the models, which may shape their
behavior, influence the advice and recommendations they offer to users, and
potentially reinforce certain viewpoints. This paper presents the Preference,
Opinion, and Belief survey (POBs), a benchmark developed to assess LLMs'
subjective inclinations across societal, cultural, ethical, and personal
domains. We applied our benchmark to evaluate leading open- and closed-source
LLMs, measuring desired properties such as reliability, neutrality, and
consistency. In addition, we investigated the effect of increasing the
test-time compute, through reasoning and self-reflection mechanisms, on those
metrics. While effective in other tasks, our results show that these mechanisms
offer only limited gains in our domain. Furthermore, we reveal that newer model
versions are becoming less consistent and more biased toward specific
viewpoints, highlighting a blind spot and a concerning trend. POBS:
https://ibm.github.io/POBS
683ea7297e58553a7f73c210
George Kour
claimed_verified
2025-06-17T07:24:38.913Z
64540de813dba495b2e0354e
https://cdn-avatars.huggingface.co/v1/production/uploads/64540de813dba495b2e0354e/w87XdCBRpNw6Kd2xn4AkY.png
George Kour
user
gkour
683ea7297e58553a7f73c211
Itay Nakash
admin_assigned
2025-06-03T12:44:49.716Z
671f8106d677d3a764a6f9a5
/avatars/90b4b00058aac30c060c5eac8debb1c7.svg
itay nakash
user
itaynakash
683ea7297e58553a7f73c212
Ateret Anaby-Tavor
683ea7297e58553a7f73c213
Michal Shmueli-Scheuer
Think Again! The Effect of Test-Time Compute on Preferences, Opinions,
  and Beliefs of Large Language Models
As Large Language Models (LLMs) become deeply integrated into human life and
increasingly influence decision-making, it's crucial to evaluate whether and to
what extent they exhibit subjective preferences, opinions, and beliefs. These
tendencies may stem from biases within the models, which may shape their
behavior, influence the advice and recommendations they offer to users, and
potentially reinforce certain viewpoints. This paper presents the Preference,
Opinion, and Belief survey (POBs), a benchmark developed to assess LLMs'
subjective inclinations across societal, cultural, ethical, and personal
domains. We applied our benchmark to evaluate leading open- and closed-source
LLMs, measuring desired properties such as reliability, neutrality, and
consistency. In addition, we investigated the effect of increasing the
test-time compute, through reasoning and self-reflection mechanisms, on those
metrics. While effective in other tasks, our results show that these mechanisms
offer only limited gains in our domain. Furthermore, we reveal that newer model
versions are becoming less consistent and more biased toward specific
viewpoints, highlighting a blind spot and a concerning trend. POBS:
https://ibm.github.io/POBS
The Preference, Opinion, and Belief survey assesses the subjective tendencies and biases of Large Language Models across various domains and highlights a trend of increased bias in newer model versions.
Large Language Models Preference Opinion and Belief survey reliability neutrality consistency reasoning mechanisms self-reflection mechanisms