AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language
  Models
Recent benchmarks have probed factual consistency and rhetorical robustness
in Large Language Models (LLMs). However, a knowledge gap exists regarding how
directional framing of factually true statements influences model agreement, a
common scenario for LLM users. AssertBench addresses this by sampling
evidence-supported facts from FEVEROUS, a fact verification dataset. For each
(evidence-backed) fact, we construct two framing prompts: one where the user
claims the statement is factually correct, and another where the user claims it
is incorrect. We then record the model's agreement and reasoning. The desired
outcome is that the model asserts itself, maintaining consistent truth
evaluation across both framings, rather than switching its evaluation to agree
with the user. AssertBench isolates framing-induced variability from the
model's underlying factual knowledge by stratifying results based on the
model's accuracy on the same claims when presented neutrally. In doing so, this
benchmark aims to measure an LLM's ability to "stick to its guns" when
presented with contradictory user assertions about the same fact. The complete
source code is available at https://github.com/achowd32/assert-bench.
685197990164cd1316710352
Jaeho Lee
claimed_verified
2025-06-17T21:13:00.114Z
663826366169f2a25f10fada
https://cdn-avatars.huggingface.co/v1/production/uploads/663826366169f2a25f10fada/3wdDH7hI7Zg55f6dCem-5.jpeg
Jaeho Lee
user
jlee24
685197990164cd1316710353
Atharv Chowdhary
claimed_verified
2025-06-17T21:13:03.070Z
67196f2652c32a1397b6674b
/avatars/0115bee40d5fc80a1413ec9ae08f35f6.svg
Atharv Chowdhary
user
achowd32
AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language
  Models
Recent benchmarks have probed factual consistency and rhetorical robustness
in Large Language Models (LLMs). However, a knowledge gap exists regarding how
directional framing of factually true statements influences model agreement, a
common scenario for LLM users. AssertBench addresses this by sampling
evidence-supported facts from FEVEROUS, a fact verification dataset. For each
(evidence-backed) fact, we construct two framing prompts: one where the user
claims the statement is factually correct, and another where the user claims it
is incorrect. We then record the model's agreement and reasoning. The desired
outcome is that the model asserts itself, maintaining consistent truth
evaluation across both framings, rather than switching its evaluation to agree
with the user. AssertBench isolates framing-induced variability from the
model's underlying factual knowledge by stratifying results based on the
model's accuracy on the same claims when presented neutrally. In doing so, this
benchmark aims to measure an LLM's ability to "stick to its guns" when
presented with contradictory user assertions about the same fact. The complete
source code is available at https://github.com/achowd32/assert-bench.
AssertBench evaluates Large Language Models' ability to maintain consistent truth evaluation when faced with contradictory user assertions about factually true statements by analyzing framing-induced variability.
Large Language Models LLMS fact verification FEVEROUS framing model agreement reasoning neutrality truth evaluation contradictory assertions