Style over Substance: Failure Modes of LLM Judges in Alignment
  Benchmarking
The release of ChatGPT in November 2022 sparked an explosion of interest in
post-training and an avalanche of new preference optimization (PO) methods.
These methods claim superior alignment by virtue of better correspondence with
human pairwise preferences, often measured by LLM judges. In this work, we
attempt to answer the following question -- do LLM-judge preferences translate
to progress on other, more concrete metrics for alignment, and if not, why not?
We define a concrete metric for alignment, and introduce SOS-Bench, the largest
standardized, reproducible LLM meta-benchmark to date. We find that (1)
LLM-judgments do not correlate with concrete measures of safety, world
knowledge, and instruction following; (2) LLM judges have powerful implicit
biases, prioritizing style over factuality and safety; and (3) the supervised
fine-tuning (SFT) stage of post-training, and not the PO stage, has the
greatest impact on alignment, with data scaling and prompt diversity as the
driving factors. Our codebase and complete results can be found at
https://github.com/penfever/sos-bench.
66f27f4a7967278698d39638
Benjamin Feuer
claimed_verified
2024-09-24T10:20:04.948Z
62f7f4efe7c1c9bf10c81465
https://cdn-avatars.huggingface.co/v1/production/uploads/62f7f4efe7c1c9bf10c81465/AYlOg0fkP1o4GAP-8Y3xt.jpeg
Benjamin Feuer
user
penfever
66f27f4a7967278698d39639
Micah Goldblum
66f27f4a7967278698d3963a
Teresa Datta
66f27f4a7967278698d3963b
Sanjana Nambiar
66f27f4a7967278698d3963c
Raz Besaleli
66f27f4a7967278698d3963d
Samuel Dooley
66f27f4a7967278698d3963e
Max Cembalest
66f27f4a7967278698d3963f
John P. Dickerson
Style over Substance: Failure Modes of LLM Judges in Alignment
  Benchmarking
The release of ChatGPT in November 2022 sparked an explosion of interest in
post-training and an avalanche of new preference optimization (PO) methods.
These methods claim superior alignment by virtue of better correspondence with
human pairwise preferences, often measured by LLM judges. In this work, we
attempt to answer the following question -- do LLM-judge preferences translate
to progress on other, more concrete metrics for alignment, and if not, why not?
We define a concrete metric for alignment, and introduce SOS-Bench, the largest
standardized, reproducible LLM meta-benchmark to date. We find that (1)
LLM-judgments do not correlate with concrete measures of safety, world
knowledge, and instruction following; (2) LLM judges have powerful implicit
biases, prioritizing style over factuality and safety; and (3) the supervised
fine-tuning (SFT) stage of post-training, and not the PO stage, has the
greatest impact on alignment, with data scaling and prompt diversity as the
driving factors. Our codebase and complete results can be found at
https://github.com/penfever/sos-bench.
SOS-Bench evaluates the effectiveness of preference optimization methods in aligning LLMs, finding that LLM-judgment preferences do not correlate well with safety, knowledge, and instruction following, and that supervised fine-tuning has the greatest impact on alignment.
preference optimization LLM judges SOS-Bench LLM meta-benchmark world knowledge instruction following supervised fine-tuning prompt diversity