title: Style over Substance: Failure Modes of LLM Judges in Alignment
    Benchmarking
summary: The release of ChatGPT in November 2022 sparked an explosion of interest in
  post-training and an avalanche of new preference optimization (PO) methods.
  These methods claim superior alignment by virtue of better correspondence with
  human pairwise preferences, often measured by LLM judges. In this work, we
  attempt to answer the following question -- do LLM-judge preferences translate
  to progress on other, more concrete metrics for alignment, and if not, why not?
  We define a concrete metric for alignment, and introduce SOS-Bench, the largest
  standardized, reproducible LLM meta-benchmark to date. We find that (1)
  LLM-judgments do not correlate with concrete measures of safety, world
  knowledge, and instruction following; (2) LLM judges have powerful implicit
  biases, prioritizing style over factuality and safety; and (3) the supervised
  fine-tuning (SFT) stage of post-training, and not the PO stage, has the
  greatest impact on alignment, with data scaling and prompt diversity as the
  driving factors. Our codebase and complete results can be found at
  https://github.com/penfever/sos-bench.
paper_authors._id: 66f27f4a7967278698d39638
paper_authors.name: Benjamin Feuer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T10:20:04.948Z
paper_authors._id: 62f7f4efe7c1c9bf10c81465
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f7f4efe7c1c9bf10c81465/AYlOg0fkP1o4GAP-8Y3xt.jpeg
paper_authors.fullname: Benjamin Feuer
paper_authors.type: user
paper_authors.user: penfever
paper_authors._id: 66f27f4a7967278698d39639
paper_authors.name: Micah Goldblum
paper_authors._id: 66f27f4a7967278698d3963a
paper_authors.name: Teresa Datta
paper_authors._id: 66f27f4a7967278698d3963b
paper_authors.name: Sanjana Nambiar
paper_authors._id: 66f27f4a7967278698d3963c
paper_authors.name: Raz Besaleli
paper_authors._id: 66f27f4a7967278698d3963d
paper_authors.name: Samuel Dooley
paper_authors._id: 66f27f4a7967278698d3963e
paper_authors.name: Max Cembalest
paper_authors._id: 66f27f4a7967278698d3963f
paper_authors.name: John P. Dickerson
paper_title: Style over Substance: Failure Modes of LLM Judges in Alignment
    Benchmarking
paper_summary: The release of ChatGPT in November 2022 sparked an explosion of interest in
  post-training and an avalanche of new preference optimization (PO) methods.
  These methods claim superior alignment by virtue of better correspondence with
  human pairwise preferences, often measured by LLM judges. In this work, we
  attempt to answer the following question -- do LLM-judge preferences translate
  to progress on other, more concrete metrics for alignment, and if not, why not?
  We define a concrete metric for alignment, and introduce SOS-Bench, the largest
  standardized, reproducible LLM meta-benchmark to date. We find that (1)
  LLM-judgments do not correlate with concrete measures of safety, world
  knowledge, and instruction following; (2) LLM judges have powerful implicit
  biases, prioritizing style over factuality and safety; and (3) the supervised
  fine-tuning (SFT) stage of post-training, and not the PO stage, has the
  greatest impact on alignment, with data scaling and prompt diversity as the
  driving factors. Our codebase and complete results can be found at
  https://github.com/penfever/sos-bench.
paper_ai_keywords: post-training preference optimization (PO) human pairwise preferences LLM judges concrete metric for alignment SOS-Bench LLM meta-benchmark supervised fine-tuning (SFT) data scaling prompt diversity