title: BLEUBERI: BLEU is a surprisingly effective reward for instruction
    following
summary: Reward models are central to aligning LLMs with human preferences, but they
  are costly to train, requiring large-scale human-labeled preference data and
  powerful pretrained LLM backbones. Meanwhile, the increasing availability of
  high-quality synthetic instruction-following datasets raises the question: can
  simpler, reference-based metrics serve as viable alternatives to reward models
  during RL-based alignment? In this paper, we show first that BLEU, a basic
  string-matching metric, surprisingly matches strong reward models in agreement
  with human preferences on general instruction-following datasets. Based on this
  insight, we develop BLEUBERI, a method that first identifies challenging
  instructions and then applies Group Relative Policy Optimization (GRPO) using
  BLEU directly as the reward function. We demonstrate that BLEUBERI-trained
  models are competitive with models trained via reward model-guided RL across
  four challenging instruction-following benchmarks and three different base
  language models. A human evaluation further supports that the quality of
  BLEUBERI model outputs is on par with those from reward model-aligned models.
  Moreover, BLEUBERI models generate outputs that are more factually grounded
  than competing methods. Overall, we show that given access to high-quality
  reference outputs (easily obtained via existing instruction-following datasets
  or synthetic data generation), string matching-based metrics are cheap yet
  effective proxies for reward models during alignment. We release our code and
  data at https://github.com/lilakk/BLEUBERI.
paper_authors._id: 682f9243ea94553305f6e1df
paper_authors.name: Yapei Chang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:17:04.366Z
paper_authors._id: 5f1dcc06cb8f993fa01f4775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668660782327-5f1dcc06cb8f993fa01f4775.png
paper_authors.fullname: Yapei Chang
paper_authors.type: user
paper_authors.user: yapeichang
paper_authors._id: 682f9243ea94553305f6e1e0
paper_authors.name: Yekyung Kim
paper_authors._id: 682f9243ea94553305f6e1e1
paper_authors.name: Michael Krumdick
paper_authors._id: 682f9243ea94553305f6e1e2
paper_authors.name: Amir Zadeh
paper_authors._id: 682f9243ea94553305f6e1e3
paper_authors.name: Chuan Li
paper_authors._id: 682f9243ea94553305f6e1e4
paper_authors.name: Chris Tanner
paper_authors._id: 682f9243ea94553305f6e1e5
paper_authors.name: Mohit Iyyer
paper_title: BLEUBERI: BLEU is a surprisingly effective reward for instruction
    following
paper_summary: Reward models are central to aligning LLMs with human preferences, but they
  are costly to train, requiring large-scale human-labeled preference data and
  powerful pretrained LLM backbones. Meanwhile, the increasing availability of
  high-quality synthetic instruction-following datasets raises the question: can
  simpler, reference-based metrics serve as viable alternatives to reward models
  during RL-based alignment? In this paper, we show first that BLEU, a basic
  string-matching metric, surprisingly matches strong reward models in agreement
  with human preferences on general instruction-following datasets. Based on this
  insight, we develop BLEUBERI, a method that first identifies challenging
  instructions and then applies Group Relative Policy Optimization (GRPO) using
  BLEU directly as the reward function. We demonstrate that BLEUBERI-trained
  models are competitive with models trained via reward model-guided RL across
  four challenging instruction-following benchmarks and three different base
  language models. A human evaluation further supports that the quality of
  BLEUBERI model outputs is on par with those from reward model-aligned models.
  Moreover, BLEUBERI models generate outputs that are more factually grounded
  than competing methods. Overall, we show that given access to high-quality
  reference outputs (easily obtained via existing instruction-following datasets
  or synthetic data generation), string matching-based metrics are cheap yet
  effective proxies for reward models during alignment. We release our code and
  data at https://github.com/lilakk/BLEUBERI.
paper_ai_summary: BLEUBERI uses BLEU as a reward function to optimize instruction-following in Language Models, achieving quality comparable to models aligned with full reward models.
paper_ai_keywords: BLEU Group Relative Policy Optimization (GRPO) reward models instruction-following benchmarks reference-based metrics