title: reWordBench: Benchmarking and Improving the Robustness of Reward Models
    with Transformed Inputs
summary: Reward models have become a staple in modern NLP, serving as not only a
  scalable text evaluator, but also an indispensable component in many alignment
  recipes and inference-time algorithms. However, while recent reward models
  increase performance on standard benchmarks, this may partly be due to
  overfitting effects, which would confound an understanding of their true
  capability. In this work, we scrutinize the robustness of reward models and the
  extent of such overfitting. We build **reWordBench**, which systematically
  transforms reward model inputs in meaning- or ranking-preserving ways. We show
  that state-of-the-art reward models suffer from substantial performance
  degradation even with minor input transformations, sometimes dropping to
  significantly below-random accuracy, suggesting brittleness. To improve reward
  model robustness, we propose to explicitly train them to assign similar scores
  to paraphrases, and find that this approach also improves robustness to other
  distinct kinds of transformations. For example, our robust reward model reduces
  such degradation by roughly half for the Chat Hard subset in RewardBench.
  Furthermore, when used in alignment, our robust reward models demonstrate
  better utility and lead to higher-quality outputs, winning in up to 59% of
  instances against a standardly trained RM.
paper_authors._id: 67d8e861fa59a8b15a921052
paper_authors.name: Zhaofeng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:27:29.372Z
paper_authors._id: 6351712b40dffad651f128c7
paper_authors.avatarUrl: /avatars/87708c86c1baef548ef556f5d32dca71.svg
paper_authors.fullname: Zhaofeng Wu
paper_authors.type: user
paper_authors.user: ZhaofengWu
paper_authors._id: 67d8e861fa59a8b15a921053
paper_authors.name: Michihiro Yasunaga
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:27:35.318Z
paper_authors._id: 621e9388345a1d9ab65391c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e9388345a1d9ab65391c3/RxurNzyAWJOUdgeSHQi1R.jpeg
paper_authors.fullname: Michihiro Yasunaga
paper_authors.type: user
paper_authors.user: michiyasunaga
paper_authors._id: 67d8e861fa59a8b15a921054
paper_authors.name: Andrew Cohen
paper_authors._id: 67d8e861fa59a8b15a921055
paper_authors.name: Yoon Kim
paper_authors._id: 67d8e861fa59a8b15a921056
paper_authors.name: Asli Celikyilmaz
paper_authors._id: 67d8e861fa59a8b15a921057
paper_authors.name: Marjan Ghazvininejad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:28:15.386Z
paper_authors._id: 660f0fd377a1e2509aa5a679
paper_authors.avatarUrl: /avatars/e04ef05bed0bf6cefdc7e3e39674e2f9.svg
paper_authors.fullname: Marjan Ghazvininejad
paper_authors.type: user
paper_authors.user: mghazvininejad
paper_title: reWordBench: Benchmarking and Improving the Robustness of Reward Models
    with Transformed Inputs
paper_summary: Reward models have become a staple in modern NLP, serving as not only a
  scalable text evaluator, but also an indispensable component in many alignment
  recipes and inference-time algorithms. However, while recent reward models
  increase performance on standard benchmarks, this may partly be due to
  overfitting effects, which would confound an understanding of their true
  capability. In this work, we scrutinize the robustness of reward models and the
  extent of such overfitting. We build **reWordBench**, which systematically
  transforms reward model inputs in meaning- or ranking-preserving ways. We show
  that state-of-the-art reward models suffer from substantial performance
  degradation even with minor input transformations, sometimes dropping to
  significantly below-random accuracy, suggesting brittleness. To improve reward
  model robustness, we propose to explicitly train them to assign similar scores
  to paraphrases, and find that this approach also improves robustness to other
  distinct kinds of transformations. For example, our robust reward model reduces
  such degradation by roughly half for the Chat Hard subset in RewardBench.
  Furthermore, when used in alignment, our robust reward models demonstrate
  better utility and lead to higher-quality outputs, winning in up to 59% of
  instances against a standardly trained RM.
paper_ai_keywords: reward models NLP text evaluator alignment inference-time algorithms overfitting reWordBench meaning-preserving ranking-preserving state-of-the-art performance degradation below-random accuracy brittleness paraphrases robust reward model Chat Hard subset RewardBench utility higher-quality outputs