title: M-RewardBench: Evaluating Reward Models in Multilingual Settings
summary: Reward models (RMs) have driven the state-of-the-art performance of LLMs
  today by enabling the integration of human feedback into the language modeling
  process. However, RMs are primarily trained and evaluated in English, and their
  capabilities in multilingual settings remain largely understudied. In this
  work, we conduct a systematic evaluation of several reward models in
  multilingual settings. We first construct the first-of-its-kind multilingual RM
  evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances
  for 23 typologically diverse languages, that tests the chat, safety, reasoning,
  and translation capabilities of RMs. We then rigorously evaluate a wide range
  of reward models on M-RewardBench, offering fresh insights into their
  performance across diverse languages. We identify a significant gap in RMs'
  performances between English and non-English languages and show that RM
  preferences can change substantially from one language to another. We also
  present several findings on how different multilingual aspects impact RM
  performance. Specifically, we show that the performance of RMs is improved with
  improved translation quality. Similarly, we demonstrate that the models exhibit
  better performance for high-resource languages. We release M-RewardBench
  dataset and the codebase in this study to facilitate a better understanding of
  RM evaluation in multilingual settings.
paper_authors._id: 671917223aa9749caf1004d1
paper_authors.name: Srishti Gureja
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-23T16:35:20.509Z
paper_authors._id: 625d6ed9c4086370158824eb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625d6ed9c4086370158824eb/KG_Plv-2rcydKzVf-9AAj.jpeg
paper_authors.fullname: Srishti Gureja
paper_authors.type: user
paper_authors.user: srishti-hf1110
paper_authors._id: 671917223aa9749caf1004d2
paper_authors.name: Lester James V. Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:03.286Z
paper_authors._id: 634e20a0c1ce28f1de920cc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
paper_authors.fullname: Lj V. Miranda
paper_authors.type: user
paper_authors.user: ljvmiranda921
paper_authors._id: 671917223aa9749caf1004d3
paper_authors.name: Shayekh Bin Islam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:41.072Z
paper_authors._id: 5e4b943a37cb5b49818287b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e4b943a37cb5b49818287b5/Sd8QOGFY-_B1z4UlFTuPf.jpeg
paper_authors.fullname: Shayekh Islam
paper_authors.type: user
paper_authors.user: shayekh
paper_authors._id: 671917223aa9749caf1004d4
paper_authors.name: Rishabh Maheshwary
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:24:26.868Z
paper_authors._id: 645c26d423ed9b7788d5e24b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/cZMUluWpYUlSLcn6yoC7c.jpeg
paper_authors.fullname: Rishabh Maheshwary
paper_authors.type: user
paper_authors.user: rmahesh
paper_authors._id: 671917223aa9749caf1004d5
paper_authors.name: Drishti Sharma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:24:20.823Z
paper_authors._id: 618c1ad1c74578e0a4a4d074
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618c1ad1c74578e0a4a4d074/8u_AkeHt4d6xtQ8hzaffU.jpeg
paper_authors.fullname: Drishti Sharma
paper_authors.type: user
paper_authors.user: DrishtiSharma
paper_authors._id: 671917223aa9749caf1004d6
paper_authors.name: Gusti Winata
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:14.521Z
paper_authors._id: 61e3d0829d24288616e5205e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e3d0829d24288616e5205e/bwsQBncSA5oGGP8H--Wi-.jpeg
paper_authors.fullname: Gusti Triandi Winata
paper_authors.type: user
paper_authors.user: sanggusti
paper_authors._id: 671917223aa9749caf1004d7
paper_authors.name: Nathan Lambert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:24:04.907Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 671917223aa9749caf1004d8
paper_authors.name: Sebastian Ruder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:23:31.119Z
paper_authors._id: 624575789dc26c59bae6a19e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648719221694-noauth.jpeg
paper_authors.fullname: Sebastian Ruder
paper_authors.type: user
paper_authors.user: ruder
paper_authors._id: 671917223aa9749caf1004d9
paper_authors.name: Sara Hooker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:23:23.838Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_authors._id: 671917223aa9749caf1004da
paper_authors.name: Marzieh Fadaee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:23:17.162Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_title: M-RewardBench: Evaluating Reward Models in Multilingual Settings
paper_summary: Reward models (RMs) have driven the state-of-the-art performance of LLMs
  today by enabling the integration of human feedback into the language modeling
  process. However, RMs are primarily trained and evaluated in English, and their
  capabilities in multilingual settings remain largely understudied. In this
  work, we conduct a systematic evaluation of several reward models in
  multilingual settings. We first construct the first-of-its-kind multilingual RM
  evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances
  for 23 typologically diverse languages, that tests the chat, safety, reasoning,
  and translation capabilities of RMs. We then rigorously evaluate a wide range
  of reward models on M-RewardBench, offering fresh insights into their
  performance across diverse languages. We identify a significant gap in RMs'
  performances between English and non-English languages and show that RM
  preferences can change substantially from one language to another. We also
  present several findings on how different multilingual aspects impact RM
  performance. Specifically, we show that the performance of RMs is improved with
  improved translation quality. Similarly, we demonstrate that the models exhibit
  better performance for high-resource languages. We release M-RewardBench
  dataset and the codebase in this study to facilitate a better understanding of
  RM evaluation in multilingual settings.
paper_ai_keywords: reward models RAG state-of-the-art language modeling human feedback multRM multilingual M-RewardBench preference instances typologically diverse chat safety reasoning translation benchmark region models language capabilities translation quality resource languages RM evaluation reward models RAG state-of-the-art language modeling human feedback multilingual M-RewardBench preference instances typologically diverse chat safety reasoning translation benchmark region models language capabilities translation quality resource languages RM evaluation