title: Linguistic Generalizability of Test-Time Scaling in Mathematical
    Reasoning
summary: Scaling pre-training compute has proven effective for achieving
  mulitlinguality, but does the same hold for test-time scaling? In this work, we
  introduce MCLM, a multilingual math benchmark featuring competition-level
  problems in 55 languages. We test three test-time scaling methods-Outcome
  Reward Modeling (ORM), Process Reward Modeling (ORM), and Budget Forcing
  (BF)-on both Qwen2.5-1.5B Math and MR1-1.5B, a multilingual LLM we trained for
  extended reasoning. Our experiments show that using Qwen2.5-1.5B Math with ORM
  achieves a score of 35.8 on MCLM, while BF on MR1-1.5B attains 35.2. Although
  "thinking LLMs" have recently garnered significant attention, we find that
  their performance is comparable to traditional scaling methods like best-of-N
  once constrained to similar levels of inference FLOPs. Moreover, while BF
  yields a 20-point improvement on English AIME, it provides only a 1.94-point
  average gain across other languages-a pattern consistent across the other
  test-time scaling methods we studied-higlighting that test-time scaling may not
  generalize as effectively to multilingual tasks. To foster further research, we
  release MCLM, MR1-1.5B, and evaluation results.
paper_authors._id: 67bd48d4becb766415a5d19d
paper_authors.name: Guijin Son
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:21:47.182Z
paper_authors._id: 60d3e619b8448e1785bbda2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60d3e619b8448e1785bbda2a/q2re5u1HNwsCCyIMtid_I.jpeg
paper_authors.fullname: GUIJIN SON
paper_authors.type: user
paper_authors.user: amphora
paper_authors._id: 67bd48d4becb766415a5d19e
paper_authors.name: Jiwoo Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:21:56.398Z
paper_authors._id: 678b61b6c7491f8b7065f68d
paper_authors.avatarUrl: /avatars/2168e7a0c58076126fcb41b01d01e622.svg
paper_authors.fullname: Jiwoo Hong
paper_authors.type: user
paper_authors.user: hongmush
paper_authors._id: 67bd48d4becb766415a5d19f
paper_authors.name: Hyunwoo Ko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:39:12.933Z
paper_authors._id: 63e087b6a98d931aa90c1b9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e087b6a98d931aa90c1b9c/96c6IT3f1pWGLbRdRDB2U.png
paper_authors.fullname: Hyunwoo Ko
paper_authors.type: user
paper_authors.user: Cartinoe5930
paper_authors._id: 67bd48d4becb766415a5d1a0
paper_authors.name: James Thorne
paper_title: Linguistic Generalizability of Test-Time Scaling in Mathematical
    Reasoning
paper_summary: Scaling pre-training compute has proven effective for achieving
  mulitlinguality, but does the same hold for test-time scaling? In this work, we
  introduce MCLM, a multilingual math benchmark featuring competition-level
  problems in 55 languages. We test three test-time scaling methods-Outcome
  Reward Modeling (ORM), Process Reward Modeling (ORM), and Budget Forcing
  (BF)-on both Qwen2.5-1.5B Math and MR1-1.5B, a multilingual LLM we trained for
  extended reasoning. Our experiments show that using Qwen2.5-1.5B Math with ORM
  achieves a score of 35.8 on MCLM, while BF on MR1-1.5B attains 35.2. Although
  "thinking LLMs" have recently garnered significant attention, we find that
  their performance is comparable to traditional scaling methods like best-of-N
  once constrained to similar levels of inference FLOPs. Moreover, while BF
  yields a 20-point improvement on English AIME, it provides only a 1.94-point
  average gain across other languages-a pattern consistent across the other
  test-time scaling methods we studied-higlighting that test-time scaling may not
  generalize as effectively to multilingual tasks. To foster further research, we
  release MCLM, MR1-1.5B, and evaluation results.
paper_ai_summary: Multilingual math benchmark MCLM tests various test-time scaling methods on multilingual LLMs, showing that performance gains from these methods are limited outside of English.
paper_ai_keywords: MCLM Outcome Reward Modeling Process Reward Modeling Budget Forcing Qwen2.5-1.5B Math MR1-1.5B multilingual LLM inference FLOPs AIME