title: TTRL: Test-Time Reinforcement Learning
summary: This paper investigates Reinforcement Learning (RL) on data without explicit
  labels for reasoning tasks in Large Language Models (LLMs). The core challenge
  of the problem is reward estimation during inference while not having access to
  ground-truth information. While this setting appears elusive, we find that
  common practices in Test-Time Scaling (TTS), such as majority voting, yield
  surprisingly effective rewards suitable for driving RL training. In this work,
  we introduce Test-Time Reinforcement Learning (TTRL), a novel method for
  training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs
  by utilizing the priors in the pre-trained models. Our experiments demonstrate
  that TTRL consistently improves performance across a variety of tasks and
  models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by
  approximately 159% on the AIME 2024 with only unlabeled test data. Furthermore,
  although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated
  performance to consistently surpass the upper limit of the initial model, and
  approach the performance of models trained directly on test data with
  ground-truth labels. Our experimental findings validate the general
  effectiveness of TTRL across various tasks, and highlight TTRL's potential for
  broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL
paper_authors._id: 6808558a07e80b69b2e351b5
paper_authors.name: Yuxin Zuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:01:58.206Z
paper_authors._id: 622474f38dc6b0b64f5e903d
paper_authors.avatarUrl: /avatars/d6b60a014277a8ec7d564163c5f644aa.svg
paper_authors.fullname: Yuxin Zuo
paper_authors.type: user
paper_authors.user: yuxinzuo
paper_authors._id: 6808558a07e80b69b2e351b6
paper_authors.name: Kaiyan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:27:51.438Z
paper_authors._id: 60bc94cd85a3ab33829b6211
paper_authors.avatarUrl: /avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
paper_authors.fullname: Kaiyan Zhang
paper_authors.type: user
paper_authors.user: iseesaw
paper_authors._id: 6808558a07e80b69b2e351b7
paper_authors.name: Shang Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T09:56:48.821Z
paper_authors._id: 65597738deee83130a1301d5
paper_authors.avatarUrl: /avatars/9bcc40aebe4db079927675d95c00463c.svg
paper_authors.fullname: Shang (Lindsay) Qu
paper_authors.type: user
paper_authors.user: lindsay-qu
paper_authors._id: 6808558a07e80b69b2e351b8
paper_authors.name: Li Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T13:00:56.119Z
paper_authors._id: 65c874daa3ea4f6d8df75dd1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c874daa3ea4f6d8df75dd1/k-2Ng8pZvEwu6IcY9omYo.jpeg
paper_authors.fullname: li sheng
paper_authors.type: user
paper_authors.user: bambisheng
paper_authors._id: 6808558a07e80b69b2e351b9
paper_authors.name: Xuekai Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:02:11.848Z
paper_authors._id: 647ffddeb82adfa7cc1a10d9
paper_authors.avatarUrl: /avatars/26aa168d6b2068298ebb16584aa52b6c.svg
paper_authors.fullname: zhu
paper_authors.type: user
paper_authors.user: xuekai
paper_authors._id: 6808558a07e80b69b2e351ba
paper_authors.name: Biqing Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:02:18.316Z
paper_authors._id: 645d9c3058f9ee315148116d
paper_authors.avatarUrl: /avatars/165e18f27b5a50738bf1d22857118478.svg
paper_authors.fullname: Biqing Qi
paper_authors.type: user
paper_authors.user: jackqi7
paper_authors._id: 6808558a07e80b69b2e351bb
paper_authors.name: Youbang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:02:24.351Z
paper_authors._id: 679ce8c048ebd7903d76a832
paper_authors.avatarUrl: /avatars/5f3fecaacfee6e2d5a72dd19fe87055a.svg
paper_authors.fullname: Youbang Sun
paper_authors.type: user
paper_authors.user: Youbang
paper_authors._id: 6808558a07e80b69b2e351bc
paper_authors.name: Ganqu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:02:30.607Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 6808558a07e80b69b2e351bd
paper_authors.name: Ning Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:02:51.039Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 6808558a07e80b69b2e351be
paper_authors.name: Bowen Zhou
paper_title: TTRL: Test-Time Reinforcement Learning
paper_summary: This paper investigates Reinforcement Learning (RL) on data without explicit
  labels for reasoning tasks in Large Language Models (LLMs). The core challenge
  of the problem is reward estimation during inference while not having access to
  ground-truth information. While this setting appears elusive, we find that
  common practices in Test-Time Scaling (TTS), such as majority voting, yield
  surprisingly effective rewards suitable for driving RL training. In this work,
  we introduce Test-Time Reinforcement Learning (TTRL), a novel method for
  training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs
  by utilizing the priors in the pre-trained models. Our experiments demonstrate
  that TTRL consistently improves performance across a variety of tasks and
  models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by
  approximately 159% on the AIME 2024 with only unlabeled test data. Furthermore,
  although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated
  performance to consistently surpass the upper limit of the initial model, and
  approach the performance of models trained directly on test data with
  ground-truth labels. Our experimental findings validate the general
  effectiveness of TTRL across various tasks, and highlight TTRL's potential for
  broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL
paper_ai_summary: Test-Time Reinforcement Learning (TTRL) enhances Large Language Models (LLMs) using unlabeled data through reinforcement learning, improving performance across tasks.
paper_ai_keywords: Reinforcement Learning (RL) Large Language Models (LLMs) reward estimation Test-Time Scaling (TTS) Test-Time Reinforcement Learning (TTRL) pre-trained models Qwen-2.5-Math-7B AIME 2024 Maj@N metric