TTRL: Test-Time Reinforcement Learning
This paper investigates Reinforcement Learning (RL) on data without explicit
labels for reasoning tasks in Large Language Models (LLMs). The core challenge
of the problem is reward estimation during inference while not having access to
ground-truth information. While this setting appears elusive, we find that
common practices in Test-Time Scaling (TTS), such as majority voting, yield
surprisingly effective rewards suitable for driving RL training. In this work,
we introduce Test-Time Reinforcement Learning (TTRL), a novel method for
training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs
by utilizing the priors in the pre-trained models. Our experiments demonstrate
that TTRL consistently improves performance across a variety of tasks and
models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by
approximately 159% on the AIME 2024 with only unlabeled test data. Furthermore,
although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated
performance to consistently surpass the upper limit of the initial model, and
approach the performance of models trained directly on test data with
ground-truth labels. Our experimental findings validate the general
effectiveness of TTRL across various tasks, and highlight TTRL's potential for
broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL
6808558a07e80b69b2e351b5
Yuxin Zuo
admin_assigned
2025-04-23T13:01:58.206Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
user
yuxinzuo
6808558a07e80b69b2e351b6
Kaiyan Zhang
claimed_verified
2025-04-23T08:27:51.438Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
6808558a07e80b69b2e351b7
Shang Qu
claimed_verified
2025-04-30T09:56:48.821Z
65597738deee83130a1301d5
/avatars/9bcc40aebe4db079927675d95c00463c.svg
Shang (Lindsay) Qu
user
lindsay-qu
6808558a07e80b69b2e351b8
Li Sheng
claimed_verified
2025-04-23T13:00:56.119Z
65c874daa3ea4f6d8df75dd1
https://cdn-avatars.huggingface.co/v1/production/uploads/65c874daa3ea4f6d8df75dd1/k-2Ng8pZvEwu6IcY9omYo.jpeg
li sheng
user
bambisheng
6808558a07e80b69b2e351b9
Xuekai Zhu
admin_assigned
2025-04-23T13:02:11.848Z
647ffddeb82adfa7cc1a10d9
/avatars/26aa168d6b2068298ebb16584aa52b6c.svg
zhu
user
xuekai
6808558a07e80b69b2e351ba
Biqing Qi
admin_assigned
2025-04-23T13:02:18.316Z
645d9c3058f9ee315148116d
/avatars/165e18f27b5a50738bf1d22857118478.svg
Biqing Qi
user
jackqi7
6808558a07e80b69b2e351bb
Youbang Sun
admin_assigned
2025-04-23T13:02:24.351Z
679ce8c048ebd7903d76a832
/avatars/5f3fecaacfee6e2d5a72dd19fe87055a.svg
Youbang Sun
user
Youbang
6808558a07e80b69b2e351bc
Ganqu Cui
admin_assigned
2025-04-23T13:02:30.607Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
6808558a07e80b69b2e351bd
Ning Ding
admin_assigned
2025-04-23T13:02:51.039Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
6808558a07e80b69b2e351be
Bowen Zhou
TTRL: Test-Time Reinforcement Learning
This paper investigates Reinforcement Learning (RL) on data without explicit
labels for reasoning tasks in Large Language Models (LLMs). The core challenge
of the problem is reward estimation during inference while not having access to
ground-truth information. While this setting appears elusive, we find that
common practices in Test-Time Scaling (TTS), such as majority voting, yield
surprisingly effective rewards suitable for driving RL training. In this work,
we introduce Test-Time Reinforcement Learning (TTRL), a novel method for
training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs
by utilizing the priors in the pre-trained models. Our experiments demonstrate
that TTRL consistently improves performance across a variety of tasks and
models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by
approximately 159% on the AIME 2024 with only unlabeled test data. Furthermore,
although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated
performance to consistently surpass the upper limit of the initial model, and
approach the performance of models trained directly on test data with
ground-truth labels. Our experimental findings validate the general
effectiveness of TTRL across various tasks, and highlight TTRL's potential for
broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL
Test-Time Reinforcement Learning (TTRL) enhances Large Language Models (LLMs) using unlabeled data through reinforcement learning, improving performance across tasks.
Reinforcement Learning (RL) Large Language Models (LLMs) reward estimation Test-Time Scaling (TTS) Test-Time Reinforcement Learning (TTRL) pre-trained models Qwen-2.5-Math-7B AIME 2024 Maj@N metric