s1: Simple test-time scaling
Test-time scaling is a promising new approach to language modeling that uses
extra test-time compute to improve performance. Recently, OpenAI's o1 model
showed this capability but did not publicly share its methodology, leading to
many replication efforts. We seek the simplest approach to achieve test-time
scaling and strong reasoning performance. First, we curate a small dataset s1K
of 1,000 questions paired with reasoning traces relying on three criteria we
validate through ablations: difficulty, diversity, and quality. Second, we
develop budget forcing to control test-time compute by forcefully terminating
the model's thinking process or lengthening it by appending "Wait" multiple
times to the model's generation when it tries to end. This can lead the model
to double-check its answer, often fixing incorrect reasoning steps. After
supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and
equipping it with budget forcing, our model s1 exceeds o1-preview on
competition math questions by up to 27% (MATH and AIME24). Further, scaling s1
with budget forcing allows extrapolating beyond its performance without
test-time intervention: from 50% to 57% on AIME24. Our model, data, and code
are open-source at https://github.com/simplescaling/s1.
67a02dd80e751b0476a1bcc6
Niklas Muennighoff
admin_assigned
2025-02-03T11:07:41.451Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
67a02dd80e751b0476a1bcc7
Zitong Yang
admin_assigned
2025-02-03T11:09:05.373Z
65a5b721f6cfc4b24a75732b
https://cdn-avatars.huggingface.co/v1/production/uploads/65a5b721f6cfc4b24a75732b/Hr2AZi3uC6nVl_x3er2Hn.png
Zitong Yang
user
zitongyang
67a02dd80e751b0476a1bcc8
Weijia Shi
admin_assigned
2025-02-03T11:09:20.064Z
6400f2ed568dbe30c9161e47
/avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
Weijia Shi
user
swj0419
67a02dd80e751b0476a1bcc9
Xiang Lisa Li
67a02dd80e751b0476a1bcca
Li Fei-Fei
67a02dd80e751b0476a1bccb
Hannaneh Hajishirzi
67a02dd80e751b0476a1bccc
Luke Zettlemoyer
67a02dd80e751b0476a1bccd
Percy Liang
admin_assigned
2025-02-03T11:10:17.521Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
67a02dd80e751b0476a1bcce
Emmanuel Candès
67a02dd80e751b0476a1bccf
Tatsunori Hashimoto
s1: Simple test-time scaling
Test-time scaling is a promising new approach to language modeling that uses
extra test-time compute to improve performance. Recently, OpenAI's o1 model
showed this capability but did not publicly share its methodology, leading to
many replication efforts. We seek the simplest approach to achieve test-time
scaling and strong reasoning performance. First, we curate a small dataset s1K
of 1,000 questions paired with reasoning traces relying on three criteria we
validate through ablations: difficulty, diversity, and quality. Second, we
develop budget forcing to control test-time compute by forcefully terminating
the model's thinking process or lengthening it by appending "Wait" multiple
times to the model's generation when it tries to end. This can lead the model
to double-check its answer, often fixing incorrect reasoning steps. After
supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and
equipping it with budget forcing, our model s1 exceeds o1-preview on
competition math questions by up to 27% (MATH and AIME24). Further, scaling s1
with budget forcing allows extrapolating beyond its performance without
test-time intervention: from 50% to 57% on AIME24. Our model, data, and code
are open-source at https://github.com/simplescaling/s1.
The use of budget forcing during test time improves reasoning performance in language models, as demonstrated by the s1 model which outperforms the o1-preview on competition math questions.
test-time scaling language modeling budget forcing Qwen2.5-32B-Instruct supervised finetuning reasoning traces