S*: Test Time Scaling for Code Generation
Increasing test-time compute for LLMs shows promise across domains but
remains underexplored in code generation, despite extensive study in math. In
this paper, we propose S*, the first hybrid test-time scaling framework that
substantially improves the coverage and selection accuracy of generated code.
S* extends the existing parallel scaling paradigm with sequential scaling to
push performance boundaries. It further leverages a novel selection mechanism
that adaptively generates distinguishing inputs for pairwise comparison,
combined with execution-grounded information to robustly identify correct
solutions. We evaluate across 12 Large Language Models and Large Reasoning
Model and show: (1) S* consistently improves performance across model families
and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables
non-reasoning models to surpass reasoning models - GPT-4o-mini with S*
outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts
state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S*
achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be
available under https://github.com/NovaSky-AI/SkyThought.
67b7ed3e58f6b70b18ddb4bc
Dacheng Li
admin_assigned
2025-02-21T14:45:13.558Z
63715b25ffc0489ed7d1f415
https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
Dacheng Li
user
DachengLi
67b7ed3e58f6b70b18ddb4bd
Shiyi Cao
claimed_verified
2025-02-21T09:58:43.358Z
64ebbae6895a36ab28de811a
https://cdn-avatars.huggingface.co/v1/production/uploads/64ebbae6895a36ab28de811a/gBiaQP4paS4L13eu-yRm7.jpeg
Shiyi Cao
user
eva98
67b7ed3e58f6b70b18ddb4be
Chengkun Cao
67b7ed3e58f6b70b18ddb4bf
Xiuyu Li
admin_assigned
2025-02-21T14:45:28.150Z
644570ba2d91b15b4c7f6311
/avatars/d5e66012066d0c330b8f23718b1499d8.svg
Xiuyu Li
user
xiuyul
67b7ed3e58f6b70b18ddb4c0
Shangyin Tan
admin_assigned
2025-02-21T14:45:33.730Z
663cfbd9b0f659de3db65c1a
/avatars/b7d82d281026ee04a9932b44a770b840.svg
Shangyin Tan
user
shangyint
67b7ed3e58f6b70b18ddb4c1
Kurt Keutzer
admin_assigned
2025-02-21T14:45:39.902Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
67b7ed3e58f6b70b18ddb4c2
Jiarong Xing
admin_assigned
2025-02-21T14:45:45.774Z
66ff8aa43a31c499dc48fdd6
/avatars/060dc90fb13991bd013ce8173f12ae3e.svg
Jiarong Xing
user
JerryPotter
67b7ed3e58f6b70b18ddb4c3
Joseph E. Gonzalez
admin_assigned
2025-02-21T14:45:52.578Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
67b7ed3e58f6b70b18ddb4c4
Ion Stoica
S*: Test Time Scaling for Code Generation
Increasing test-time compute for LLMs shows promise across domains but
remains underexplored in code generation, despite extensive study in math. In
this paper, we propose S*, the first hybrid test-time scaling framework that
substantially improves the coverage and selection accuracy of generated code.
S* extends the existing parallel scaling paradigm with sequential scaling to
push performance boundaries. It further leverages a novel selection mechanism
that adaptively generates distinguishing inputs for pairwise comparison,
combined with execution-grounded information to robustly identify correct
solutions. We evaluate across 12 Large Language Models and Large Reasoning
Model and show: (1) S* consistently improves performance across model families
and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables
non-reasoning models to surpass reasoning models - GPT-4o-mini with S*
outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts
state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S*
achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be
available under https://github.com/NovaSky-AI/SkyThought.
A hybrid test-time scaling framework improves code generation coverage and accuracy across various models and domains.
hybrid test-time scaling framework parallel scaling sequential scaling selection mechanism distinguishing inputs execution-grounded information Large Language Models Large Reasoning Models LiveCodeBench DeepSeek