|
--- |
|
library_name: transformers |
|
tags: [] |
|
--- |
|
|
|
# Introduction |
|
|
|
We release **STILL-3-1.5B-preview**, a slow-thinking reasoning model achieves 39.33% accuracy on AIME benchmark! We adapt reinforcement learning on 1.5B model and observe the continuous performance improvement as the number of training steps increased. For better reproducing our work and advancing research progress, we open-source our code, model, and data. |
|
|
|
Code: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs |
|
|
|
# Evaluation |
|
|
|
We evaluated the model on four benchmarks: MATH, AIME, OMNI, and LiveAOPS. For MATH and AIME, we employed a sampling decoding setup with a sampling temperature of 0.6 and a top-p sampling probability of 0.95. Each question was sampled 64 times, and the average score was calculated. For OMNI and LiveAOPS (August-November 2024), we randomly sampled a subset of answers as integers to facilitate automated evaluation, and used greedy search decoding for the evaluation. The trained model, STILL-3-1.5B-preview, achieved significant improvement. The accuracy on the AIME task increased from 28.67% to 39.33%, resulting in a relative improvement of 37.18%. |
|
|
|
| | MATH | AIME | OMNI | LiveAOPS | Avg. | |
|
| --- | :---: | :---: | :---: | :---: | :---: | |
|
| Backbone | 84.04 | 28.67 | 25.60 | 33.33 | 42.91 | |
|
| STILL-3-1.5B-preview | **85.48** | **39.33** | **33.00** | **39.50** | **49.33** | |
|
|
|
|
|
# Quick Start |
|
|
|
``` |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
from vllm import LLM, SamplingParams |
|
|
|
# Load model and tokenizer |
|
tokenizer = AutoTokenizer.from_pretrained("RUC-AIBOX/STILL-3-1.5B-preview") |
|
model = AutoModelForCausalLM.from_pretrained("RUC-AIBOX/STILL-3-1.5B-preview") |
|
|
|
# Input text |
|
question = "Convert the point $(0,3)$ in rectangular coordinates to polar coordinates. Enter your answer in the form $(r,\\theta),$ where $r > 0$ and $0 \\le \\theta < 2 \\pi.$" |
|
|
|
input_prompts = tokenizer.apply_chat_template( |
|
[ |
|
{"role": "user", "content": question}], |
|
tokenize=False, |
|
add_generation_prompt=True |
|
) |
|
|
|
|
|
# Params |
|
llm = LLM(model=model_path, tensor_parallel_size=1, dtype='bfloat16') |
|
|
|
sampling_params_gs = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=32768, stop=stop_words, seed=42, skip_special_tokens=False) |
|
|
|
|
|
# Completion |
|
responses = model.generate(input_prompts, sampling_params) |
|
print(responses[0].outputs[0].text) |
|
``` |
|
|
|
# Reference |
|
|
|
Please kindly cite our reports if they are helpful for your research. |
|
|
|
|
|
``` |
|
@article{Slow_Thinking_with_LLMs_3_Preview, |
|
title={STILL-3-1.5B-preview: Enhancing Slow Thinking Abilities of Small Models through Reinforcement Learning |
|
}, |
|
author={RUCAIBox STILL Team}, |
|
url={https://github.com/RUCAIBox/Slow_Thinking_with_LLMs}, |
|
year={2025} |
|
} |
|
``` |
|
|
|
``` |
|
@article{Slow_Thinking_with_LLMs_1, |
|
title={Enhancing LLM Reasoning with Reward-guided Tree Search}, |
|
author={Jiang, Jinhao and Chen, Zhipeng and Min, Yingqian and Chen, Jie and Cheng, Xiaoxue and Wang, Jiapeng and Tang, Yiru and Sun, Haoxiang and Deng, Jia and Zhao, Wayne Xin and Liu, Zheng and Yan, Dong and Xie, Jian and Wang, Zhongyuan and Wen, Ji-Rong}, |
|
journal={arXiv preprint arXiv:2411.11694}, |
|
year={2024} |
|
} |
|
``` |
|
|
|
``` |
|
@article{Slow_Thinking_with_LLMs_2, |
|
title={Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems}, |
|
author={Min, Yingqian and Chen, Zhipeng and Jiang, Jinhao and Chen, Jie and Deng, Jia and Hu, Yiwen and Tang, Yiru and Wang, Jiapeng and Cheng, Xiaoxue and Song, Huatong and Zhao, Wayne Xin and Liu, Zheng and Wang, Zhongyuan and Wen, Ji-Rong}, |
|
journal={arXiv preprint arXiv:2412.09413}, |
|
year={2024} |
|
} |
|
``` |
|
|
