LIMO: Less is More for Reasoning
We present a fundamental discovery that challenges our understanding of how
complex reasoning emerges in large language models. While conventional wisdom
suggests that sophisticated reasoning tasks demand extensive training data
(>100,000 examples), we demonstrate that complex mathematical reasoning
abilities can be effectively elicited with surprisingly few examples. Through
comprehensive experiments, our proposed model LIMO demonstrates unprecedented
performance in mathematical reasoning. With merely 817 curated training
samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from
previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of
the training data required by previous approaches. LIMO demonstrates
exceptional out-of-distribution generalization, achieving 40.5% absolute
improvement across 10 diverse benchmarks, outperforming models trained on 100x
more data, challenging the notion that SFT leads to memorization rather than
generalization. Based on these results, we propose the Less-Is-More Reasoning
Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has
been comprehensively encoded during pre-training, sophisticated reasoning
capabilities can emerge through minimal but precisely orchestrated
demonstrations of cognitive processes. This hypothesis posits that the
elicitation threshold for complex reasoning is determined by two key factors:
(1) the completeness of the model's encoded knowledge foundation during
pre-training, and (2) the effectiveness of post-training examples as "cognitive
templates" that show the model how to utilize its knowledge base to solve
complex reasoning tasks. To facilitate reproducibility and future research in
data-efficient reasoning, we release LIMO as a comprehensive open-source suite
at https://github.com/GAIR-NLP/LIMO.
67a445ccbdd74b63b4e52a7d
Yixin Ye
67a445ccbdd74b63b4e52a7e
Zhen Huang
admin_assigned
2025-02-06T14:37:03.643Z
643581a4f3b08e267d990499
https://cdn-avatars.huggingface.co/v1/production/uploads/643581a4f3b08e267d990499/KRhB-48W4IPuB0bX16Ahj.png
Zhen Huang
user
ZhenHuang
67a445ccbdd74b63b4e52a7f
Yang Xiao
67a445ccbdd74b63b4e52a80
Ethan Chern
admin_assigned
2025-02-06T14:36:45.045Z
64bb5f9d8e051085bace4d1e
/avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
Ethan Chern
user
ethanchern
67a445ccbdd74b63b4e52a81
Shijie Xia
claimed_verified
2025-02-06T14:14:59.334Z
65900d4ff5a209eeac08b463
https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
shijie xia
user
seven-cat
67a445ccbdd74b63b4e52a82
Pengfei Liu
admin_assigned
2025-02-06T14:36:38.049Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
LIMO: Less is More for Reasoning
We present a fundamental discovery that challenges our understanding of how
complex reasoning emerges in large language models. While conventional wisdom
suggests that sophisticated reasoning tasks demand extensive training data
(>100,000 examples), we demonstrate that complex mathematical reasoning
abilities can be effectively elicited with surprisingly few examples. Through
comprehensive experiments, our proposed model LIMO demonstrates unprecedented
performance in mathematical reasoning. With merely 817 curated training
samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from
previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of
the training data required by previous approaches. LIMO demonstrates
exceptional out-of-distribution generalization, achieving 40.5% absolute
improvement across 10 diverse benchmarks, outperforming models trained on 100x
more data, challenging the notion that SFT leads to memorization rather than
generalization. Based on these results, we propose the Less-Is-More Reasoning
Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has
been comprehensively encoded during pre-training, sophisticated reasoning
capabilities can emerge through minimal but precisely orchestrated
demonstrations of cognitive processes. This hypothesis posits that the
elicitation threshold for complex reasoning is determined by two key factors:
(1) the completeness of the model's encoded knowledge foundation during
pre-training, and (2) the effectiveness of post-training examples as "cognitive
templates" that show the model how to utilize its knowledge base to solve
complex reasoning tasks. To facilitate reproducibility and future research in
data-efficient reasoning, we release LIMO as a comprehensive open-source suite
at https://github.com/GAIR-NLP/LIMO.
LIMO, a new model, achieves high mathematical reasoning performance using minimal training data, challenging the notion that extensive datasets are necessary for complex reasoning.
LIMO LIMO Hypothesis pre-training foundation models encoded knowledge cognitive templates data-efficient reasoning