title: LIMO: Less is More for Reasoning
summary: We present a fundamental discovery that challenges our understanding of how
  complex reasoning emerges in large language models. While conventional wisdom
  suggests that sophisticated reasoning tasks demand extensive training data
  (>100,000 examples), we demonstrate that complex mathematical reasoning
  abilities can be effectively elicited with surprisingly few examples. Through
  comprehensive experiments, our proposed model LIMO demonstrates unprecedented
  performance in mathematical reasoning. With merely 817 curated training
  samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from
  previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of
  the training data required by previous approaches. LIMO demonstrates
  exceptional out-of-distribution generalization, achieving 40.5% absolute
  improvement across 10 diverse benchmarks, outperforming models trained on 100x
  more data, challenging the notion that SFT leads to memorization rather than
  generalization. Based on these results, we propose the Less-Is-More Reasoning
  Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has
  been comprehensively encoded during pre-training, sophisticated reasoning
  capabilities can emerge through minimal but precisely orchestrated
  demonstrations of cognitive processes. This hypothesis posits that the
  elicitation threshold for complex reasoning is determined by two key factors:
  (1) the completeness of the model's encoded knowledge foundation during
  pre-training, and (2) the effectiveness of post-training examples as "cognitive
  templates" that show the model how to utilize its knowledge base to solve
  complex reasoning tasks. To facilitate reproducibility and future research in
  data-efficient reasoning, we release LIMO as a comprehensive open-source suite
  at https://github.com/GAIR-NLP/LIMO.
paper_authors._id: 67a445ccbdd74b63b4e52a7d
paper_authors.name: Yixin Ye
paper_authors._id: 67a445ccbdd74b63b4e52a7e
paper_authors.name: Zhen Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:37:03.643Z
paper_authors._id: 643581a4f3b08e267d990499
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643581a4f3b08e267d990499/KRhB-48W4IPuB0bX16Ahj.png
paper_authors.fullname: Zhen Huang
paper_authors.type: user
paper_authors.user: ZhenHuang
paper_authors._id: 67a445ccbdd74b63b4e52a7f
paper_authors.name: Yang Xiao
paper_authors._id: 67a445ccbdd74b63b4e52a80
paper_authors.name: Ethan Chern
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:36:45.045Z
paper_authors._id: 64bb5f9d8e051085bace4d1e
paper_authors.avatarUrl: /avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
paper_authors.fullname: Ethan Chern
paper_authors.type: user
paper_authors.user: ethanchern
paper_authors._id: 67a445ccbdd74b63b4e52a81
paper_authors.name: Shijie Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:59.334Z
paper_authors._id: 65900d4ff5a209eeac08b463
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
paper_authors.fullname: shijie xia
paper_authors.type: user
paper_authors.user: seven-cat
paper_authors._id: 67a445ccbdd74b63b4e52a82
paper_authors.name: Pengfei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:36:38.049Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: LIMO: Less is More for Reasoning
paper_summary: We present a fundamental discovery that challenges our understanding of how
  complex reasoning emerges in large language models. While conventional wisdom
  suggests that sophisticated reasoning tasks demand extensive training data
  (>100,000 examples), we demonstrate that complex mathematical reasoning
  abilities can be effectively elicited with surprisingly few examples. Through
  comprehensive experiments, our proposed model LIMO demonstrates unprecedented
  performance in mathematical reasoning. With merely 817 curated training
  samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from
  previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of
  the training data required by previous approaches. LIMO demonstrates
  exceptional out-of-distribution generalization, achieving 40.5% absolute
  improvement across 10 diverse benchmarks, outperforming models trained on 100x
  more data, challenging the notion that SFT leads to memorization rather than
  generalization. Based on these results, we propose the Less-Is-More Reasoning
  Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has
  been comprehensively encoded during pre-training, sophisticated reasoning
  capabilities can emerge through minimal but precisely orchestrated
  demonstrations of cognitive processes. This hypothesis posits that the
  elicitation threshold for complex reasoning is determined by two key factors:
  (1) the completeness of the model's encoded knowledge foundation during
  pre-training, and (2) the effectiveness of post-training examples as "cognitive
  templates" that show the model how to utilize its knowledge base to solve
  complex reasoning tasks. To facilitate reproducibility and future research in
  data-efficient reasoning, we release LIMO as a comprehensive open-source suite
  at https://github.com/GAIR-NLP/LIMO.
paper_ai_keywords: LIMO AIME MATH SFT Less-Is-More Reasoning Hypothesis (LIMO Hypothesis) cognitive templates