title: Reinforcement Learning for Reasoning in Small LLMs: What Works and What
    Doesn't
summary: Enhancing the reasoning capabilities of large language models (LLMs)
  typically relies on massive computational resources and extensive datasets,
  limiting accessibility for resource-constrained settings. Our study
  investigates the potential of reinforcement learning (RL) to improve reasoning
  in small LLMs, focusing on a 1.5-billion-parameter model,
  DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA
  A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy
  Optimization (GRPO) algorithm and curating a compact, high-quality mathematical
  reasoning dataset, we conducted three experiments to explore model behavior and
  performance. Our results demonstrate rapid reasoning gains - e.g., AMC23
  accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing
  o1-preview - using only 7,000 samples and a $42 training cost, compared to
  thousands of dollars for baseline models. However, challenges such as
  optimization instability and length constraints emerged with prolonged
  training. These findings highlight the efficacy of RL-based fine-tuning for
  small LLMs, offering a cost-effective alternative to large-scale approaches. We
  release our code and datasets as open-source resources, providing insights into
  trade-offs and laying a foundation for scalable, reasoning-capable LLMs in
  resource-limited environments. All are available at
  https://github.com/knoveleng/open-rs.
paper_authors._id: 67dd1a9cfa598c90d14e9b47
paper_authors.name: Quy-Anh Dang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-21T07:52:47.629Z
paper_authors._id: 645b663eca5d8a297712f2e1
paper_authors.avatarUrl: /avatars/c61dcba43feb879088b15b525e441cb9.svg
paper_authors.fullname: Quy-Anh Dang
paper_authors.type: user
paper_authors.user: quyanh
paper_authors._id: 67dd1a9cfa598c90d14e9b48
paper_authors.name: Chris Ngo
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-22T02:28:23.821Z
paper_authors._id: 63105f7463b70252b4775783
paper_authors.avatarUrl: /avatars/89cfe71302cde6c2834f58d44bbcdbd5.svg
paper_authors.fullname: Chris Ngo
paper_authors.type: user
paper_authors.user: tnngo2
paper_title: Reinforcement Learning for Reasoning in Small LLMs: What Works and What
    Doesn't
paper_summary: Enhancing the reasoning capabilities of large language models (LLMs)
  typically relies on massive computational resources and extensive datasets,
  limiting accessibility for resource-constrained settings. Our study
  investigates the potential of reinforcement learning (RL) to improve reasoning
  in small LLMs, focusing on a 1.5-billion-parameter model,
  DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA
  A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy
  Optimization (GRPO) algorithm and curating a compact, high-quality mathematical
  reasoning dataset, we conducted three experiments to explore model behavior and
  performance. Our results demonstrate rapid reasoning gains - e.g., AMC23
  accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing
  o1-preview - using only 7,000 samples and a $42 training cost, compared to
  thousands of dollars for baseline models. However, challenges such as
  optimization instability and length constraints emerged with prolonged
  training. These findings highlight the efficacy of RL-based fine-tuning for
  small LLMs, offering a cost-effective alternative to large-scale approaches. We
  release our code and datasets as open-source resources, providing insights into
  trade-offs and laying a foundation for scalable, reasoning-capable LLMs in
  resource-limited environments. All are available at
  https://github.com/knoveleng/open-rs.
paper_ai_summary: Reinforcement learning enhances reasoning in small language models with limited resources, demonstrating significant performance improvements in mathematical reasoning tasks.
paper_ai_keywords: reinforcement learning Group Relative Policy Optimization GRPO mathematical reasoning AMC23 AIME24 parameter-efficient fine-tuning optimization instability length constraints scalable LLMs