Warm Up Before You Train: Unlocking General Reasoning in
  Resource-Constrained Settings
Designing effective reasoning-capable LLMs typically requires training using
Reinforcement Learning with Verifiable Rewards (RLVR) or distillation with
carefully curated Long Chain of Thoughts (CoT), both of which depend heavily on
extensive training data. This creates a major challenge when the amount of
quality training data is scarce. We propose a sample-efficient, two-stage
training strategy to develop reasoning LLMs under limited supervision. In the
first stage, we "warm up" the model by distilling Long CoTs from a toy domain,
namely, Knights \& Knaves (K\&K) logic puzzles to acquire general reasoning
skills. In the second stage, we apply RLVR to the warmed-up model using a
limited set of target-domain examples. Our experiments demonstrate that this
two-phase approach offers several benefits: (i) the warmup phase alone
facilitates generalized reasoning, leading to performance improvements across a
range of tasks, including MATH, HumanEval^{+}, and MMLU-Pro. (ii) When both
the base model and the warmed-up model are RLVR trained on the same small
dataset (leq100 examples), the warmed-up model consistently outperforms the
base model; (iii) Warming up before RLVR training allows a model to maintain
cross-domain generalizability even after training on a specific domain; (iv)
Introducing warmup in the pipeline improves not only accuracy but also overall
sample efficiency during RLVR training. The results in this paper highlight the
promise of warmup for building robust reasoning LLMs in data-scarce
environments.
682d6ccc26146f27d1ab3d83
Safal Shrestha
claimed_verified
2025-05-21T08:40:07.374Z
64cb922ec7f30fbf7b91a9a7
/avatars/457eae5e56b9641ee5543146447d1755.svg
Safal Shrestha
user
safal312
682d6ccc26146f27d1ab3d84
Minwu Kim
admin_assigned
2025-05-21T10:27:16.736Z
64e77b47d96966317b45eeb3
/avatars/6b67eba3f15d6cd86ac3ad55c1daf166.svg
Minwu Kim
user
guactastesgood
682d6ccc26146f27d1ab3d85
Aadim Nepal
admin_assigned
2025-05-21T10:27:23.895Z
67b5b0800918c8645fea09d7
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_6pKdkxjsC0dOBbKAN3ok.png
Aadim Nepal
user
AadimNepal
682d6ccc26146f27d1ab3d86
Anubhav Shrestha
admin_assigned
2025-05-21T10:27:30.010Z
66a0394001cfae79a11b54ea
/avatars/e135cca9b9027cfb762e7ff86bd5ab5a.svg
Anubhav Shrestha
user
xanubhav81
682d6ccc26146f27d1ab3d87
Keith Ross
Warm Up Before You Train: Unlocking General Reasoning in
  Resource-Constrained Settings
Designing effective reasoning-capable LLMs typically requires training using
Reinforcement Learning with Verifiable Rewards (RLVR) or distillation with
carefully curated Long Chain of Thoughts (CoT), both of which depend heavily on
extensive training data. This creates a major challenge when the amount of
quality training data is scarce. We propose a sample-efficient, two-stage
training strategy to develop reasoning LLMs under limited supervision. In the
first stage, we "warm up" the model by distilling Long CoTs from a toy domain,
namely, Knights \& Knaves (K\&K) logic puzzles to acquire general reasoning
skills. In the second stage, we apply RLVR to the warmed-up model using a
limited set of target-domain examples. Our experiments demonstrate that this
two-phase approach offers several benefits: (i) the warmup phase alone
facilitates generalized reasoning, leading to performance improvements across a
range of tasks, including MATH, HumanEval^{+}, and MMLU-Pro. (ii) When both
the base model and the warmed-up model are RLVR trained on the same small
dataset (leq100 examples), the warmed-up model consistently outperforms the
base model; (iii) Warming up before RLVR training allows a model to maintain
cross-domain generalizability even after training on a specific domain; (iv)
Introducing warmup in the pipeline improves not only accuracy but also overall
sample efficiency during RLVR training. The results in this paper highlight the
promise of warmup for building robust reasoning LLMs in data-scarce
environments.
A two-stage training strategy involving a warm-up phase with distillation of logic puzzles followed by RLVR training on limited domain-specific data improves reasoning capabilities and sample efficiency in LLMs.
Reinforcement Learning with Verifiable Rewards (RLVR) Long Chain of Thoughts (CoT) Knights & Knaves (K&K) logic puzzles warm-up phase generalized reasoning MATH HumanEval+ MMLU-Pro sample efficiency cross-domain generalizability