title: SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better
    Reasoning in SLMs
summary: We present SmolTulu-1.7b-Instruct, referenced in this report as
  SmolTulu-DPO-1130, an instruction-tuned language model that adapts AllenAI's
  Tulu 3 post-training pipeline to enhance Huggingface's SmolLM2-1.7B base model.
  Through comprehensive empirical analysis using a 135M parameter model, we
  demonstrate that the relationship between learning rate and batch size
  significantly impacts model performance in a task-dependent manner. Our
  findings reveal a clear split: reasoning tasks like ARC and GSM8K benefit from
  higher learning rate to batch size ratios, while pattern recognition tasks such
  as HellaSwag and IFEval show optimal performance with lower ratios. These
  insights informed the development of SmolTulu, which achieves state-of-the-art
  performance among sub-2B parameter models on instruction following, scoring
  67.7% on IFEval (Delta11%), and mathematical reasoning with 51.6% on GSM8K
  (Delta3.4%), with an alternate version achieving scoring 57.1% on ARC
  (Delta5.4%). We release our model, training recipes, and ablation studies to
  facilitate further research in efficient model alignment, demonstrating that
  careful adaptation of optimization dynamics can help bridge the capability gap
  between small and large language models.
paper_authors._id: 675ac92733f6ac95ba7d6336
paper_authors.name: Sultan Alrashed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T13:41:00.392Z
paper_authors._id: 63fc685a4c57549ad5e5c419
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63fc685a4c57549ad5e5c419/0oySJfCD7BUVD59kXyhsV.jpeg
paper_authors.fullname: Sultan Alrashed
paper_authors.type: user
paper_authors.user: SultanR
paper_title: SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better
    Reasoning in SLMs
paper_summary: We present SmolTulu-1.7b-Instruct, referenced in this report as
  SmolTulu-DPO-1130, an instruction-tuned language model that adapts AllenAI's
  Tulu 3 post-training pipeline to enhance Huggingface's SmolLM2-1.7B base model.
  Through comprehensive empirical analysis using a 135M parameter model, we
  demonstrate that the relationship between learning rate and batch size
  significantly impacts model performance in a task-dependent manner. Our
  findings reveal a clear split: reasoning tasks like ARC and GSM8K benefit from
  higher learning rate to batch size ratios, while pattern recognition tasks such
  as HellaSwag and IFEval show optimal performance with lower ratios. These
  insights informed the development of SmolTulu, which achieves state-of-the-art
  performance among sub-2B parameter models on instruction following, scoring
  67.7% on IFEval (Delta11%), and mathematical reasoning with 51.6% on GSM8K
  (Delta3.4%), with an alternate version achieving scoring 57.1% on ARC
  (Delta5.4%). We release our model, training recipes, and ablation studies to
  facilitate further research in efficient model alignment, demonstrating that
  careful adaptation of optimization dynamics can help bridge the capability gap
  between small and large language models.
paper_ai_keywords: SmolTulu-1.7b-Instruct Tulu 3 Huggingface's SmolLM2-1.7B learning rate batch size ARC GSM8K HellaSwag IFEval instruction following sub-2B parameter models state-of-the-art performance efficient model alignment optimization dynamics