title: Improving Large Language Model Fine-tuning for Solving Math Problems
summary: Despite their success in many natural language tasks, solving math problems
  remains a significant challenge for large language models (LLMs). A large gap
  exists between LLMs' pass-at-one and pass-at-N performance in solving math
  problems, suggesting LLMs might be close to finding correct solutions,
  motivating our exploration of fine-tuning methods to unlock LLMs' performance.
  Using the challenging MATH dataset, we investigate three fine-tuning
  strategies: (1) solution fine-tuning, where we fine-tune to generate a detailed
  solution for a given math problem; (2) solution-cluster re-ranking, where the
  LLM is fine-tuned as a solution verifier/evaluator to choose among generated
  candidate solution clusters; (3) multi-task sequential fine-tuning, which
  integrates both solution generation and evaluation tasks together efficiently
  to enhance the LLM performance. With these methods, we present a thorough
  empirical study on a series of PaLM 2 models and find: (1) The quality and
  style of the step-by-step solutions used for fine-tuning can make a significant
  impact on the model performance; (2) While solution re-ranking and majority
  voting are both effective for improving the model performance when used
  separately, they can also be used together for an even greater performance
  boost; (3) Multi-task fine-tuning that sequentially separates the solution
  generation and evaluation tasks can offer improved performance compared with
  the solution fine-tuning baseline. Guided by these insights, we design a
  fine-tuning recipe that yields approximately 58.8% accuracy on the MATH dataset
  with fine-tuned PaLM 2-L models, an 11.2% accuracy improvement over the
  few-shot performance of pre-trained PaLM 2-L model with majority voting.
paper_authors._id: 652df3d40c60de928ef8feaf
paper_authors.name: Yixin Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-17T02:45:40.403Z
paper_authors._id: 6244de1c1c560fb11edfca44
paper_authors.avatarUrl: /avatars/36558928bd04be7f49837d4c603681d7.svg
paper_authors.fullname: Yixin Liu
paper_authors.type: user
paper_authors.user: henryL7
paper_authors._id: 652df3d40c60de928ef8feb0
paper_authors.name: Avi Singh
paper_authors._id: 652df3d40c60de928ef8feb1
paper_authors.name: C. Daniel Freeman
paper_authors._id: 652df3d40c60de928ef8feb2
paper_authors.name: John D. Co-Reyes
paper_authors._id: 652df3d40c60de928ef8feb3
paper_authors.name: Peter J. Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-02T20:32:48.394Z
paper_authors._id: 62ded294dcdc9043efaa4fc3
paper_authors.avatarUrl: /avatars/1294078b8cd5c1901e4021253d877357.svg
paper_authors.fullname: Peter J Liu
paper_authors.type: user
paper_authors.user: peterjliu
paper_title: Improving Large Language Model Fine-tuning for Solving Math Problems
paper_summary: Despite their success in many natural language tasks, solving math problems
  remains a significant challenge for large language models (LLMs). A large gap
  exists between LLMs' pass-at-one and pass-at-N performance in solving math
  problems, suggesting LLMs might be close to finding correct solutions,
  motivating our exploration of fine-tuning methods to unlock LLMs' performance.
  Using the challenging MATH dataset, we investigate three fine-tuning
  strategies: (1) solution fine-tuning, where we fine-tune to generate a detailed
  solution for a given math problem; (2) solution-cluster re-ranking, where the
  LLM is fine-tuned as a solution verifier/evaluator to choose among generated
  candidate solution clusters; (3) multi-task sequential fine-tuning, which
  integrates both solution generation and evaluation tasks together efficiently
  to enhance the LLM performance. With these methods, we present a thorough
  empirical study on a series of PaLM 2 models and find: (1) The quality and
  style of the step-by-step solutions used for fine-tuning can make a significant
  impact on the model performance; (2) While solution re-ranking and majority
  voting are both effective for improving the model performance when used
  separately, they can also be used together for an even greater performance
  boost; (3) Multi-task fine-tuning that sequentially separates the solution
  generation and evaluation tasks can offer improved performance compared with
  the solution fine-tuning baseline. Guided by these insights, we design a
  fine-tuning recipe that yields approximately 58.8% accuracy on the MATH dataset
  with fine-tuned PaLM 2-L models, an 11.2% accuracy improvement over the
  few-shot performance of pre-trained PaLM 2-L model with majority voting.
paper_ai_keywords: large language models (LLMs) pass-at-one pass-at-N fine-tuning MATH dataset solution fine-tuning solution-cluster re-ranking multi-task sequential fine-tuning reinforcement fine-tuning solution verifier/evaluator candidate solution clusters step-by-step solutions majority voting multi-task fine-tuning pre-trained PaLM 2-L model few-shot performance accuracy improvement