title: Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks
summary: We introduce Goat, a fine-tuned LLaMA model that significantly outperforms
  GPT-4 on a range of arithmetic tasks. Fine-tuned on a synthetically generated
  dataset, Goat achieves state-of-the-art performance on BIG-bench arithmetic
  sub-task. In particular, the zero-shot Goat-7B matches or even surpasses the
  accuracy achieved by the few-shot PaLM-540B. Surprisingly, Goat can achieve
  near-perfect accuracy on large-number addition and subtraction through
  supervised fine-tuning only, which is almost impossible with previous
  pretrained language models, such as Bloom, OPT, GPT-NeoX, etc. We attribute
  Goat's exceptional performance to LLaMA's consistent tokenization of numbers.
  To tackle more challenging tasks like large-number multiplication and division,
  we propose an approach that classifies tasks based on their learnability, and
  subsequently decomposes unlearnable tasks, such as multi-digit multiplication
  and division, into a series of learnable tasks by leveraging basic arithmetic
  principles. We thoroughly examine the performance of our model, offering a
  comprehensive evaluation of the effectiveness of our proposed decomposition
  steps. Additionally, Goat-7B can be easily trained using LoRA on a 24GB VRAM
  GPU, facilitating reproducibility for other researchers. We release our model,
  dataset, and the Python script for dataset generation.
paper_authors._id: 646d622b1aca2e9d9c706aed
paper_authors.name: Tiedong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T08:16:01.839Z
paper_authors._id: 646476b76d621be0dd08e67a
paper_authors.avatarUrl: /avatars/ff269254e818c488e58f2d514305c0c1.svg
paper_authors.fullname: Liu Tiedong
paper_authors.type: user
paper_authors.user: tiedong
paper_authors._id: 646d622b1aca2e9d9c706aee
paper_authors.name: Bryan Kian Hsiang Low
paper_title: Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks
paper_summary: We introduce Goat, a fine-tuned LLaMA model that significantly outperforms
  GPT-4 on a range of arithmetic tasks. Fine-tuned on a synthetically generated
  dataset, Goat achieves state-of-the-art performance on BIG-bench arithmetic
  sub-task. In particular, the zero-shot Goat-7B matches or even surpasses the
  accuracy achieved by the few-shot PaLM-540B. Surprisingly, Goat can achieve
  near-perfect accuracy on large-number addition and subtraction through
  supervised fine-tuning only, which is almost impossible with previous
  pretrained language models, such as Bloom, OPT, GPT-NeoX, etc. We attribute
  Goat's exceptional performance to LLaMA's consistent tokenization of numbers.
  To tackle more challenging tasks like large-number multiplication and division,
  we propose an approach that classifies tasks based on their learnability, and
  subsequently decomposes unlearnable tasks, such as multi-digit multiplication
  and division, into a series of learnable tasks by leveraging basic arithmetic
  principles. We thoroughly examine the performance of our model, offering a
  comprehensive evaluation of the effectiveness of our proposed decomposition
  steps. Additionally, Goat-7B can be easily trained using LoRA on a 24GB VRAM
  GPU, facilitating reproducibility for other researchers. We release our model,
  dataset, and the Python script for dataset generation.
paper_ai_keywords: fine-tuned LLAPT model BIG-bench arithmetic sub-task zero-shot few-shot PaLM-540B large-number addition large-number subtraction supervised fine-tuning Bloom OPT GPT-NeoX consistent tokenization of numbers learnability decomposition multi-digit multiplication multi-digit division basic arithmetic principles LoRA VRAM qualitative evaluation