title: Teaching Arithmetic to Small Transformers
summary: Large language models like GPT-4 exhibit emergent capabilities across
  general-purpose tasks, such as basic arithmetic, when trained on extensive text
  data, even though these tasks are not explicitly encoded by the unsupervised,
  next-token prediction objective. This study investigates how small
  transformers, trained from random initialization, can efficiently learn
  arithmetic operations such as addition, multiplication, and elementary
  functions like square root, using the next-token prediction objective. We first
  demonstrate that conventional training data is not the most effective for
  arithmetic learning, and simple formatting changes can significantly improve
  accuracy. This leads to sharp phase transitions as a function of training data
  scale, which, in some cases, can be explained through connections to low-rank
  matrix completion. Building on prior work, we then train on chain-of-thought
  style data that includes intermediate step results. Even in the complete
  absence of pretraining, this approach significantly and simultaneously improves
  accuracy, sample complexity, and convergence speed. We also study the interplay
  between arithmetic and text data during training and examine the effects of
  few-shot prompting, pretraining, and model scale. Additionally, we discuss
  length generalization challenges. Our work highlights the importance of
  high-quality, instructive data that considers the particular characteristics of
  the next-word prediction objective for rapidly eliciting arithmetic
  capabilities.
paper_authors._id: 64ab8e5803cd5dee2ef57fa6
paper_authors.name: Nayoung Lee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-10T16:51:49.286Z
paper_authors._id: 64ac36de7b5ff76277ec02a5
paper_authors.avatarUrl: /avatars/e128d7bf886cdc29a56751c3a75dd8d5.svg
paper_authors.fullname: Nayoung Lee
paper_authors.type: user
paper_authors.user: nayoungl95
paper_authors._id: 64ab8e5803cd5dee2ef57fa7
paper_authors.name: Kartik Sreenivasan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-07-13T04:14:37.143Z
paper_authors._id: 648372e241796f46df60833c
paper_authors.avatarUrl: /avatars/0ec44f753d4d2af73b1dafe3e324935a.svg
paper_authors.fullname: Kartik Sreenivasan
paper_authors.type: user
paper_authors.user: kartikmosaicml
paper_authors._id: 64ab8e5803cd5dee2ef57fa8
paper_authors.name: Jason D. Lee
paper_authors._id: 64ab8e5803cd5dee2ef57fa9
paper_authors.name: Kangwook Lee
paper_authors._id: 64ab8e5803cd5dee2ef57faa
paper_authors.name: Dimitris Papailiopoulos
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-10T04:59:03.003Z
paper_authors._id: 64428cf07feb866811b0be6a
paper_authors.avatarUrl: /avatars/f30d339a02ce070c7e77315b59e20359.svg
paper_authors.fullname: Dimitris Papailiopoulos
paper_authors.type: user
paper_authors.user: anadim
paper_title: Teaching Arithmetic to Small Transformers
paper_summary: Large language models like GPT-4 exhibit emergent capabilities across
  general-purpose tasks, such as basic arithmetic, when trained on extensive text
  data, even though these tasks are not explicitly encoded by the unsupervised,
  next-token prediction objective. This study investigates how small
  transformers, trained from random initialization, can efficiently learn
  arithmetic operations such as addition, multiplication, and elementary
  functions like square root, using the next-token prediction objective. We first
  demonstrate that conventional training data is not the most effective for
  arithmetic learning, and simple formatting changes can significantly improve
  accuracy. This leads to sharp phase transitions as a function of training data
  scale, which, in some cases, can be explained through connections to low-rank
  matrix completion. Building on prior work, we then train on chain-of-thought
  style data that includes intermediate step results. Even in the complete
  absence of pretraining, this approach significantly and simultaneously improves
  accuracy, sample complexity, and convergence speed. We also study the interplay
  between arithmetic and text data during training and examine the effects of
  few-shot prompting, pretraining, and model scale. Additionally, we discuss
  length generalization challenges. Our work highlights the importance of
  high-quality, instructive data that considers the particular characteristics of
  the next-word prediction objective for rapidly eliciting arithmetic
  capabilities.
paper_ai_keywords: transformers next-token prediction chain-of-thought style data low-rank matrix completion few-shot prompting pretraining length generalization