Teaching Arithmetic to Small Transformers
Large language models like GPT-4 exhibit emergent capabilities across
general-purpose tasks, such as basic arithmetic, when trained on extensive text
data, even though these tasks are not explicitly encoded by the unsupervised,
next-token prediction objective. This study investigates how small
transformers, trained from random initialization, can efficiently learn
arithmetic operations such as addition, multiplication, and elementary
functions like square root, using the next-token prediction objective. We first
demonstrate that conventional training data is not the most effective for
arithmetic learning, and simple formatting changes can significantly improve
accuracy. This leads to sharp phase transitions as a function of training data
scale, which, in some cases, can be explained through connections to low-rank
matrix completion. Building on prior work, we then train on chain-of-thought
style data that includes intermediate step results. Even in the complete
absence of pretraining, this approach significantly and simultaneously improves
accuracy, sample complexity, and convergence speed. We also study the interplay
between arithmetic and text data during training and examine the effects of
few-shot prompting, pretraining, and model scale. Additionally, we discuss
length generalization challenges. Our work highlights the importance of
high-quality, instructive data that considers the particular characteristics of
the next-word prediction objective for rapidly eliciting arithmetic
capabilities.
64ab8e5803cd5dee2ef57fa6
Nayoung Lee
extracted_confirmed
2023-07-10T16:51:49.286Z
64ac36de7b5ff76277ec02a5
/avatars/e128d7bf886cdc29a56751c3a75dd8d5.svg
Nayoung Lee
user
nayoungl95
64ab8e5803cd5dee2ef57fa7
Kartik Sreenivasan
extracted_pending
2023-07-13T04:14:37.143Z
648372e241796f46df60833c
/avatars/0ec44f753d4d2af73b1dafe3e324935a.svg
Kartik Sreenivasan
user
kartikmosaicml
64ab8e5803cd5dee2ef57fa8
Jason D. Lee
64ab8e5803cd5dee2ef57fa9
Kangwook Lee
64ab8e5803cd5dee2ef57faa
Dimitris Papailiopoulos
extracted_confirmed
2023-07-10T04:59:03.003Z
64428cf07feb866811b0be6a
/avatars/f30d339a02ce070c7e77315b59e20359.svg
Dimitris Papailiopoulos
user
anadim
Teaching Arithmetic to Small Transformers
Large language models like GPT-4 exhibit emergent capabilities across
general-purpose tasks, such as basic arithmetic, when trained on extensive text
data, even though these tasks are not explicitly encoded by the unsupervised,
next-token prediction objective. This study investigates how small
transformers, trained from random initialization, can efficiently learn
arithmetic operations such as addition, multiplication, and elementary
functions like square root, using the next-token prediction objective. We first
demonstrate that conventional training data is not the most effective for
arithmetic learning, and simple formatting changes can significantly improve
accuracy. This leads to sharp phase transitions as a function of training data
scale, which, in some cases, can be explained through connections to low-rank
matrix completion. Building on prior work, we then train on chain-of-thought
style data that includes intermediate step results. Even in the complete
absence of pretraining, this approach significantly and simultaneously improves
accuracy, sample complexity, and convergence speed. We also study the interplay
between arithmetic and text data during training and examine the effects of
few-shot prompting, pretraining, and model scale. Additionally, we discuss
length generalization challenges. Our work highlights the importance of
high-quality, instructive data that considers the particular characteristics of
the next-word prediction objective for rapidly eliciting arithmetic
capabilities.
Small transformers trained with chain-of-thought data demonstrate improved arithmetic learning, sample complexity, and convergence speed, highlighting the importance of high-quality, instructive training data.
transformers next-token prediction arithmetic learning chain-of-thought data few-shot prompting pretraining model scale low-rank matrix completion length generalization