Transformers Can Do Arithmetic with the Right Embeddings
The poor performance of transformers on arithmetic tasks seems to stem in
large part from their inability to keep track of the exact position of each
digit inside of a large span of digits. We mend this problem by adding an
embedding to each digit that encodes its position relative to the start of the
number. In addition to the boost these embeddings provide on their own, we show
that this fix enables architectural modifications such as input injection and
recurrent layers to improve performance even further.
  With positions resolved, we can study the logical extrapolation ability of
transformers. Can they solve arithmetic problems that are larger and more
complex than those in their training data? We find that training on only 20
digit numbers with a single GPU for one day, we can reach state-of-the-art
performance, achieving up to 99% accuracy on 100 digit addition problems.
Finally, we show that these gains in numeracy also unlock improvements on other
multi-step reasoning tasks including sorting and multiplication.
66555b26480d7e1ccba35ded
Sean McLeish
extracted_confirmed
2024-05-28T12:27:29.320Z
65255f1073a043e50d043641
/avatars/257085f01c439d7c84787a4e6d085b3d.svg
Sean McLeish
user
smcleish
66555b26480d7e1ccba35dee
Arpit Bansal
extracted_confirmed
2024-05-28T04:32:15.010Z
631b6f605b628ccf3ce676f3
/avatars/047cb8ec70b89d732340ac2c559552d8.svg
Arpit Bansal
user
ArpitB
66555b26480d7e1ccba35def
Alex Stein
claimed_verified
2024-05-28T15:21:37.490Z
63b88eaaaaa8cf17f097c205
/avatars/c9f6f3c3ac86def2650d1ecc473cd486.svg
Alex Stein
user
astein0
66555b26480d7e1ccba35df0
Neel Jain
admin_assigned
2024-05-28T09:56:33.803Z
63e2b1ec282ee5f9624cfbcb
https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
Neel Jain
user
nsjain
66555b26480d7e1ccba35df1
John Kirchenbauer
admin_assigned
2024-05-28T09:56:41.349Z
63d98af1897746d6496177df
/avatars/c5d0031c796a3c11bcb0d01b959168dc.svg
John Kirchenbauer
user
jwkirchenbauer
66555b26480d7e1ccba35df2
Brian R. Bartoldson
admin_assigned
2024-05-28T09:57:35.518Z
641a7ecb91e3376a057adf89
/avatars/f407adae3e9df964a469cfb350115eb8.svg
brian bartoldson
user
bartoldson
66555b26480d7e1ccba35df3
Bhavya Kailkhura
admin_assigned
2024-05-28T09:57:42.408Z
65cb79db6427380bc21261e2
/avatars/a003eb5d0955417329c1a4170ae65879.svg
Bhavya Kailkhura
user
bhavyakailkhura
66555b26480d7e1ccba35df4
Abhinav Bhatele
admin_assigned
2024-05-28T09:57:48.718Z
6361d9ce6bd72c97d005b4db
https://cdn-avatars.huggingface.co/v1/production/uploads/6361d9ce6bd72c97d005b4db/vjKaW2JFavVffoRxwgFwn.jpeg
Abhinav Bhatele
user
bhatele
66555b26480d7e1ccba35df5
Jonas Geiping
admin_assigned
2024-05-28T09:57:55.630Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
66555b26480d7e1ccba35df6
Avi Schwarzschild
admin_assigned
2024-05-28T09:58:01.343Z
6570917c0ea91e592aff0b8c
/avatars/529e9713e6ac835e11599ea7070a9603.svg
Avi Schwarzschild
user
schwarzschild
66555b26480d7e1ccba35df7
Tom Goldstein
admin_assigned
2024-05-28T09:58:06.890Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
Transformers Can Do Arithmetic with the Right Embeddings
The poor performance of transformers on arithmetic tasks seems to stem in
large part from their inability to keep track of the exact position of each
digit inside of a large span of digits. We mend this problem by adding an
embedding to each digit that encodes its position relative to the start of the
number. In addition to the boost these embeddings provide on their own, we show
that this fix enables architectural modifications such as input injection and
recurrent layers to improve performance even further.
  With positions resolved, we can study the logical extrapolation ability of
transformers. Can they solve arithmetic problems that are larger and more
complex than those in their training data? We find that training on only 20
digit numbers with a single GPU for one day, we can reach state-of-the-art
performance, achieving up to 99% accuracy on 100 digit addition problems.
Finally, we show that these gains in numeracy also unlock improvements on other
multi-step reasoning tasks including sorting and multiplication.
Transformers achieve state-of-the-art performance on large arithmetic tasks and other reasoning tasks by addressing positional tracking with embeddings and integrating architectural modifications.
transformers positional tracking embeddings input injection recurrent layers logical extrapolation multi-step reasoning tasks