title: Transformers Can Do Arithmetic with the Right Embeddings
summary: The poor performance of transformers on arithmetic tasks seems to stem in
  large part from their inability to keep track of the exact position of each
  digit inside of a large span of digits. We mend this problem by adding an
  embedding to each digit that encodes its position relative to the start of the
  number. In addition to the boost these embeddings provide on their own, we show
  that this fix enables architectural modifications such as input injection and
  recurrent layers to improve performance even further.
    With positions resolved, we can study the logical extrapolation ability of
  transformers. Can they solve arithmetic problems that are larger and more
  complex than those in their training data? We find that training on only 20
  digit numbers with a single GPU for one day, we can reach state-of-the-art
  performance, achieving up to 99% accuracy on 100 digit addition problems.
  Finally, we show that these gains in numeracy also unlock improvements on other
  multi-step reasoning tasks including sorting and multiplication.
paper_authors._id: 66555b26480d7e1ccba35ded
paper_authors.name: Sean McLeish
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-28T12:27:29.320Z
paper_authors._id: 65255f1073a043e50d043641
paper_authors.avatarUrl: /avatars/257085f01c439d7c84787a4e6d085b3d.svg
paper_authors.fullname: Sean McLeish
paper_authors.type: user
paper_authors.user: smcleish
paper_authors._id: 66555b26480d7e1ccba35dee
paper_authors.name: Arpit Bansal
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-28T04:32:15.010Z
paper_authors._id: 631b6f605b628ccf3ce676f3
paper_authors.avatarUrl: /avatars/047cb8ec70b89d732340ac2c559552d8.svg
paper_authors.fullname: Arpit Bansal
paper_authors.type: user
paper_authors.user: ArpitB
paper_authors._id: 66555b26480d7e1ccba35def
paper_authors.name: Alex Stein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T15:21:37.490Z
paper_authors._id: 63b88eaaaaa8cf17f097c205
paper_authors.avatarUrl: /avatars/c9f6f3c3ac86def2650d1ecc473cd486.svg
paper_authors.fullname: Alex Stein
paper_authors.type: user
paper_authors.user: astein0
paper_authors._id: 66555b26480d7e1ccba35df0
paper_authors.name: Neel Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:56:33.803Z
paper_authors._id: 63e2b1ec282ee5f9624cfbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
paper_authors.fullname: Neel Jain
paper_authors.type: user
paper_authors.user: nsjain
paper_authors._id: 66555b26480d7e1ccba35df1
paper_authors.name: John Kirchenbauer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:56:41.349Z
paper_authors._id: 63d98af1897746d6496177df
paper_authors.avatarUrl: /avatars/c5d0031c796a3c11bcb0d01b959168dc.svg
paper_authors.fullname: John Kirchenbauer
paper_authors.type: user
paper_authors.user: jwkirchenbauer
paper_authors._id: 66555b26480d7e1ccba35df2
paper_authors.name: Brian R. Bartoldson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:57:35.518Z
paper_authors._id: 641a7ecb91e3376a057adf89
paper_authors.avatarUrl: /avatars/f407adae3e9df964a469cfb350115eb8.svg
paper_authors.fullname: brian bartoldson
paper_authors.type: user
paper_authors.user: bartoldson
paper_authors._id: 66555b26480d7e1ccba35df3
paper_authors.name: Bhavya Kailkhura
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:57:42.408Z
paper_authors._id: 65cb79db6427380bc21261e2
paper_authors.avatarUrl: /avatars/a003eb5d0955417329c1a4170ae65879.svg
paper_authors.fullname: Bhavya Kailkhura
paper_authors.type: user
paper_authors.user: bhavyakailkhura
paper_authors._id: 66555b26480d7e1ccba35df4
paper_authors.name: Abhinav Bhatele
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:57:48.718Z
paper_authors._id: 6361d9ce6bd72c97d005b4db
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6361d9ce6bd72c97d005b4db/vjKaW2JFavVffoRxwgFwn.jpeg
paper_authors.fullname: Abhinav Bhatele
paper_authors.type: user
paper_authors.user: bhatele
paper_authors._id: 66555b26480d7e1ccba35df5
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:57:55.630Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 66555b26480d7e1ccba35df6
paper_authors.name: Avi Schwarzschild
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:58:01.343Z
paper_authors._id: 6570917c0ea91e592aff0b8c
paper_authors.avatarUrl: /avatars/529e9713e6ac835e11599ea7070a9603.svg
paper_authors.fullname: Avi Schwarzschild
paper_authors.type: user
paper_authors.user: schwarzschild
paper_authors._id: 66555b26480d7e1ccba35df7
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:58:06.890Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_title: Transformers Can Do Arithmetic with the Right Embeddings
paper_summary: The poor performance of transformers on arithmetic tasks seems to stem in
  large part from their inability to keep track of the exact position of each
  digit inside of a large span of digits. We mend this problem by adding an
  embedding to each digit that encodes its position relative to the start of the
  number. In addition to the boost these embeddings provide on their own, we show
  that this fix enables architectural modifications such as input injection and
  recurrent layers to improve performance even further.
    With positions resolved, we can study the logical extrapolation ability of
  transformers. Can they solve arithmetic problems that are larger and more
  complex than those in their training data? We find that training on only 20
  digit numbers with a single GPU for one day, we can reach state-of-the-art
  performance, achieving up to 99% accuracy on 100 digit addition problems.
  Finally, we show that these gains in numeracy also unlock improvements on other
  multi-step reasoning tasks including sorting and multiplication.
paper_ai_summary: Transformers achieve state-of-the-art performance on large arithmetic tasks and other reasoning tasks by addressing positional tracking with embeddings and integrating architectural modifications.
paper_ai_keywords: transformers positional tracking embeddings input injection recurrent layers logical extrapolation multi-step reasoning tasks