title: Length Generalization in Arithmetic Transformers
summary: We examine how transformers cope with two challenges: learning basic integer
  arithmetic, and generalizing to longer sequences than seen during training. We
  find that relative position embeddings enable length generalization for simple
  tasks, such as addition: models trained on 5-digit numbers can perform
  15-digit sums. However, this method fails for multiplication, and we propose
  train set priming: adding a few (10 to 50) long sequences to the training
  set. We show that priming allows models trained on 5-digit times 3-digit
  multiplications to generalize to 35times 3 examples. We also show that
  models can be primed for different generalization lengths, and that the priming
  sample size scales as the logarithm of the training set size. Finally, we
  discuss potential applications of priming beyond arithmetic.
paper_authors._id: 649b9a6bf0ea0134c5789c88
paper_authors.name: Samy Jelassi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-28T07:57:42.031Z
paper_authors._id: 643740cab8ea86e503291f18
paper_authors.avatarUrl: /avatars/46a7e6ba97c89215aa89ba67e61e17e7.svg
paper_authors.fullname: Samy Jelassi
paper_authors.type: user
paper_authors.user: sjelassi
paper_authors._id: 649b9a6bf0ea0134c5789c89
paper_authors.name: Stéphane d'Ascoli
paper_authors._id: 649b9a6bf0ea0134c5789c8a
paper_authors.name: Carles Domingo-Enrich
paper_authors._id: 649b9a6bf0ea0134c5789c8b
paper_authors.name: Yuhuai Wu
paper_authors._id: 649b9a6bf0ea0134c5789c8c
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-28T07:58:15.478Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_authors._id: 649b9a6bf0ea0134c5789c8d
paper_authors.name: François Charton
paper_title: Length Generalization in Arithmetic Transformers
paper_summary: We examine how transformers cope with two challenges: learning basic integer
  arithmetic, and generalizing to longer sequences than seen during training. We
  find that relative position embeddings enable length generalization for simple
  tasks, such as addition: models trained on 5-digit numbers can perform
  15-digit sums. However, this method fails for multiplication, and we propose
  train set priming: adding a few (10 to 50) long sequences to the training
  set. We show that priming allows models trained on 5-digit times 3-digit
  multiplications to generalize to 35times 3 examples. We also show that
  models can be primed for different generalization lengths, and that the priming
  sample size scales as the logarithm of the training set size. Finally, we
  discuss potential applications of priming beyond arithmetic.
paper_ai_keywords: transformers relative position embeddings length generalization integer arithmetic addition multiplication train set priming