Length Generalization in Arithmetic Transformers
We examine how transformers cope with two challenges: learning basic integer
arithmetic, and generalizing to longer sequences than seen during training. We
find that relative position embeddings enable length generalization for simple
tasks, such as addition: models trained on 5-digit numbers can perform
15-digit sums. However, this method fails for multiplication, and we propose
train set priming: adding a few (10 to 50) long sequences to the training
set. We show that priming allows models trained on 5-digit times 3-digit
multiplications to generalize to 35times 3 examples. We also show that
models can be primed for different generalization lengths, and that the priming
sample size scales as the logarithm of the training set size. Finally, we
discuss potential applications of priming beyond arithmetic.
649b9a6bf0ea0134c5789c88
Samy Jelassi
admin_assigned
2023-06-28T07:57:42.031Z
643740cab8ea86e503291f18
/avatars/46a7e6ba97c89215aa89ba67e61e17e7.svg
Samy Jelassi
user
sjelassi
649b9a6bf0ea0134c5789c89
Stéphane d'Ascoli
649b9a6bf0ea0134c5789c8a
Carles Domingo-Enrich
649b9a6bf0ea0134c5789c8b
Yuhuai Wu
649b9a6bf0ea0134c5789c8c
Yuanzhi Li
admin_assigned
2023-06-28T07:58:15.478Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
649b9a6bf0ea0134c5789c8d
François Charton
Length Generalization in Arithmetic Transformers
We examine how transformers cope with two challenges: learning basic integer
arithmetic, and generalizing to longer sequences than seen during training. We
find that relative position embeddings enable length generalization for simple
tasks, such as addition: models trained on 5-digit numbers can perform
15-digit sums. However, this method fails for multiplication, and we propose
train set priming: adding a few (10 to 50) long sequences to the training
set. We show that priming allows models trained on 5-digit times 3-digit
multiplications to generalize to 35times 3 examples. We also show that
models can be primed for different generalization lengths, and that the priming
sample size scales as the logarithm of the training set size. Finally, we
discuss potential applications of priming beyond arithmetic.
Transformers use relative position embeddings for length generalization in simple tasks like addition and propose train set priming to achieve generalization in complex tasks like multiplication.
transformers relative position embeddings length generalization addition multiplication train set priming