Randomized Positional Encodings Boost Length Generalization of
  Transformers
Transformers have impressive generalization capabilities on tasks with a
fixed context length. However, they fail to generalize to sequences of
arbitrary length, even for seemingly simple tasks such as duplicating a string.
Moreover, simply training on longer sequences is inefficient due to the
quadratic computation complexity of the global attention mechanism. In this
work, we demonstrate that this failure mode is linked to positional encodings
being out-of-distribution for longer sequences (even for relative encodings)
and introduce a novel family of positional encodings that can overcome this
problem. Concretely, our randomized positional encoding scheme simulates the
positions of longer sequences and randomly selects an ordered subset to fit the
sequence's length. Our large-scale empirical evaluation of 6000 models across
15 algorithmic reasoning tasks shows that our method allows Transformers to
generalize to sequences of unseen length (increasing test accuracy by 12.0% on
average).
647431747be71eb8b1bad177
Anian Ruoss
claimed_verified
2024-02-09T08:06:51.927Z
65c51758522ee3a4edd917d0
/avatars/2ea3b64b5eeb34e2bfe821e4210f1b3e.svg
Anian Ruoss
user
anianruoss
647431747be71eb8b1bad178
Grégoire Delétang
647431747be71eb8b1bad179
Tim Genewein
647431747be71eb8b1bad17a
Jordi Grau-Moya
647431747be71eb8b1bad17b
Róbert Csordás
claimed_verified
2025-03-11T08:26:23.823Z
64e5d1bef450dca5ca0d05ed
https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
Robert Csordas
user
robertcsordas
647431747be71eb8b1bad17c
Mehdi Bennani
647431747be71eb8b1bad17d
Shane Legg
647431747be71eb8b1bad17e
Joel Veness
Randomized Positional Encodings Boost Length Generalization of
  Transformers
Transformers have impressive generalization capabilities on tasks with a
fixed context length. However, they fail to generalize to sequences of
arbitrary length, even for seemingly simple tasks such as duplicating a string.
Moreover, simply training on longer sequences is inefficient due to the
quadratic computation complexity of the global attention mechanism. In this
work, we demonstrate that this failure mode is linked to positional encodings
being out-of-distribution for longer sequences (even for relative encodings)
and introduce a novel family of positional encodings that can overcome this
problem. Concretely, our randomized positional encoding scheme simulates the
positions of longer sequences and randomly selects an ordered subset to fit the
sequence's length. Our large-scale empirical evaluation of 6000 models across
15 algorithmic reasoning tasks shows that our method allows Transformers to
generalize to sequences of unseen length (increasing test accuracy by 12.0% on
average).
A new positional encoding scheme enables Transformers to generalize to longer sequences by simulating and randomly selecting positions, improving test accuracy on algorithmic reasoning tasks.
Transformers positional encodings global attention mechanism out-of-distribution randomized positional encoding