title: Randomized Positional Encodings Boost Length Generalization of
    Transformers
summary: Transformers have impressive generalization capabilities on tasks with a
  fixed context length. However, they fail to generalize to sequences of
  arbitrary length, even for seemingly simple tasks such as duplicating a string.
  Moreover, simply training on longer sequences is inefficient due to the
  quadratic computation complexity of the global attention mechanism. In this
  work, we demonstrate that this failure mode is linked to positional encodings
  being out-of-distribution for longer sequences (even for relative encodings)
  and introduce a novel family of positional encodings that can overcome this
  problem. Concretely, our randomized positional encoding scheme simulates the
  positions of longer sequences and randomly selects an ordered subset to fit the
  sequence's length. Our large-scale empirical evaluation of 6000 models across
  15 algorithmic reasoning tasks shows that our method allows Transformers to
  generalize to sequences of unseen length (increasing test accuracy by 12.0% on
  average).
paper_authors._id: 647431747be71eb8b1bad177
paper_authors.name: Anian Ruoss
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:06:51.927Z
paper_authors._id: 65c51758522ee3a4edd917d0
paper_authors.avatarUrl: /avatars/2ea3b64b5eeb34e2bfe821e4210f1b3e.svg
paper_authors.fullname: Anian Ruoss
paper_authors.type: user
paper_authors.user: anianruoss
paper_authors._id: 647431747be71eb8b1bad178
paper_authors.name: Grégoire Delétang
paper_authors._id: 647431747be71eb8b1bad179
paper_authors.name: Tim Genewein
paper_authors._id: 647431747be71eb8b1bad17a
paper_authors.name: Jordi Grau-Moya
paper_authors._id: 647431747be71eb8b1bad17b
paper_authors.name: Róbert Csordás
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:26:23.823Z
paper_authors._id: 64e5d1bef450dca5ca0d05ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
paper_authors.fullname: Robert Csordas
paper_authors.type: user
paper_authors.user: robertcsordas
paper_authors._id: 647431747be71eb8b1bad17c
paper_authors.name: Mehdi Bennani
paper_authors._id: 647431747be71eb8b1bad17d
paper_authors.name: Shane Legg
paper_authors._id: 647431747be71eb8b1bad17e
paper_authors.name: Joel Veness
paper_title: Randomized Positional Encodings Boost Length Generalization of
    Transformers
paper_summary: Transformers have impressive generalization capabilities on tasks with a
  fixed context length. However, they fail to generalize to sequences of
  arbitrary length, even for seemingly simple tasks such as duplicating a string.
  Moreover, simply training on longer sequences is inefficient due to the
  quadratic computation complexity of the global attention mechanism. In this
  work, we demonstrate that this failure mode is linked to positional encodings
  being out-of-distribution for longer sequences (even for relative encodings)
  and introduce a novel family of positional encodings that can overcome this
  problem. Concretely, our randomized positional encoding scheme simulates the
  positions of longer sequences and randomly selects an ordered subset to fit the
  sequence's length. Our large-scale empirical evaluation of 6000 models across
  15 algorithmic reasoning tasks shows that our method allows Transformers to
  generalize to sequences of unseen length (increasing test accuracy by 12.0% on
  average).
paper_ai_keywords: Transformers generalization capabilities multimodal length duplicating a string global attention mechanism positional encodings relative encodings randomized positional encoding sequence's length downstream accuracy destructive generalization