title: Transformers Can Achieve Length Generalization But Not Robustly
summary: Length generalization, defined as the ability to extrapolate from shorter
  training sequences to longer test ones, is a significant challenge for language
  models. This issue persists even with large-scale Transformers handling
  relatively straightforward tasks. In this paper, we test the Transformer's
  ability of length generalization using the task of addition of two integers. We
  show that the success of length generalization is intricately linked to the
  data format and the type of position encoding. Using the right combination of
  data format and position encodings, we show for the first time that standard
  Transformers can extrapolate to a sequence length that is 2.5x the input
  length. Nevertheless, unlike in-distribution generalization, length
  generalization remains fragile, significantly influenced by factors like random
  weight initialization and training data order, leading to large variances
  across different random seeds.
paper_authors._id: 65cd97177b7a844001f5d803
paper_authors.name: Yongchao Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:54:19.590Z
paper_authors._id: 60df063fbb3e1aaa18b005bc
paper_authors.avatarUrl: /avatars/f7210ffd942c124266845bc4d4d89148.svg
paper_authors.fullname: Yongchao Zhou
paper_authors.type: user
paper_authors.user: zycluke
paper_authors._id: 65cd97177b7a844001f5d804
paper_authors.name: Uri Alon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:54:36.778Z
paper_authors._id: 62cd8153248f9e6bc20ab250
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657635637504-62cd8153248f9e6bc20ab250.jpeg
paper_authors.fullname: Uri Alon
paper_authors.type: user
paper_authors.user: urialon
paper_authors._id: 65cd97177b7a844001f5d805
paper_authors.name: Xinyun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:55:04.137Z
paper_authors._id: 64d0268001931c60161e026a
paper_authors.avatarUrl: /avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
paper_authors.fullname: Xinyun Chen
paper_authors.type: user
paper_authors.user: xinyunchen
paper_authors._id: 65cd97177b7a844001f5d806
paper_authors.name: Xuezhi Wang
paper_authors._id: 65cd97177b7a844001f5d807
paper_authors.name: Rishabh Agarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:55:37.591Z
paper_authors._id: 65318c1022350ffeaf7726ef
paper_authors.avatarUrl: /avatars/9359e63d38cb69d08da2de07ae933885.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: rishabhagarwal
paper_authors._id: 65cd97177b7a844001f5d808
paper_authors.name: Denny Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T18:26:33.392Z
paper_authors._id: 65d4d62d493aeaf0bb295f1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
paper_authors.fullname: Denny Zhou
paper_authors.type: user
paper_authors.user: dennyzhou
paper_title: Transformers Can Achieve Length Generalization But Not Robustly
paper_summary: Length generalization, defined as the ability to extrapolate from shorter
  training sequences to longer test ones, is a significant challenge for language
  models. This issue persists even with large-scale Transformers handling
  relatively straightforward tasks. In this paper, we test the Transformer's
  ability of length generalization using the task of addition of two integers. We
  show that the success of length generalization is intricately linked to the
  data format and the type of position encoding. Using the right combination of
  data format and position encodings, we show for the first time that standard
  Transformers can extrapolate to a sequence length that is 2.5x the input
  length. Nevertheless, unlike in-distribution generalization, length
  generalization remains fragile, significantly influenced by factors like random
  weight initialization and training data order, leading to large variances
  across different random seeds.
paper_ai_keywords: Length generalization Transformers Addition of two integers Data format Position encoding Sequence length In-distribution generalization Random weight initialization Training data order