title: LM-Infinite: Simple On-the-Fly Length Generalization for Large Language
    Models
summary: In recent years, there have been remarkable advancements in the performance
  of Transformer-based Large Language Models (LLMs) across various domains. As
  these LLMs are deployed for increasingly complex tasks, they often face the
  needs to conduct longer reasoning processes or understanding larger contexts.
  In these situations, the length generalization failure of LLMs on long
  sequences become more prominent. Most pre-training schemes truncate training
  sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
  generate fluent texts, let alone carry out downstream tasks, after longer
  contexts, even with relative positional encoding which is designed to cope with
  this problem. Common solutions such as finetuning on longer corpora often
  involves daunting hardware and time costs and requires careful training process
  design. To more efficiently leverage the generation capacity of existing LLMs,
  we theoretically and empirically investigate the main out-of-distribution (OOD)
  factors contributing to this problem. Inspired by this diagnosis, we propose a
  simple yet effective solution for on-the-fly length generalization,
  LM-Infinite, which involves only a Lambda-shaped attention mask and a
  distance limit while requiring no parameter updates or learning. We find it
  applicable to a variety of LLMs using relative-position encoding methods.
  LM-Infinite is computational efficient with O(n) time and space, and
  demonstrates consistent fluency and generation quality to as long as 32k tokens
  on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
  task such as passkey retrieval, it continues to work on inputs much longer than
  training lengths where vanilla models fail immediately.
paper_authors._id: 64f0077f50dae59768e5bd9a
paper_authors.name: Chi Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-31T14:45:31.933Z
paper_authors._id: 645b11ff5987734177a7cdcf
paper_authors.avatarUrl: /avatars/3b04d2ca166eb28f227fc8d800e55c1c.svg
paper_authors.fullname: Chi Han
paper_authors.type: user
paper_authors.user: Glaciohound
paper_authors._id: 64f0077f50dae59768e5bd9b
paper_authors.name: Qifan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-31T12:07:44.613Z
paper_authors._id: 64b65c84c991daa61404abd7
paper_authors.avatarUrl: /avatars/3592aa8774ea4e2836099e0701b4b78d.svg
paper_authors.fullname: WangQifan
paper_authors.type: user
paper_authors.user: Bayees
paper_authors._id: 64f0077f50dae59768e5bd9c
paper_authors.name: Wenhan Xiong
paper_authors._id: 64f0077f50dae59768e5bd9d
paper_authors.name: Yu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-11T06:54:47.451Z
paper_authors._id: 647e90b31a1fcad2fdc52eb6
paper_authors.avatarUrl: /avatars/d389261a14c3262fd95e241e66c44530.svg
paper_authors.fullname: Hugo Chen
paper_authors.type: user
paper_authors.user: hugochen
paper_authors._id: 64f0077f50dae59768e5bd9e
paper_authors.name: Heng Ji
paper_authors._id: 64f0077f50dae59768e5bd9f
paper_authors.name: Sinong Wang
paper_title: LM-Infinite: Simple On-the-Fly Length Generalization for Large Language
    Models
paper_summary: In recent years, there have been remarkable advancements in the performance
  of Transformer-based Large Language Models (LLMs) across various domains. As
  these LLMs are deployed for increasingly complex tasks, they often face the
  needs to conduct longer reasoning processes or understanding larger contexts.
  In these situations, the length generalization failure of LLMs on long
  sequences become more prominent. Most pre-training schemes truncate training
  sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
  generate fluent texts, let alone carry out downstream tasks, after longer
  contexts, even with relative positional encoding which is designed to cope with
  this problem. Common solutions such as finetuning on longer corpora often
  involves daunting hardware and time costs and requires careful training process
  design. To more efficiently leverage the generation capacity of existing LLMs,
  we theoretically and empirically investigate the main out-of-distribution (OOD)
  factors contributing to this problem. Inspired by this diagnosis, we propose a
  simple yet effective solution for on-the-fly length generalization,
  LM-Infinite, which involves only a Lambda-shaped attention mask and a
  distance limit while requiring no parameter updates or learning. We find it
  applicable to a variety of LLMs using relative-position encoding methods.
  LM-Infinite is computational efficient with O(n) time and space, and
  demonstrates consistent fluency and generation quality to as long as 32k tokens
  on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
  task such as passkey retrieval, it continues to work on inputs much longer than
  training lengths where vanilla models fail immediately.
paper_ai_keywords: Transformer Transformer-based Large Language Models (LLMs) sequence length generalization pre-training positional encoding finetuning OOD factors attention mask distance limit LM-Infinite computational efficiency decoding speedup downstream tasks passkey retrieval