title: LongNet: Scaling Transformers to 1,000,000,000 Tokens
summary: Scaling sequence length has become a critical demand in the era of large
  language models. However, existing methods struggle with either computational
  complexity or model expressivity, rendering the maximum sequence length
  restricted. In this work, we introduce LongNet, a Transformer variant that can
  scale sequence length to more than 1 billion tokens, without sacrificing the
  performance on shorter sequences. Specifically, we propose dilated attention,
  which expands the attentive field exponentially as the distance grows. LongNet
  has significant advantages: 1) it has a linear computation complexity and a
  logarithm dependency between tokens; 2) it can be served as a distributed
  trainer for extremely long sequences; 3) its dilated attention is a drop-in
  replacement for standard attention, which can be seamlessly integrated with the
  existing Transformer-based optimization. Experiments results demonstrate that
  LongNet yields strong performance on both long-sequence modeling and general
  language tasks. Our work opens up new possibilities for modeling very long
  sequences, e.g., treating a whole corpus or even the entire Internet as a
  sequence.
paper_authors._id: 64a6234ba3227ff52076ea25
paper_authors.name: Jiayu Ding
paper_authors._id: 64a6234ba3227ff52076ea26
paper_authors.name: Shuming Ma
paper_authors._id: 64a6234ba3227ff52076ea27
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:20:01.888Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 64a6234ba3227ff52076ea28
paper_authors.name: Xingxing Zhang
paper_authors._id: 64a6234ba3227ff52076ea29
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-06T12:15:34.559Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 64a6234ba3227ff52076ea2a
paper_authors.name: Wenhui Wang
paper_authors._id: 64a6234ba3227ff52076ea2b
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-06T12:17:15.965Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: LongNet: Scaling Transformers to 1,000,000,000 Tokens
paper_summary: Scaling sequence length has become a critical demand in the era of large
  language models. However, existing methods struggle with either computational
  complexity or model expressivity, rendering the maximum sequence length
  restricted. In this work, we introduce LongNet, a Transformer variant that can
  scale sequence length to more than 1 billion tokens, without sacrificing the
  performance on shorter sequences. Specifically, we propose dilated attention,
  which expands the attentive field exponentially as the distance grows. LongNet
  has significant advantages: 1) it has a linear computation complexity and a
  logarithm dependency between tokens; 2) it can be served as a distributed
  trainer for extremely long sequences; 3) its dilated attention is a drop-in
  replacement for standard attention, which can be seamlessly integrated with the
  existing Transformer-based optimization. Experiments results demonstrate that
  LongNet yields strong performance on both long-sequence modeling and general
  language tasks. Our work opens up new possibilities for modeling very long
  sequences, e.g., treating a whole corpus or even the entire Internet as a
  sequence.
paper_ai_keywords: LongNet Transformers dilated attention attentive field linear computation complexity logarithm dependency distributed sequence trainer standard attention drop-in replacement long-sequence modeling