title: Effective Long-Context Scaling of Foundation Models
summary: We present a series of long-context LLMs that support effective context
  windows of up to 32,768 tokens. Our model series are built through continual
  pretraining from Llama 2 with longer training sequences and on a dataset where
  long texts are upsampled. We perform extensive evaluation on language modeling,
  synthetic context probing tasks, and a wide range of research benchmarks. On
  research benchmarks, our models achieve consistent improvements on most regular
  tasks and significant improvements on long-context tasks over Llama 2. Notably,
  with a cost-effective instruction tuning procedure that does not require
  human-annotated long instruction data, the 70B variant can already surpass
  gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks.
  Alongside these results, we provide an in-depth analysis on the individual
  components of our method. We delve into Llama's position encodings and discuss
  its limitation in modeling long dependencies. We also examine the impact of
  various design choices in the pretraining process, including the data mix and
  the training curriculum of sequence lengths -- our ablation experiments suggest
  that having abundant long texts in the pretrain dataset is not the key to
  achieving strong performance, and we empirically verify that long context
  continual pretraining is more efficient and similarly effective compared to
  pretraining from scratch with long sequences.
paper_authors._id: 6516292ec2aedaa7e69f9ba5
paper_authors.name: Wenhan Xiong
paper_authors._id: 6516292ec2aedaa7e69f9ba6
paper_authors.name: Jingyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:50:14.333Z
paper_authors._id: 6278fbf13503c4f835c7ff70
paper_authors.avatarUrl: /avatars/e19960973132ccb0fc0b60a76328b5d8.svg
paper_authors.fullname: Jingyu Liu
paper_authors.type: user
paper_authors.user: Jingyu6
paper_authors._id: 6516292ec2aedaa7e69f9ba7
paper_authors.name: Igor Molybog
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:50:25.671Z
paper_authors._id: 64b7f901fa7eabaae50c5a74
paper_authors.avatarUrl: /avatars/2519dab16608a60bfdc565be5e173779.svg
paper_authors.fullname: Igor Molybog
paper_authors.type: user
paper_authors.user: igormolybog
paper_authors._id: 6516292ec2aedaa7e69f9ba8
paper_authors.name: Hejia Zhang
paper_authors._id: 6516292ec2aedaa7e69f9ba9
paper_authors.name: Prajjwal Bhargava
paper_authors.statusLastChangedAt: 2023-09-29T11:50:50.208Z
paper_authors._id: 6516292ec2aedaa7e69f9baa
paper_authors.name: Rui Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:51:42.158Z
paper_authors._id: 6407d236a7bc7c3865adcc3a
paper_authors.avatarUrl: /avatars/8b75cc07fd049acde5363d268c5b88d4.svg
paper_authors.fullname: Rui Hou
paper_authors.type: user
paper_authors.user: rayhou
paper_authors._id: 6516292ec2aedaa7e69f9bab
paper_authors.name: Louis Martin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:52:00.488Z
paper_authors._id: 64b900dc339adc8f30c683da
paper_authors.avatarUrl: /avatars/823d54764e1a8a5b850fd369262d59e0.svg
paper_authors.fullname: louis Martin
paper_authors.type: user
paper_authors.user: louismartin
paper_authors._id: 6516292ec2aedaa7e69f9bac
paper_authors.name: Rashi Rungta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:52:09.650Z
paper_authors._id: 6227b906e78328793ae0edd9
paper_authors.avatarUrl: /avatars/7ed1773abb419d7862940005b7970d03.svg
paper_authors.fullname: Rashi Rungta
paper_authors.type: user
paper_authors.user: rr654
paper_authors._id: 6516292ec2aedaa7e69f9bad
paper_authors.name: Karthik Abinav Sankararaman
paper_authors._id: 6516292ec2aedaa7e69f9bae
paper_authors.name: Barlas Oguz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:52:31.520Z
paper_authors._id: 6317cb5b212fce5a3cd97227
paper_authors.avatarUrl: /avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
paper_authors.fullname: Barlas Oguz
paper_authors.type: user
paper_authors.user: barlaso
paper_authors._id: 6516292ec2aedaa7e69f9baf
paper_authors.name: Madian Khabsa
paper_authors._id: 6516292ec2aedaa7e69f9bb0
paper_authors.name: Han Fang
paper_authors._id: 6516292ec2aedaa7e69f9bb1
paper_authors.name: Yashar Mehdad
paper_authors._id: 6516292ec2aedaa7e69f9bb2
paper_authors.name: Sharan Narang
paper_authors._id: 6516292ec2aedaa7e69f9bb3
paper_authors.name: Kshitiz Malik
paper_authors._id: 6516292ec2aedaa7e69f9bb4
paper_authors.name: Angela Fan
paper_authors._id: 6516292ec2aedaa7e69f9bb5
paper_authors.name: Shruti Bhosale
paper_authors._id: 6516292ec2aedaa7e69f9bb6
paper_authors.name: Sergey Edunov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:53:08.499Z
paper_authors._id: 639a1b1857595ea3838b4ed7
paper_authors.avatarUrl: /avatars/1cda3a496726dbf50ce0bc00ca1004a7.svg
paper_authors.fullname: Sergey Edunov
paper_authors.type: user
paper_authors.user: edunov
paper_authors._id: 6516292ec2aedaa7e69f9bb7
paper_authors.name: Mike Lewis
paper_authors._id: 6516292ec2aedaa7e69f9bb8
paper_authors.name: Sinong Wang
paper_authors._id: 6516292ec2aedaa7e69f9bb9
paper_authors.name: Hao Ma
paper_title: Effective Long-Context Scaling of Foundation Models
paper_summary: We present a series of long-context LLMs that support effective context
  windows of up to 32,768 tokens. Our model series are built through continual
  pretraining from Llama 2 with longer training sequences and on a dataset where
  long texts are upsampled. We perform extensive evaluation on language modeling,
  synthetic context probing tasks, and a wide range of research benchmarks. On
  research benchmarks, our models achieve consistent improvements on most regular
  tasks and significant improvements on long-context tasks over Llama 2. Notably,
  with a cost-effective instruction tuning procedure that does not require
  human-annotated long instruction data, the 70B variant can already surpass
  gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks.
  Alongside these results, we provide an in-depth analysis on the individual
  components of our method. We delve into Llama's position encodings and discuss
  its limitation in modeling long dependencies. We also examine the impact of
  various design choices in the pretraining process, including the data mix and
  the training curriculum of sequence lengths -- our ablation experiments suggest
  that having abundant long texts in the pretrain dataset is not the key to
  achieving strong performance, and we empirically verify that long context
  continual pretraining is more efficient and similarly effective compared to
  pretraining from scratch with long sequences.
paper_ai_summary: Long-context LLMs achieve significant advancements in handling extended contexts through continual pretraining and efficient instruction tuning, surpassing existing models on various benchmarks.
paper_ai_keywords: LLMs long-context Llama 2 continual pretraining usample language modeling synthetic context probing research benchmarks instruction tuning gpt-3.5-turbo-16k position encodings data mix training curriculum long context continual pretraining