Effective Long-Context Scaling of Foundation Models
We present a series of long-context LLMs that support effective context
windows of up to 32,768 tokens. Our model series are built through continual
pretraining from Llama 2 with longer training sequences and on a dataset where
long texts are upsampled. We perform extensive evaluation on language modeling,
synthetic context probing tasks, and a wide range of research benchmarks. On
research benchmarks, our models achieve consistent improvements on most regular
tasks and significant improvements on long-context tasks over Llama 2. Notably,
with a cost-effective instruction tuning procedure that does not require
human-annotated long instruction data, the 70B variant can already surpass
gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks.
Alongside these results, we provide an in-depth analysis on the individual
components of our method. We delve into Llama's position encodings and discuss
its limitation in modeling long dependencies. We also examine the impact of
various design choices in the pretraining process, including the data mix and
the training curriculum of sequence lengths -- our ablation experiments suggest
that having abundant long texts in the pretrain dataset is not the key to
achieving strong performance, and we empirically verify that long context
continual pretraining is more efficient and similarly effective compared to
pretraining from scratch with long sequences.
6516292ec2aedaa7e69f9ba5
Wenhan Xiong
6516292ec2aedaa7e69f9ba6
Jingyu Liu
admin_assigned
2023-09-29T11:50:14.333Z
6278fbf13503c4f835c7ff70
/avatars/e19960973132ccb0fc0b60a76328b5d8.svg
Jingyu Liu
user
Jingyu6
6516292ec2aedaa7e69f9ba7
Igor Molybog
admin_assigned
2023-09-29T11:50:25.671Z
64b7f901fa7eabaae50c5a74
/avatars/2519dab16608a60bfdc565be5e173779.svg
Igor Molybog
user
igormolybog
6516292ec2aedaa7e69f9ba8
Hejia Zhang
6516292ec2aedaa7e69f9ba9
Prajjwal Bhargava
2023-09-29T11:50:50.208Z
6516292ec2aedaa7e69f9baa
Rui Hou
admin_assigned
2023-09-29T11:51:42.158Z
6407d236a7bc7c3865adcc3a
/avatars/8b75cc07fd049acde5363d268c5b88d4.svg
Rui Hou
user
rayhou
6516292ec2aedaa7e69f9bab
Louis Martin
admin_assigned
2023-09-29T11:52:00.488Z
64b900dc339adc8f30c683da
/avatars/823d54764e1a8a5b850fd369262d59e0.svg
louis Martin
user
louismartin
6516292ec2aedaa7e69f9bac
Rashi Rungta
admin_assigned
2023-09-29T11:52:09.650Z
6227b906e78328793ae0edd9
/avatars/7ed1773abb419d7862940005b7970d03.svg
Rashi Rungta
user
rr654
6516292ec2aedaa7e69f9bad
Karthik Abinav Sankararaman
6516292ec2aedaa7e69f9bae
Barlas Oguz
admin_assigned
2023-09-29T11:52:31.520Z
6317cb5b212fce5a3cd97227
/avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
Barlas Oguz
user
barlaso
6516292ec2aedaa7e69f9baf
Madian Khabsa
6516292ec2aedaa7e69f9bb0
Han Fang
6516292ec2aedaa7e69f9bb1
Yashar Mehdad
6516292ec2aedaa7e69f9bb2
Sharan Narang
6516292ec2aedaa7e69f9bb3
Kshitiz Malik
6516292ec2aedaa7e69f9bb4
Angela Fan
6516292ec2aedaa7e69f9bb5
Shruti Bhosale
6516292ec2aedaa7e69f9bb6
Sergey Edunov
admin_assigned
2023-09-29T11:53:08.499Z
639a1b1857595ea3838b4ed7
/avatars/1cda3a496726dbf50ce0bc00ca1004a7.svg
Sergey Edunov
user
edunov
6516292ec2aedaa7e69f9bb7
Mike Lewis
6516292ec2aedaa7e69f9bb8
Sinong Wang
6516292ec2aedaa7e69f9bb9
Hao Ma
Effective Long-Context Scaling of Foundation Models
We present a series of long-context LLMs that support effective context
windows of up to 32,768 tokens. Our model series are built through continual
pretraining from Llama 2 with longer training sequences and on a dataset where
long texts are upsampled. We perform extensive evaluation on language modeling,
synthetic context probing tasks, and a wide range of research benchmarks. On
research benchmarks, our models achieve consistent improvements on most regular
tasks and significant improvements on long-context tasks over Llama 2. Notably,
with a cost-effective instruction tuning procedure that does not require
human-annotated long instruction data, the 70B variant can already surpass
gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks.
Alongside these results, we provide an in-depth analysis on the individual
components of our method. We delve into Llama's position encodings and discuss
its limitation in modeling long dependencies. We also examine the impact of
various design choices in the pretraining process, including the data mix and
the training curriculum of sequence lengths -- our ablation experiments suggest
that having abundant long texts in the pretrain dataset is not the key to
achieving strong performance, and we empirically verify that long context
continual pretraining is more efficient and similarly effective compared to
pretraining from scratch with long sequences.
Long-context LLMs achieve significant advancements in handling extended contexts through continual pretraining and efficient instruction tuning, surpassing existing models on various benchmarks.
LLMs long-context Llama 2 continual pretraining usample language modeling synthetic context probing research benchmarks instruction tuning gpt-3.5-turbo-16k position encodings data mix training curriculum long context continual pretraining