SkyLadder: Better and Faster Pretraining via Context Window Scheduling
Recent advancements in LLM pretraining have featured ever-expanding context
windows to process longer sequences. However, our pilot study reveals that
models pretrained with shorter context windows consistently outperform their
long-context counterparts under a fixed token budget. This finding motivates us
to explore an optimal context window scheduling strategy to better balance
long-context capability with pretraining efficiency. To this end, we propose
SkyLadder, a simple yet effective approach that implements a short-to-long
context window transition. SkyLadder preserves strong standard benchmark
performance, while matching or exceeding baseline results on long context
tasks. Through extensive experiments, we pre-train 1B-parameter models (up to
32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating
that SkyLadder yields consistent gains of up to 3.7% on common benchmarks,
while achieving up to 22% faster training speeds compared to baselines. The
code is at https://github.com/sail-sg/SkyLadder.
67dc44a3a19fb08b924a9ea6
Tongyao Zhu
claimed_verified
2025-03-25T08:21:27.446Z
647db42711084fb58318b053
/avatars/901a4db8e99f6e0c7e7e596c251b315c.svg
Tongyao
user
tyzhu
67dc44a3a19fb08b924a9ea7
Qian Liu
claimed_verified
2025-03-20T19:16:58.740Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
67dc44a3a19fb08b924a9ea8
Haonan Wang
67dc44a3a19fb08b924a9ea9
Shiqi Chen
claimed_verified
2025-03-21T11:41:42.509Z
62bf99cae140faaa85335ab8
/avatars/bcd2aa1823995b385096e2a68ce3e071.svg
Shiqi Chen
user
ShiqiChen
67dc44a3a19fb08b924a9eaa
Xiangming Gu
67dc44a3a19fb08b924a9eab
Tianyu Pang
admin_assigned
2025-03-20T19:30:51.650Z
661a4a556fb488fa078c60aa
/avatars/c77401fa9c6d2db896b4a337bb3f8add.svg
Tianyu Pang
user
TIanyupang
67dc44a3a19fb08b924a9eac
Min-Yen Kan
admin_assigned
2025-03-20T19:30:38.894Z
628f137053028c04a3b750e4
/avatars/85d174cc041ab91e724bb9d9d4e46c88.svg
Min-Yen Kan
user
knmnyn
SkyLadder: Better and Faster Pretraining via Context Window Scheduling
Recent advancements in LLM pretraining have featured ever-expanding context
windows to process longer sequences. However, our pilot study reveals that
models pretrained with shorter context windows consistently outperform their
long-context counterparts under a fixed token budget. This finding motivates us
to explore an optimal context window scheduling strategy to better balance
long-context capability with pretraining efficiency. To this end, we propose
SkyLadder, a simple yet effective approach that implements a short-to-long
context window transition. SkyLadder preserves strong standard benchmark
performance, while matching or exceeding baseline results on long context
tasks. Through extensive experiments, we pre-train 1B-parameter models (up to
32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating
that SkyLadder yields consistent gains of up to 3.7% on common benchmarks,
while achieving up to 22% faster training speeds compared to baselines. The
code is at https://github.com/sail-sg/SkyLadder.
SkyLadder, a short-to-long context window transition strategy, enhances LLM pretraining efficiency while maintaining or surpassing long-context performance.
SkyLadder context window LLM pretraining standard benchmark long-context tasks parameter-efficient fine-tuning token budget pretraining efficiency