title: SkyLadder: Better and Faster Pretraining via Context Window Scheduling
summary: Recent advancements in LLM pretraining have featured ever-expanding context
  windows to process longer sequences. However, our pilot study reveals that
  models pretrained with shorter context windows consistently outperform their
  long-context counterparts under a fixed token budget. This finding motivates us
  to explore an optimal context window scheduling strategy to better balance
  long-context capability with pretraining efficiency. To this end, we propose
  SkyLadder, a simple yet effective approach that implements a short-to-long
  context window transition. SkyLadder preserves strong standard benchmark
  performance, while matching or exceeding baseline results on long context
  tasks. Through extensive experiments, we pre-train 1B-parameter models (up to
  32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating
  that SkyLadder yields consistent gains of up to 3.7% on common benchmarks,
  while achieving up to 22% faster training speeds compared to baselines. The
  code is at https://github.com/sail-sg/SkyLadder.
paper_authors._id: 67dc44a3a19fb08b924a9ea6
paper_authors.name: Tongyao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:21:27.446Z
paper_authors._id: 647db42711084fb58318b053
paper_authors.avatarUrl: /avatars/901a4db8e99f6e0c7e7e596c251b315c.svg
paper_authors.fullname: Tongyao
paper_authors.type: user
paper_authors.user: tyzhu
paper_authors._id: 67dc44a3a19fb08b924a9ea7
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T19:16:58.740Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 67dc44a3a19fb08b924a9ea8
paper_authors.name: Haonan Wang
paper_authors._id: 67dc44a3a19fb08b924a9ea9
paper_authors.name: Shiqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:41:42.509Z
paper_authors._id: 62bf99cae140faaa85335ab8
paper_authors.avatarUrl: /avatars/bcd2aa1823995b385096e2a68ce3e071.svg
paper_authors.fullname: Shiqi Chen
paper_authors.type: user
paper_authors.user: ShiqiChen
paper_authors._id: 67dc44a3a19fb08b924a9eaa
paper_authors.name: Xiangming Gu
paper_authors._id: 67dc44a3a19fb08b924a9eab
paper_authors.name: Tianyu Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:30:51.650Z
paper_authors._id: 661a4a556fb488fa078c60aa
paper_authors.avatarUrl: /avatars/c77401fa9c6d2db896b4a337bb3f8add.svg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: TIanyupang
paper_authors._id: 67dc44a3a19fb08b924a9eac
paper_authors.name: Min-Yen Kan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:30:38.894Z
paper_authors._id: 628f137053028c04a3b750e4
paper_authors.avatarUrl: /avatars/85d174cc041ab91e724bb9d9d4e46c88.svg
paper_authors.fullname: Min-Yen Kan
paper_authors.type: user
paper_authors.user: knmnyn
paper_title: SkyLadder: Better and Faster Pretraining via Context Window Scheduling
paper_summary: Recent advancements in LLM pretraining have featured ever-expanding context
  windows to process longer sequences. However, our pilot study reveals that
  models pretrained with shorter context windows consistently outperform their
  long-context counterparts under a fixed token budget. This finding motivates us
  to explore an optimal context window scheduling strategy to better balance
  long-context capability with pretraining efficiency. To this end, we propose
  SkyLadder, a simple yet effective approach that implements a short-to-long
  context window transition. SkyLadder preserves strong standard benchmark
  performance, while matching or exceeding baseline results on long context
  tasks. Through extensive experiments, we pre-train 1B-parameter models (up to
  32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating
  that SkyLadder yields consistent gains of up to 3.7% on common benchmarks,
  while achieving up to 22% faster training speeds compared to baselines. The
  code is at https://github.com/sail-sg/SkyLadder.
paper_ai_summary: SkyLadder, a short-to-long context window transition strategy, enhances LLM pretraining efficiency while maintaining or surpassing long-context performance.
paper_ai_keywords: SkyLadder context window LLM pretraining standard benchmark long-context tasks parameter-efficient fine-tuning token budget pretraining efficiency