title: Efficient Pretraining Length Scaling
summary: Recent advances in large language models have demonstrated the effectiveness
  of length scaling during post-training, yet its potential in pre-training
  remains underexplored. We present the Parallel Hidden Decoding Transformer
  (PHD-Transformer), a novel framework that enables efficient length
  scaling during pre-training while maintaining inference efficiency.
  PHD-Transformer achieves this through an innovative KV cache
  management strategy that distinguishes between original tokens and hidden
  decoding tokens. By retaining only the KV cache of original tokens for
  long-range dependencies while immediately discarding hidden decoding tokens
  after use, our approach maintains the same KV cache size as the vanilla
  transformer while enabling effective length scaling. To further enhance
  performance, we introduce two optimized variants: PHD-SWA employs
  sliding window attention to preserve local dependencies, while
  PHD-CSWA implements chunk-wise sliding window attention to eliminate
  linear growth in pre-filling time. Extensive experiments demonstrate consistent
  improvements across multiple benchmarks.
paper_authors._id: 68074ed102571b837f03463c
paper_authors.name: Bohong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:50:15.811Z
paper_authors._id: 64722a616facfb01d8ae8349
paper_authors.avatarUrl: /avatars/1dce23ae5ebd9996770cf5efe910b857.svg
paper_authors.fullname: Wu Bohong
paper_authors.type: user
paper_authors.user: bongbohong
paper_authors._id: 68074ed102571b837f03463d
paper_authors.name: Shen Yan
paper_authors._id: 68074ed102571b837f03463e
paper_authors.name: Sijun Zhang
paper_authors._id: 68074ed102571b837f03463f
paper_authors.name: Jianqiao Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:50:21.244Z
paper_authors._id: 64d59835bcab729cb48e3258
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d59835bcab729cb48e3258/cDTTOozta7zCkjkHmbPCQ.png
paper_authors.fullname: lujianqiao
paper_authors.type: user
paper_authors.user: rookiemango
paper_authors._id: 68074ed102571b837f034640
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:50:18.659Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 68074ed102571b837f034641
paper_authors.name: Ya Wang
paper_authors._id: 68074ed102571b837f034642
paper_authors.name: Xun Zhou
paper_title: Efficient Pretraining Length Scaling
paper_summary: Recent advances in large language models have demonstrated the effectiveness
  of length scaling during post-training, yet its potential in pre-training
  remains underexplored. We present the Parallel Hidden Decoding Transformer
  (PHD-Transformer), a novel framework that enables efficient length
  scaling during pre-training while maintaining inference efficiency.
  PHD-Transformer achieves this through an innovative KV cache
  management strategy that distinguishes between original tokens and hidden
  decoding tokens. By retaining only the KV cache of original tokens for
  long-range dependencies while immediately discarding hidden decoding tokens
  after use, our approach maintains the same KV cache size as the vanilla
  transformer while enabling effective length scaling. To further enhance
  performance, we introduce two optimized variants: PHD-SWA employs
  sliding window attention to preserve local dependencies, while
  PHD-CSWA implements chunk-wise sliding window attention to eliminate
  linear growth in pre-filling time. Extensive experiments demonstrate consistent
  improvements across multiple benchmarks.
paper_ai_summary: A novel PHD-Transformer framework enables efficient length scaling during pre-training with optimized KV cache management, achieving improvements across multiple benchmarks.
paper_ai_keywords: Parallel Hidden Decoding Transformer PHD-Transformer KV cache management sliding window attention chunk-wise sliding window attention