Efficient Pretraining Length Scaling
Recent advances in large language models have demonstrated the effectiveness
of length scaling during post-training, yet its potential in pre-training
remains underexplored. We present the Parallel Hidden Decoding Transformer
(PHD-Transformer), a novel framework that enables efficient length
scaling during pre-training while maintaining inference efficiency.
PHD-Transformer achieves this through an innovative KV cache
management strategy that distinguishes between original tokens and hidden
decoding tokens. By retaining only the KV cache of original tokens for
long-range dependencies while immediately discarding hidden decoding tokens
after use, our approach maintains the same KV cache size as the vanilla
transformer while enabling effective length scaling. To further enhance
performance, we introduce two optimized variants: PHD-SWA employs
sliding window attention to preserve local dependencies, while
PHD-CSWA implements chunk-wise sliding window attention to eliminate
linear growth in pre-filling time. Extensive experiments demonstrate consistent
improvements across multiple benchmarks.
68074ed102571b837f03463c
Bohong Wu
claimed_verified
2025-04-22T09:50:15.811Z
64722a616facfb01d8ae8349
/avatars/1dce23ae5ebd9996770cf5efe910b857.svg
Wu Bohong
user
bongbohong
68074ed102571b837f03463d
Shen Yan
68074ed102571b837f03463e
Sijun Zhang
68074ed102571b837f03463f
Jianqiao Lu
admin_assigned
2025-04-23T13:50:21.244Z
64d59835bcab729cb48e3258
https://cdn-avatars.huggingface.co/v1/production/uploads/64d59835bcab729cb48e3258/cDTTOozta7zCkjkHmbPCQ.png
lujianqiao
user
rookiemango
68074ed102571b837f034640
Yutao Zeng
claimed_verified
2025-04-22T09:50:18.659Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
68074ed102571b837f034641
Ya Wang
68074ed102571b837f034642
Xun Zhou
Efficient Pretraining Length Scaling
Recent advances in large language models have demonstrated the effectiveness
of length scaling during post-training, yet its potential in pre-training
remains underexplored. We present the Parallel Hidden Decoding Transformer
(PHD-Transformer), a novel framework that enables efficient length
scaling during pre-training while maintaining inference efficiency.
PHD-Transformer achieves this through an innovative KV cache
management strategy that distinguishes between original tokens and hidden
decoding tokens. By retaining only the KV cache of original tokens for
long-range dependencies while immediately discarding hidden decoding tokens
after use, our approach maintains the same KV cache size as the vanilla
transformer while enabling effective length scaling. To further enhance
performance, we introduce two optimized variants: PHD-SWA employs
sliding window attention to preserve local dependencies, while
PHD-CSWA implements chunk-wise sliding window attention to eliminate
linear growth in pre-filling time. Extensive experiments demonstrate consistent
improvements across multiple benchmarks.
A novel PHD-Transformer framework enables efficient length scaling during pre-training with optimized KV cache management, achieving improvements across multiple benchmarks.
Parallel Hidden Decoding Transformer PHD-Transformer KV cache management sliding window attention chunk-wise sliding window attention