Parallel Scaling Law for Language Models
It is commonly believed that scaling language models should commit a
significant space or time cost, by increasing the parameters (parameter
scaling) or output tokens (inference-time scaling). We introduce the third and
more inference-efficient scaling paradigm: increasing the model's parallel
computation during both training and inference time. We apply P diverse and
learnable transformations to the input, execute forward passes of the model in
parallel, and dynamically aggregate the P outputs. This method, namely
parallel scaling (ParScale), scales parallel computation by reusing existing
parameters and can be applied to any model structure, optimization procedure,
data, or task. We theoretically propose a new scaling law and validate it
through large-scale pre-training, which shows that a model with P parallel
streams is similar to scaling the parameters by O(log P) while showing
superior inference efficiency. For example, ParScale can use up to 22times
less memory increase and 6times less latency increase compared to parameter
scaling that achieves the same performance improvement. It can also recycle an
off-the-shelf pre-trained model into a parallelly scaled one by post-training
on a small amount of tokens, further reducing the training budget. The new
scaling law we discovered potentially facilitates the deployment of more
powerful models in low-resource scenarios, and provides an alternative
perspective for the role of computation in machine learning.
68271b1e228dc6f5fd90b10f
Mouxiang Chen
claimed_verified
2025-05-16T13:23:18.035Z
66d58df54b87a685ccb8e4a0
/avatars/2566c20d79088ba761215b9a0197cb8e.svg
Mouxiang Chen
user
chenmouxiang
68271b1e228dc6f5fd90b110
Binyuan Hui
admin_assigned
2025-05-16T14:11:11.082Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
68271b1e228dc6f5fd90b111
Zeyu Cui
admin_assigned
2025-05-16T14:11:59.338Z
672c25ca8cfb61188128eb6f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
Zeyu Cui
user
misakamage
68271b1e228dc6f5fd90b112
Jiaxi Yang
68271b1e228dc6f5fd90b113
Dayiheng Liu
admin_assigned
2025-05-16T14:11:40.391Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
68271b1e228dc6f5fd90b114
Jianling Sun
68271b1e228dc6f5fd90b115
Junyang Lin
admin_assigned
2025-05-16T14:11:27.444Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
68271b1e228dc6f5fd90b116
Zhongxin Liu
Parallel Scaling Law for Language Models
It is commonly believed that scaling language models should commit a
significant space or time cost, by increasing the parameters (parameter
scaling) or output tokens (inference-time scaling). We introduce the third and
more inference-efficient scaling paradigm: increasing the model's parallel
computation during both training and inference time. We apply P diverse and
learnable transformations to the input, execute forward passes of the model in
parallel, and dynamically aggregate the P outputs. This method, namely
parallel scaling (ParScale), scales parallel computation by reusing existing
parameters and can be applied to any model structure, optimization procedure,
data, or task. We theoretically propose a new scaling law and validate it
through large-scale pre-training, which shows that a model with P parallel
streams is similar to scaling the parameters by O(log P) while showing
superior inference efficiency. For example, ParScale can use up to 22times
less memory increase and 6times less latency increase compared to parameter
scaling that achieves the same performance improvement. It can also recycle an
off-the-shelf pre-trained model into a parallelly scaled one by post-training
on a small amount of tokens, further reducing the training budget. The new
scaling law we discovered potentially facilitates the deployment of more
powerful models in low-resource scenarios, and provides an alternative
perspective for the role of computation in machine learning.
Parallel scaling (ParScale) improves inference efficiency by reusing existing parameters and executing multiple transformations in parallel, offering superior performance with reduced memory and latency compared to parameter scaling.
parallel computation parallel scaling ParScale parameter scaling inference-time scaling scaling law inference efficiency memory increase latency increase post-training