title: Parallel Scaling Law for Language Models
summary: It is commonly believed that scaling language models should commit a
  significant space or time cost, by increasing the parameters (parameter
  scaling) or output tokens (inference-time scaling). We introduce the third and
  more inference-efficient scaling paradigm: increasing the model's parallel
  computation during both training and inference time. We apply P diverse and
  learnable transformations to the input, execute forward passes of the model in
  parallel, and dynamically aggregate the P outputs. This method, namely
  parallel scaling (ParScale), scales parallel computation by reusing existing
  parameters and can be applied to any model structure, optimization procedure,
  data, or task. We theoretically propose a new scaling law and validate it
  through large-scale pre-training, which shows that a model with P parallel
  streams is similar to scaling the parameters by O(log P) while showing
  superior inference efficiency. For example, ParScale can use up to 22times
  less memory increase and 6times less latency increase compared to parameter
  scaling that achieves the same performance improvement. It can also recycle an
  off-the-shelf pre-trained model into a parallelly scaled one by post-training
  on a small amount of tokens, further reducing the training budget. The new
  scaling law we discovered potentially facilitates the deployment of more
  powerful models in low-resource scenarios, and provides an alternative
  perspective for the role of computation in machine learning.
paper_authors._id: 68271b1e228dc6f5fd90b10f
paper_authors.name: Mouxiang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T13:23:18.035Z
paper_authors._id: 66d58df54b87a685ccb8e4a0
paper_authors.avatarUrl: /avatars/2566c20d79088ba761215b9a0197cb8e.svg
paper_authors.fullname: Mouxiang Chen
paper_authors.type: user
paper_authors.user: chenmouxiang
paper_authors._id: 68271b1e228dc6f5fd90b110
paper_authors.name: Binyuan Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T14:11:11.082Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 68271b1e228dc6f5fd90b111
paper_authors.name: Zeyu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T14:11:59.338Z
paper_authors._id: 672c25ca8cfb61188128eb6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
paper_authors.fullname: Zeyu Cui
paper_authors.type: user
paper_authors.user: misakamage
paper_authors._id: 68271b1e228dc6f5fd90b112
paper_authors.name: Jiaxi Yang
paper_authors._id: 68271b1e228dc6f5fd90b113
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T14:11:40.391Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 68271b1e228dc6f5fd90b114
paper_authors.name: Jianling Sun
paper_authors._id: 68271b1e228dc6f5fd90b115
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T14:11:27.444Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 68271b1e228dc6f5fd90b116
paper_authors.name: Zhongxin Liu
paper_title: Parallel Scaling Law for Language Models
paper_summary: It is commonly believed that scaling language models should commit a
  significant space or time cost, by increasing the parameters (parameter
  scaling) or output tokens (inference-time scaling). We introduce the third and
  more inference-efficient scaling paradigm: increasing the model's parallel
  computation during both training and inference time. We apply P diverse and
  learnable transformations to the input, execute forward passes of the model in
  parallel, and dynamically aggregate the P outputs. This method, namely
  parallel scaling (ParScale), scales parallel computation by reusing existing
  parameters and can be applied to any model structure, optimization procedure,
  data, or task. We theoretically propose a new scaling law and validate it
  through large-scale pre-training, which shows that a model with P parallel
  streams is similar to scaling the parameters by O(log P) while showing
  superior inference efficiency. For example, ParScale can use up to 22times
  less memory increase and 6times less latency increase compared to parameter
  scaling that achieves the same performance improvement. It can also recycle an
  off-the-shelf pre-trained model into a parallelly scaled one by post-training
  on a small amount of tokens, further reducing the training budget. The new
  scaling law we discovered potentially facilitates the deployment of more
  powerful models in low-resource scenarios, and provides an alternative
  perspective for the role of computation in machine learning.
paper_ai_summary: Parallel scaling (ParScale) improves inference efficiency by reusing existing parameters and executing multiple transformations in parallel, offering superior performance with reduced memory and latency compared to parameter scaling.
paper_ai_keywords: parallel computation parallel scaling ParScale parameter scaling inference-time scaling scaling law inference efficiency memory increase latency increase post-training