AMSP: Super-Scaling LLM Training via Advanced Model States Partitioning
Large Language Models (LLMs) have demonstrated impressive performance across
various downstream tasks. When training these models, there is a growing
inclination to process more tokens on larger training scales but with
relatively smaller model sizes. Zero Redundancy Optimizer (ZeRO), although
effective in conventional training environments, grapples with scaling
challenges when confronted with this emerging paradigm. To this end, we propose
a novel LLM training framework AMSP, which undertakes a granular partitioning
of model states, encompassing parameters (P), gradient (G), and optimizer
states (OS). Specifically, AMSP(1) builds a unified partitioning space,
enabling independent partitioning strategies for P, G, and OS; (2)
incorporates a scale-aware partitioner to autonomously search for optimal
partitioning strategies: (3) designs a dedicated communication optimizer to
ensure proficient management of data placement discrepancies arising from
diverse partitioning strategies. Our evaluations show that AMSP achieves up to
90.3% scaling efficiency across 1024 GPUs.
6543072e5670dd57b76d66ff
Qiaoling Chen
admin_assigned
2023-11-02T13:28:04.329Z
654372ee91a22f5a118fdeb9
/avatars/9a93f7a8ee000cf7b02ab061a4756216.svg
QiaolingChen
user
QiaolingChen
6543072e5670dd57b76d6700
Qinghao Hu
admin_assigned
2023-11-02T13:28:11.156Z
62fb8412610dae1bcd036db7
https://cdn-avatars.huggingface.co/v1/production/uploads/62fb8412610dae1bcd036db7/LIaEoDrvrq8hDl8509ZuF.jpeg
Qinghao Hu
user
Qinghao
6543072e5670dd57b76d6701
Zhisheng Ye
admin_assigned
2023-11-02T13:28:18.603Z
6384e0d5b2906edaf835d56c
https://cdn-avatars.huggingface.co/v1/production/uploads/6384e0d5b2906edaf835d56c/tEo0rap-hd02bVy_u1cJb.jpeg
Zhisheng Ye
user
yzs981130
6543072e5670dd57b76d6702
Guoteng Wang
admin_assigned
2023-11-02T13:28:29.284Z
6542391f3fcd1aee202383d2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5O7dZAVGmwhsx1THSJ-gn.jpeg
wang
user
guoteng
6543072e5670dd57b76d6703
Peng Sun
6543072e5670dd57b76d6704
Yonggang Wen
6543072e5670dd57b76d6705
Tianwei Zhang
AMSP: Super-Scaling LLM Training via Advanced Model States Partitioning
Large Language Models (LLMs) have demonstrated impressive performance across
various downstream tasks. When training these models, there is a growing
inclination to process more tokens on larger training scales but with
relatively smaller model sizes. Zero Redundancy Optimizer (ZeRO), although
effective in conventional training environments, grapples with scaling
challenges when confronted with this emerging paradigm. To this end, we propose
a novel LLM training framework AMSP, which undertakes a granular partitioning
of model states, encompassing parameters (P), gradient (G), and optimizer
states (OS). Specifically, AMSP(1) builds a unified partitioning space,
enabling independent partitioning strategies for P, G, and OS; (2)
incorporates a scale-aware partitioner to autonomously search for optimal
partitioning strategies: (3) designs a dedicated communication optimizer to
ensure proficient management of data placement discrepancies arising from
diverse partitioning strategies. Our evaluations show that AMSP achieves up to
90.3% scaling efficiency across 1024 GPUs.
A novel LLM training framework, AMSP, addresses scaling challenges for large-scale token processing using model state partitioning and a scale-aware partitioner.
Large Language Models (LLMs) Zero Redundancy Optimizer (ZeRO) model states parameters gradient optimizer states partitioning space scale-aware partitioner communication optimizer scaling efficiency