Understanding LLMs: A Comprehensive Overview from Training to Inference
The introduction of ChatGPT has led to a significant increase in the
utilization of Large Language Models (LLMs) for addressing downstream tasks.
There's an increasing focus on cost-efficient training and deployment within
this context. Low-cost training and deployment of LLMs represent the future
development trend. This paper reviews the evolution of large language model
training techniques and inference deployment technologies aligned with this
emerging trend. The discussion on training includes various aspects, including
data preprocessing, training architecture, pre-training tasks, parallel
training, and relevant content related to model fine-tuning. On the inference
side, the paper covers topics such as model compression, parallel computation,
memory scheduling, and structural optimization. It also explores LLMs'
utilization and provides insights into their future development.
659770e9420dcc6850000a6f
Yiheng Liu
659770e9420dcc6850000a70
Hao He
659770e9420dcc6850000a71
Tianle Han
659770e9420dcc6850000a72
Xu Zhang
659770e9420dcc6850000a73
Mengyuan Liu
659770e9420dcc6850000a74
Jiaming Tian
659770e9420dcc6850000a75
Yutong Zhang
659770e9420dcc6850000a76
Jiaqi Wang
659770e9420dcc6850000a77
Xiaohui Gao
659770e9420dcc6850000a78
Tianyang Zhong
659770e9420dcc6850000a79
Yi Pan
659770e9420dcc6850000a7a
Shaochen Xu
659770e9420dcc6850000a7b
Zihao Wu
659770e9420dcc6850000a7c
Zhengliang Liu
659770e9420dcc6850000a7d
Xin Zhang
659770e9420dcc6850000a7e
Shu Zhang
659770e9420dcc6850000a7f
Xintao Hu
659770e9420dcc6850000a80
Tuo Zhang
659770e9420dcc6850000a81
Ning Qiang
659770e9420dcc6850000a82
Tianming Liu
659770e9420dcc6850000a83
Bao Ge
Understanding LLMs: A Comprehensive Overview from Training to Inference
The introduction of ChatGPT has led to a significant increase in the
utilization of Large Language Models (LLMs) for addressing downstream tasks.
There's an increasing focus on cost-efficient training and deployment within
this context. Low-cost training and deployment of LLMs represent the future
development trend. This paper reviews the evolution of large language model
training techniques and inference deployment technologies aligned with this
emerging trend. The discussion on training includes various aspects, including
data preprocessing, training architecture, pre-training tasks, parallel
training, and relevant content related to model fine-tuning. On the inference
side, the paper covers topics such as model compression, parallel computation,
memory scheduling, and structural optimization. It also explores LLMs'
utilization and provides insights into their future development.
The paper reviews techniques for cost-efficient training and deployment of large language models, covering aspects like data preprocessing, parallel training, model fine-tuning, and inference optimizations including model compression and memory scheduling.
Large Language Models pre-training tasks parallel training model fine-tuning model compression parallel computation memory scheduling structural optimization