title: Understanding LLMs: A Comprehensive Overview from Training to Inference
summary: The introduction of ChatGPT has led to a significant increase in the
  utilization of Large Language Models (LLMs) for addressing downstream tasks.
  There's an increasing focus on cost-efficient training and deployment within
  this context. Low-cost training and deployment of LLMs represent the future
  development trend. This paper reviews the evolution of large language model
  training techniques and inference deployment technologies aligned with this
  emerging trend. The discussion on training includes various aspects, including
  data preprocessing, training architecture, pre-training tasks, parallel
  training, and relevant content related to model fine-tuning. On the inference
  side, the paper covers topics such as model compression, parallel computation,
  memory scheduling, and structural optimization. It also explores LLMs'
  utilization and provides insights into their future development.
paper_authors._id: 659770e9420dcc6850000a6f
paper_authors.name: Yiheng Liu
paper_authors._id: 659770e9420dcc6850000a70
paper_authors.name: Hao He
paper_authors._id: 659770e9420dcc6850000a71
paper_authors.name: Tianle Han
paper_authors._id: 659770e9420dcc6850000a72
paper_authors.name: Xu Zhang
paper_authors._id: 659770e9420dcc6850000a73
paper_authors.name: Mengyuan Liu
paper_authors._id: 659770e9420dcc6850000a74
paper_authors.name: Jiaming Tian
paper_authors._id: 659770e9420dcc6850000a75
paper_authors.name: Yutong Zhang
paper_authors._id: 659770e9420dcc6850000a76
paper_authors.name: Jiaqi Wang
paper_authors._id: 659770e9420dcc6850000a77
paper_authors.name: Xiaohui Gao
paper_authors._id: 659770e9420dcc6850000a78
paper_authors.name: Tianyang Zhong
paper_authors._id: 659770e9420dcc6850000a79
paper_authors.name: Yi Pan
paper_authors._id: 659770e9420dcc6850000a7a
paper_authors.name: Shaochen Xu
paper_authors._id: 659770e9420dcc6850000a7b
paper_authors.name: Zihao Wu
paper_authors._id: 659770e9420dcc6850000a7c
paper_authors.name: Zhengliang Liu
paper_authors._id: 659770e9420dcc6850000a7d
paper_authors.name: Xin Zhang
paper_authors._id: 659770e9420dcc6850000a7e
paper_authors.name: Shu Zhang
paper_authors._id: 659770e9420dcc6850000a7f
paper_authors.name: Xintao Hu
paper_authors._id: 659770e9420dcc6850000a80
paper_authors.name: Tuo Zhang
paper_authors._id: 659770e9420dcc6850000a81
paper_authors.name: Ning Qiang
paper_authors._id: 659770e9420dcc6850000a82
paper_authors.name: Tianming Liu
paper_authors._id: 659770e9420dcc6850000a83
paper_authors.name: Bao Ge
paper_title: Understanding LLMs: A Comprehensive Overview from Training to Inference
paper_summary: The introduction of ChatGPT has led to a significant increase in the
  utilization of Large Language Models (LLMs) for addressing downstream tasks.
  There's an increasing focus on cost-efficient training and deployment within
  this context. Low-cost training and deployment of LLMs represent the future
  development trend. This paper reviews the evolution of large language model
  training techniques and inference deployment technologies aligned with this
  emerging trend. The discussion on training includes various aspects, including
  data preprocessing, training architecture, pre-training tasks, parallel
  training, and relevant content related to model fine-tuning. On the inference
  side, the paper covers topics such as model compression, parallel computation,
  memory scheduling, and structural optimization. It also explores LLMs'
  utilization and provides insights into their future development.
paper_ai_summary: The paper reviews techniques for cost-efficient training and deployment of large language models, covering aspects like data preprocessing, parallel training, model fine-tuning, and inference optimizations including model compression and memory scheduling.
paper_ai_keywords: Large Language Models pre-training tasks parallel training model fine-tuning model compression parallel computation memory scheduling structural optimization