title: Pre-training Distillation for Large Language Models: A Design Space
    Exploration
summary: Knowledge distillation (KD) aims to transfer knowledge from a large teacher
  model to a smaller student model. Previous work applying KD in the field of
  large language models (LLMs) typically focused on the post-training phase,
  where the student LLM learns directly from instructions and corresponding
  responses generated by the teacher model. In this paper, we extend KD to the
  pre-training phase of LLMs, named pre-training distillation (PD). We first
  conduct a preliminary experiment using GLM-4-9B as the teacher LLM to distill a
  1.9B parameter student LLM, validating the effectiveness of PD. Considering the
  key impact factors of distillation, we systematically explore the design space
  of pre-training distillation across four aspects: logits processing, loss
  selection, scaling law, and offline or online logits. We conduct extensive
  experiments to explore the design space of pre-training distillation and find
  better configurations and interesting conclusions, such as larger student LLMs
  generally benefiting more from pre-training distillation, while a larger
  teacher LLM does not necessarily guarantee better results. We hope our
  exploration of the design space will inform future practices in pre-training
  distillation.
paper_authors._id: 67173a397994589e3f441da8
paper_authors.name: Hao Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T09:48:33.508Z
paper_authors._id: 625a5446f1063e7085d5178a
paper_authors.avatarUrl: /avatars/5e78186f13f74b14e01583e06ff6c4dc.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: Wesleythu
paper_authors._id: 67173a397994589e3f441da9
paper_authors.name: Xin Lv
paper_authors._id: 67173a397994589e3f441daa
paper_authors.name: Yushi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:55:46.101Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 67173a397994589e3f441dab
paper_authors.name: Zijun Yao
paper_authors._id: 67173a397994589e3f441dac
paper_authors.name: Jiajie Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T09:14:57.280Z
paper_authors._id: 66cdd285c51a915bd5f2d017
paper_authors.avatarUrl: /avatars/14e5794307e4672b1b51d26b31227e0f.svg
paper_authors.fullname: Jiajie Zhang
paper_authors.type: user
paper_authors.user: NeoZ123
paper_authors._id: 67173a397994589e3f441dad
paper_authors.name: Lei Hou
paper_authors._id: 67173a397994589e3f441dae
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:13:58.645Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: Pre-training Distillation for Large Language Models: A Design Space
    Exploration
paper_summary: Knowledge distillation (KD) aims to transfer knowledge from a large teacher
  model to a smaller student model. Previous work applying KD in the field of
  large language models (LLMs) typically focused on the post-training phase,
  where the student LLM learns directly from instructions and corresponding
  responses generated by the teacher model. In this paper, we extend KD to the
  pre-training phase of LLMs, named pre-training distillation (PD). We first
  conduct a preliminary experiment using GLM-4-9B as the teacher LLM to distill a
  1.9B parameter student LLM, validating the effectiveness of PD. Considering the
  key impact factors of distillation, we systematically explore the design space
  of pre-training distillation across four aspects: logits processing, loss
  selection, scaling law, and offline or online logits. We conduct extensive
  experiments to explore the design space of pre-training distillation and find
  better configurations and interesting conclusions, such as larger student LLMs
  generally benefiting more from pre-training distillation, while a larger
  teacher LLM does not necessarily guarantee better results. We hope our
  exploration of the design space will inform future practices in pre-training
  distillation.
paper_ai_keywords: Knowledge distillation (KD) Large language models (LLMs) Pre-training phase Post-training phase GLM-4-9B Parameter-efficient fine-tuning Logits processing Loss selection Scaling law Offline logits Online logits Pre-training distillation (PD) DyVo DyVo training distillation City-training datasets