title: DDK: Distilling Domain Knowledge for Efficient Large Language Models
summary: Despite the advanced intelligence abilities of large language models (LLMs)
  in various applications, they still face significant computational and storage
  demands. Knowledge Distillation (KD) has emerged as an effective strategy to
  improve the performance of a smaller LLM (i.e., the student model) by
  transferring knowledge from a high-performing LLM (i.e., the teacher model).
  Prevailing techniques in LLM distillation typically use a black-box model API
  to generate high-quality pretrained and aligned datasets, or utilize white-box
  distillation by altering the loss function to better transfer knowledge from
  the teacher LLM. However, these methods ignore the knowledge differences
  between the student and teacher LLMs across domains. This results in excessive
  focus on domains with minimal performance gaps and insufficient attention to
  domains with large gaps, reducing overall performance. In this paper, we
  introduce a new LLM distillation framework called DDK, which dynamically
  adjusts the composition of the distillation dataset in a smooth manner
  according to the domain performance differences between the teacher and student
  models, making the distillation process more stable and effective. Extensive
  evaluations show that DDK significantly improves the performance of student
  models, outperforming both continuously pretrained baselines and existing
  knowledge distillation methods by a large margin.
paper_authors._id: 66a1b60ce5b6ab831cadf737
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T08:22:37.042Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 66a1b60ce5b6ab831cadf738
paper_authors.name: Chenchen Zhang
paper_authors._id: 66a1b60ce5b6ab831cadf739
paper_authors.name: Jinyang Guo
paper_authors._id: 66a1b60ce5b6ab831cadf73a
paper_authors.name: Yuanxing Zhang
paper_authors._id: 66a1b60ce5b6ab831cadf73b
paper_authors.name: Haoran Que
paper_authors._id: 66a1b60ce5b6ab831cadf73c
paper_authors.name: Ken Deng
paper_authors._id: 66a1b60ce5b6ab831cadf73d
paper_authors.name: Zhiqi Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:06:09.763Z
paper_authors._id: 650d40a8747f5328a2253dbc
paper_authors.avatarUrl: /avatars/5d9186658fb8ab853dff6d2d75d03efb.svg
paper_authors.fullname: Zhiqi Bai
paper_authors.type: user
paper_authors.user: bzq
paper_authors._id: 66a1b60ce5b6ab831cadf73e
paper_authors.name: Jie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:06:18.388Z
paper_authors._id: 639be86b59473c6ae02ef9c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
paper_authors.fullname: Jie Liu
paper_authors.type: user
paper_authors.user: jieliu
paper_authors._id: 66a1b60ce5b6ab831cadf73f
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:06:31.914Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66a1b60ce5b6ab831cadf740
paper_authors.name: Jiakai Wang
paper_authors._id: 66a1b60ce5b6ab831cadf741
paper_authors.name: Yanan Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:07:17.713Z
paper_authors._id: 65a33a99db5c00652eaa5e4c
paper_authors.avatarUrl: /avatars/23346ff87dfd2db2463c11137de9fbf2.svg
paper_authors.fullname: Yanan Wu
paper_authors.type: user
paper_authors.user: wuyanan513
paper_authors._id: 66a1b60ce5b6ab831cadf742
paper_authors.name: Congnan Liu
paper_authors._id: 66a1b60ce5b6ab831cadf743
paper_authors.name: Wenbo Su
paper_authors._id: 66a1b60ce5b6ab831cadf744
paper_authors.name: Jiamang Wang
paper_authors._id: 66a1b60ce5b6ab831cadf745
paper_authors.name: Lin Qu
paper_authors._id: 66a1b60ce5b6ab831cadf746
paper_authors.name: Bo Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:08:32.628Z
paper_authors._id: 62c695ad5aae1c624ca992e2
paper_authors.avatarUrl: /avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
paper_authors.fullname: Bo Zheng
paper_authors.type: user
paper_authors.user: bzheng
paper_title: DDK: Distilling Domain Knowledge for Efficient Large Language Models
paper_summary: Despite the advanced intelligence abilities of large language models (LLMs)
  in various applications, they still face significant computational and storage
  demands. Knowledge Distillation (KD) has emerged as an effective strategy to
  improve the performance of a smaller LLM (i.e., the student model) by
  transferring knowledge from a high-performing LLM (i.e., the teacher model).
  Prevailing techniques in LLM distillation typically use a black-box model API
  to generate high-quality pretrained and aligned datasets, or utilize white-box
  distillation by altering the loss function to better transfer knowledge from
  the teacher LLM. However, these methods ignore the knowledge differences
  between the student and teacher LLMs across domains. This results in excessive
  focus on domains with minimal performance gaps and insufficient attention to
  domains with large gaps, reducing overall performance. In this paper, we
  introduce a new LLM distillation framework called DDK, which dynamically
  adjusts the composition of the distillation dataset in a smooth manner
  according to the domain performance differences between the teacher and student
  models, making the distillation process more stable and effective. Extensive
  evaluations show that DDK significantly improves the performance of student
  models, outperforming both continuously pretrained baselines and existing
  knowledge distillation methods by a large margin.
paper_ai_summary: The DDK framework dynamically adjusts the distillation dataset to improve performance differences between teacher and student LLMs across various domains, outperforming existing methods.
paper_ai_keywords: large language models LLMs Knowledge Distillation KD black-box model API white-box distillation loss function domain performance differences distillation dataset DDK continuously pretrained baselines