title: Direct Preference Knowledge Distillation for Large Language Models
summary: In the field of large language models (LLMs), Knowledge Distillation (KD) is
  a critical technique for transferring capabilities from teacher models to
  student models. However, existing KD methods face limitations and challenges in
  distillation of LLMs, including efficiency and insufficient measurement
  capabilities of traditional KL divergence. It is shown that LLMs can serve as
  an implicit reward function, which we define as a supplement to KL divergence.
  In this work, we propose Direct Preference Knowledge Distillation (DPKD) for
  LLMs. DPKD utilizes distribution divergence to represent the preference loss
  and implicit reward function. We re-formulate KD of LLMs into two stages: first
  optimizing and objective consisting of implicit reward and reverse KL
  divergence and then improving the preference probability of teacher outputs
  over student outputs. We conducted experiments and analysis on various datasets
  with LLM parameters ranging from 120M to 13B and demonstrate the broad
  applicability and effectiveness of our DPKD approach. Meanwhile, we prove the
  value and effectiveness of the introduced implicit reward and output preference
  in KD through experiments and theoretical analysis. The DPKD method outperforms
  the baseline method in both output response precision and exact match
  percentage. Code and data are available at https://aka.ms/dpkd.
paper_authors._id: 6682377ad892ee6886aab6f6
paper_authors.name: Yixing Li
paper_authors._id: 6682377ad892ee6886aab6f7
paper_authors.name: Yuxian Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:00:21.943Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 6682377ad892ee6886aab6f8
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-01T07:46:49.916Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 6682377ad892ee6886aab6f9
paper_authors.name: Dequan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:01:03.279Z
paper_authors._id: 61ad24836da53246bd6ac410
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61ad24836da53246bd6ac410/o-FL-C6B77iB94wyAtTuO.png
paper_authors.fullname: Dequan Wang
paper_authors.type: user
paper_authors.user: dqwang
paper_authors._id: 6682377ad892ee6886aab6fa
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T16:26:37.846Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 6682377ad892ee6886aab6fb
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:09:15.680Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Direct Preference Knowledge Distillation for Large Language Models
paper_summary: In the field of large language models (LLMs), Knowledge Distillation (KD) is
  a critical technique for transferring capabilities from teacher models to
  student models. However, existing KD methods face limitations and challenges in
  distillation of LLMs, including efficiency and insufficient measurement
  capabilities of traditional KL divergence. It is shown that LLMs can serve as
  an implicit reward function, which we define as a supplement to KL divergence.
  In this work, we propose Direct Preference Knowledge Distillation (DPKD) for
  LLMs. DPKD utilizes distribution divergence to represent the preference loss
  and implicit reward function. We re-formulate KD of LLMs into two stages: first
  optimizing and objective consisting of implicit reward and reverse KL
  divergence and then improving the preference probability of teacher outputs
  over student outputs. We conducted experiments and analysis on various datasets
  with LLM parameters ranging from 120M to 13B and demonstrate the broad
  applicability and effectiveness of our DPKD approach. Meanwhile, we prove the
  value and effectiveness of the introduced implicit reward and output preference
  in KD through experiments and theoretical analysis. The DPKD method outperforms
  the baseline method in both output response precision and exact match
  percentage. Code and data are available at https://aka.ms/dpkd.
paper_ai_summary: Direct Preference Knowledge Distillation (DPKD) improves the knowledge transfer from large language model teachers to students by incorporating an implicit reward function and distribution divergence.
paper_ai_keywords: Knowledge Distillation Direct Preference Knowledge Distillation LLMs implicit reward function distribution divergence preference loss reverse KL divergence output preference