Direct Preference Knowledge Distillation for Large Language Models
In the field of large language models (LLMs), Knowledge Distillation (KD) is
a critical technique for transferring capabilities from teacher models to
student models. However, existing KD methods face limitations and challenges in
distillation of LLMs, including efficiency and insufficient measurement
capabilities of traditional KL divergence. It is shown that LLMs can serve as
an implicit reward function, which we define as a supplement to KL divergence.
In this work, we propose Direct Preference Knowledge Distillation (DPKD) for
LLMs. DPKD utilizes distribution divergence to represent the preference loss
and implicit reward function. We re-formulate KD of LLMs into two stages: first
optimizing and objective consisting of implicit reward and reverse KL
divergence and then improving the preference probability of teacher outputs
over student outputs. We conducted experiments and analysis on various datasets
with LLM parameters ranging from 120M to 13B and demonstrate the broad
applicability and effectiveness of our DPKD approach. Meanwhile, we prove the
value and effectiveness of the introduced implicit reward and output preference
in KD through experiments and theoretical analysis. The DPKD method outperforms
the baseline method in both output response precision and exact match
percentage. Code and data are available at https://aka.ms/dpkd.
6682377ad892ee6886aab6f6
Yixing Li
6682377ad892ee6886aab6f7
Yuxian Gu
admin_assigned
2024-07-01T09:00:21.943Z
624ac662102fcdff87be51b9
https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
Yuxian Gu
user
t1101675
6682377ad892ee6886aab6f8
Li Dong
claimed_verified
2024-07-01T07:46:49.916Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
6682377ad892ee6886aab6f9
Dequan Wang
admin_assigned
2024-07-01T09:01:03.279Z
61ad24836da53246bd6ac410
https://cdn-avatars.huggingface.co/v1/production/uploads/61ad24836da53246bd6ac410/o-FL-C6B77iB94wyAtTuO.png
Dequan Wang
user
dqwang
6682377ad892ee6886aab6fa
Yu Cheng
claimed_verified
2024-10-09T16:26:37.846Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
6682377ad892ee6886aab6fb
Furu Wei
admin_assigned
2024-07-01T09:09:15.680Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Direct Preference Knowledge Distillation for Large Language Models
In the field of large language models (LLMs), Knowledge Distillation (KD) is
a critical technique for transferring capabilities from teacher models to
student models. However, existing KD methods face limitations and challenges in
distillation of LLMs, including efficiency and insufficient measurement
capabilities of traditional KL divergence. It is shown that LLMs can serve as
an implicit reward function, which we define as a supplement to KL divergence.
In this work, we propose Direct Preference Knowledge Distillation (DPKD) for
LLMs. DPKD utilizes distribution divergence to represent the preference loss
and implicit reward function. We re-formulate KD of LLMs into two stages: first
optimizing and objective consisting of implicit reward and reverse KL
divergence and then improving the preference probability of teacher outputs
over student outputs. We conducted experiments and analysis on various datasets
with LLM parameters ranging from 120M to 13B and demonstrate the broad
applicability and effectiveness of our DPKD approach. Meanwhile, we prove the
value and effectiveness of the introduced implicit reward and output preference
in KD through experiments and theoretical analysis. The DPKD method outperforms
the baseline method in both output response precision and exact match
percentage. Code and data are available at https://aka.ms/dpkd.
Direct Preference Knowledge Distillation (DPKD) improves the knowledge transfer from large language model teachers to students by incorporating an implicit reward function and distribution divergence.
Knowledge Distillation Direct Preference Knowledge Distillation LLMs implicit reward function distribution divergence preference loss reverse KL divergence output preference