DDK: Distilling Domain Knowledge for Efficient Large Language Models
Despite the advanced intelligence abilities of large language models (LLMs)
in various applications, they still face significant computational and storage
demands. Knowledge Distillation (KD) has emerged as an effective strategy to
improve the performance of a smaller LLM (i.e., the student model) by
transferring knowledge from a high-performing LLM (i.e., the teacher model).
Prevailing techniques in LLM distillation typically use a black-box model API
to generate high-quality pretrained and aligned datasets, or utilize white-box
distillation by altering the loss function to better transfer knowledge from
the teacher LLM. However, these methods ignore the knowledge differences
between the student and teacher LLMs across domains. This results in excessive
focus on domains with minimal performance gaps and insufficient attention to
domains with large gaps, reducing overall performance. In this paper, we
introduce a new LLM distillation framework called DDK, which dynamically
adjusts the composition of the distillation dataset in a smooth manner
according to the domain performance differences between the teacher and student
models, making the distillation process more stable and effective. Extensive
evaluations show that DDK significantly improves the performance of student
models, outperforming both continuously pretrained baselines and existing
knowledge distillation methods by a large margin.
66a1b60ce5b6ab831cadf737
Jiaheng Liu
admin_assigned
2024-07-25T08:22:37.042Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
66a1b60ce5b6ab831cadf738
Chenchen Zhang
66a1b60ce5b6ab831cadf739
Jinyang Guo
66a1b60ce5b6ab831cadf73a
Yuanxing Zhang
66a1b60ce5b6ab831cadf73b
Haoran Que
66a1b60ce5b6ab831cadf73c
Ken Deng
66a1b60ce5b6ab831cadf73d
Zhiqi Bai
admin_assigned
2024-07-25T09:06:09.763Z
650d40a8747f5328a2253dbc
/avatars/5d9186658fb8ab853dff6d2d75d03efb.svg
Zhiqi Bai
user
bzq
66a1b60ce5b6ab831cadf73e
Jie Liu
admin_assigned
2024-07-25T09:06:18.388Z
639be86b59473c6ae02ef9c4
https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
Jie Liu
user
jieliu
66a1b60ce5b6ab831cadf73f
Ge Zhang
admin_assigned
2024-07-25T09:06:31.914Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66a1b60ce5b6ab831cadf740
Jiakai Wang
66a1b60ce5b6ab831cadf741
Yanan Wu
admin_assigned
2024-07-25T09:07:17.713Z
65a33a99db5c00652eaa5e4c
/avatars/23346ff87dfd2db2463c11137de9fbf2.svg
Yanan Wu
user
wuyanan513
66a1b60ce5b6ab831cadf742
Congnan Liu
66a1b60ce5b6ab831cadf743
Wenbo Su
66a1b60ce5b6ab831cadf744
Jiamang Wang
66a1b60ce5b6ab831cadf745
Lin Qu
66a1b60ce5b6ab831cadf746
Bo Zheng
admin_assigned
2024-07-25T09:08:32.628Z
62c695ad5aae1c624ca992e2
/avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
Bo Zheng
user
bzheng
DDK: Distilling Domain Knowledge for Efficient Large Language Models
Despite the advanced intelligence abilities of large language models (LLMs)
in various applications, they still face significant computational and storage
demands. Knowledge Distillation (KD) has emerged as an effective strategy to
improve the performance of a smaller LLM (i.e., the student model) by
transferring knowledge from a high-performing LLM (i.e., the teacher model).
Prevailing techniques in LLM distillation typically use a black-box model API
to generate high-quality pretrained and aligned datasets, or utilize white-box
distillation by altering the loss function to better transfer knowledge from
the teacher LLM. However, these methods ignore the knowledge differences
between the student and teacher LLMs across domains. This results in excessive
focus on domains with minimal performance gaps and insufficient attention to
domains with large gaps, reducing overall performance. In this paper, we
introduce a new LLM distillation framework called DDK, which dynamically
adjusts the composition of the distillation dataset in a smooth manner
according to the domain performance differences between the teacher and student
models, making the distillation process more stable and effective. Extensive
evaluations show that DDK significantly improves the performance of student
models, outperforming both continuously pretrained baselines and existing
knowledge distillation methods by a large margin.
The DDK framework dynamically adjusts the distillation dataset to improve performance differences between teacher and student LLMs across various domains, outperforming existing methods.
large language models LLMs Knowledge Distillation KD black-box model API white-box distillation loss function domain performance differences distillation dataset DDK continuously pretrained baselines