DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Despite the success of distillation in large language models (LLMs), most
prior work applies identical loss functions to both teacher- and
student-generated data. These strategies overlook the synergy between loss
formulations and data types, leading to a suboptimal performance boost in
student models. To address this, we propose DistiLLM-2, a contrastive approach
that simultaneously increases the likelihood of teacher responses and decreases
that of student responses by harnessing this synergy. Our extensive experiments
show that DistiLLM-2 not only builds high-performing student models across a
wide range of tasks, including instruction-following and code generation, but
also supports diverse applications, such as preference alignment and
vision-language extensions. These findings highlight the potential of a
contrastive approach to enhance the efficacy of LLM distillation by effectively
aligning teacher and student models across varied data types.
67cfa99b7c95194db8d75468
Jongwoo Ko
claimed_verified
2025-03-11T08:22:29.622Z
64b7628af902508f0d7ae112
/avatars/83c155254486e80c1dfd14676fdf9215.svg
Jongwoo Ko
user
jongwooko
67cfa99b7c95194db8d75469
Tianyi Chen
claimed_verified
2025-03-11T08:22:27.139Z
64ad94f05a4a60156925ec96
/avatars/643bdb076e703bfcc89cec6fccb756c6.svg
Tianyi Chen
user
tianyic
67cfa99b7c95194db8d7546a
Sungnyun Kim
admin_assigned
2025-03-11T11:03:03.613Z
63f0c2ac9cf89c9ed1bdd25c
/avatars/856b2cb482250fb83c6fe793e29dfd74.svg
Sungnyun Kim
user
sungnyun
67cfa99b7c95194db8d7546b
Tianyu Ding
admin_assigned
2025-03-11T11:03:09.967Z
646b0c6d3e2a7b0659452fc7
https://cdn-avatars.huggingface.co/v1/production/uploads/646b0c6d3e2a7b0659452fc7/aqT9nbrJtOk38tDw_12xW.jpeg
Tianyu Ding
user
tding1
67cfa99b7c95194db8d7546c
Luming Liang
67cfa99b7c95194db8d7546d
Ilya Zharkov
67cfa99b7c95194db8d7546e
Se-Young Yun
DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Despite the success of distillation in large language models (LLMs), most
prior work applies identical loss functions to both teacher- and
student-generated data. These strategies overlook the synergy between loss
formulations and data types, leading to a suboptimal performance boost in
student models. To address this, we propose DistiLLM-2, a contrastive approach
that simultaneously increases the likelihood of teacher responses and decreases
that of student responses by harnessing this synergy. Our extensive experiments
show that DistiLLM-2 not only builds high-performing student models across a
wide range of tasks, including instruction-following and code generation, but
also supports diverse applications, such as preference alignment and
vision-language extensions. These findings highlight the potential of a
contrastive approach to enhance the efficacy of LLM distillation by effectively
aligning teacher and student models across varied data types.
A contrastive distillation method (DistiLLM-2) enhances student language models by optimizing teacher and student responses differently, improving performance across various tasks and applications.
distillation large language models (LLMs) teacher- and student-generated data contrastive approach instruction-following code generation preference alignment vision-language extensions