title: DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
summary: Despite the success of distillation in large language models (LLMs), most
  prior work applies identical loss functions to both teacher- and
  student-generated data. These strategies overlook the synergy between loss
  formulations and data types, leading to a suboptimal performance boost in
  student models. To address this, we propose DistiLLM-2, a contrastive approach
  that simultaneously increases the likelihood of teacher responses and decreases
  that of student responses by harnessing this synergy. Our extensive experiments
  show that DistiLLM-2 not only builds high-performing student models across a
  wide range of tasks, including instruction-following and code generation, but
  also supports diverse applications, such as preference alignment and
  vision-language extensions. These findings highlight the potential of a
  contrastive approach to enhance the efficacy of LLM distillation by effectively
  aligning teacher and student models across varied data types.
paper_authors._id: 67cfa99b7c95194db8d75468
paper_authors.name: Jongwoo Ko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:22:29.622Z
paper_authors._id: 64b7628af902508f0d7ae112
paper_authors.avatarUrl: /avatars/83c155254486e80c1dfd14676fdf9215.svg
paper_authors.fullname: Jongwoo Ko
paper_authors.type: user
paper_authors.user: jongwooko
paper_authors._id: 67cfa99b7c95194db8d75469
paper_authors.name: Tianyi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:22:27.139Z
paper_authors._id: 64ad94f05a4a60156925ec96
paper_authors.avatarUrl: /avatars/643bdb076e703bfcc89cec6fccb756c6.svg
paper_authors.fullname: Tianyi Chen
paper_authors.type: user
paper_authors.user: tianyic
paper_authors._id: 67cfa99b7c95194db8d7546a
paper_authors.name: Sungnyun Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T11:03:03.613Z
paper_authors._id: 63f0c2ac9cf89c9ed1bdd25c
paper_authors.avatarUrl: /avatars/856b2cb482250fb83c6fe793e29dfd74.svg
paper_authors.fullname: Sungnyun Kim
paper_authors.type: user
paper_authors.user: sungnyun
paper_authors._id: 67cfa99b7c95194db8d7546b
paper_authors.name: Tianyu Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T11:03:09.967Z
paper_authors._id: 646b0c6d3e2a7b0659452fc7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646b0c6d3e2a7b0659452fc7/aqT9nbrJtOk38tDw_12xW.jpeg
paper_authors.fullname: Tianyu Ding
paper_authors.type: user
paper_authors.user: tding1
paper_authors._id: 67cfa99b7c95194db8d7546c
paper_authors.name: Luming Liang
paper_authors._id: 67cfa99b7c95194db8d7546d
paper_authors.name: Ilya Zharkov
paper_authors._id: 67cfa99b7c95194db8d7546e
paper_authors.name: Se-Young Yun
paper_title: DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
paper_summary: Despite the success of distillation in large language models (LLMs), most
  prior work applies identical loss functions to both teacher- and
  student-generated data. These strategies overlook the synergy between loss
  formulations and data types, leading to a suboptimal performance boost in
  student models. To address this, we propose DistiLLM-2, a contrastive approach
  that simultaneously increases the likelihood of teacher responses and decreases
  that of student responses by harnessing this synergy. Our extensive experiments
  show that DistiLLM-2 not only builds high-performing student models across a
  wide range of tasks, including instruction-following and code generation, but
  also supports diverse applications, such as preference alignment and
  vision-language extensions. These findings highlight the potential of a
  contrastive approach to enhance the efficacy of LLM distillation by effectively
  aligning teacher and student models across varied data types.
paper_ai_summary: A contrastive distillation method (DistiLLM-2) enhances student language models by optimizing teacher and student responses differently, improving performance across various tasks and applications.
paper_ai_keywords: distillation large language models (LLMs) teacher- and student-generated data contrastive approach instruction-following code generation preference alignment vision-language extensions