On-Policy Distillation of Language Models: Learning from Self-Generated
  Mistakes
Knowledge distillation (KD) is widely used for compressing a teacher model to
reduce its inference cost and memory footprint, by training a smaller student
model. However, current KD methods for auto-regressive sequence models suffer
from distribution mismatch between output sequences seen during training and
those generated by the student during inference. To address this issue, we
introduce Generalized Knowledge Distillation (GKD). Instead of solely relying
on a fixed set of output sequences, GKD trains the student on its
self-generated output sequences by leveraging feedback from the teacher on such
sequences. Unlike supervised KD approaches, GKD also offers the flexibility to
employ alternative loss functions between the student and teacher, which can be
useful when the student lacks the expressivity to mimic the teacher's
distribution. Furthermore, GKD facilitates the seamless integration of
distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for
distilling auto-regressive language models on summarization, translation, and
arithmetic reasoning tasks, and task-agnostic distillation for
instruction-tuning.
649914c20651355205f517e4
Rishabh Agarwal
extracted_pending
2023-06-26T04:32:02.566Z
647875e37b370854241b89a8
/avatars/fd25f7e5960e4e132b9ee689f114820b.svg
Rishabh Agarwal
user
agarwl
649914c20651355205f517e5
Nino Vieillard
668267b2b491884b4e1e3531
Yongchao Zhou
649914c20651355205f517e6
Piotr Stanczyk
649914c20651355205f517e7
Sabela Ramos
649914c20651355205f517e8
Matthieu Geist
claimed_verified
2025-05-19T13:09:47.891Z
67bef153eaca0bef78ed5478
/avatars/ab838c541bd170986370a99670981c2f.svg
Matthieu Geist
user
matthieu-geist
649914c20651355205f517e9
Olivier Bachem
On-Policy Distillation of Language Models: Learning from Self-Generated
  Mistakes
Knowledge distillation (KD) is widely used for compressing a teacher model to
reduce its inference cost and memory footprint, by training a smaller student
model. However, current KD methods for auto-regressive sequence models suffer
from distribution mismatch between output sequences seen during training and
those generated by the student during inference. To address this issue, we
introduce Generalized Knowledge Distillation (GKD). Instead of solely relying
on a fixed set of output sequences, GKD trains the student on its
self-generated output sequences by leveraging feedback from the teacher on such
sequences. Unlike supervised KD approaches, GKD also offers the flexibility to
employ alternative loss functions between the student and teacher, which can be
useful when the student lacks the expressivity to mimic the teacher's
distribution. Furthermore, GKD facilitates the seamless integration of
distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for
distilling auto-regressive language models on summarization, translation, and
arithmetic reasoning tasks, and task-agnostic distillation for
instruction-tuning.
Generalized Knowledge Distillation addresses distribution mismatch in auto-regressive sequence models by training the student on its own output sequences and integrates with RL fine-tuning for effective distillation in tasks like summarization and instruction-tuning.
Knowledge distillation auto-regressive sequence models distribution mismatch Generalized Knowledge Distillation student model teacher model output sequences feedback alternative loss functions RL fine-tuning RLHF auto-regressive language models summarization translation arithmetic reasoning task-agnostic distillation instruction-tuning