title: On-Policy Distillation of Language Models: Learning from Self-Generated
    Mistakes
summary: Knowledge distillation (KD) is widely used for compressing a teacher model to
  reduce its inference cost and memory footprint, by training a smaller student
  model. However, current KD methods for auto-regressive sequence models suffer
  from distribution mismatch between output sequences seen during training and
  those generated by the student during inference. To address this issue, we
  introduce Generalized Knowledge Distillation (GKD). Instead of solely relying
  on a fixed set of output sequences, GKD trains the student on its
  self-generated output sequences by leveraging feedback from the teacher on such
  sequences. Unlike supervised KD approaches, GKD also offers the flexibility to
  employ alternative loss functions between the student and teacher, which can be
  useful when the student lacks the expressivity to mimic the teacher's
  distribution. Furthermore, GKD facilitates the seamless integration of
  distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for
  distilling auto-regressive language models on summarization, translation, and
  arithmetic reasoning tasks, and task-agnostic distillation for
  instruction-tuning.
paper_authors._id: 649914c20651355205f517e4
paper_authors.name: Rishabh Agarwal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-06-26T04:32:02.566Z
paper_authors._id: 647875e37b370854241b89a8
paper_authors.avatarUrl: /avatars/fd25f7e5960e4e132b9ee689f114820b.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: agarwl
paper_authors._id: 649914c20651355205f517e5
paper_authors.name: Nino Vieillard
paper_authors._id: 668267b2b491884b4e1e3531
paper_authors.name: Yongchao Zhou
paper_authors._id: 649914c20651355205f517e6
paper_authors.name: Piotr Stanczyk
paper_authors._id: 649914c20651355205f517e7
paper_authors.name: Sabela Ramos
paper_authors._id: 649914c20651355205f517e8
paper_authors.name: Matthieu Geist
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T13:09:47.891Z
paper_authors._id: 67bef153eaca0bef78ed5478
paper_authors.avatarUrl: /avatars/ab838c541bd170986370a99670981c2f.svg
paper_authors.fullname: Matthieu Geist
paper_authors.type: user
paper_authors.user: matthieu-geist
paper_authors._id: 649914c20651355205f517e9
paper_authors.name: Olivier Bachem
paper_title: On-Policy Distillation of Language Models: Learning from Self-Generated
    Mistakes
paper_summary: Knowledge distillation (KD) is widely used for compressing a teacher model to
  reduce its inference cost and memory footprint, by training a smaller student
  model. However, current KD methods for auto-regressive sequence models suffer
  from distribution mismatch between output sequences seen during training and
  those generated by the student during inference. To address this issue, we
  introduce Generalized Knowledge Distillation (GKD). Instead of solely relying
  on a fixed set of output sequences, GKD trains the student on its
  self-generated output sequences by leveraging feedback from the teacher on such
  sequences. Unlike supervised KD approaches, GKD also offers the flexibility to
  employ alternative loss functions between the student and teacher, which can be
  useful when the student lacks the expressivity to mimic the teacher's
  distribution. Furthermore, GKD facilitates the seamless integration of
  distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for
  distilling auto-regressive language models on summarization, translation, and
  arithmetic reasoning tasks, and task-agnostic distillation for
  instruction-tuning.
paper_ai_summary: Generalized Knowledge Distillation addresses distribution mismatch in auto-regressive sequence models by training the student on its own output sequences and integrates with RL fine-tuning for effective distillation in tasks like summarization and instruction-tuning.
paper_ai_keywords: Knowledge distillation auto-regressive sequence models distribution mismatch Generalized Knowledge Distillation student model teacher model output sequences feedback alternative loss functions RL fine-tuning RLHF auto-regressive language models summarization translation arithmetic reasoning task-agnostic distillation instruction-tuning