title: On Teacher Hacking in Language Model Distillation
summary: Post-training of language models (LMs) increasingly relies on the following
  two stages: (i) knowledge distillation, where the LM is trained to imitate a
  larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF),
  where the LM is aligned by optimizing a reward model. In the second RLHF stage,
  a well-known challenge is reward hacking, where the LM over-optimizes the
  reward model. Such phenomenon is in line with Goodhart's law and can lead to
  degraded performance on the true objective. In this paper, we investigate
  whether a similar phenomenon, that we call teacher hacking, can occur during
  knowledge distillation. This could arise because the teacher LM is itself an
  imperfect approximation of the true distribution. To study this, we propose a
  controlled experimental setup involving: (i) an oracle LM representing the
  ground-truth distribution, (ii) a teacher LM distilled from the oracle, and
  (iii) a student LM distilled from the teacher. Our experiments reveal the
  following insights. When using a fixed offline dataset for distillation,
  teacher hacking occurs; moreover, we can detect it by observing when the
  optimization process deviates from polynomial convergence laws. In contrast,
  employing online data generation techniques effectively mitigates teacher
  hacking. More precisely, we identify data diversity as the key factor in
  preventing hacking. Overall, our findings provide a deeper understanding of the
  benefits and limitations of distillation for building robust and efficient LMs.
paper_authors._id: 67a495ce0f2d0f0303a3af71
paper_authors.name: Daniil Tiapkin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:19.738Z
paper_authors._id: 6262880c5eb4fa93219f0064
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6262880c5eb4fa93219f0064/6yyBvRK4Oh7OhjaaweaVN.jpeg
paper_authors.fullname: Daniil Tiapkin
paper_authors.type: user
paper_authors.user: dtiapkin
paper_authors._id: 67a495ce0f2d0f0303a3af72
paper_authors.name: Daniele Calandriello
paper_authors._id: 67a495ce0f2d0f0303a3af73
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T16:12:15.183Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 67a495ce0f2d0f0303a3af74
paper_authors.name: Sarah Perrin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T16:12:39.678Z
paper_authors._id: 66328157b270ae503e91339b
paper_authors.avatarUrl: /avatars/ea7a52060f5360f523ca28e137e85e33.svg
paper_authors.fullname: Sarah Perrin
paper_authors.type: user
paper_authors.user: Sper42
paper_authors._id: 67a495ce0f2d0f0303a3af75
paper_authors.name: Nino Vieillard
paper_authors._id: 67a495ce0f2d0f0303a3af76
paper_authors.name: Alexandre Ramé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T16:12:53.881Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 67a495ce0f2d0f0303a3af77
paper_authors.name: Mathieu Blondel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T16:13:01.896Z
paper_authors._id: 66d093b681e0683bca48bed6
paper_authors.avatarUrl: /avatars/cc1fbccb0b6aa93d648bcbdf9c3a35e1.svg
paper_authors.fullname: Mathieu Blondel
paper_authors.type: user
paper_authors.user: mblondel
paper_title: On Teacher Hacking in Language Model Distillation
paper_summary: Post-training of language models (LMs) increasingly relies on the following
  two stages: (i) knowledge distillation, where the LM is trained to imitate a
  larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF),
  where the LM is aligned by optimizing a reward model. In the second RLHF stage,
  a well-known challenge is reward hacking, where the LM over-optimizes the
  reward model. Such phenomenon is in line with Goodhart's law and can lead to
  degraded performance on the true objective. In this paper, we investigate
  whether a similar phenomenon, that we call teacher hacking, can occur during
  knowledge distillation. This could arise because the teacher LM is itself an
  imperfect approximation of the true distribution. To study this, we propose a
  controlled experimental setup involving: (i) an oracle LM representing the
  ground-truth distribution, (ii) a teacher LM distilled from the oracle, and
  (iii) a student LM distilled from the teacher. Our experiments reveal the
  following insights. When using a fixed offline dataset for distillation,
  teacher hacking occurs; moreover, we can detect it by observing when the
  optimization process deviates from polynomial convergence laws. In contrast,
  employing online data generation techniques effectively mitigates teacher
  hacking. More precisely, we identify data diversity as the key factor in
  preventing hacking. Overall, our findings provide a deeper understanding of the
  benefits and limitations of distillation for building robust and efficient LMs.
paper_ai_keywords: knowledge distillation reinforcement learning from human feedback (RLHF) reward hacking Goodhart's law teacher hacking oracle LM ground-truth distribution teacher LM student LM offline dataset polynomial convergence laws online data generation techniques data diversity