On Teacher Hacking in Language Model Distillation
Post-training of language models (LMs) increasingly relies on the following
two stages: (i) knowledge distillation, where the LM is trained to imitate a
larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF),
where the LM is aligned by optimizing a reward model. In the second RLHF stage,
a well-known challenge is reward hacking, where the LM over-optimizes the
reward model. Such phenomenon is in line with Goodhart's law and can lead to
degraded performance on the true objective. In this paper, we investigate
whether a similar phenomenon, that we call teacher hacking, can occur during
knowledge distillation. This could arise because the teacher LM is itself an
imperfect approximation of the true distribution. To study this, we propose a
controlled experimental setup involving: (i) an oracle LM representing the
ground-truth distribution, (ii) a teacher LM distilled from the oracle, and
(iii) a student LM distilled from the teacher. Our experiments reveal the
following insights. When using a fixed offline dataset for distillation,
teacher hacking occurs; moreover, we can detect it by observing when the
optimization process deviates from polynomial convergence laws. In contrast,
employing online data generation techniques effectively mitigates teacher
hacking. More precisely, we identify data diversity as the key factor in
preventing hacking. Overall, our findings provide a deeper understanding of the
benefits and limitations of distillation for building robust and efficient LMs.
67a495ce0f2d0f0303a3af71
Daniil Tiapkin
claimed_verified
2025-02-06T14:14:19.738Z
6262880c5eb4fa93219f0064
https://cdn-avatars.huggingface.co/v1/production/uploads/6262880c5eb4fa93219f0064/6yyBvRK4Oh7OhjaaweaVN.jpeg
Daniil Tiapkin
user
dtiapkin
67a495ce0f2d0f0303a3af72
Daniele Calandriello
67a495ce0f2d0f0303a3af73
Johan Ferret
admin_assigned
2025-02-06T16:12:15.183Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
67a495ce0f2d0f0303a3af74
Sarah Perrin
admin_assigned
2025-02-06T16:12:39.678Z
66328157b270ae503e91339b
/avatars/ea7a52060f5360f523ca28e137e85e33.svg
Sarah Perrin
user
Sper42
67a495ce0f2d0f0303a3af75
Nino Vieillard
67a495ce0f2d0f0303a3af76
Alexandre Ramé
admin_assigned
2025-02-06T16:12:53.881Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
67a495ce0f2d0f0303a3af77
Mathieu Blondel
admin_assigned
2025-02-06T16:13:01.896Z
66d093b681e0683bca48bed6
/avatars/cc1fbccb0b6aa93d648bcbdf9c3a35e1.svg
Mathieu Blondel
user
mblondel
On Teacher Hacking in Language Model Distillation
Post-training of language models (LMs) increasingly relies on the following
two stages: (i) knowledge distillation, where the LM is trained to imitate a
larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF),
where the LM is aligned by optimizing a reward model. In the second RLHF stage,
a well-known challenge is reward hacking, where the LM over-optimizes the
reward model. Such phenomenon is in line with Goodhart's law and can lead to
degraded performance on the true objective. In this paper, we investigate
whether a similar phenomenon, that we call teacher hacking, can occur during
knowledge distillation. This could arise because the teacher LM is itself an
imperfect approximation of the true distribution. To study this, we propose a
controlled experimental setup involving: (i) an oracle LM representing the
ground-truth distribution, (ii) a teacher LM distilled from the oracle, and
(iii) a student LM distilled from the teacher. Our experiments reveal the
following insights. When using a fixed offline dataset for distillation,
teacher hacking occurs; moreover, we can detect it by observing when the
optimization process deviates from polynomial convergence laws. In contrast,
employing online data generation techniques effectively mitigates teacher
hacking. More precisely, we identify data diversity as the key factor in
preventing hacking. Overall, our findings provide a deeper understanding of the
benefits and limitations of distillation for building robust and efficient LMs.
Experiments reveal that knowledge distillation can lead to teacher hacking, an issue akin to reward hacking, but it can be mitigated by using online data generation techniques that enhance data diversity.
knowledge distillation reinforcement learning from human feedback RLHF reward hacking Goodhart's law oracle LM teacher hacking polynomial convergence laws data diversity