Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive
  Toxicity Reduction
Safety fine-tuning algorithms are commonly used to fine-tune language models
to reduce harmful outputs, but the exact internal mechanisms of how those
models achieve this remain unclear. In studying direct preference optimisation
(DPO) for toxicity reduction, current explanations claim that DPO works by
dampening the most toxic MLP neurons to learn an offset to avert toxic regions
in the residual stream. However, by ablating the most toxic neurons and
applying activation patching, we find this explanation incomplete. By
projecting neuron activation changes onto a toxicity probe, we find that only
31.8\% of toxicity reduction comes from dampened toxic neurons. Instead, DPO
reduces toxicity by accumulating effects across multiple neuron groups, both
reducing writing in the toxic direction and promoting anti-toxicity in the
residual stream. Moreover, DPO gives noisy adjustments to neuron activations,
with many neurons actually increasing toxicity. This indicates that DPO is a
balancing process between opposing neuron effects to achieve toxicity
reduction.
67331930fabf0f0e471a19f1
Yushi Yang
67331930fabf0f0e471a19f2
Filip Sondej
67331930fabf0f0e471a19f3
Harry Mayne
67331930fabf0f0e471a19f4
Adam Mahdi
claimed_verified
2024-11-20T10:09:19.905Z
673cf9c93af47d1d2b6584c9
/avatars/0792921a8adaa1493255abb0b5aaa4f1.svg
Adam Mahdi
user
ammaox
Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive
  Toxicity Reduction
Safety fine-tuning algorithms are commonly used to fine-tune language models
to reduce harmful outputs, but the exact internal mechanisms of how those
models achieve this remain unclear. In studying direct preference optimisation
(DPO) for toxicity reduction, current explanations claim that DPO works by
dampening the most toxic MLP neurons to learn an offset to avert toxic regions
in the residual stream. However, by ablating the most toxic neurons and
applying activation patching, we find this explanation incomplete. By
projecting neuron activation changes onto a toxicity probe, we find that only
31.8\% of toxicity reduction comes from dampened toxic neurons. Instead, DPO
reduces toxicity by accumulating effects across multiple neuron groups, both
reducing writing in the toxic direction and promoting anti-toxicity in the
residual stream. Moreover, DPO gives noisy adjustments to neuron activations,
with many neurons actually increasing toxicity. This indicates that DPO is a
balancing process between opposing neuron effects to achieve toxicity
reduction.
Direct preference optimization reduces toxicity in language models by balancing neuron effects across multiple groups rather than solely dampening the most toxic neurons.
direct preference optimisation DPO toxicity reduction MLP neurons residual stream neuron activation changes toxicity probe anti-toxicity