title: Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive
    Toxicity Reduction
summary: Safety fine-tuning algorithms are commonly used to fine-tune language models
  to reduce harmful outputs, but the exact internal mechanisms of how those
  models achieve this remain unclear. In studying direct preference optimisation
  (DPO) for toxicity reduction, current explanations claim that DPO works by
  dampening the most toxic MLP neurons to learn an offset to avert toxic regions
  in the residual stream. However, by ablating the most toxic neurons and
  applying activation patching, we find this explanation incomplete. By
  projecting neuron activation changes onto a toxicity probe, we find that only
  31.8\% of toxicity reduction comes from dampened toxic neurons. Instead, DPO
  reduces toxicity by accumulating effects across multiple neuron groups, both
  reducing writing in the toxic direction and promoting anti-toxicity in the
  residual stream. Moreover, DPO gives noisy adjustments to neuron activations,
  with many neurons actually increasing toxicity. This indicates that DPO is a
  balancing process between opposing neuron effects to achieve toxicity
  reduction.
paper_authors._id: 67331930fabf0f0e471a19f1
paper_authors.name: Yushi Yang
paper_authors._id: 67331930fabf0f0e471a19f2
paper_authors.name: Filip Sondej
paper_authors._id: 67331930fabf0f0e471a19f3
paper_authors.name: Harry Mayne
paper_authors._id: 67331930fabf0f0e471a19f4
paper_authors.name: Adam Mahdi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:09:19.905Z
paper_authors._id: 673cf9c93af47d1d2b6584c9
paper_authors.avatarUrl: /avatars/0792921a8adaa1493255abb0b5aaa4f1.svg
paper_authors.fullname: Adam Mahdi
paper_authors.type: user
paper_authors.user: ammaox
paper_title: Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive
    Toxicity Reduction
paper_summary: Safety fine-tuning algorithms are commonly used to fine-tune language models
  to reduce harmful outputs, but the exact internal mechanisms of how those
  models achieve this remain unclear. In studying direct preference optimisation
  (DPO) for toxicity reduction, current explanations claim that DPO works by
  dampening the most toxic MLP neurons to learn an offset to avert toxic regions
  in the residual stream. However, by ablating the most toxic neurons and
  applying activation patching, we find this explanation incomplete. By
  projecting neuron activation changes onto a toxicity probe, we find that only
  31.8\% of toxicity reduction comes from dampened toxic neurons. Instead, DPO
  reduces toxicity by accumulating effects across multiple neuron groups, both
  reducing writing in the toxic direction and promoting anti-toxicity in the
  residual stream. Moreover, DPO gives noisy adjustments to neuron activations,
  with many neurons actually increasing toxicity. This indicates that DPO is a
  balancing process between opposing neuron effects to achieve toxicity
  reduction.
paper_ai_summary: Direct preference optimization reduces toxicity in language models by balancing neuron effects across multiple groups rather than solely dampening the most toxic neurons.
paper_ai_keywords: direct preference optimisation DPO toxicity reduction MLP neurons residual stream neuron activation changes toxicity probe anti-toxicity