title: A Common Pitfall of Margin-based Language Model Alignment: Gradient
    Entanglement
summary: Reinforcement Learning from Human Feedback (RLHF) has become the predominant
  approach for language model (LM) alignment. At its core, RLHF uses a
  margin-based loss for preference optimization, specifying ideal LM behavior
  only by the difference between preferred and dispreferred responses. In this
  paper, we identify a common pitfall of margin-based methods -- the
  under-specification of ideal LM behavior on preferred and dispreferred
  responses individually, which leads to two unintended consequences as the
  margin increases: (1) The probability of dispreferred (e.g., unsafe) responses
  may increase, resulting in potential safety alignment failures. (2) The
  probability of preferred responses may decrease, even when those responses are
  ideal. We demystify the reasons behind these problematic behaviors:
  margin-based losses couple the change in the preferred probability to the
  gradient of the dispreferred one, and vice versa, often preventing the
  preferred probability from increasing while the dispreferred one decreases, and
  thus causing a synchronized increase or decrease in both probabilities. We term
  this effect, inherent in margin-based objectives, gradient entanglement.
  Formally, we derive conditions for general margin-based alignment objectives
  under which gradient entanglement becomes concerning: the inner product of the
  gradients of preferred and dispreferred log-probabilities is large relative to
  the individual gradient norms. We theoretically investigate why such inner
  products can be large when aligning language models and empirically validate
  our findings. Empirical implications of our framework extend to explaining
  important differences in the training dynamics of various preference
  optimization algorithms, and suggesting potential algorithm designs to mitigate
  the under-specification issue of margin-based methods and thereby improving
  language model alignment.
paper_authors._id: 6711dffcff45f0171ce2a393
paper_authors.name: Hui Yuan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-18T04:11:40.965Z
paper_authors._id: 65256f6ec6a6a53e2e693fc7
paper_authors.avatarUrl: /avatars/928f29207c196bbffd53bb3b4a3bfd17.svg
paper_authors.fullname: Hui Yuan
paper_authors.type: user
paper_authors.user: huiyuan23
paper_authors._id: 6711dffcff45f0171ce2a394
paper_authors.name: Yifan Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:09:54.399Z
paper_authors._id: 6245285af59b8d262df3321b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6245285af59b8d262df3321b/dvy__dTf-miJ60IbveDg4.jpeg
paper_authors.fullname: Yifan Zeng
paper_authors.type: user
paper_authors.user: yokey
paper_authors._id: 6711dffcff45f0171ce2a395
paper_authors.name: Yue Wu
paper_authors._id: 6711dffcff45f0171ce2a396
paper_authors.name: Huazheng Wang
paper_authors._id: 6711dffcff45f0171ce2a397
paper_authors.name: Mengdi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:17:19.140Z
paper_authors._id: 6599415e8c8ac79295e0b5e3
paper_authors.avatarUrl: /avatars/85500bc8d2cd51444adcc19b1f8db313.svg
paper_authors.fullname: Mengdi Wang
paper_authors.type: user
paper_authors.user: Edify-Kd2024
paper_authors._id: 6711dffcff45f0171ce2a398
paper_authors.name: Liu Leqi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:18:14.977Z
paper_authors._id: 62791751bba7c62cbab1076e
paper_authors.avatarUrl: /avatars/9efb4f809d8961b47d3ae687f6e8dc2f.svg
paper_authors.fullname: Liu Leqi
paper_authors.type: user
paper_authors.user: leqiliu
paper_title: A Common Pitfall of Margin-based Language Model Alignment: Gradient
    Entanglement
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has become the predominant
  approach for language model (LM) alignment. At its core, RLHF uses a
  margin-based loss for preference optimization, specifying ideal LM behavior
  only by the difference between preferred and dispreferred responses. In this
  paper, we identify a common pitfall of margin-based methods -- the
  under-specification of ideal LM behavior on preferred and dispreferred
  responses individually, which leads to two unintended consequences as the
  margin increases: (1) The probability of dispreferred (e.g., unsafe) responses
  may increase, resulting in potential safety alignment failures. (2) The
  probability of preferred responses may decrease, even when those responses are
  ideal. We demystify the reasons behind these problematic behaviors:
  margin-based losses couple the change in the preferred probability to the
  gradient of the dispreferred one, and vice versa, often preventing the
  preferred probability from increasing while the dispreferred one decreases, and
  thus causing a synchronized increase or decrease in both probabilities. We term
  this effect, inherent in margin-based objectives, gradient entanglement.
  Formally, we derive conditions for general margin-based alignment objectives
  under which gradient entanglement becomes concerning: the inner product of the
  gradients of preferred and dispreferred log-probabilities is large relative to
  the individual gradient norms. We theoretically investigate why such inner
  products can be large when aligning language models and empirically validate
  our findings. Empirical implications of our framework extend to explaining
  important differences in the training dynamics of various preference
  optimization algorithms, and suggesting potential algorithm designs to mitigate
  the under-specification issue of margin-based methods and thereby improving
  language model alignment.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) language model (LM) align margin-based loss preference optimization preferred responses dispreferred responses gradient entanglement inner product gradient norms training dynamics preference optimization algorithms