title: Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
summary: Large Language Models (LLMs) have demonstrated great potential as generalist
  assistants, showcasing powerful task understanding and problem-solving
  capabilities. To deploy LLMs as AI assistants, it is crucial that these models
  exhibit desirable behavioral traits, such as non-toxicity and resilience
  against jailbreak attempts. Current methods for detoxification or preventing
  jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement
  Learning from Human Feedback (RLHF), which requires finetuning billions of
  parameters through gradient descent with substantial computation cost.
  Furthermore, models modified through SFT and RLHF may deviate from the
  pretrained models, potentially leading to a degradation in foundational LLM
  capabilities. In this paper, we observe that surprisingly, directly editing a
  small subset of parameters can effectively modulate specific behaviors of LLMs,
  such as detoxification and resistance to jailbreaking. Specifically, for a
  behavior that we aim to avoid, we employ a linear classifier, which we term the
  behavior probe, to classify binary behavior labels within the hidden state
  space of the LLM. Using this probe, we introduce an algorithm to identify a
  critical subset of LLM parameters that significantly influence this targeted
  behavior. Then we directly edit these selected parameters by shifting them
  towards the behavior probe. Such a direct parameter editing method necessitates
  only inference-level computational resources. Experiments demonstrate that in
  the representative detoxification task, our approach achieves reductions of up
  to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen,
  while maintaining the LLM's general capabilities in areas such as common sense,
  question answering, and mathematics. Our code is available at
  https://github.com/lucywang720/model-surgery.
paper_authors._id: 6694bd9b64dc88cf5e83fb5a
paper_authors.name: Huanqian Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T08:58:14.171Z
paper_authors._id: 64ec52887e69909b1ad1ff0b
paper_authors.avatarUrl: /avatars/30bb6fda51093fc71e80d0313c13d15e.svg
paper_authors.fullname: Lucy Wang
paper_authors.type: user
paper_authors.user: Lucywang720
paper_authors._id: 6694bd9b64dc88cf5e83fb5b
paper_authors.name: Yang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:59:01.337Z
paper_authors._id: 649d475111592b1a765ac1a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
paper_authors.fullname: Yang Yue
paper_authors.type: user
paper_authors.user: Yang130
paper_authors._id: 6694bd9b64dc88cf5e83fb5c
paper_authors.name: Rui Lu
paper_authors._id: 6694bd9b64dc88cf5e83fb5d
paper_authors.name: Jingxin Shi
paper_authors._id: 6694bd9b64dc88cf5e83fb5e
paper_authors.name: Andrew Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-16T20:16:22.404Z
paper_authors._id: 630482fbce6b12280b18971d
paper_authors.avatarUrl: /avatars/b07f31fd970d736bdf574d56da7a5634.svg
paper_authors.fullname: Andrew Zhao
paper_authors.type: user
paper_authors.user: andrewzh
paper_authors._id: 6694bd9b64dc88cf5e83fb5f
paper_authors.name: Shenzhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:06.670Z
paper_authors._id: 6486dde1f74857df3f1a5828
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
paper_authors.fullname: Shenzhi Wang
paper_authors.type: user
paper_authors.user: shenzhi-wang
paper_authors._id: 6694bd9b64dc88cf5e83fb60
paper_authors.name: Shiji Song
paper_authors._id: 6694bd9b64dc88cf5e83fb61
paper_authors.name: Gao Huang
paper_title: Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
paper_summary: Large Language Models (LLMs) have demonstrated great potential as generalist
  assistants, showcasing powerful task understanding and problem-solving
  capabilities. To deploy LLMs as AI assistants, it is crucial that these models
  exhibit desirable behavioral traits, such as non-toxicity and resilience
  against jailbreak attempts. Current methods for detoxification or preventing
  jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement
  Learning from Human Feedback (RLHF), which requires finetuning billions of
  parameters through gradient descent with substantial computation cost.
  Furthermore, models modified through SFT and RLHF may deviate from the
  pretrained models, potentially leading to a degradation in foundational LLM
  capabilities. In this paper, we observe that surprisingly, directly editing a
  small subset of parameters can effectively modulate specific behaviors of LLMs,
  such as detoxification and resistance to jailbreaking. Specifically, for a
  behavior that we aim to avoid, we employ a linear classifier, which we term the
  behavior probe, to classify binary behavior labels within the hidden state
  space of the LLM. Using this probe, we introduce an algorithm to identify a
  critical subset of LLM parameters that significantly influence this targeted
  behavior. Then we directly edit these selected parameters by shifting them
  towards the behavior probe. Such a direct parameter editing method necessitates
  only inference-level computational resources. Experiments demonstrate that in
  the representative detoxification task, our approach achieves reductions of up
  to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen,
  while maintaining the LLM's general capabilities in areas such as common sense,
  question answering, and mathematics. Our code is available at
  https://github.com/lucywang720/model-surgery.
paper_ai_keywords: Supervised Fine-Tuning (SFT) Reinforcement Learning from Human Feedback (RLHF) hidden state space behavior probe direct parameter editing RealToxicityPrompts ToxiGen