Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller
<|reference_start|>Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller: We propose Self-Control, a novel method utilizing suffix gradients to control the behavior of large language models (LLMs) without explicit human annotations. Given a guideline expressed in suffix string and the model's self-assessment of adherence, Self-Control computes the gradient of this self-judgment concerning the model's hidden states, directly influencing the auto-regressive generation process towards desired behaviors. To enhance efficiency, we introduce Self-Control_{prefix}, a compact module that encapsulates the learned representations from suffix gradients into a Prefix Controller, facilitating inference-time control for various LLM behaviors. Our experiments demonstrate Self-Control's efficacy across multiple domains, including emotional modulation, ensuring harmlessness, and enhancing complex reasoning. Especially, Self-Control_{prefix} enables a plug-and-play control and jointly controls multiple attributes, improving model outputs without altering model parameters or increasing inference-time costs.<|reference_end|>
@article{cai2024self-control,
    title={Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix
  Controller},
    author={Min Cai and Yuchen Zhang and Shichang Zhang and Fan Yin and Dan Zhang
  and Difan Zou and Yisong Yue and Ziniu Hu},
    journal={arXiv preprint arXiv:2406.02721},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02721},
    primaryClass={cs.CL cs.AI}
}