Course-Correction: Safety Alignment Using Synthetic Preferences
<|reference_start|>Course-Correction: Safety Alignment Using Synthetic Preferences: The risk of harmful content generated by large language models (LLMs) becomes a critical concern. This paper presents a systematic study on assessing and improving LLMs' capability to perform the task of \textbf{course-correction}, \ie, the model can steer away from generating harmful content autonomously. To start with, we introduce the \textsc{C$^2$-Eval} benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction. To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create \textsc{C$^2$-Syn}, a synthetic dataset with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven preference learning. Experiments on 2 LLMs, \textsc{Llama2-Chat 7B} and \textsc{Qwen2 7B}, show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs' safety, particularly in resisting jailbreak attacks.<|reference_end|>
@article{xu2024course-correction:,
    title={Course-Correction: Safety Alignment Using Synthetic Preferences},
    author={Rongwu Xu, Yishuo Cai, Zhenhong Zhou, Renjie Gu, Haiqin Weng, Yan Liu,
  Tianwei Zhang, Wei Xu, Han Qiu},
    journal={arXiv preprint arXiv:2407.16637},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.16637},
    primaryClass={cs.CL cs.AI cs.LG}
}