Course-Correction: Safety Alignment Using Synthetic Preferences
The risk of harmful content generated by large language models (LLMs) becomes
a critical concern. This paper presents a systematic study on assessing and
improving LLMs' capability to perform the task of course-correction,
\ie, the model can steer away from generating harmful content autonomously. To
start with, we introduce the C^2-Eval benchmark for quantitative
assessment and analyze 10 popular LLMs, revealing varying proficiency of
current safety-tuned LLMs in course-correction. To improve, we propose
fine-tuning LLMs with preference learning, emphasizing the preference for
timely course-correction. Using an automated pipeline, we create
C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
teach models the concept of timely course-correction through data-driven
preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
Qwen2 7B, show that our method effectively enhances course-correction
skills without affecting general performance. Additionally, it effectively
improves LLMs' safety, particularly in resisting jailbreak attacks.
66a378e49b6c8a5bd7ddf2b4
Rongwu Xu
claimed_verified
2024-07-26T10:26:39.119Z
64ed86a5b6d4b4b02a9190cf
/avatars/8b92a24566ecb3489a3adac0586d4dd6.svg
Rongwu Xu
user
pillowsofwind
66a378e49b6c8a5bd7ddf2b5
Yishuo Cai
66a378e49b6c8a5bd7ddf2b6
Zhenhong Zhou
66a378e49b6c8a5bd7ddf2b7
Renjie Gu
66a378e49b6c8a5bd7ddf2b8
Haiqin Weng
admin_assigned
2024-07-26T12:20:26.728Z
63fef648c0ec83fda43da041
/avatars/a24ce8fc339f9b36237d241f741472ea.svg
Haiqin Weng
user
haiqin
66a378e49b6c8a5bd7ddf2b9
Yan Liu
66a378e49b6c8a5bd7ddf2ba
Tianwei Zhang
66a378e49b6c8a5bd7ddf2bb
Wei Xu
66a378e49b6c8a5bd7ddf2bc
Han Qiu
Course-Correction: Safety Alignment Using Synthetic Preferences
The risk of harmful content generated by large language models (LLMs) becomes
a critical concern. This paper presents a systematic study on assessing and
improving LLMs' capability to perform the task of course-correction,
\ie, the model can steer away from generating harmful content autonomously. To
start with, we introduce the C^2-Eval benchmark for quantitative
assessment and analyze 10 popular LLMs, revealing varying proficiency of
current safety-tuned LLMs in course-correction. To improve, we propose
fine-tuning LLMs with preference learning, emphasizing the preference for
timely course-correction. Using an automated pipeline, we create
C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
teach models the concept of timely course-correction through data-driven
preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
Qwen2 7B, show that our method effectively enhances course-correction
skills without affecting general performance. Additionally, it effectively
improves LLMs' safety, particularly in resisting jailbreak attacks.
The study assesses and enhances large language models' capability to autonomously avoid harmful content generation through a benchmark and preference learning fine-tuning.
large language models LLMs course-correction C$^2$-Eval preference learning C$^2$-Syn jailbreak attacks