title: Course-Correction: Safety Alignment Using Synthetic Preferences
summary: The risk of harmful content generated by large language models (LLMs) becomes
  a critical concern. This paper presents a systematic study on assessing and
  improving LLMs' capability to perform the task of course-correction,
  \ie, the model can steer away from generating harmful content autonomously. To
  start with, we introduce the C^2-Eval benchmark for quantitative
  assessment and analyze 10 popular LLMs, revealing varying proficiency of
  current safety-tuned LLMs in course-correction. To improve, we propose
  fine-tuning LLMs with preference learning, emphasizing the preference for
  timely course-correction. Using an automated pipeline, we create
  C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
  teach models the concept of timely course-correction through data-driven
  preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
  Qwen2 7B, show that our method effectively enhances course-correction
  skills without affecting general performance. Additionally, it effectively
  improves LLMs' safety, particularly in resisting jailbreak attacks.
paper_authors._id: 66a378e49b6c8a5bd7ddf2b4
paper_authors.name: Rongwu Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T10:26:39.119Z
paper_authors._id: 64ed86a5b6d4b4b02a9190cf
paper_authors.avatarUrl: /avatars/8b92a24566ecb3489a3adac0586d4dd6.svg
paper_authors.fullname: Rongwu Xu
paper_authors.type: user
paper_authors.user: pillowsofwind
paper_authors._id: 66a378e49b6c8a5bd7ddf2b5
paper_authors.name: Yishuo Cai
paper_authors._id: 66a378e49b6c8a5bd7ddf2b6
paper_authors.name: Zhenhong Zhou
paper_authors._id: 66a378e49b6c8a5bd7ddf2b7
paper_authors.name: Renjie Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T12:20:33.035Z
paper_authors._id: 668657b4c9ffbedc86831006
paper_authors.avatarUrl: /avatars/0d766ccde2e9890c1d61cc5d04e1d692.svg
paper_authors.fullname: Renjie Gu
paper_authors.type: user
paper_authors.user: csuJC
paper_authors._id: 66a378e49b6c8a5bd7ddf2b8
paper_authors.name: Haiqin Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T12:20:26.728Z
paper_authors._id: 63fef648c0ec83fda43da041
paper_authors.avatarUrl: /avatars/a24ce8fc339f9b36237d241f741472ea.svg
paper_authors.fullname: Haiqin Weng
paper_authors.type: user
paper_authors.user: haiqin
paper_authors._id: 66a378e49b6c8a5bd7ddf2b9
paper_authors.name: Yan Liu
paper_authors._id: 66a378e49b6c8a5bd7ddf2ba
paper_authors.name: Tianwei Zhang
paper_authors._id: 66a378e49b6c8a5bd7ddf2bb
paper_authors.name: Wei Xu
paper_authors._id: 66a378e49b6c8a5bd7ddf2bc
paper_authors.name: Han Qiu
paper_title: Course-Correction: Safety Alignment Using Synthetic Preferences
paper_summary: The risk of harmful content generated by large language models (LLMs) becomes
  a critical concern. This paper presents a systematic study on assessing and
  improving LLMs' capability to perform the task of course-correction,
  \ie, the model can steer away from generating harmful content autonomously. To
  start with, we introduce the C^2-Eval benchmark for quantitative
  assessment and analyze 10 popular LLMs, revealing varying proficiency of
  current safety-tuned LLMs in course-correction. To improve, we propose
  fine-tuning LLMs with preference learning, emphasizing the preference for
  timely course-correction. Using an automated pipeline, we create
  C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
  teach models the concept of timely course-correction through data-driven
  preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
  Qwen2 7B, show that our method effectively enhances course-correction
  skills without affecting general performance. Additionally, it effectively
  improves LLMs' safety, particularly in resisting jailbreak attacks.
paper_ai_keywords: large language models (LLMs) course-correction C$^2$-Eval preference learning C$^2$-Syn jailbreak attacks