title: Principle-Driven Self-Alignment of Language Models from Scratch with
    Minimal Human Supervision
summary: Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised
  fine-tuning (SFT) with human annotations and reinforcement learning from human
  feedback (RLHF) to align the output of large language models (LLMs) with human
  intentions, ensuring they are helpful, ethical, and reliable. However, this
  dependence can significantly constrain the true potential of AI-assistant
  agents due to the high cost of obtaining human supervision and the related
  issues on quality, reliability, diversity, self-consistency, and undesirable
  biases. To address these challenges, we propose a novel approach called
  SELF-ALIGN, which combines principle-driven reasoning and the generative power
  of LLMs for the self-alignment of AI agents with minimal human supervision. Our
  approach encompasses four stages: first, we use an LLM to generate synthetic
  prompts, and a topic-guided method to augment the prompt diversity; second, we
  use a small set of human-written principles for AI models to follow, and guide
  the LLM through in-context learning from demonstrations (of principles
  application) to produce helpful, ethical, and reliable responses to user's
  queries; third, we fine-tune the original LLM with the high-quality
  self-aligned responses so that the resulting model can generate desirable
  responses for each query directly without the principle set and the
  demonstrations anymore; and finally, we offer a refinement step to address the
  issues of overly-brief or indirect responses. Applying SELF-ALIGN to the
  LLaMA-65b base language model, we develop an AI assistant named Dromedary. With
  fewer than 300 lines of human annotations (including < 200 seed prompts, 16
  generic principles, and 5 exemplars for in-context learning). Dromedary
  significantly surpasses the performance of several state-of-the-art AI systems,
  including Text-Davinci-003 and Alpaca, on benchmark datasets with various
  settings.
paper_authors._id: 64545b4736821f6860fef22a
paper_authors.name: Zhiqing Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-08T09:27:42.694Z
paper_authors._id: 5ee24411636bdb3834e2da24
paper_authors.avatarUrl: /avatars/0e12a0b19ada4138ddf808501ba68423.svg
paper_authors.fullname: Zhiqing Sun
paper_authors.type: user
paper_authors.user: zhiqings
paper_authors._id: 64545b4736821f6860fef22b
paper_authors.name: Yikang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:05:14.032Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_authors._id: 64545b4736821f6860fef22c
paper_authors.name: Qinhong Zhou
paper_authors._id: 64545b4736821f6860fef22d
paper_authors.name: Hongxin Zhang
paper_authors._id: 64545b4736821f6860fef22e
paper_authors.name: Zhenfang Chen
paper_authors._id: 64545b4736821f6860fef22f
paper_authors.name: David Cox
paper_authors._id: 64545b4736821f6860fef230
paper_authors.name: Yiming Yang
paper_authors._id: 64545b4736821f6860fef231
paper_authors.name: Chuang Gan
paper_title: Principle-Driven Self-Alignment of Language Models from Scratch with
    Minimal Human Supervision
paper_summary: Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised
  fine-tuning (SFT) with human annotations and reinforcement learning from human
  feedback (RLHF) to align the output of large language models (LLMs) with human
  intentions, ensuring they are helpful, ethical, and reliable. However, this
  dependence can significantly constrain the true potential of AI-assistant
  agents due to the high cost of obtaining human supervision and the related
  issues on quality, reliability, diversity, self-consistency, and undesirable
  biases. To address these challenges, we propose a novel approach called
  SELF-ALIGN, which combines principle-driven reasoning and the generative power
  of LLMs for the self-alignment of AI agents with minimal human supervision. Our
  approach encompasses four stages: first, we use an LLM to generate synthetic
  prompts, and a topic-guided method to augment the prompt diversity; second, we
  use a small set of human-written principles for AI models to follow, and guide
  the LLM through in-context learning from demonstrations (of principles
  application) to produce helpful, ethical, and reliable responses to user's
  queries; third, we fine-tune the original LLM with the high-quality
  self-aligned responses so that the resulting model can generate desirable
  responses for each query directly without the principle set and the
  demonstrations anymore; and finally, we offer a refinement step to address the
  issues of overly-brief or indirect responses. Applying SELF-ALIGN to the
  LLaMA-65b base language model, we develop an AI assistant named Dromedary. With
  fewer than 300 lines of human annotations (including < 200 seed prompts, 16
  generic principles, and 5 exemplars for in-context learning). Dromedary
  significantly surpasses the performance of several state-of-the-art AI systems,
  including Text-Davinci-003 and Alpaca, on benchmark datasets with various
  settings.
paper_ai_summary: SELF-ALIGN enhances AI assistants with minimal human supervision by leveraging LLMs for synthetic prompts, in-context learning, and self-alignment to improve helpfulness, ethics, and reliability.
paper_ai_keywords: supervised fine-tuning (SFT) reinforcement learning from human feedback (RLHF) large language models (LLMs) principle-driven reasoning synthetic prompts topic-guided method in-context learning fine-tuning self-aligned responses