title: Aligning Large Language Models through Synthetic Feedback
summary: Aligning large language models (LLMs) to human values has become increasingly
  important as it enables sophisticated steering of LLMs, e.g., making them
  follow given instructions while keeping them less toxic. However, it requires a
  significant amount of human demonstrations and feedback. Recently, open-sourced
  models have attempted to replicate the alignment learning process by distilling
  data from already aligned LLMs like InstructGPT or ChatGPT. While this process
  reduces human efforts, constructing these datasets has a heavy dependency on
  the teacher models. In this work, we propose a novel framework for alignment
  learning with almost no human labor and no dependency on pre-aligned LLMs.
  First, we perform reward modeling (RM) with synthetic feedback by contrasting
  responses from vanilla LLMs with various sizes and prompts. Then, we use the RM
  for simulating high-quality demonstrations to train a supervised policy and for
  further optimizing the model with reinforcement learning. Our resulting model,
  Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms
  open-sourced models, including Alpaca, Dolly, and OpenAssistant, which are
  trained on the outputs of InstructGPT or human-annotated instructions. Our
  7B-sized model outperforms the 12-13B models in the A/B tests using GPT-4 as
  the judge with about 75% winning rate on average.
paper_authors._id: 646d603a4e59b82995e24fd0
paper_authors.name: Sungdong Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T10:45:45.558Z
paper_authors._id: 6084ebe956f7623a0b849153
paper_authors.avatarUrl: /avatars/c56fb056800c145e5a8234e27b6b5a31.svg
paper_authors.fullname: Sungdong kim
paper_authors.type: user
paper_authors.user: dsksd
paper_authors._id: 646d603a4e59b82995e24fd1
paper_authors.name: Sanghwan Bae
paper_authors._id: 646d603a4e59b82995e24fd2
paper_authors.name: Jamin Shin
paper_authors._id: 646d603a4e59b82995e24fd3
paper_authors.name: Soyoung Kang
paper_authors._id: 646d603a4e59b82995e24fd4
paper_authors.name: Donghyun Kwak
paper_authors._id: 646d603a4e59b82995e24fd5
paper_authors.name: Kang Min Yoo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:57:42.269Z
paper_authors._id: 5f2294832128ba01bdf08c66
paper_authors.avatarUrl: /avatars/334082608dd69a44b826fac00b36aa0a.svg
paper_authors.fullname: Kang Min Yoo
paper_authors.type: user
paper_authors.user: kmyoo
paper_authors._id: 646d603a4e59b82995e24fd6
paper_authors.name: Minjoon Seo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-08T09:15:44.085Z
paper_authors._id: 621f05ba970615ad5861ceb1
paper_authors.avatarUrl: /avatars/7e1902aa71369a524afda9b0a9e88e22.svg
paper_authors.fullname: Minjoon Seo
paper_authors.type: user
paper_authors.user: minjoon
paper_title: Aligning Large Language Models through Synthetic Feedback
paper_summary: Aligning large language models (LLMs) to human values has become increasingly
  important as it enables sophisticated steering of LLMs, e.g., making them
  follow given instructions while keeping them less toxic. However, it requires a
  significant amount of human demonstrations and feedback. Recently, open-sourced
  models have attempted to replicate the alignment learning process by distilling
  data from already aligned LLMs like InstructGPT or ChatGPT. While this process
  reduces human efforts, constructing these datasets has a heavy dependency on
  the teacher models. In this work, we propose a novel framework for alignment
  learning with almost no human labor and no dependency on pre-aligned LLMs.
  First, we perform reward modeling (RM) with synthetic feedback by contrasting
  responses from vanilla LLMs with various sizes and prompts. Then, we use the RM
  for simulating high-quality demonstrations to train a supervised policy and for
  further optimizing the model with reinforcement learning. Our resulting model,
  Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms
  open-sourced models, including Alpaca, Dolly, and OpenAssistant, which are
  trained on the outputs of InstructGPT or human-annotated instructions. Our
  7B-sized model outperforms the 12-13B models in the A/B tests using GPT-4 as
  the judge with about 75% winning rate on average.
paper_ai_keywords: reward modeling (RM) synthetic feedback vanilla LLMs supervised policy reinforcement learning Aligned Language Model with Synthetic Training dataset (ALMoST) A/B tests