Aligning Large Language Models through Synthetic Feedback
Aligning large language models (LLMs) to human values has become increasingly
important as it enables sophisticated steering of LLMs, e.g., making them
follow given instructions while keeping them less toxic. However, it requires a
significant amount of human demonstrations and feedback. Recently, open-sourced
models have attempted to replicate the alignment learning process by distilling
data from already aligned LLMs like InstructGPT or ChatGPT. While this process
reduces human efforts, constructing these datasets has a heavy dependency on
the teacher models. In this work, we propose a novel framework for alignment
learning with almost no human labor and no dependency on pre-aligned LLMs.
First, we perform reward modeling (RM) with synthetic feedback by contrasting
responses from vanilla LLMs with various sizes and prompts. Then, we use the RM
for simulating high-quality demonstrations to train a supervised policy and for
further optimizing the model with reinforcement learning. Our resulting model,
Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms
open-sourced models, including Alpaca, Dolly, and OpenAssistant, which are
trained on the outputs of InstructGPT or human-annotated instructions. Our
7B-sized model outperforms the 12-13B models in the A/B tests using GPT-4 as
the judge with about 75% winning rate on average.
646d603a4e59b82995e24fd0
Sungdong Kim
claimed_verified
2023-05-24T10:45:45.558Z
6084ebe956f7623a0b849153
/avatars/c56fb056800c145e5a8234e27b6b5a31.svg
Sungdong kim
user
dsksd
646d603a4e59b82995e24fd1
Sanghwan Bae
646d603a4e59b82995e24fd2
Jamin Shin
claimed_verified
2025-04-24T12:41:32.713Z
6188cf3293317afcd1c2df7f
/avatars/95621801d5b3f3c1a681f1ad6cc66c6a.svg
Jay Shin
user
jshin49
646d603a4e59b82995e24fd3
Soyoung Kang
646d603a4e59b82995e24fd4
Donghyun Kwak
646d603a4e59b82995e24fd5
Kang Min Yoo
claimed_verified
2024-10-30T09:57:42.269Z
5f2294832128ba01bdf08c66
/avatars/334082608dd69a44b826fac00b36aa0a.svg
Kang Min Yoo
user
kmyoo
646d603a4e59b82995e24fd6
Minjoon Seo
claimed_verified
2024-05-08T09:15:44.085Z
621f05ba970615ad5861ceb1
/avatars/7e1902aa71369a524afda9b0a9e88e22.svg
Minjoon Seo
user
minjoon
Aligning Large Language Models through Synthetic Feedback
Aligning large language models (LLMs) to human values has become increasingly
important as it enables sophisticated steering of LLMs, e.g., making them
follow given instructions while keeping them less toxic. However, it requires a
significant amount of human demonstrations and feedback. Recently, open-sourced
models have attempted to replicate the alignment learning process by distilling
data from already aligned LLMs like InstructGPT or ChatGPT. While this process
reduces human efforts, constructing these datasets has a heavy dependency on
the teacher models. In this work, we propose a novel framework for alignment
learning with almost no human labor and no dependency on pre-aligned LLMs.
First, we perform reward modeling (RM) with synthetic feedback by contrasting
responses from vanilla LLMs with various sizes and prompts. Then, we use the RM
for simulating high-quality demonstrations to train a supervised policy and for
further optimizing the model with reinforcement learning. Our resulting model,
Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms
open-sourced models, including Alpaca, Dolly, and OpenAssistant, which are
trained on the outputs of InstructGPT or human-annotated instructions. Our
7B-sized model outperforms the 12-13B models in the A/B tests using GPT-4 as
the judge with about 75% winning rate on average.
A novel framework achieves LLM alignment with minimal human labor by using synthetic feedback for reward modeling and reinforcement learning, outperforming models trained on human-annotated data.
reward modeling reinforcement learning alignment learning synthetic feedback human demonstrations policy training Aligned Language Model with Synthetic Training dataset (ALMoST) A/B tests