Better Alignment with Instruction Back-and-Forth Translation
We propose a new method, instruction back-and-forth translation, to construct
high-quality synthetic data grounded in world knowledge for aligning large
language models (LLMs). Given documents from a web corpus, we generate and
curate synthetic instructions using the backtranslation approach proposed by Li
et al.(2023a), and rewrite the responses to improve their quality further based
on the initial documents. Fine-tuning with the resulting (backtranslated
instruction, rewritten response) pairs yields higher win rates on AlpacaEval
than using other common instruction datasets such as Humpback, ShareGPT, Open
Orca, Alpaca-GPT4 and Self-instruct. We also demonstrate that rewriting the
responses with an LLM outperforms direct distillation, and the two generated
text distributions exhibit significant distinction in embedding space. Further
analysis shows that our backtranslated instructions are of higher quality than
other sources of synthetic instructions, while our responses are more diverse
and complex than those obtained from distillation. Overall we find that
instruction back-and-forth translation combines the best of both worlds --
making use of the information diversity and quantity found on the web, while
ensuring the quality of the responses which is necessary for effective
alignment.
66b58c938533192be928639b
Thao Nguyen
claimed_verified
2024-08-09T07:47:27.838Z
64ba4923921dbcbc96006fb8
/avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
Thao Nguyen
user
thaottn
66b58c938533192be928639c
Jeffrey Li
admin_assigned
2024-08-09T10:10:05.779Z
63ca51a82ffd350b485f9fa5
https://cdn-avatars.huggingface.co/v1/production/uploads/1674203513156-noauth.jpeg
Jeffrey Li
user
JeffreyLi
66b58c938533192be928639d
Sewoong Oh
66b58c938533192be928639e
Ludwig Schmidt
admin_assigned
2024-08-09T10:10:26.914Z
6252375c818a5dc29ab9d945
/avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
Ludwig Schmidt
user
ludwigschmidt
66b58c938533192be928639f
Jason Weston
admin_assigned
2024-08-09T10:10:35.766Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
66b58c938533192be92863a0
Luke Zettlemoyer
66b58c938533192be92863a1
Xian Li
Better Alignment with Instruction Back-and-Forth Translation
We propose a new method, instruction back-and-forth translation, to construct
high-quality synthetic data grounded in world knowledge for aligning large
language models (LLMs). Given documents from a web corpus, we generate and
curate synthetic instructions using the backtranslation approach proposed by Li
et al.(2023a), and rewrite the responses to improve their quality further based
on the initial documents. Fine-tuning with the resulting (backtranslated
instruction, rewritten response) pairs yields higher win rates on AlpacaEval
than using other common instruction datasets such as Humpback, ShareGPT, Open
Orca, Alpaca-GPT4 and Self-instruct. We also demonstrate that rewriting the
responses with an LLM outperforms direct distillation, and the two generated
text distributions exhibit significant distinction in embedding space. Further
analysis shows that our backtranslated instructions are of higher quality than
other sources of synthetic instructions, while our responses are more diverse
and complex than those obtained from distillation. Overall we find that
instruction back-and-forth translation combines the best of both worlds --
making use of the information diversity and quantity found on the web, while
ensuring the quality of the responses which is necessary for effective
alignment.
The proposed instruction back-and-forth translation method enhances synthetic data for large language models, improving win rates and quality compared to existing datasets.
instruction back-and-forth translation synthetic data world knowledge large language models backtranslation fine-tuning AlpacaEval Humpback ShareGPT Open Orca Alpaca-GPT4 Self-instruct response rewriting direct distillation embedding space information diversity quality of responses effective alignment