title: Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs
    with Nothing
summary: High-quality instruction data is critical for aligning large language models
  (LLMs). Although some models, such as Llama-3-Instruct, have open weights,
  their alignment data remain private, which hinders the democratization of AI.
  High human labor costs and a limited, predefined scope for prompting prevent
  existing open-source data creation methods from scaling effectively,
  potentially limiting the diversity and quality of public alignment datasets. Is
  it possible to synthesize high-quality instruction data at scale by extracting
  it directly from an aligned LLM? We present a self-synthesis method for
  generating large-scale alignment data named Magpie. Our key observation is that
  aligned LLMs like Llama-3-Instruct can generate a user query when we input only
  the left-side templates up to the position reserved for user messages, thanks
  to their auto-regressive nature. We use this method to prompt Llama-3-Instruct
  and generate 4 million instructions along with their corresponding responses.
  We perform a comprehensive analysis of the extracted data and select 300K
  high-quality instances. To compare Magpie data with other public instruction
  datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the
  performance of the fine-tuned models. Our results indicate that in some tasks,
  models fine-tuned with Magpie perform comparably to the official
  Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data
  points through supervised fine-tuning (SFT) and subsequent feedback learning.
  We also show that using Magpie solely for SFT can surpass the performance of
  previous public datasets utilized for both SFT and preference optimization,
  such as direct preference optimization with UltraFeedback. This advantage is
  evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
paper_authors._id: 666a4c1165a06c5745d71a5c
paper_authors.name: Zhangchen Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:24:06.092Z
paper_authors._id: 653df1323479e9ebbe3eb6cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653df1323479e9ebbe3eb6cc/K_g-r1iMRNKj99LXPuYF3.jpeg
paper_authors.fullname: Zhangchen Xu
paper_authors.type: user
paper_authors.user: zhangchenxu
paper_authors._id: 666a4c1165a06c5745d71a5d
paper_authors.name: Fengqing Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:14.539Z
paper_authors._id: 6531e1021dd8ebbdc1a6fd8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6531e1021dd8ebbdc1a6fd8e/lIcl7zCPtzRsfiUh6uY1o.jpeg
paper_authors.fullname: Fengqing Jiang
paper_authors.type: user
paper_authors.user: fqjiang
paper_authors._id: 666a4c1165a06c5745d71a5e
paper_authors.name: Luyao Niu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T21:29:06.971Z
paper_authors._id: 666dfd4770f5a2cb4aefd12f
paper_authors.avatarUrl: /avatars/fa0e0dbc203a21e58dda8fdb4cbc67ad.svg
paper_authors.fullname: Luyao Niu
paper_authors.type: user
paper_authors.user: LNIU
paper_authors._id: 666a4c1165a06c5745d71a5f
paper_authors.name: Yuntian Deng
paper_authors._id: 666a4c1165a06c5745d71a60
paper_authors.name: Radha Poovendran
paper_authors._id: 666a4c1165a06c5745d71a61
paper_authors.name: Yejin Choi
paper_authors._id: 666a4c1165a06c5745d71a62
paper_authors.name: Bill Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:16.691Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_title: Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs
    with Nothing
paper_summary: High-quality instruction data is critical for aligning large language models
  (LLMs). Although some models, such as Llama-3-Instruct, have open weights,
  their alignment data remain private, which hinders the democratization of AI.
  High human labor costs and a limited, predefined scope for prompting prevent
  existing open-source data creation methods from scaling effectively,
  potentially limiting the diversity and quality of public alignment datasets. Is
  it possible to synthesize high-quality instruction data at scale by extracting
  it directly from an aligned LLM? We present a self-synthesis method for
  generating large-scale alignment data named Magpie. Our key observation is that
  aligned LLMs like Llama-3-Instruct can generate a user query when we input only
  the left-side templates up to the position reserved for user messages, thanks
  to their auto-regressive nature. We use this method to prompt Llama-3-Instruct
  and generate 4 million instructions along with their corresponding responses.
  We perform a comprehensive analysis of the extracted data and select 300K
  high-quality instances. To compare Magpie data with other public instruction
  datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the
  performance of the fine-tuned models. Our results indicate that in some tasks,
  models fine-tuned with Magpie perform comparably to the official
  Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data
  points through supervised fine-tuning (SFT) and subsequent feedback learning.
  We also show that using Magpie solely for SFT can surpass the performance of
  previous public datasets utilized for both SFT and preference optimization,
  such as direct preference optimization with UltraFeedback. This advantage is
  evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
paper_ai_keywords: Large language models (LLMs) Aligned LLMs Llama-3-Instruct High-quality instruction data Alignment data Open-source data creation methods User query generation Auto-regressive nature Magpie Vectorization Template-based prompting Fine-tuning Supervised fine-tuning (SFT) Direct preference optimization AlpacaEval ArenaHard WildBench UltraFeedback