title: MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic
    Data Generation
summary: Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data
  generated using larger Language models. Questions remain about leveraging
  synthetic data for other use cases, such as adapting LLMs to specific domains.
  A key limitation of synthetic data is low diversity, which negatively impacts
  its downstream applicability for improving other models. To address this, we
  propose MetaSynth, a method for generating synthetic data that enhances
  diversity through meta-prompting, where a language model orchestrates multiple
  "expert" LLM agents to collaboratively generate data. Using only 25 million
  tokens of synthetic data generated with MetaSynth, we successfully adapt a
  well-trained LLM (Mistral-7B-v0.3) to two specialized domains-Finance and
  Biomedicine-without compromising the capabilities of the resulting model in
  general tasks. In addition, we evaluate the diversity of our synthetic data
  using seven automated metrics, and find that it approaches the diversity of LLM
  pre-training corpora.
    Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms
  the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in
  Biomedicine. The same model shows degraded performance when trained on data
  generated using a template prompt, even when the template includes prior
  generations and varying In-Context exemplars of real data. Our findings suggest
  that a few million tokens of diverse synthetic data without mixing any real
  data, is sufficient for effective domain adaptation when using MetaSynth.
paper_authors._id: 68025a2274429c60ea9d501b
paper_authors.name: Haris Riaz
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-18T14:19:12.226Z
paper_authors._id: 6463739b51fa6e63060491b1
paper_authors.avatarUrl: /avatars/f8ee64b3768ef46e847cfa15fcd689f0.svg
paper_authors.fullname: Haris Riaz
paper_authors.type: user
paper_authors.user: hriaz
paper_authors._id: 68025a2274429c60ea9d501c
paper_authors.name: Sourav Bhabesh
paper_authors._id: 68025a2274429c60ea9d501d
paper_authors.name: Vinayak Arannil
paper_authors._id: 68025a2274429c60ea9d501e
paper_authors.name: Miguel Ballesteros
paper_authors._id: 68025a2274429c60ea9d501f
paper_authors.name: Graham Horwood
paper_title: MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic
    Data Generation
paper_summary: Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data
  generated using larger Language models. Questions remain about leveraging
  synthetic data for other use cases, such as adapting LLMs to specific domains.
  A key limitation of synthetic data is low diversity, which negatively impacts
  its downstream applicability for improving other models. To address this, we
  propose MetaSynth, a method for generating synthetic data that enhances
  diversity through meta-prompting, where a language model orchestrates multiple
  "expert" LLM agents to collaboratively generate data. Using only 25 million
  tokens of synthetic data generated with MetaSynth, we successfully adapt a
  well-trained LLM (Mistral-7B-v0.3) to two specialized domains-Finance and
  Biomedicine-without compromising the capabilities of the resulting model in
  general tasks. In addition, we evaluate the diversity of our synthetic data
  using seven automated metrics, and find that it approaches the diversity of LLM
  pre-training corpora.
    Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms
  the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in
  Biomedicine. The same model shows degraded performance when trained on data
  generated using a template prompt, even when the template includes prior
  generations and varying In-Context exemplars of real data. Our findings suggest
  that a few million tokens of diverse synthetic data without mixing any real
  data, is sufficient for effective domain adaptation when using MetaSynth.
paper_ai_summary: MetaSynth, a method using meta-prompting, generates diverse synthetic data to adapt large language models to specific domains while maintaining general capabilities.
paper_ai_keywords: meta-prompting synthetic data domain adaptation LLM agents Mistral-7B-v0.3 automated metrics template prompt in-context exemplars