MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic
  Data Generation
Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data
generated using larger Language models. Questions remain about leveraging
synthetic data for other use cases, such as adapting LLMs to specific domains.
A key limitation of synthetic data is low diversity, which negatively impacts
its downstream applicability for improving other models. To address this, we
propose MetaSynth, a method for generating synthetic data that enhances
diversity through meta-prompting, where a language model orchestrates multiple
"expert" LLM agents to collaboratively generate data. Using only 25 million
tokens of synthetic data generated with MetaSynth, we successfully adapt a
well-trained LLM (Mistral-7B-v0.3) to two specialized domains-Finance and
Biomedicine-without compromising the capabilities of the resulting model in
general tasks. In addition, we evaluate the diversity of our synthetic data
using seven automated metrics, and find that it approaches the diversity of LLM
pre-training corpora.
  Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms
the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in
Biomedicine. The same model shows degraded performance when trained on data
generated using a template prompt, even when the template includes prior
generations and varying In-Context exemplars of real data. Our findings suggest
that a few million tokens of diverse synthetic data without mixing any real
data, is sufficient for effective domain adaptation when using MetaSynth.
68025a2274429c60ea9d501b
Haris Riaz
extracted_confirmed
2025-04-18T14:19:12.226Z
6463739b51fa6e63060491b1
/avatars/f8ee64b3768ef46e847cfa15fcd689f0.svg
Haris Riaz
user
hriaz
68025a2274429c60ea9d501c
Sourav Bhabesh
68025a2274429c60ea9d501d
Vinayak Arannil
68025a2274429c60ea9d501e
Miguel Ballesteros
68025a2274429c60ea9d501f
Graham Horwood
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic
  Data Generation
Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data
generated using larger Language models. Questions remain about leveraging
synthetic data for other use cases, such as adapting LLMs to specific domains.
A key limitation of synthetic data is low diversity, which negatively impacts
its downstream applicability for improving other models. To address this, we
propose MetaSynth, a method for generating synthetic data that enhances
diversity through meta-prompting, where a language model orchestrates multiple
"expert" LLM agents to collaboratively generate data. Using only 25 million
tokens of synthetic data generated with MetaSynth, we successfully adapt a
well-trained LLM (Mistral-7B-v0.3) to two specialized domains-Finance and
Biomedicine-without compromising the capabilities of the resulting model in
general tasks. In addition, we evaluate the diversity of our synthetic data
using seven automated metrics, and find that it approaches the diversity of LLM
pre-training corpora.
  Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms
the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in
Biomedicine. The same model shows degraded performance when trained on data
generated using a template prompt, even when the template includes prior
generations and varying In-Context exemplars of real data. Our findings suggest
that a few million tokens of diverse synthetic data without mixing any real
data, is sufficient for effective domain adaptation when using MetaSynth.
MetaSynth, a method using meta-prompting, generates diverse synthetic data to adapt large language models to specific domains while maintaining general capabilities.
meta-prompting synthetic data domain adaptation LLM agents Mistral-7B-v0.3 automated metrics template prompt in-context exemplars