title: Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large
    Language Models by Extrapolating Errors from Small Models
summary: *Data Synthesis* is a promising way to train a small model with very little
  labeled data. One approach for data synthesis is to leverage the rich knowledge
  from large language models to synthesize pseudo training examples for small
  models, making it possible to achieve both data and compute efficiency at the
  same time. However, a key challenge in data synthesis is that the synthesized
  dataset often suffers from a large distributional discrepancy from the *real
  task* data distribution. Thus, in this paper, we propose *Synthesis Step by
  Step* (**S3**), a data synthesis framework that shrinks this distribution gap
  by iteratively extrapolating the errors made by a small model trained on the
  synthesized dataset on a small real-world validation dataset using a large
  language model. Extensive experiments on multiple NLP tasks show that our
  approach improves the performance of a small model by reducing the gap between
  the synthetic dataset and the real data, resulting in significant improvement
  compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73%
  compared to GoldGen, and at most 15.17% improvement compared to the small model
  trained on human-annotated data.
paper_authors._id: 6535f2ff9860c1cb37002ae2
paper_authors.name: Ruida Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T15:12:45.338Z
paper_authors._id: 6618d1c3c1167c8d8702b19d
paper_authors.avatarUrl: /avatars/24611ca6c4158f4978ac8476a87d8d9c.svg
paper_authors.fullname: Ruida WANG
paper_authors.type: user
paper_authors.user: RickyDeSkywalker
paper_authors._id: 6535f2ff9860c1cb37002ae3
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:33:48.670Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 6535f2ff9860c1cb37002ae4
paper_authors.name: Mrinmaya Sachan
paper_title: Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large
    Language Models by Extrapolating Errors from Small Models
paper_summary: *Data Synthesis* is a promising way to train a small model with very little
  labeled data. One approach for data synthesis is to leverage the rich knowledge
  from large language models to synthesize pseudo training examples for small
  models, making it possible to achieve both data and compute efficiency at the
  same time. However, a key challenge in data synthesis is that the synthesized
  dataset often suffers from a large distributional discrepancy from the *real
  task* data distribution. Thus, in this paper, we propose *Synthesis Step by
  Step* (**S3**), a data synthesis framework that shrinks this distribution gap
  by iteratively extrapolating the errors made by a small model trained on the
  synthesized dataset on a small real-world validation dataset using a large
  language model. Extensive experiments on multiple NLP tasks show that our
  approach improves the performance of a small model by reducing the gap between
  the synthetic dataset and the real data, resulting in significant improvement
  compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73%
  compared to GoldGen, and at most 15.17% improvement compared to the small model
  trained on human-annotated data.
paper_ai_summary: Synthesis Step by Step (S3) iteratively refines synthesized pseudo training examples using a large language model to reduce distributional discrepancies and improve small model performance on NLP tasks.
paper_ai_keywords: data synthesis large language models pseudo training examples distributional discrepancy Synthesis Step by Step (S3) iterative extrapolation real-world validation dataset ZeroGen GoldGen