Better Synthetic Data by Retrieving and Transforming Existing Datasets
<|reference_start|>Better Synthetic Data by Retrieving and Transforming Existing Datasets: Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, DataTune, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49% and improves over existing methods that use synthetic or retrieved training data by 34%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community: https://github.com/neulab/prompt2model.<|reference_end|>
@article{gandhi2024better,
    title={Better Synthetic Data by Retrieving and Transforming Existing Datasets},
    author={Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, Graham
  Neubig},
    journal={arXiv preprint arXiv:2404.14361},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.14361},
    primaryClass={cs.CL}
}