title: DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM
    Workflows
summary: Large language models (LLMs) have become a dominant and important tool for
  NLP researchers in a wide range of tasks. Today, many researchers use LLMs in
  synthetic data generation, task evaluation, fine-tuning, distillation, and
  other model-in-the-loop research workflows. However, challenges arise when
  using these models that stem from their scale, their closed source nature, and
  the lack of standardized tooling for these new and emerging workflows. The
  rapid rise to prominence of these models and these unique challenges has had
  immediate adverse impacts on open science and on the reproducibility of work
  that uses them. In this paper, we introduce DataDreamer, an open source Python
  library that allows researchers to write simple code to implement powerful LLM
  workflows. DataDreamer also helps researchers adhere to best practices that we
  propose to encourage open science and reproducibility. The library and
  documentation are available at https://github.com/datadreamer-dev/DataDreamer .
paper_authors._id: 65d2d88d6236ca85a40313c8
paper_authors.name: Ajay Patel
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-19T04:29:30.810Z
paper_authors._id: 61c40eeb727d1257bf3cf5ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c40eeb727d1257bf3cf5ba/hVNbcFjsvwWqWarcGTOdI.jpeg
paper_authors.fullname: Ajay Patel
paper_authors.type: user
paper_authors.user: AjayP13
paper_authors._id: 65d2d88d6236ca85a40313c9
paper_authors.name: Colin Raffel
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-19T04:26:53.817Z
paper_authors._id: 6079c29765b9d0165cb18392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
paper_authors.fullname: Colin Raffel
paper_authors.type: user
paper_authors.user: craffel
paper_authors._id: 65d2d88d6236ca85a40313ca
paper_authors.name: Chris Callison-Burch
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-19T12:02:44.275Z
paper_authors._id: 6303ce25fc783bfc744216af
paper_authors.avatarUrl: /avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
paper_authors.fullname: Chris Callison-Burch
paper_authors.type: user
paper_authors.user: CCB
paper_title: DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM
    Workflows
paper_summary: Large language models (LLMs) have become a dominant and important tool for
  NLP researchers in a wide range of tasks. Today, many researchers use LLMs in
  synthetic data generation, task evaluation, fine-tuning, distillation, and
  other model-in-the-loop research workflows. However, challenges arise when
  using these models that stem from their scale, their closed source nature, and
  the lack of standardized tooling for these new and emerging workflows. The
  rapid rise to prominence of these models and these unique challenges has had
  immediate adverse impacts on open science and on the reproducibility of work
  that uses them. In this paper, we introduce DataDreamer, an open source Python
  library that allows researchers to write simple code to implement powerful LLM
  workflows. DataDreamer also helps researchers adhere to best practices that we
  propose to encourage open science and reproducibility. The library and
  documentation are available at https://github.com/datadreamer-dev/DataDreamer .