title: Increasing Diversity While Maintaining Accuracy: Text Data Generation
    with Large Language Models and Human Interventions
summary: Large language models (LLMs) can be used to generate text data for training
  and evaluating other models. However, creating high-quality datasets with LLMs
  can be challenging. In this work, we explore human-AI partnerships to
  facilitate high diversity and accuracy in LLM-based text data generation. We
  first examine two approaches to diversify text generation: 1) logit
  suppression, which minimizes the generation of languages that have already been
  frequently generated, and 2) temperature sampling, which flattens the token
  sampling probability. We found that diversification approaches can increase
  data diversity but often at the cost of data accuracy (i.e., text and labels
  being appropriate for the target domain). To address this issue, we examined
  two human interventions, 1) label replacement (LR), correcting misaligned
  labels, and 2) out-of-scope filtering (OOSF), removing instances that are out
  of the user's domain of interest or to which no considered label applies. With
  oracle studies, we found that LR increases the absolute accuracy of models
  trained with diversified datasets by 14.4%. Moreover, we found that some models
  trained with data generated with LR interventions outperformed LLM-based
  few-shot classification. In contrast, OOSF was not effective in increasing
  model accuracy, implying the need for future work in human-in-the-loop text
  data generation.
paper_authors._id: 64812ec63fb124fc984fdcda
paper_authors.name: John Joon Young Chung
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-08T01:29:28.157Z
paper_authors._id: 630c92f6c169245d78fcfa19
paper_authors.avatarUrl: /avatars/a4dd46b1b61f43530a1c934de29420af.svg
paper_authors.fullname: John Joon Young Chung
paper_authors.type: user
paper_authors.user: johnr0
paper_authors._id: 64812ec63fb124fc984fdcdb
paper_authors.name: Ece Kamar
paper_authors._id: 64812ec63fb124fc984fdcdc
paper_authors.name: Saleema Amershi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-02T22:47:10.572Z
paper_authors._id: 665cf655d2b102a5bbb21150
paper_authors.avatarUrl: /avatars/9cd4926132844f1900aae9b2903e0f37.svg
paper_authors.fullname: Saleema Amershi
paper_authors.type: user
paper_authors.user: samershi
paper_title: Increasing Diversity While Maintaining Accuracy: Text Data Generation
    with Large Language Models and Human Interventions
paper_summary: Large language models (LLMs) can be used to generate text data for training
  and evaluating other models. However, creating high-quality datasets with LLMs
  can be challenging. In this work, we explore human-AI partnerships to
  facilitate high diversity and accuracy in LLM-based text data generation. We
  first examine two approaches to diversify text generation: 1) logit
  suppression, which minimizes the generation of languages that have already been
  frequently generated, and 2) temperature sampling, which flattens the token
  sampling probability. We found that diversification approaches can increase
  data diversity but often at the cost of data accuracy (i.e., text and labels
  being appropriate for the target domain). To address this issue, we examined
  two human interventions, 1) label replacement (LR), correcting misaligned
  labels, and 2) out-of-scope filtering (OOSF), removing instances that are out
  of the user's domain of interest or to which no considered label applies. With
  oracle studies, we found that LR increases the absolute accuracy of models
  trained with diversified datasets by 14.4%. Moreover, we found that some models
  trained with data generated with LR interventions outperformed LLM-based
  few-shot classification. In contrast, OOSF was not effective in increasing
  model accuracy, implying the need for future work in human-in-the-loop text
  data generation.
paper_ai_summary: Human-AI partnerships improve text data generation quality and accuracy through label replacement, while diversification introduces some accuracy trade-offs.
paper_ai_keywords: logit suppression temperature sampling label replacement out-of-scope filtering