title: Are AI Detectors Good Enough? A Survey on Quality of Datasets With
    Machine-Generated Texts
summary: The rapid development of autoregressive Large Language Models (LLMs) has
  significantly improved the quality of generated texts, necessitating reliable
  machine-generated text detectors. A huge number of detectors and collections
  with AI fragments have emerged, and several detection methods even showed
  recognition quality up to 99.9% according to the target metrics in such
  collections. However, the quality of such detectors tends to drop dramatically
  in the wild, posing a question: Are detectors actually highly trustworthy or do
  their high benchmark scores come from the poor quality of evaluation datasets?
  In this paper, we emphasise the need for robust and qualitative methods for
  evaluating generated data to be secure against bias and low generalising
  ability of future model. We present a systematic review of datasets from
  competitions dedicated to AI-generated content detection and propose methods
  for evaluating the quality of datasets containing AI-generated fragments. In
  addition, we discuss the possibility of using high-quality generated data to
  achieve two goals: improving the training of detection models and improving the
  training datasets themselves. Our contribution aims to facilitate a better
  understanding of the dynamics between human and machine text, which will
  ultimately support the integrity of information in an increasingly automated
  world.
paper_authors._id: 6715f7ffef625f4d17fbe97a
paper_authors.name: German Gritsai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T14:20:57.911Z
paper_authors._id: 63e5017caf814a2149ec772d
paper_authors.avatarUrl: /avatars/21ad963e969e4f55ab37ec4d0d86e5c3.svg
paper_authors.fullname: German G
paper_authors.type: user
paper_authors.user: grgera
paper_authors._id: 6715f7ffef625f4d17fbe97b
paper_authors.name: Anastasia Voznyuk
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T11:25:59.696Z
paper_authors._id: 64cbfc39a81988d0735867a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64cbfc39a81988d0735867a1/JLlYTaRa70H4MzLXOHhEB.jpeg
paper_authors.fullname: Anastasia Voznyuk
paper_authors.type: user
paper_authors.user: natriistorm
paper_authors._id: 6715f7ffef625f4d17fbe97c
paper_authors.name: Andrey Grabovoy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:40:47.194Z
paper_authors._id: 670786e1b21dabc40ce5e702
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/670786e1b21dabc40ce5e702/Q_P5PoqmQr7b9tBuGZfls.jpeg
paper_authors.fullname: Andrey Grabovoy
paper_authors.type: user
paper_authors.user: andriygav
paper_authors._id: 6715f7ffef625f4d17fbe97d
paper_authors.name: Yury Chekhovich
paper_title: Are AI Detectors Good Enough? A Survey on Quality of Datasets With
    Machine-Generated Texts
paper_summary: The rapid development of autoregressive Large Language Models (LLMs) has
  significantly improved the quality of generated texts, necessitating reliable
  machine-generated text detectors. A huge number of detectors and collections
  with AI fragments have emerged, and several detection methods even showed
  recognition quality up to 99.9% according to the target metrics in such
  collections. However, the quality of such detectors tends to drop dramatically
  in the wild, posing a question: Are detectors actually highly trustworthy or do
  their high benchmark scores come from the poor quality of evaluation datasets?
  In this paper, we emphasise the need for robust and qualitative methods for
  evaluating generated data to be secure against bias and low generalising
  ability of future model. We present a systematic review of datasets from
  competitions dedicated to AI-generated content detection and propose methods
  for evaluating the quality of datasets containing AI-generated fragments. In
  addition, we discuss the possibility of using high-quality generated data to
  achieve two goals: improving the training of detection models and improving the
  training datasets themselves. Our contribution aims to facilitate a better
  understanding of the dynamics between human and machine text, which will
  ultimately support the integrity of information in an increasingly automated
  world.
paper_ai_keywords: autoregressive Large Language Models (LLMs) machine-generated text detectors synthetic content detection recognition quality evaluation datasets robust and qualitative methods multi-modal evaluation AI-generated content detection competitions high-quality generated datasets training of detection models dynamic between human and machine text information integrity