Are AI Detectors Good Enough? A Survey on Quality of Datasets With
  Machine-Generated Texts
The rapid development of autoregressive Large Language Models (LLMs) has
significantly improved the quality of generated texts, necessitating reliable
machine-generated text detectors. A huge number of detectors and collections
with AI fragments have emerged, and several detection methods even showed
recognition quality up to 99.9% according to the target metrics in such
collections. However, the quality of such detectors tends to drop dramatically
in the wild, posing a question: Are detectors actually highly trustworthy or do
their high benchmark scores come from the poor quality of evaluation datasets?
In this paper, we emphasise the need for robust and qualitative methods for
evaluating generated data to be secure against bias and low generalising
ability of future model. We present a systematic review of datasets from
competitions dedicated to AI-generated content detection and propose methods
for evaluating the quality of datasets containing AI-generated fragments. In
addition, we discuss the possibility of using high-quality generated data to
achieve two goals: improving the training of detection models and improving the
training datasets themselves. Our contribution aims to facilitate a better
understanding of the dynamics between human and machine text, which will
ultimately support the integrity of information in an increasingly automated
world.
6715f7ffef625f4d17fbe97a
German Gritsai
claimed_verified
2024-10-22T14:20:57.911Z
63e5017caf814a2149ec772d
/avatars/21ad963e969e4f55ab37ec4d0d86e5c3.svg
German G
user
grgera
6715f7ffef625f4d17fbe97b
Anastasia Voznyuk
claimed_verified
2024-10-25T11:25:59.696Z
64cbfc39a81988d0735867a1
https://cdn-avatars.huggingface.co/v1/production/uploads/64cbfc39a81988d0735867a1/JLlYTaRa70H4MzLXOHhEB.jpeg
Anastasia Voznyuk
user
natriistorm
6715f7ffef625f4d17fbe97c
Andrey Grabovoy
claimed_verified
2024-10-21T07:40:47.194Z
670786e1b21dabc40ce5e702
https://cdn-avatars.huggingface.co/v1/production/uploads/670786e1b21dabc40ce5e702/Q_P5PoqmQr7b9tBuGZfls.jpeg
Andrey Grabovoy
user
andriygav
6715f7ffef625f4d17fbe97d
Yury Chekhovich
Are AI Detectors Good Enough? A Survey on Quality of Datasets With
  Machine-Generated Texts
The rapid development of autoregressive Large Language Models (LLMs) has
significantly improved the quality of generated texts, necessitating reliable
machine-generated text detectors. A huge number of detectors and collections
with AI fragments have emerged, and several detection methods even showed
recognition quality up to 99.9% according to the target metrics in such
collections. However, the quality of such detectors tends to drop dramatically
in the wild, posing a question: Are detectors actually highly trustworthy or do
their high benchmark scores come from the poor quality of evaluation datasets?
In this paper, we emphasise the need for robust and qualitative methods for
evaluating generated data to be secure against bias and low generalising
ability of future model. We present a systematic review of datasets from
competitions dedicated to AI-generated content detection and propose methods
for evaluating the quality of datasets containing AI-generated fragments. In
addition, we discuss the possibility of using high-quality generated data to
achieve two goals: improving the training of detection models and improving the
training datasets themselves. Our contribution aims to facilitate a better
understanding of the dynamics between human and machine text, which will
ultimately support the integrity of information in an increasingly automated
world.
A systematic review and evaluation of datasets and methods for AI-generated text detection are presented to improve robustness and reliability of detection models.
autoregressive Large Language Models AI-generated text detectors evaluation datasets bias generalising ability AI-generated content detection