RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
<|reference_start|>RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors: Many commercial and open-source models claim to detect machine-generated text with extremely high accuracy (99% or more). However, very few of these detectors are evaluated on shared benchmark datasets and even when they are, the datasets used for evaluation are insufficiently challenging-lacking variations in sampling strategy, adversarial attacks, and open-source generative models. In this work we present RAID: the largest and most challenging benchmark dataset for machine-generated text detection. RAID includes over 6 million generations spanning 11 models, 8 domains, 11 adversarial attacks and 4 decoding strategies. Using RAID, we evaluate the out-of-domain and adversarial robustness of 8 open- and 4 closed-source detectors and find that current detectors are easily fooled by adversarial attacks, variations in sampling strategies, repetition penalties, and unseen generative models. We release our data along with a leaderboard to encourage future research.<|reference_end|>
@article{dugan2024raid:,
    title={RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text
  Detectors},
    author={Liam Dugan, Alyssa Hwang, Filip Trhlik, Josh Magnus Ludan, Andrew Zhu,
  Hainiu Xu, Daphne Ippolito, Chris Callison-Burch},
    journal={arXiv preprint arXiv:2405.07940},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.07940},
    primaryClass={cs.CL}
}