title: People who frequently use ChatGPT for writing tasks are accurate and
    robust detectors of AI-generated text
summary: In this paper, we study how well humans can detect text generated by
  commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300
  non-fiction English articles, label them as either human-written or
  AI-generated, and provide paragraph-length explanations for their decisions.
  Our experiments show that annotators who frequently use LLMs for writing tasks
  excel at detecting AI-generated text, even without any specialized training or
  feedback. In fact, the majority vote among five such "expert" annotators
  misclassifies only 1 of 300 articles, significantly outperforming most
  commercial and open-source detectors we evaluated even in the presence of
  evasion tactics like paraphrasing and humanization. Qualitative analysis of the
  experts' free-form explanations shows that while they rely heavily on specific
  lexical clues ('AI vocabulary'), they also pick up on more complex phenomena
  within the text (e.g., formality, originality, clarity) that are challenging to
  assess for automatic detectors. We release our annotated dataset and code to
  spur future research into both human and automated detection of AI-generated
  text.
paper_authors._id: 679a7a7ea3ffd2887d76a1e7
paper_authors.name: Jenna Russell
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T21:06:15.282Z
paper_authors._id: 637519809ef4e3b87a5e88fb
paper_authors.avatarUrl: /avatars/4465c9194c17f9b5e5a5a5e88d4a4656.svg
paper_authors.fullname: Jenna Russell
paper_authors.type: user
paper_authors.user: jjrussell10
paper_authors._id: 679a7a7ea3ffd2887d76a1e8
paper_authors.name: Marzena Karpinska
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-31T03:42:57.813Z
paper_authors._id: 62293b03acd5bef90e55c4ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667593978543-62293b03acd5bef90e55c4ae.png
paper_authors.fullname: Marzena Karpinska
paper_authors.type: user
paper_authors.user: marzena
paper_authors._id: 679a7a7ea3ffd2887d76a1e9
paper_authors.name: Mohit Iyyer
paper_title: People who frequently use ChatGPT for writing tasks are accurate and
    robust detectors of AI-generated text
paper_summary: In this paper, we study how well humans can detect text generated by
  commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300
  non-fiction English articles, label them as either human-written or
  AI-generated, and provide paragraph-length explanations for their decisions.
  Our experiments show that annotators who frequently use LLMs for writing tasks
  excel at detecting AI-generated text, even without any specialized training or
  feedback. In fact, the majority vote among five such "expert" annotators
  misclassifies only 1 of 300 articles, significantly outperforming most
  commercial and open-source detectors we evaluated even in the presence of
  evasion tactics like paraphrasing and humanization. Qualitative analysis of the
  experts' free-form explanations shows that while they rely heavily on specific
  lexical clues ('AI vocabulary'), they also pick up on more complex phenomena
  within the text (e.g., formality, originality, clarity) that are challenging to
  assess for automatic detectors. We release our annotated dataset and code to
  spur future research into both human and automated detection of AI-generated
  text.
paper_ai_keywords: LLMs GPT-4 Claude human-written AI-generated paragraph-length explanations majority vote specialized training evasion tactics paraphrasing humanization lexical clues AI vocabulary formality originality clarity automatic detectors