title: Stress-testing Machine Generated Text Detection: Shifting Language
    Models Writing Style to Fool Detectors
summary: Recent advancements in Generative AI and Large Language Models (LLMs) have
  enabled the creation of highly realistic synthetic content, raising concerns
  about the potential for malicious use, such as misinformation and manipulation.
  Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the
  lack of robust benchmarks that assess generalization to real-world scenarios.
  In this work, we present a pipeline to test the resilience of state-of-the-art
  MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed
  adversarial attacks. To challenge the detectors, we fine-tune language models
  using Direct Preference Optimization (DPO) to shift the MGT style toward
  human-written text (HWT). This exploits the detectors' reliance on stylistic
  clues, making new generations more challenging to detect. Additionally, we
  analyze the linguistic shifts induced by the alignment and which features are
  used by detectors to detect MGT texts. Our results show that detectors can be
  easily fooled with relatively few examples, resulting in a significant drop in
  detection performance. This highlights the importance of improving detection
  methods and making them robust to unseen in-domain texts.
paper_authors._id: 683eae20ff848ec97ef4d377
paper_authors.name: Andrea Pedrotti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:36:50.501Z
paper_authors._id: 616a9f40ebe369e3e0209c00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616a9f40ebe369e3e0209c00/lBzeeugOSWz-VkhGMr25Z.png
paper_authors.fullname: Andrea Pedrotti
paper_authors.type: user
paper_authors.user: andreapdr
paper_authors._id: 683eae20ff848ec97ef4d378
paper_authors.name: Michele Papucci
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T12:33:42.976Z
paper_authors._id: 630f299da119d49bc1df7633
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671839656656-630f299da119d49bc1df7633.jpeg
paper_authors.fullname: Michele Papucci
paper_authors.type: user
paper_authors.user: mpapucci
paper_authors._id: 683eae20ff848ec97ef4d379
paper_authors.name: Cristiano Ciaccio
paper_authors._id: 683eae20ff848ec97ef4d37a
paper_authors.name: Alessio Miaschi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T12:33:45.653Z
paper_authors._id: 62e3fe792a8df5b22feed15e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e3fe792a8df5b22feed15e/iy8J4oGhW8Nr9QAgoH8ES.jpeg
paper_authors.fullname: Alessio Miaschi
paper_authors.type: user
paper_authors.user: alemiaschi
paper_authors._id: 683eae20ff848ec97ef4d37b
paper_authors.name: Giovanni Puccetti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:02:58.151Z
paper_authors._id: 60c72bad2c6fd4f486d25b4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60c72bad2c6fd4f486d25b4b/H-hZfI-td4Ngyn7rORyx2.jpeg
paper_authors.fullname: Giovanni Puccetti
paper_authors.type: user
paper_authors.user: gpucce
paper_authors._id: 683eae20ff848ec97ef4d37c
paper_authors.name: Felice Dell'Orletta
paper_authors._id: 683eae20ff848ec97ef4d37d
paper_authors.name: Andrea Esuli
paper_title: Stress-testing Machine Generated Text Detection: Shifting Language
    Models Writing Style to Fool Detectors
paper_summary: Recent advancements in Generative AI and Large Language Models (LLMs) have
  enabled the creation of highly realistic synthetic content, raising concerns
  about the potential for malicious use, such as misinformation and manipulation.
  Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the
  lack of robust benchmarks that assess generalization to real-world scenarios.
  In this work, we present a pipeline to test the resilience of state-of-the-art
  MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed
  adversarial attacks. To challenge the detectors, we fine-tune language models
  using Direct Preference Optimization (DPO) to shift the MGT style toward
  human-written text (HWT). This exploits the detectors' reliance on stylistic
  clues, making new generations more challenging to detect. Additionally, we
  analyze the linguistic shifts induced by the alignment and which features are
  used by detectors to detect MGT texts. Our results show that detectors can be
  easily fooled with relatively few examples, resulting in a significant drop in
  detection performance. This highlights the importance of improving detection
  methods and making them robust to unseen in-domain texts.
paper_ai_summary: Adversarial attacks using Direct Preference Optimization fine-tune language models to evade detection, leading to a significant drop in the performance of existing MGT detectors.
paper_ai_keywords: Generative AI Large Language Models Machine-Generated Text MGT detectors Mage Radar LLM-DetectAIve Direct Preference Optimization linguistic shifts stylistic clues