Stress-testing Machine Generated Text Detection: Shifting Language
  Models Writing Style to Fool Detectors
Recent advancements in Generative AI and Large Language Models (LLMs) have
enabled the creation of highly realistic synthetic content, raising concerns
about the potential for malicious use, such as misinformation and manipulation.
Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the
lack of robust benchmarks that assess generalization to real-world scenarios.
In this work, we present a pipeline to test the resilience of state-of-the-art
MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed
adversarial attacks. To challenge the detectors, we fine-tune language models
using Direct Preference Optimization (DPO) to shift the MGT style toward
human-written text (HWT). This exploits the detectors' reliance on stylistic
clues, making new generations more challenging to detect. Additionally, we
analyze the linguistic shifts induced by the alignment and which features are
used by detectors to detect MGT texts. Our results show that detectors can be
easily fooled with relatively few examples, resulting in a significant drop in
detection performance. This highlights the importance of improving detection
methods and making them robust to unseen in-domain texts.
683eae20ff848ec97ef4d377
Andrea Pedrotti
claimed_verified
2025-06-03T08:36:50.501Z
616a9f40ebe369e3e0209c00
https://cdn-avatars.huggingface.co/v1/production/uploads/616a9f40ebe369e3e0209c00/lBzeeugOSWz-VkhGMr25Z.png
Andrea Pedrotti
user
andreapdr
683eae20ff848ec97ef4d378
Michele Papucci
claimed_verified
2025-06-03T12:33:42.976Z
630f299da119d49bc1df7633
https://cdn-avatars.huggingface.co/v1/production/uploads/1671839656656-630f299da119d49bc1df7633.jpeg
Michele Papucci
user
mpapucci
683eae20ff848ec97ef4d379
Cristiano Ciaccio
683eae20ff848ec97ef4d37a
Alessio Miaschi
claimed_verified
2025-06-03T12:33:45.653Z
62e3fe792a8df5b22feed15e
https://cdn-avatars.huggingface.co/v1/production/uploads/62e3fe792a8df5b22feed15e/iy8J4oGhW8Nr9QAgoH8ES.jpeg
Alessio Miaschi
user
alemiaschi
683eae20ff848ec97ef4d37b
Giovanni Puccetti
claimed_verified
2025-06-04T09:02:58.151Z
60c72bad2c6fd4f486d25b4b
https://cdn-avatars.huggingface.co/v1/production/uploads/60c72bad2c6fd4f486d25b4b/H-hZfI-td4Ngyn7rORyx2.jpeg
Giovanni Puccetti
user
gpucce
683eae20ff848ec97ef4d37c
Felice Dell'Orletta
683eae20ff848ec97ef4d37d
Andrea Esuli
Stress-testing Machine Generated Text Detection: Shifting Language
  Models Writing Style to Fool Detectors
Recent advancements in Generative AI and Large Language Models (LLMs) have
enabled the creation of highly realistic synthetic content, raising concerns
about the potential for malicious use, such as misinformation and manipulation.
Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the
lack of robust benchmarks that assess generalization to real-world scenarios.
In this work, we present a pipeline to test the resilience of state-of-the-art
MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed
adversarial attacks. To challenge the detectors, we fine-tune language models
using Direct Preference Optimization (DPO) to shift the MGT style toward
human-written text (HWT). This exploits the detectors' reliance on stylistic
clues, making new generations more challenging to detect. Additionally, we
analyze the linguistic shifts induced by the alignment and which features are
used by detectors to detect MGT texts. Our results show that detectors can be
easily fooled with relatively few examples, resulting in a significant drop in
detection performance. This highlights the importance of improving detection
methods and making them robust to unseen in-domain texts.
Adversarial attacks using Direct Preference Optimization fine-tune language models to evade detection, leading to a significant drop in the performance of existing MGT detectors.
Generative AI Large Language Models Machine-Generated Text MGT detectors Mage Radar LLM-DetectAIve Direct Preference Optimization linguistic shifts stylistic clues