One Token to Fool LLM-as-a-Judge
Generative reward models (also known as LLMs-as-judges), which use large
language models (LLMs) to evaluate answer quality, are increasingly adopted in
reinforcement learning with verifiable rewards (RLVR). They are often preferred
over rigid rule-based metrics, especially for complex reasoning tasks involving
free-form outputs. In this paradigm, an LLM is typically prompted to compare a
candidate answer against a ground-truth reference and assign a binary reward
indicating correctness. Despite the seeming simplicity of this comparison task,
we find that generative reward models exhibit surprising vulnerabilities to
superficial manipulations: non-word symbols (e.g., ":" or ".") or reasoning
openers like "Thought process:" and "Let's solve this problem step by step."
can often lead to false positive rewards. We demonstrate that this weakness is
widespread across LLMs, datasets, and prompt formats, posing a serious threat
for core algorithmic paradigms that rely on generative reward models, such as
rejection sampling, preference optimization, and RLVR. To mitigate this issue,
we introduce a simple yet effective data augmentation strategy and train a new
generative reward model with substantially improved robustness. Our findings
highlight the urgent need for more reliable LLM-based evaluation methods. We
release our robust, general-domain reward model and its synthetic training data
at https://huggingface.co/sarosavo/Master-RM and
https://huggingface.co/datasets/sarosavo/Master-RM.
68747a58257d4f04353702f9
Yulai Zhao
claimed_verified
2025-07-15T19:10:35.003Z
6575171654d1749612e21eed
/avatars/c032c1b942b3cb9450a49db88fce5c70.svg
Yulai Zhao
user
sarosavo
68747a58257d4f04353702fa
Haolin Liu
claimed_verified
2025-07-18T08:32:12.721Z
6850ab3fb73a72cdc6159f6f
/avatars/74c58336655cdfe7d3fb9854d426240d.svg
Haolin Liu
user
lhl616
68747a58257d4f04353702fb
Dian Yu
claimed_verified
2025-07-18T07:50:42.571Z
62d58fd53bf5e059f7cc3245
/avatars/7a4f3ee4a37245f67efd26749d66a706.svg
Dian Yu
user
yudian
68747a58257d4f04353702fc
S. Y. Kung
68747a58257d4f04353702fd
Haitao Mi
68747a58257d4f04353702fe
Dong Yu
One Token to Fool LLM-as-a-Judge
Generative reward models (also known as LLMs-as-judges), which use large
language models (LLMs) to evaluate answer quality, are increasingly adopted in
reinforcement learning with verifiable rewards (RLVR). They are often preferred
over rigid rule-based metrics, especially for complex reasoning tasks involving
free-form outputs. In this paradigm, an LLM is typically prompted to compare a
candidate answer against a ground-truth reference and assign a binary reward
indicating correctness. Despite the seeming simplicity of this comparison task,
we find that generative reward models exhibit surprising vulnerabilities to
superficial manipulations: non-word symbols (e.g., ":" or ".") or reasoning
openers like "Thought process:" and "Let's solve this problem step by step."
can often lead to false positive rewards. We demonstrate that this weakness is
widespread across LLMs, datasets, and prompt formats, posing a serious threat
for core algorithmic paradigms that rely on generative reward models, such as
rejection sampling, preference optimization, and RLVR. To mitigate this issue,
we introduce a simple yet effective data augmentation strategy and train a new
generative reward model with substantially improved robustness. Our findings
highlight the urgent need for more reliable LLM-based evaluation methods. We
release our robust, general-domain reward model and its synthetic training data
at https://huggingface.co/sarosavo/Master-RM and
https://huggingface.co/datasets/sarosavo/Master-RM.
Generative reward models using large language models are vulnerable to superficial manipulations, leading to false positive rewards, and a new data augmentation strategy improves their robustness.
generative reward models LLMs-as-judges large language models reinforcement learning with verifiable rewards RLVR binary reward rejection sampling preference optimization data augmentation