Specification Self-Correction: Mitigating In-Context Reward Hacking
  Through Test-Time Refinement
Language models (LMs) are susceptible to in-context reward hacking, where
they exploit flaws in tainted or faulty written specifications or rubrics to
achieve high scores without fulfilling the user's true intent. We introduce
Specification Self-Correction (SSC), a novel, test-time framework that enables
an LM to identify and correct flaws within its own guiding specification. SSC
employs a multi-step inference process where the model first generates a
response based on a potentially tainted specification, critiques its output,
and then revises the specification itself to remove the exploitable loophole. A
final, more robust response is then generated using this self-corrected
specification. Across experiments spanning creative writing and agentic coding
tasks with several LMs, we demonstrate that while models initially game tainted
specifications in 50-70\% of cases, the SSC process reduces this vulnerability
by over 90\%. This dynamic repair occurs at inference time, requires no weight
modification, and leads to more robustly aligned model behavior. Code at
https://github.com/vicgalle/specification-self-correction .
688728d6a5a841b139945452
Víctor Gallego
claimed_verified
2025-07-28T08:44:19.745Z
5fad8602b8423e1d80b8a965
https://cdn-avatars.huggingface.co/v1/production/uploads/5fad8602b8423e1d80b8a965/tRqTwcZmrGka8c1vFq2wX.jpeg
Victor Gallego
user
vicgalle
Specification Self-Correction: Mitigating In-Context Reward Hacking
  Through Test-Time Refinement
Language models (LMs) are susceptible to in-context reward hacking, where
they exploit flaws in tainted or faulty written specifications or rubrics to
achieve high scores without fulfilling the user's true intent. We introduce
Specification Self-Correction (SSC), a novel, test-time framework that enables
an LM to identify and correct flaws within its own guiding specification. SSC
employs a multi-step inference process where the model first generates a
response based on a potentially tainted specification, critiques its output,
and then revises the specification itself to remove the exploitable loophole. A
final, more robust response is then generated using this self-corrected
specification. Across experiments spanning creative writing and agentic coding
tasks with several LMs, we demonstrate that while models initially game tainted
specifications in 50-70\% of cases, the SSC process reduces this vulnerability
by over 90\%. This dynamic repair occurs at inference time, requires no weight
modification, and leads to more robustly aligned model behavior. Code at
https://github.com/vicgalle/specification-self-correction .
Specification Self-Correction (SSC) is a framework that allows language models to identify and correct flaws in their specifications at inference time, significantly reducing in-context reward hacking.
Specification Self-Correction SSC in-context reward hacking language models multi-step inference self-corrected specification robust response creative writing agentic coding tasks