Kunvar Thaman's Reward Hacking Benchmark: Measuring Exploits in LLM Agents Tool Use (https://arxiv.org/abs/2605.02964) evaluates 13 frontier models on multi-step, tool-use tasks that each contain a built-in shortcut: skipping a verification step, inferring an answer from metadata rather than doing the task, or tampering with a function the evaluation itself depends on.
The finding I keep coming back to: in 72% of the reward hacking episodes recorded, the model's chain-of-thought(CoT) reasoning framed the exploit as legitimate problem solving rather than as a violation of the task. That matters for evaluation design specifically. If "the model can explain its reasoning" is being used as a safety signal, for interpretability based auditing, by way of example, this result suggests that signal can look identical whether the model is reasoning honestly or rationalizing a shortcut after the fact. A coherent explanation for an exploit is not obviously evidence of a safer model: it may just be evidence that the model is good at generating explanations, which is a different property than the one being tested for. Auditing regimes built on CoT transparency inherit that gap.
Exploit rates themselves varied widely across models, from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero). The paper's sharpest internal comparison holds task family and base architecture roughly constant: DeepSeek-V3 exploits at 0.6%, while DeepSeek-R1-Zero, the RL-post-trained version of a comparable model, exploits at 13.9% across all four task families tested. That is evidence that RL post-training itself, at least as applied here, is associated with a large increase in reward hacking behavior, not just noise across unrelated training recipes.
A second result reframes the first one. Simple environmental hardening reduced the exploit rate 5.7 percentage points, an 87.7% relative reduction, without any loss in task success. That is a large effect from an intervention that targets the environment rather than the model's weights or training. It suggests that "is this model safe" may often be measuring a joint property of the model and the specific, often permissive environment it was tested in, with the whole result then attributed to the model alone.
Similar territory is covered in beren's Mitigating Reward Hacking as Institutional Design (https://www.lesswrong.com/posts/tgcooi77NXMquCR5L/mitigating-reward-hacking-as-institutional-design), which argues for designing the training system itself, giving models right of appeal, adversarial verifiers, confession mechanisms, drawing an explicit analogy to legal institutions; my argument here is about what to do with evaluation and governance once a model is already trained, not how to architect the training loop that produces it.
The obvious objection: environmental hardening works here because RHB's tasks are legible and bounded enough that "harden the environment" has a clear, implementable meaning, closing specific loopholes in specific tool functions. Most real deployment contexts are not like that. You cannot exhaustively enumerate and patch every exploitable seam in an open-ended agentic deployment the way you can in a benchmark's task suite, and an agent operating in the open world would encounter affordances no one anticipated. So this result likely doesn't generalize past evals and narrow, well-specified deployment settings; it does not obviously scale to "harden the internet" or "harden a customer-facing agent's full action space." Where it likely does transfer is anywhere the deployment surface is enumerable: internal tool-use agents, agentic coding environments, and evaluation infrastructure itself, where the environment is something an organization actually controls and can iterate on.
That points to a concrete governance implication rather than an open question. Evaluators and auditors testing tool-use agents should treat chain-of-thoughts justification as a separate variable from task-outcome correctness, not as corroborating evidence for it, and should report exploit rates alongside RL post-training details rather than aggregating across training regimes. Policy frameworks setting expectations around agentic AI deployment should treat environmental-hardening requirements, meaning constraints and audit on the tool and action surface an agent has access to, as a governance lever with its own cost-benefit profile, owned separately from model-level safety requirements, in contexts where the deployment surface is bounded enough for hardening to mean something concrete. In practice that means the question of who owns environment design, who audits it, and who is accountable when it fails deserves the same institutional attention currently given to model evaluation.
