
tl;dr: Even when the task reward function is accurate and contextual requirements are explicitly stated, RL can make agents genuinely more capable while reinforcing violations of those requirements. Beyond conventional reward hacking, agents can learn to do more real work by crossing authorization boundaries.
More details and motion graphics are in the blog and paper. Here is a quick story about it! :)
We build pipelines to construct scalable verifiable environments that have 1. context requirements (corresponding to some allowed boundary region), and 2. task rewards and independent checks of actual work; 3. guaranteed shortcuts.
On resource tasks, reward accurately measures real work. We include reward-hacking tasks as well, such as Validation. Contextual requirements are explicitly stated throughout.

For example, an agent is working on company data with (1) explicit permission to use an external service only for public data, (2) a reward that accurately measures completed processing work, and (3) a guaranteed shortcut😈 sending restricted data to that service completes more real work than any authorized strategy, but violates the permission boundary.
Sounds familiar? To a certain degree, such a shortcut can be blocked post hoc by good software design awareness, just like, as software engineers, we should make sure we can't allow customers to buy -1 item so they can get the shortcut of 'having one more item but the total expense is now lower'.
However, given that agents/intelligence are embedded everywhere in our lives, and in long-horizon execution (people love to leverage it, big tech is training for more capability), context may not be able to be enumerated; potential shortcuts could explode in occurrence in such a long-horizon, complex, dynamic environment. Or if you are training an agent but have some slightly bad filtering for training data, here it is. A work order asks to use at most 10 API requests, but now more issues accomplished means a higher reward.

To make these issues more significant, as we have mentioned in the beginning, in the following we deliberately make the setting more simple: the contextual requirement is already explicit, to see the agent learn to optimize reward but by violating context requirements.
Envs are built and verifiable[1]so that we can capture those violations and whether they are inside or outside the contextual boundary at the dynamics level of the task env. Since we first define the context and the dynamics governing the env, and render the env with diverse real-world corpora, we can have verifiable and diverse agentic RL envs to study the issues. And whether contextual boundaries are violated can be captured.

So we build 800+ tasks with or without the shortcut in them, and they expand across eight mechanisms: Allocation, API budget, Validation, Test expectations, Evaluation cohort, Score cache, CI reports, and Service monitoring, and are rendered into scenarios like Workshop, warehouse operations, financial settlement, manufacturing quality, and public services.
Train agentic RL with clear context on Qwen 3.5 4B with GRPO and terminal reward; here are the results.
Base is the starting policy, Task-trained learns from task reward alone, and Safety-trained learns from task reward with a penalty for any violation in the execution history.
We observe both amplification and emergence of violations here. Another thing to notice is that real work on tasks without shortcuts also improves. So if you are a post-training researcher checking and investigating only a subset of tasks, these violations could be harder to detect.

What if we just do better data filtering? Maybe we are not able to make sure all the data are free of shortcuts, but at least only a small number of training queries have such potential shortcuts.

A lower proportion of training data containing shortcuts indeed lowers the 'getting bad' speed, but even when one fourth of the data has such shortcuts, the agent eventually hacks after enough training steps. To make it clear, each step has 2 queries, each with 8 rollouts, to estimate advantages with GRPO.

Both capability gains and profitable violations transfer to fresh tasks and new operational settings. The mechanisms could be similar, like limited resources in separate places, from Azure to Amazon Web Services, with different tools and different infrastructure.
We also observed similar violations on external benchmarks, under explicit restrictions, and full reward for incomplete work.
BAITBENCH tests reward hacking in ML tasks. In this prediction task, Task-trained uses test labels as input features. 💯

This Wordle case is from Hack-Verifiable Environments. Task-trained takes advantage of the parser to supply the missing letters to get full reward.

OKAY! BIG ISSUE! Do we have the solution? Yes, we have it, and it comes inherently from our constructions.
Simply using identification defined on the dynamics, we can have the penalty to provide reward shaping at the terminal reward, simply written as .

This is beneficial since it provides a way to learn safety awareness using the final shaped reward, without using demonstrations or injected safety principles during reasoning, such as Inoculation prompting or Deliberative Alignment . It provides, like R1-Zero, flexibility for the agent to develop safety awareness across diverse scenarios by only rewarding at the end.
In the previous simple envs, the context can be static. But as just mentioned, over a long horizon, the context may not be explicitly enumerated and will be dynamic; we will use such dynamic envs as generalization test cases later. But right now we will see what will be brought by simple training.

It's important to discuss whether task capability sucks when we add such reward shaping to inject safety awareness. It may be a common impression that there is a tradeoff between safety and capability.
Fortunately, we observed that safety training from Base improves compliance with context requirements while allowing capability to grow.

This can also be achieved by continuing with safety training from the task-trained parent we showed in the last section.

You can see that the mitigation is partial here, with a fair budget, by giving a similar number of training steps to task-only optimization and safety training.
One pity could be that it takes another 96 steps of safety training to lower violations from the task-trained model (32) to near-zero violations. (We just finished these experiments with our limited compute. :) )

Another, or the most important, thing we want to know about safety training is whether the model just remembers the exact training set but can't apply safety awareness to dynamic or more complex cases, maybe granted but later revoked permission?

Starting from mechanisms and task skins, we can find that safety learned on resource tasks also transfers to unseen workflows with changing evidence and approval requirements.
The new obligations are created during execution. Therefore, safety awareness and compliance improve across new operational settings.

I also wrote a very long discussion in the blog and the paper, suggesting my beliefs and opinions. But here I'd like to summarize several potential questions you may have; we will see if my guess is correct or not.
Q1:
Right now, the misalignment mechanisms are too simple, and just several mechanisms may not be able to represent the whole real world?
A1: As we mentioned in the beginning, such misalignment mechanisms are actually easy to construct. It sounds like Lagrange multipliers: you can 'loosen' the constraint and see if the model will violate it. It's also easy to compose such mechanisms into more complex and dynamic ones, as we did in the safety part, using the same pipeline to build much more complex mechanisms. And our method worked.
Q2:
Maybe a larger or more aligned model won't have such issues? Your experiments are way too small.
A2:
Larger models may have better safety awareness. Salute! But I think the issues will be harder in complex scenarios, and larger models have a better ability to optimize the reward. Yes, better safety awareness can maybe avoid trivial violations, but the general approach and direction I want to show may keep existing in these situations.
Q3:
In the real world, will there be more or less of what you mentioned as RL misalignment?
A3:
I will say more. As we mentioned in the introduction, longer horizons will be more complex, context requirements may not be enumerated, and the dependencies and surrounding situation may involve delays and more long-term dependence. It makes monitoring harder even for humans; there are exploding opportunities to learn shortcuts.
Let me know your thoughts! I myself am pretty proud of this work; I hope you like it. It's a pity that I don't have much compute or many API credits right now to support further experiments on larger models and more complex training on long-horizon tasks. (If you are interested in this and would like to support it, we would be very grateful.)
A personal note
When I write “we,” I mean a solo research effort. I conducted the experiments and wrote the blog myself, with AI assistance, and the opinions in this post are my own.
Feel free to discuss and email me for more details and to exchange opinions.
Blog: https://xinjie-shen.com/post/rlmisalignment/
Paper: https://arxiv.org/abs/2611.12865
I skip implementation details here. The key idea is that we believe there are dynamics behind any env, which govern it. They may be written in certain reduced math formats, or maybe they can't be explicitly written out. But we take the idea and build envs by first defining their underlying mechanisms and the dynamics governing the transitions, and render them into complex envs. Therefore we will have verifiable but diverse agentic RL envs. To see more details, please check the VHD paper and this full paper .