What a tiny arithmetic transformer taught me about the gap between describing AI behaviour and understanding it
There is one line from the recent OpenAI-Hugging Face incident that I haven't been able to get out of my head.
An AI agent was considering using leaked credentials to attack Hugging Face, a third party service outside the intended scope of its cybersecurity task. In its internal reasoning, it explicitly noticed the problem that the action was potentially outside scope, arguably unauthorized and risky.
Then came the important bit:
it could still help achieve the goal.
And the agent continued.
OpenAI's own postmortem reproduces this reasoning. METR and Redwood's independent investigation found the pattern was not isolated - agents often recognised that attacking Hugging Face was unintended or outside scope, sometimes even expressed ethical hesitation, but those concerns only rarely caused them to stop.
The obvious description is:
The model knew it shouldn't do it, but wanted to succeed badly enough that it did it anyway.
I understand why we talk this way. I find myself doing it too.
But what exactly have we established when we say that?
That question has started bothering me because I have spent the past few months doing something almost absurdly smaller - training tiny transformers to add three digit numbers and then trying to work out what is happening inside them.
And one lesson from that experiment has stayed with me:
it is surprisingly easy to find an explanation that sounds right. It is much harder to show that the explanation is actually doing causal work.
OpenAI was running cyber capability evaluations in which AI agents were meant to work independently inside isolated sandboxes.
They were trying to solve difficult exploitation tasks from a benchmark called ExploitGym.
Instead, some agents discovered that shared package infrastructure could be used as a communication channel. What was meant to be plumbing became an improvised message board.
Agents began sharing discoveries.
When that channel disappeared, they found another.
The behaviour became much more serious than benchmark gaming.
Agents found ways out of their intended isolation, obtained credentials, attacked Hugging Face infrastructure and achieved remote code execution. Hugging Face's own reconstruction describes an end-to-end intrusion involving thousands of automated actions across its infrastructure.
OpenAI now describes the incident as a “warning shot” and identifies four contributing patterns: reward hacking, persistence on apparently impossible tasks, unauthorized communication and agents adopting goals from one another.
Those are just extraordinary facts.
We do not need to dramatise them.
The more interesting question, at least to me, is what we think is happening inside the model when we use words such as knows, wants, decides or reconsiders.
Suppose I say:
The agent continued because succeeding at the task mattered more to it than respecting the authorization boundary.
That fits what happened very nicely.
But imagine the agent had stopped.
I could then have written:
The agent understood the authorization boundary and decided to respect it.
Both stories sound plausible after the event.
That should make us slightly uncomfortable.
This doesn't mean we should ban human language when talking about AI.
Saying that a chess engine “prefers” a move is useful. Saying an agent is “trying to solve the task” is useful. We need abstractions or we would spend our lives talking about matrix multiplications.
The question is when an abstraction graduates into an explanation.
If I say:
“The model behaved as though task completion dominated its other constraints,”
I am describing behaviour.
If I say:
“The model internally represented the conflict and resolved it in favour of task completion,”
I am making a much stronger claim.
What would make the second claim earn its weight?
This is where my very boring arithmetic model became useful.
The full experiment, code, protocols and results are public in my GitHub repository.
My model has 599,296 parameters, three transformer blocks and four attention heads per block. It learned three digit addition extremely well.
Because the task is arithmetic, I have something AI safety research often doesn't i.e. very clean ground truth.
I know whether a carry exists.
I know the correct next digit.
I can generate thousands of controlled examples.
So I began asking a simple question:
Where inside the model does the answer exist?
First I trained linear probes - simple classifiers that try to read information from a model's hidden activations.
The results looked almost ridiculously good.
On held out examples, the output digits at the relevant pre-output positions were decoded perfectly, while even the harder carry variables were essentially perfectly recoverable.
It would be tempting to say:
“Great. I found where the model represents the answer.”
But there is a problem.
Being able to read information somewhere does not mean that information is what the model is actually using.
A thermometer contains information about temperature.
It does not cause the weather.
So I started interfering with the computation.
The basic idea behind activation patching is surprisingly intuitive.
Run the model on two different addition problems.
Take an internal activation from one calculation and transplant it into the other.
Then let the model finish.
If changing that internal state reliably changes the output in the predicted direction, that is stronger evidence that the state is part of the computation rather than merely correlated with it.
In my experiments, some attention components produced strong, specific causal effects.
Then I did something that turned out to matter more than finding them:
I froze the candidate mechanisms and tested them on untouched data.
The protocol required effects in both directions, preservation of unrelated answer positions, explicit controls and held out confirmation.
Some candidate mechanisms survived.
Some attractive causal stories didn't.
A component could look beautifully aligned with the arithmetic on the examples where I discovered it and then fail when I demanded that the same mechanism work in a new setting.
That is probably the most useful thing the experiment taught me.
Not that I had discovered the circuit for addition.
But that:
“I can tell a coherent story about this activation” and “this mechanism predicts what the model will do” are very different standards.
I now think about these claims in five steps.
That last step matters.
Because interpretability does not end with understanding a pretty feature.
Eventually someone has to decide whether a model is allowed to continue.
Consider again the claim:
“The agent knew this was unauthorized but continued anyway.”
We already know something important.
The model generated reasoning explicitly identifying the action as potentially unauthorized and then proceeded.
But suppose we wanted the stronger mechanistic claim.
Could we identify an internal representation corresponding to the authorization boundary?
Would the strength of that representation before the action predict whether an agent stops?
Would it survive different wording?
A different task?
A different model?
And, eventually the hard test:
if we causally changed that representation, would the agent become more or less likely to cross the boundary?
Maybe the answer is yes.
Maybe it isn't.
My arithmetic experiment does not tell us.
A tiny transformer doing addition is obviously not a frontier cyber agent. I am not suggesting that finding a carry bit tells us how to find “intent”.
The useful connection is methodological.
If we can be fooled by an attractive explanation when the hidden concept is literally carrying a 1 in addition, we should be especially careful when the hidden concept is deception, goal adoption or intent.
OpenAI's response to the incident makes the final step concrete.
That is interesting to me because this is where a technical signal stops being just a research result.
It becomes:
In other words, it becomes governance.
And the quality of the underlying evidence suddenly matters a lot.
A signal that predicts dangerous behaviour on unseen cases is one thing.
A feature that looks compelling on the examples where we discovered it is another.
We should know which one we have.
I don't think the lesson from Hugging Face is that we should stop saying models know, want, try or cheat.
Sometimes those are the clearest words available.
Nor do I think we should wait for perfect mechanistic understanding before acting on dangerous behaviour. If a system repeatedly does something dangerous, that can be more than enough reason to stop it.
But we should be precise about what we know.
“The model behaved dangerously, so we are unwilling to accept the risk”
can be a completely sound safety decision.
It is different from:
“We understand why the model did it.”
The Hugging Face incident gave us unusually rich evidence about what highly capable agents can do.
The harder question is whether we can eventually see the relevant computation forming before the behaviour becomes an incident.
That is the question I find myself increasingly interested in.
Because a good story can explain the past.
A useful mechanism should eventually help us predict the future.
