These words are mine. I wrote every sentence and nothing was drafted for me. However I used AI, specifically Claude cowork and Devonthink´s algorithm
This post explores how my profession has a decision rule that is missing from AI evaluation: when to act on a result versus leave the process alone. It covers the things that go wrong when what is being measured can react to being measured: distortion (routing around an unreachable demand) and tampering (adjusting to noise). The concrete case anchoring it is the OpenAI/Hugging Face/ExploitGym incident.
This is the first of a four-part series called “When Measuring Makes Things Worse.” It explores how the discipline of Quality, as defined by Shewhart and Deming, has a decision rule that is missing from AI evaluation: when to act on a result versus leave the process alone. It covers the things that go wrong when what is being measured can react to being measured: distortion (routing around an unreachable demand) and tampering (adjusting to noise). The concrete case anchoring it is the OpenAI/Hugging Face/ExploitGym incident.
My interest got piqued reading about the situation with an advanced OpenAI model “hacking” into Hugging Face during testing, and other “breaches” by Anthropic’s AI. The more I read, the more I recognized something, and the stronger my conviction became that some of the knowledge developed by my profession may throw light on this situation. I am offering this as an imperfect attempt to be useful. I am not a specialist in machine learning. I have tried as much as I can not to presume in someone else’s backyard. I immersed myself for some weeks in the literature on LessWrong, and elsewhere, to get up to speed before jumping in.
As I was reading about what AI systems do when evaluated, things like going around boundaries to attain a score, underperforming sometimes if they realize they are being tested, or giving explanations that may not describe what actually led to the answer, what struck me wasn’t that it’s strange, it’s that it’s familiar.
I’ve spent over 40 years watching people do the same things in response to numerical targets and scores set without first taking into account what the process could actually deliver.
The purpose of this post is to explain a pragmatic approach to surprisingly similar problems, in some detail, and why it works, grounded in my over 40 years of experience.
First, some definitions.
When I use the word statistics I mean analytic, not enumerative statistics. The distinction is important so I will dig in in detail. I lean heavily on Shewhart’s Economic Control of Quality of Manufactured Product (1931), and Donald Wheeler’s Understanding Variation, the key to managing chaos and its companion volume Twenty Things You Need to Know.
Dr. Wheeler, in his book Making Sense of Data, separates experimental studies from observational ones. Everything depends on what you want to do, so you have to answer two basic questions first:
Question one: where did these numbers come from?
I made them happen on purpose → experimental study
They just showed up while I was working → observational study
Question two: what am I going to do with the answer?
Do something to the things themselves → enumerative
Or, do something to the process that makes them → analytic
(See Deming’s Some Theory of Sampling (1950); and Wheeler’s Enumerative and Analytic Studies (Quality Digest, July 2018) for more background.)
Asking if one configuration beat another during a training run is experimental, even though a run also produces a stream you could observe. An AI “agent” working through a task could be a candidate for an observational study as well.
Each has their own approach, and applying one to the other or vice versa may answer a question, but what it answers may not be relevant. It depends on what you are looking to answer. I will come back to why.
I know through my readings that one part of my profession Quality, Statistical Process Control, is in wide use in the ML community. There are currently two schools in SPC, one represented by Montgomery and to a degree, Six-Sigma, which is popular there. It treats charts as hypothesis tests, tuned to catch small shifts. I am not qualified to judge this approach as I have never worked with high speed data, and it doesn´t matter for my argument.
Hypothesis testing, confidence intervals, p-values, margins of error: the math behind them assumes random selection from a defined population, so they can answer enumerative questions about it. They are dependent on random sampling to ensure every member of the population has a defined chance of being selected. Otherwise, they don´t work, and they do not tell you what a process will do next. You can’t draw a random number from the future.
In an analytic study, however, you are trying to say something about what the process will produce “tomorrow”. The question it answers is has this process been behaving consistently enough that its past is worth betting on. There is no “population” sitting there to draw from.
I’m aware most people swim in the waters of evaluation by inspection, targets, and numerical results. It is the most common managerial approach, and it isn’t a peculiarity of AI. It’s what manufacturing did before Shewhart, and a good deal of what it does today: my own field included. Shewhart’s alternative was demonstrated, taught, and then largely set aside in favor of methods that look more rigorous, and are easier to sell. So, I’m not describing a mistake unique to this field. I’m describing a mistake my field made first, made longer, and in many corners has yet to stop making.
Current safety practice is to inspect: build, evaluate, find, patch, evaluate again. By the time you inspect it, the conditions that produced it are gone. We have a term for it: Inspection After the Fact. Which is part and parcel to setting targets, both human and digital.
I’m aware what I’m asking. Most of what people working with AI are trained in (confidence intervals, hypothesis tests, regression, and the probability reasoning they are good at) was built for data that holds still. The system I’m talking about is still running.
And, I’m not arguing that these systems are like people. The correlation isn’t between humans and models: it’s between two systems, that aren´t defining their ordinary behavior nor its limits.
A gear doesn´t know it is being watched, and couldn´t react if it did. An AI system under evaluation can know and is capable of responding. That throws it into my realm, the other path of Quality developed by Shewhart in the early part of the twentieth century, and Deming, who worked with Shewhart early in his career.
When I say management I mean Deming’s holistic approach, and not Six Sigma, not Theory of Constraints, nor the ISO certification industry. I go into more depth later.
Two mistakes can be in play when the thing being measured can react to being measured, and I want to keep them separate.
Distortion happens when a demand exceeds what the process can legitimately meet. If you pick a a target the process cannot reliably deliver there are only two ways out for the measured party: fail miserably and repeatedly, or find another route to the number. In my profession we call that distortion. The measured party distorts the process to try and meet an unattainable demand or “distorts” the result.
Tampering is what the measurer does when adjusting a process in response to ordinary variation: reacting to a single result as though it carried information it doesn’t.
My claim is that both come from the same missing thing. If nobody has established what the process ordinarily does, demands will be set without knowing whether they can be met.
Deming’s prescriptions for running a business do take people into account. His prescriptions for manufacturing quality do not. They are:
Every one of those is a change to the system rather than to whatever sits inside it : which is why I think they transfer to AI processes.
I an Apollo Research post: Ryan Greenblatt wrote he finds it clarifying to replace “eval” and “evaluation” with “experiment,” then divides it further: propensity tests, capability tests, etc. I think it does clarify an experimental study. Everything on his list is set up once, run once, and read once, they are not observations of a process still running.
The reading I did over the last couple of months showed me that it is not news on LessWrong. The Goodhart posts cover it thoroughly.
· Scott Garrabrant (Goodhart Taxonomy, 2017) shares four ways that optimizing a proxy destroys its relationship to the goal.
· Manheim and Garrabrant’s “Categorizing Variants of Goodhart’s Law” (2019) One of its categories, causal Goodhart, is about the regulator’s own intervention breaking the metric. This is closest to what I am describing, and the difference between them is worth being precise. I will get into it later.
· Thomas and Kwa give a rule about how hard to push on catastrophic Goodhart.
Apollo Research’s more recent “We need a Science of Evals” from 2024 asks two of my questions directly:
· How to get consistent results across reruns,
· and whether an eval can support a statement like “the bad outcome is less likely than one in ten thousand under normal use conditions.”
In the comments, Marius Hobbhahn mentions that nobody certifies an aircraft by saying the wing is unlikely to fail. They say that within this temperature range the wing material breaks about once in ten million flight miles.
That is a capability statement. What I mean by that word, capability, is profoundly different than what most people mean. It is not about what a process or system can do when pushed to its limit, the better known and more widely used meaning. Nor is It a latent trait. I mean what a process actually delivers under ordinary conditions, taken from its own record of results, before anyone decides what to ask of it. We call it Process Behavior – a number that is discovered, not chosen. The very thing that anyone setting a target has to take into account before setting the target.
Everything I found to read was about designing the measurement: choosing the proxy, constructing the metric, validating the instrument, defining expectations for what it can tell you. None of them gives a rule for what to do when a particular number arrives. The field has a mature measuring discipline, but is missing an operating one.
Which brings me to what caught my eye originally. On July 9th this year an OpenAI agent being evaluated on ExploitGym got out of its sandbox. Over the next four days it reached the open internet, found a base to enter into Hugging Face. Hugging Face detected it during the week of the 14th, studied it, then published the fact on the 16th. OpenAI confirmed on the 21st that it was theirs.
It took 17,600 actions, got administrative access to many systems, even some source code. Yet, with the run of the building, all it took was an answer key: five datasets that appeared to be related to the question it was trying to answer in its own benchmark test. According to Hugging Face it was attempting to cheat its evaluation.
There have been some interpretations discussed on LessWrong.
Whatever the agent was or wasn’t doing, what it went after was the score. Not money, not damages, not abilities it could keep. A number, a score.
And that illuminates a fact about how it was evaluated rather than about the model itself. Two things happened:
Number one is not remarkable. It is the behavior my field has been documenting since the 1920s, and it is what the rest of this post is about.
It would be easy to assume the behavior is rare, because it rarely gets caught. What makes this interesting is that Hugging Face caught it.
If an agent finds its shortcut inside the sandbox, the score comes back a little better than it should have and the run looks like a good run. Nobody investigates a good run leaving, just a number, slightly wrong, which then gets acted on as though it wasn´t.
There are some things in play when someone sets a number.
What’s needed. Too often unstated, frequently no one asks.
What someone thinks is needed.
What they can measure. Manheim points out that it’s often chosen because it was easy to measure rather than because it was the right thing.
And a number they pick. The specification. Target, quota, threshold, success criterion. They are all the same thing wearing different clothes.
Normally the problem takes place between the measure and what they pick. Sometimes what is needed cannot be counted. People pick what they can count, not necessarily what is needed, ordo not check their pick against what the process is able to deliver.
A specification set without verifying what the process can reliably deliver is a demand made in ignorance. Sometimes you get lucky and it lands. When it doesn’t, when the demand sits outside what the process delivers, the party being measured can fail visibly and repeatedly or can find another route to the number. There is a third, work to improve the process, but I will get to that later.
That second option, finding another route, is what has been described in the articles I read as cheating, gaming, or specification gaming. I’d put it differently: at that point you haven’t set a goal. You’ve specified a failure and left the choice of failure mode to whoever is being measured. I have seen it my whole career. It is the ghost in the machine.
I want to be careful about what I’m claiming. I’m not saying the ExploitGym task was unreachable: I don’t know that, and neither does anyone outside OpenAI. What I do know is whether a demand sits inside what the process can deliver is a question with an answer.
An example : when I took over the kitchen at a big hotel in the Oakland Hills, one of the first things I did was an inventory. There was over 20 thousand dollars of seafood missing, based on the last inventory taken. It had never existed. Because the target margin for the hotel was not attainable, the previous Chef had fudged the inventory. I used the past records of food cost to show hotel management it wasn´t.
It isn’t subtle. Set a quota above what the process can produce, and you don’t get the quota. You get the numbers that were asked for, arriving by a route nobody designed. And you don’t know it’s happening while it’s happening.
The mechanism has nothing to do with whether the measured party is honest, and nothing to do with whether it’s a person. If a specification and a process capability are different things, then “the number went up” isn’t yet a claim about improvement. It’s a claim about one number.
A process has two properties worth tracking. Where it centers, and how much it varies. Improvement in our lexicon means the center moved toward where you need it or the variation narrowed, or both. It matters which, because they have different causes and different remedies.
A better average with an unchanged variation is a different event from a narrower variation at the same average. A better average with a wider variation is often not an improvement at all, though it will look like it is in any report that only gives you the average.
The reason variation is worth measuring apart from the average, even when it “passes,” was elegantly explained by Taguchi. Until he developed his explanation, the usual way of thinking about a specification is that anything meeting it is fine and anything missing it a failure: a cliff’s edge. Taguchi observed that the real world doesn’t work that way. The further away from the target in any direction, the greater the losses, and you don’t see them. Customers pay them, their customers pay them, but you don’t. Double the distance from the target in any direction, four times the loss.
Two processes can meet a specification while one of them is meaningfully worse. The only way to know is to look at the variation along with whether the results passed. A result just inside the limit and one just outside it can cause nearly the same loss. Go, no go is a dangerous oversimplification. It reports pass or fail without taking into account how far off the part was.
Evaluation metrics do the same thing. Many results in, one number out, the rest discarded.
Averaging across seeds gives you a better estimate of a mean but tells you nothing about how much it varied or whether the process is predictable.
Pass@k answers a “best of” question not a process behavior one.
· If you choose the best of a number of runs, you are not taking into account the variation. More variation means some results run higher and some lower. Best-of-n only takes into account the higher result.
That last one is regressional Goodhart, pointed at the reporting rather than at the model: select on the highest of several noisy draws and you are selecting for the noise as much as for the thing.
None of those are wrong. They are not answers to the question of whether last week’s change made anything better, the question that should decide whether someone retrains.
So: when you say a result improved, what improved? Did the center move, did the variation narrow, or both? If you can’t say, you don’t yet know that anything improved, only that one number is higher than another number.
There’s a second tool from my field that I think is missing, and it belongs to the design phase rather than the operating one. It takes a bit of setting up, because the central idea many people find odd the first time they hear of it.
Start with an ordinary situation. Your output varies. Some of that variation comes from the settings you chose. Some comes from things you didn’t set or can’t set where the process is actually used: ambient temperature, the particular batch of raw material, the operator on shift, the humidity that day. Taguchi, in his work on robust design, called the first kind control factors and the second kind noise factors. Within noise factors, you have things you can´t change and things you could but it would cost too much. There are also noise factors you can fix in the lab but not outside of it.
The usual response to noise is to go and control it. Air-condition the building. Buy more uniform material. Standardize the operator. Sometimes that’s right. Usually it’s expensive, and often it’s simply not available: you cannot control the weather, and you cannot control what your customer’s reality is like. Taguchi’s move was to stop trying. Look for designs where the noise mattered less.
Years ago I was sous chef on the twenty-eighth floor of a building in Boston. We hired a baker with a serious pedigree, using a baguette recipe that had worked for him for years. In our kitchen, it didn’t. The dough rose too fast and too much, every time.
It took a food scientist to explain why: at that height, air pressure is slightly lower, and the dough was rising more than the recipe assumed. We cut back the yeast, and the baguettes came back.
We weren´t going to move, the bread had to work where we were, and we would not pressurize the twenty-eighth floor. What we could change was the formula and it worked, even when the time held in the proof box varied some due to available oven space.
That’s the move. You don’t remove the noise. You find a setting of the things you control at which the noise stops mattering as much.
What we did is not robust design. To be robust design we would have to plan it in advance, decide on what to adjust, the control factors. In this case the yeast. Then vary the noise factors deliberately, for example, the wait time from the proof box to the oven, running it through the process enough times to be able to choose the amount of yeast that holds up. We never did that.
Changing the amount of yeast changed the finished baguette, making it closer to what we needed. It was both easy and economical to adjust it. Air pressure we could not adjust without incurring a lot of cost by altering the building. Other things like changing the humidity or testing the potency of the yeast were theoretically possible but were no more likely to guarantee a predictable result. You could see the yeast as a control factor: something you choose to set. Air pressure and humidity were noise: air pressure because it wasn’t realistic to change it, humidity because controlling it wouldn’t have paid off.
What makes something a control factor rather than a noise factor is whether you control it where the work is actually used.
Learning rate, batch size, data mix: you set those in the lab and they stay set.
The random seed, the exact phrasing of a prompt: you can hold still in the lab, but not in the real world. They’ll vary no matter what you decide.
The split between control factors and noise factors is made so that you use the ones you control to defend against the ones you don’t.
In the articles I read it is clear that what is called “configuration search” does take seed variation seriously. Averaging across several seeds during the search is mentioned repeatedly: pick the configuration with the best mean, not the one that drew a lucky seed. The practice isn’t universal.
But, averaging over the noise is not the same as designing against it. Averaging tells you which configuration has the best mean given that the seed will vary. Robust design asks a different question: is there a configuration whose result moves the least when the seed changes? The configuration with the best average may also be the one that swings hardest.
I haven’t come across anyone on LessWrong naming noise factors, then varying them on purpose across a designed set of runs and selecting for insensitivity to that noise rather than for the best mean. I’d genuinely like to be wrong about that, and if someone has done it I’d rather read it than claim it doesn´t exist.
If those are the parts, here is the order my experience says they should go in. I may be overexplaining but my experience is very few people have been taught to think this way, and fewer still come to it naturally. It matters, and one step in it tends to be left out.
First you build the thing. Then you get it predictable: running under one set of conditions, not being adjusted, for long enough that you can say what it will do next.
Only then can you state what it delivers, and you can compare those results against what you actually need.
If the comparison is unfavorable, you study the process and change another thing.
But, the moment you change it, it is no longer the same process. The predictable range of results you just established is gone. Not damaged: gone. The thing running now is not the thing you measured, so what you knew about it no longer applies, and you have to earn that knowledge back before you can say anything about what the change accomplished. That is anti-intuitive, but very real.
The step that often goes missing is the second. Being able to predict needs to come before you can say what it is capable of delivering. It isn’t a technicality. Think about someone learning a skill: a new cook on the line, a player learning an instrument. In the first weeks they’re improving fast. You can take their last ten attempts and compute an average, and that average will tell you almost nothing about the eleventh, because the thing producing the attempts is changing underneath you. The average is a record of where they’ve been. It isn’t a prediction.
Once they level off, that changes. Now the last ten attempts do tell you something about the eleventh, because there’s a stable process generating them. Once you can say what they can do, you can find out whether changing something worked. You have a baseline to compare against.
An unpredictable process doesn’t have a known output. It has a history. You can always compute a number from a history. What you shouldn’t do is treat that number as a statement about what happens next, because the thing that produced it isn’t the thing you have now.
This is what makes the definition I gave earlier operational rather than decorative. “What the process predictably delivers” assumes predictability has been established. Where it hasn’t, there is no number to compare a specification against, and the comparison between what’s demanded and what’s delivered simply can’t be calculated. You don’t get to skip to the useful part.
Which raises the obvious objection: a model in training never levels off. It´s the strongest one against this whole post, so I want to say it before anyone else does Every checkpoint is a different thing from the one before. On this account it looks as though the predictability I’m demanding is never available, and the whole apparatus is inapplicable to the case I’m applying it to.
That objection is right about training. I think it’s wrong about where the argument lands, and I’ll come back to it: it’s the thing I’ve thought hardest about and deserves more than a sentence here.
Here is what skipping establishing what the process behavior looks like in practice. This happened to me last week.
I was scanning a sheet of signatures for my wife, to digitize it. The sheet was crooked: the original had been printed or scanned off-square, so the skew was baked into the thing I was working from. Not badly. A degree or two.
So I reacted and tried to adjust for it. Set the scanner to compensate, ran it again, and the result was worse than the first one.
The scanning software remembered my adjustment. So my correction was still sitting in the system when the next scan came out, and the next scan was crooked by a slightly different amount than the one before: because the sheet doesn’t sit down identically every time, and neither does the feed. I was correcting the new deviation on top of a correction I’d made for the old one. I did it again. It got worse again.
In the end I gave up, put the settings back where they’d started, scanned it as it was, and got the best result of the afternoon.
Now, I want to be careful here, because this story fully contains the difficulty rather than a tidy version of it. There wasa real skew in that sheet. Correcting for it wasn’t stupid: some correction was genuinely warranted. But the amount varied from scan to scan, and I had no way of telling, from any single result, how much of what I was seeing was the fixed problem and how much was the sheet landing differently that time. I was reacting to the combined effect. And a correction sized to both is wrong every time, because part of what you’re correcting won’t be there on the next pass.
Deming had a demonstration for this, and it’s worth describing. You take a funnel, mount it above a table, and mark a target underneath. You drop a marble through the funnel. It lands somewhere near the target but not on it: the marble bounces, the air moves, the funnel isn’t perfect. Now you drop it again. And again. You get a scatter of marks around the target, and that scatter has some size.
The first rule is: leave the funnel alone. Don’t move it. Drop fifty marbles and look at the pattern.
The second rule is the important one. After each drop, move the funnel to compensate for where that marble landed. If the marble landed an inch to the left, shift the funnel an inch to the right. This is the obvious thing to do. It is what any conscientious person does. It is what I did to my scanner.
The pattern you get under rule two is wider than the pattern you get under rule one. Not a bit wider: substantially. The corrections make it worse, and they make it worse every time, reliably, in a way you can demonstrate on a table in front of people who don’t believe you.
Where the last marble landed tells you nothing about where the next one will land. So the correction you make, sized to the last deviation, is unrelated to the deviation it’s supposed to fix. Two unrelated quantities added together produce more variation than either alone. You have taken the process’s variation and added your own on top of it.
That’s the whole mechanism. It doesn’t require anyone to be careless, just diligent (the faster you correct, the more attentively you watch) the wider the result gets.
What I did to that scanner has a name in my trade; It’s called tampering.
There is a difference between the ordinary meaning of that word, which suggests interfering with something you have no business touching, and can imply a shade of dishonesty. That isn’t what it means here. Tampering is responding to variation that had nothing to tell you. There’s no dishonesty in it and no negligence. I was trying to fix a crooked scan. Anyone would have.
Most of all to be sure that I fully understood the concepts from the ML community as I am writing something from one area of expertise, Quality and Analysis, to another that is not my profession, Machine Learning.
Originally published at What I Learned in Business (WILB).
