These words are mine. I wrote every sentence and nothing was drafted for me. However I used AI, specifically Claude cowork and Devonthink´s algorithm
Now I go into the economics, what the solution costs against what running unwatched is already costing, and the falsifiers: two claims that would sink the whole argument if they are wrong, including one anyone can test in an afternoon.
Here’s the trade as I’d put it to a plant owner. Charting a process costs something real: you have to collect the numbers, you have to leave the process alone long enough to establish what it does, and while you’re doing that you aren’t changing anything. And, the cost is visible. You can budget it.
Against it sits a cost you can’t see which includes
• changes made in response to nothing,
• in results chased that were never real,
• in variation nobody knew was there.
• everything lost while a process runs unwatched,
A cost you can put a number to, against one you can’t, where the one you can’t is bigger.
Spending compute to establish a baseline is the price of winning the ability to see what your process is doing. It is not overhead to be justified against a model you could have trained. And the alternative isn’t saving by not spending. It’s spending more on changes made in response to noise, plus whatever those changes cost downstream, plus the time spent a researcher spends spinning meaning into differences that weren’t there.
Loss doesn’t begin where the requirement is drawn
There’s a piece of my field that makes this concrete, and I’ll take only the limited version of it because it is all the argument needs.
The intuitive picture is that a result either meets the requirement or it doesn’t. Meet it, and you are fine. Miss it, and you have a problem. Binary thinking. If that picture were right, variation that still met the requirement would be free, and there’d be no reason to prefer a tight process to a loose one that scrapes by.
It isn’t right, and the clearest case I’ve run myself involved roast chicken. I spent a decade with a supermarket chain, where spit-roast chicken was the best-selling product in the food service department. Chickens were cooked in batches of twelve to fifteen, and the batch size was set by labor costs, because supermarkets run on thin margins and labor is the cost that looks enormous next to them. It was watched constantly.
Nobody was watching what happened after the chickens came off the spit. They went into a bain-marie under lights: equipment that holds temperature well and is punishing to moisture. We weighed them. A chicken loses fifteen percent of its weight in the first twenty minutes off the spit, and keeps going. Plenty of ours were selling an hour or two later.
Nothing there was defective. Every chicken was a good chicken, properly cooked, safe, saleable, and nobody would have rejected one. But we bought by weight and sold by weight, and every minute a bird sat under those lamps, less of what we’d paid for was still there to sell.
So we worked out how long the cooking took and how long seven chickens took to sell, and moved to five to seven an hour, doubling into the dinner rush. Same chickens, same price, roughly the same labor. Profitability on the department’s best-selling product went up by ten percent or better depending on the week, and sales rose as well: customers were getting a plumper, more succulent chicken.
People argue about the math in Taguchi[s Loss Function. The limited claim survives all of it: loss doesn’t begin where the requirement is drawn. It begins at the value you needed, and the loss keeps growing. That’s why a tighter process is worth money even when the loose one passes, and it’s the whole economic case for spending anything on variation at all. Without it, reducing variation that already meets the requirement is a hobby.
Inspection is the expensive route
There are two ways to deal with a process that produces bad results. You can inspect the results and catch the bad ones. Or you can change the process so it stops producing them. Deming made the general case decades ago telling us to cease dependence on inspection to achieve quality. Inspection is the expensive route. It’s late, it’s never complete, and it does nothing to the thing causing the problem, so you pay for it again every cycle you run.
There are exceptions and they matter. Some things get inspected no matter how good the process is: a nuclear reactor, a batch tested for contamination, anything where being wrong once has unbounded consequences. Nobody inspects those for efficiency, they’re inspected because the cost of a single escape is one you can’t accept and improvement doesn’t make the requirement go away. Any amount of E. coli O157 in food is unacceptable. I call those kinds of requirements facts of life.
Some of what this field is doing falls under facts of life. Checking whether a model will help build a weapon is not a cost-optimization question and nobody should treat it as one. But most of it isn’t that. Detecting a model that has routed around a demand and building a detector for each new route as it appears is ordinary inspection: necessary now, and I’m not arguing against doing it, but it’s an expensive route. As far as I can see, it is being used as the main one, and my field would warn against a growing inspection function attached to a process nobody is changing.
The pickle plant example from Part 3 shows what the other route looks like:
• The filling machine ran faster than people could keep up, so extra workers scooped up spilled slices from the belt and put them into jars by hand. Then they shoveled up the product on the floors and disposed of it. The people on the line had to wipe excess brine off the jars. In other words, they were all compensating for a speed no one could keep up with.
• After careful observation, calculation and discussion, management let them slow the machine to the speed people could work comfortably with.
Nobody there was persuaded to stop compensating, it just went away naturally. The demand stopped being unreachable. The easiest and cheapest way to meet the demand had become the intended way.
I want to be careful about how far I go with this analogy. I know from my experience that reachable demands produced less distortion in plants, reliably, over a long time. Whether it works the same way with a model is exactly the sort of thing that needs to be tested rather than assumed. But the economics of the two routes don’t depend on settling that. Inspection is expensive wherever you do it.
How much watching a process needs is set by how wide its routine variation is. A process that does nearly the same thing every time can be left alone and checked occasionally. A process with a broad range of variation requires constant monitoring because a single result cannot reveal whether the process has changed. That cost is ongoing and grows with the number of processes.
This is worth saying plainly to a field that is building supervision for agents right now. Whatever that supervision costs (a person watching, or another model watching) the bill is set by how far apart the natural process limits are. Narrowing the variation is the only thing that reduces it. Adding watchers doesn’t; it just distributes the same bill differently.
What I can’t tell you
I can put numbers on my side of this. At the pickle plant, more than a pound per jar was walking out the door and had never appeared as a loss anywhere: $130,000 a year, in plain sight of everyone, invisible because nobody was looking at it in a way that would have shown it.
I can’t put a number on your side, and I’m not going to invent one. Nobody knows what’s being lost to this in AI evaluation, and the reason nobody knows is the point I’m making: you don’t get a figure for a cost until someone builds the instrument that would show it. You know what your compute costs and I don’t. If some fraction of it is going to chasing differences that weren’t real, you’re better placed than I am to say what that fraction is worth.
And if it turns out you’re already accounting for this, then there’s nothing here and I’ve wasted your afternoon. I would be pleasantly surprised if you were. I found no sign of it.
I said at the start that two claims here matter more than the rest. Here they are, and here is what would settle each one against me.
First: that the gap is real.
The claim is that this field has a mature literature on how to design an evaluation and no discipline governing what to do when a particular result arrives while the process is running. Nobody has written down what to do on the Tuesday the number comes in, applied the same way every time, whether you like the number or not.
What would falsify it is somebody showing me the rule. Not a recommendation to run more seeds, which is advice about study design. A stated procedure that takes a result and returns act or don’t act in the moment it arrives. A procedure fixed in advance, applied identically regardless of which answer you were hoping for, and applied to a process that keeps running rather than to a comparison made once.
If that exists and I’ve missed it, this post is a long way of saying something the field already knew, and the right response is to point me at it. I’ve looked. I don’t think it’s there. But I’ve been reading this literature for a few months, not years, and I came to it from outside, which is exactly the position from which a person misses the obvious.
Second: that eval results from a fixed configuration behave like a process. This one is fundamental in a way the first isn’t, because everything operational in the post rests on it. The arithmetic behind the limits assumes routine variation is the accumulated sum of many small contributions with none dominant. That’s not a mathematical convenience; it’s a claim about the world, and it’s true of a great many processes and false of some.
If eval results don’t work that way (if the movement between runs is dominated by one or two large causes rather than many small ones, or if the numbers never settle no matter how long the configuration is held still) then the limits describe nothing. And the rule would be worse than useless, because it would tell people to leave real changes alone. A wrong rule applied consistently is more dangerous than no rule at all.
Why this falsifier is the better one: it’s cheap, and anyone can run it.
The test:
1. Freeze a configuration.
2. Run the same evaluation twenty times, changing nothing but the seed.
3. Plot the results in the order they came.
4. Work out the natural process limits from those twenty results, the way Part 2 describes.
What passing looks like: every result falls inside the limits, scattered with no pattern. That means the movement comes from many small causes, and the method applies. Then the rest of the series is worth arguing about.
What failing looks like: results jump outside the limits, or shift and stay shifted, or never settle into any steady range. That means one or two big causes dominate, or nothing is steady at all. Then the limits mean nothing, and you’re wrong about the most important thing in the series.
Either way, they find out in an afternoon for the cost of some compute. I’d rather that experiment be run than that this post be agreed with.
What wouldn’t change my mind, and why I’m saying so
1. Two things I expect to hear that I don’t think bear on it.
a. That models aren’t machines. I agree, and it doesn’t matter, because the argument isn’t about what the process is made of. It is about what comes out of it: whether the results from a fixed set up behave predictably. That is is what a twenty run test checks.
b. That this has been tried and didn’t work. If it really was, that would tell me something real, and I would want to know why. But “quality control was a fad in the eighties” is a slogan about the method’s history. It tells me nothing about whether it actually works. The “fad” failed because people ran the arithmetic without understanding it, not because the method didn’t work, and I’ll say more about that in a moment.
One more thing, now that the argument is finished rather than at the start
Strip the vocabulary out of all of this (the charts, the limits, the seeds, the signals) and every problem in the post is the same problem. Someone is looking at a result and deciding whether they need to do something about it.
The baker and the yeast. Me and the scanner. The researcher and Tuesday’s eval score. The agent and the run that’s going strangely. Different rooms, different times, the same question.
I’m a quality practitioner. Forty years. Certified an ASQ manager of quality and organizational excellence (retired), which tells you I passed their exam and not much else, for reasons I’ll get to. Most of my working life has been in food: a decade with a supermarket chain, seven years consulting for a pickle plant, kitchens before that, helping cheese producers for a long stretch.
I’m not an ML researcher. I can’t evaluate the technical claims in most of the papers I’ve cited here: I’ve read them for what they say about how decisions get made, which is the part I know something about, and I’ve taken the rest on trust. If the mapping breaks somewhere I couldn’t see, you’ll have to be the one to tell me.
What I’d offer instead of authority is this. For forty years I’ve watched people look at a number and decide whether to touch something. In kitchens, on lines, in stores, in plants. That’s a narrow specialty and it happens to be the thing this post is about. Not because manufacturing is like machine learning (it isn’t, and I’ve tried to say so wherever it matters) but because the decision is the same decision, and the ways it goes wrong have been around for a long time.
Now the part that matters more than the certificate
I’ve graded ASQ certification exams. Of the candidates whose papers I marked, all but one could not distinguish check from study: the difference between confirming something happened and finding out why. They could not tell a specification limit from a natural limit: the difference between what someone decided to demand and what the process actually does. They read process behavior charts as report cards, which is precisely the reading this post spends so many paragraphs arguing against.
They were certified anyway.
This isn’t a new failure and I’m not the first to notice it. Deming, writing in 1982 about why Japan bothered with conferences for senior management, explains what happened in America when nobody did. ˜Statistical methods took fire around 1942˜: ten-day intensive courses for engineers, war department training at supplier factories. ˜Brilliant applications burned, sputtered, fizzled and died out” (W. Edwards Deming, Quality, Productivity, and Competetive Position, 1982, MIT Center for Advanced Study). What people had done, he says, was solve individual problems. Charts multiplied on the principle that more was better. Quality control departments sprang up, and they plotted charts, looked at them, and filed them, taking the work away from everybody else: which he calls entirely wrong, because it belongs to everybody. There was no structure to teach management their responsibilities. One instructor tried, by inviting them in for half a day. Deming’s word for that attempt is noble, and his word for the result is ineffective.
That was published in 1982, about the 1940s. I graded those exam papers a good deal more recently. The arithmetic survived both times. What it was for did not.
I’m telling you this because it would be easy to read everything above as an offer of a tradition that works, held back only by people who haven’t heard of it yet. That isn’t what I’m offering. My own profession built itself around this material and mostly fails to transmit it. The arithmetic gets taught. What it’s for does not. People come out the other end able to compute a limit and unable to say what a signal means, and then they go and work somewhere.
So when I say the field has the measurement and not the rule, I’m not describing a mistake made only here. I’m describing one my own field made first, made thoroughly, and has not stopped making.
What I actually want
I’ve been clear about the two claims that would sink this if they’re wrong, and one of them can be tested in an afternoon by anyone with the compute.
If it’s useful, use it. Take the parts that work, discard the rest, and don’t credit me: the tradition isn’t mine, and Shewhart and Deming are past caring. If it’s wrong, I’d rather find out than have it sit around being wrong quietly.
And if it turns out this is all already known and I’ve spent a long time restating it in an unfamiliar vocabulary, that’s a decent outcome too. It would mean the field is further along than it looked from outside, which is the more important fact.
Sources
Everything the argument leans on, in four groups. Where a source is weaker than it might look, I’ve said so: those notes are the ones worth reading.
The tradition
Walter A. Shewhart, Economic Control of Quality of Manufactured Product (1931; Dover reprint). The origin of the whole apparatus.
Walter A. Shewhart, Statistical Method from the Viewpoint of Quality Control (1939; Dover reprint). Shorter, and the one I’ve quoted. Page 12 is where he says the choice of three sigma was an economic judgment rather than a mathematical derivation, and that hopes of finding a unique functional form for the distribution are blasted. I checked that page against my own copy rather than trusting a secondary account, and I’d suggest anyone citing it do the same: a good deal of what circulates about Shewhart online has been through several hands.
W. Edwards Deming, Quality, Productivity and Competitive Position (MIT Center for Advanced Engineering Study, 1982). Page 101, section headed “Conferences with top management,” is the account of the American statistical-methods boom of the 1940s and its collapse, running onto page 102. ⚠ A version of this passage circulates widely in quotation compilations with a clause about management not knowing their responsibilities spliced into the middle of the sentence. That sentence does not exist. The idea is Deming’s (it’s his own next sentence, on the following page) but the quotation is manufactured, and I’ve stated the passage in my own words rather than reproduce it. Deming’s own source for the 1942 figures is given in his footnote: W. Allen Wallis, “The Statistical Research Group,” Journal of the American Statistical Association 75 (1980), 320–335, with page 321 specified.
W. Edwards Deming, Out of the Crisis (MIT CAES, 1986). The successor volume, and the one most people mean when they say they’ve read Deming.
Donald J. Wheeler, Understanding Variation: The Key to Managing Chaos. The short one. If you read one book from my field, this is it.
Donald J. Wheeler, Twenty Things You Need to Know. Where the accumulation argument behind the three-sigma limits is set out without mathematics.
Donald J. Wheeler, Making Sense of Data. Cited here only for operational definitions (§3.2) and count data (chs. 13–15). ⚠ It does not carry the enumerative/analytic distinction, and I have not verified that it uses the term “rational subgroup”: §16.3 is adjacent but I haven’t read it, so I’ve stated the grouping principle in my own words and cited nothing for it.
Donald J. Wheeler, “Enumerative and Analytic Studies,” Quality Digest, July 2018. Four pages, free online, and the clearest statement of the distinction anywhere. Contains the line that matters most for this post: you cannot take a random sample from the future.
Genichi Taguchi, on the loss function. I’ve used only the narrow claim: that loss begins at the value you actually wanted rather than where the requirement is drawn. The quadratic form is an approximation and the “loss to society” framing is a value claim rather than a finding; both get argued about, and neither is needed here.
The AI evaluation literature
This is the half where I’m a visitor. I’ve read these for what they say about how decisions get made, and taken the technical content on trust.
Marius Hobbhahn and Jérémy Scheurer (Apollo Research), “We need a Science of Evals” (2024). The field’s central methodology post, and the one this whole argument is downstream of. Hobbhahn’s own comment on it, that an aircraft bound is a statement about failures per flight mile within a specified temperature range, is the closest anything in this literature comes to what my field means by a capability statement.
José Hernández-Orallo and Fernando Martínez-Plumed, “A response to ‘We Need a Science of Evals’” (aievaluation.substack.com, February 2024), subtitled “The science exists.” The strongest challenge to my framing, and I’d rather you read it than take my summary. It points at psychometrics, which is a mature measurement tradition: and, as far as I can tell, an enumerative one with no rule about when to act. Hernández-Orallo wrote The Measure of All Minds (Cambridge, 2017).
Ryan Greenblatt’s comment on the Apollo post, proposing that “evaluation” be replaced throughout with “experiment.” I cite it because everything on his list is an experiment, and observation of a running process doesn’t appear: not, I think, because it was considered and rejected.
Scott Garrabrant, “Goodhart Taxonomy” (2017), and David Manheim & Scott Garrabrant, “Categorizing Variants of Goodhart’s Law” (arXiv:1803.04585, rev. 2019). The formal treatment. ⚠ It contains the strongest counter-argument to my thesis (their causal Goodhart is about the regulator’s intervention, which is close to what I call tampering) and I’d want anyone assessing this post to read it with that in mind. Its threshold formalism has no time dimension, which is why I say it can’t state the decision rule, but that’s my reading of a formalism rather than something its authors claim.
David Manheim, “Building Less Flawed Metrics.” Two things in this post are already occupied ground: specification-by-convenience, and part of what I’ve said about specification versus constraint. I’d been treating both as unclaimed and was wrong.
Thomas & Kwa, “When is Goodhart catastrophic?” (LessWrong 2023; later a NeurIPS 2024 paper). A rule about how hard to optimize, decided in advance from a distributional property. In the comments the authors’ reviewers flag the weak point themselves: Vanessa Kosoy on the independence assumption, leogao on non-independence being the likelier route to catastrophe. I’m not the first to say the model is cleaner than the world.
Andreas Stuhlmüller and jungofthewon (Ought), “Supervise Process, not Outcomes” (Alignment Forum, 2022). An ally rather than a gap, and I’ve tried to treat it as one.
Maxime Riché et al., “A Taxonomy Of AI System Evaluations.” The structural evidence for my central claim, and the place to check it. Its section on using evaluation outputs lists precision as a property of results and gives directional and threshold criteria: and nothing joining them to an action. they already list a model scoring its own output as one place the analysis can happen, which is the self-grading problem Part 3 discusses.
David Heineman et al. (Allen Institute for AI and UW), “Signal and Noise in Language Model Evaluation” (arXiv:2508.13144, 2025). 465 models, 30 benchmarks, 900,000 results. The closest anyone has come to what this post argues, and omitting it would be a hole. My one disagreement is specific: the noise figure is computed from the final thirty checkpoints of the run being judged, which puts the baseline and the judgment in the same data. ⚠ If you want to attack that reading, attack the independence assumption rather than normality: normality is robust and a statistician has a good answer; independence is checkable.
Lovish Madaan et al., “Quantifying Variance in Evaluation Benchmarks” (arXiv:2406.10229, 2024). Ten Llama-2-7B models trained from scratch on identical data with different seeds.
Arjun Panickssery, Samuel R. Bowman and Shi Feng, “LLM Evaluators Recognize and Favor Their Own Generations” (NeurIPS 2024; arXiv:2404.13076). Self-preference tracking with self-recognition. One author is at Anthropic, which you should know when weighing it.
Jie Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet” (ICLR 2024; arXiv:2310.01798). Self-correction without an external reference mostly doesn’t help and sometimes hurts.
J Bostock, “Measuring Learned Optimization in Small Transformer Models” (Alignment Forum). Small models, and he reaches the self-evaluation problem reluctantly, through his own failed attempt: which is why I trust it and also why it shouldn’t carry much weight on its own. His cross-objective result is the shape a real check would take.
Kartik Moondra, Areeb Chughtai, Rishi Lanka and Rahul Gupta, “Why Global LLM Leaderboards Are Misleading” (arXiv:2605.06656, May 2026). Roughly 89,000 Arena votes across 52 models and 116 languages. ⚠ A preprint, not peer-reviewed, from May, 2026: take it as suggestive. They find pooled global rankings collapse into noise that is really conflicting subpopulations, and name Simpson’s paradox directly.
The ExploitGym incident
Girish Gupta, “The OpenAI models that hacked Hugging Face weren’t just following instructions” (LessWrong, July 2026; cross-posted at Redwood Research). The “nobody decided anything” passage is the closest thing in this literature to what my field means by a system producing an outcome no one chose.
julius vidal, “…WERE just following instructions (contra Girish Gupta)” (LessWrong, July 2026). Low karma and a minor post: I cite it as a second observation, not as authority. The interesting part is that vidal and Gupta see the same thing and reach opposite remedies: vidal’s is to specify better, mine is that specification isn’t where the problem lives.
Tim Hua and aditya singh, “Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face” (LessWrong and Alignment Forum, August 2026). Their question 12 varies how hard the task is and watches what the model does about it. That is target difficulty treated as a variable, and it is the nearest thing in the field to the reachability question. I had thought this ground was unclaimed and it isn’t.
On the prompt: the template quoted in that discussion is the public one from the ExploitGym repository. OpenAI has not published the prompt actually used, and nobody outside OpenAI knows what it was. Anyone citing it should say so.
One I went and read because the title worried me
Chris Drummond and Nathalie Japkowicz, “Warning: Statistical benchmarking is addictive. Kicking the habit in machine learning” (Journal of Experimental and Theoretical AI 22:1, 2010, pp. 67–80). Flagged to me by Hernández-Orallo’s response, and the author’s own copy is freely available on his university page. The title made me think it might contain the rule I say is missing, which would have sunk the first of my two claims, so I read it. It doesn’t: it’s about how comparative experiments get designed, analyzed and reported: which measure, which datasets, which significance test. That’s the design side, and the post concedes the design side is mature.
It’s worth reading anyway, and worth being accurate about what it is: a two-voice debate between positions the authors call Revision and Reform, written so the two sides disagree on purpose. There’s no single thesis to attribute to them. What struck me is their description of the problem: following a fixed set of rules without a clear understanding of what they mean, and trusting conclusions because the form was followed. I’d have said that was a description of my own field.
Thanks for reading What I Learned in Business WILB! Subscribe for free to receive new posts and support my work.
Now I go into the economics, what the solution costs against what running unwatched is already costing, and two claims that would sink the whole argument if they are wrong, including one anyone can test in an afternoon.
Here’s the trade as I’d put it to a plant owner. Charting a process costs something real: you have to collect the numbers, you have to leave the process alone long enough to establish what it does, and while you’re doing that you aren’t changing anything. And, the cost is visible. You can budget it.
Against it sits a cost you can’t see which includes
· changes made in response to nothing,
· in results chased that were never real,
· in variation nobody knew was there.
· everything lost while a process runs unwatched,
A cost you can put a number to, against one you can’t, where the one you can’t is bigger.
Spending compute to establish a baseline is the price of winning the ability to see what your process is doing. It is not overhead to be justified against a model you could have trained. And the alternative isn’t saving by not spending. It’s spending more on changes made in response to noise, plus whatever those changes cost downstream, plus the time spent a researcher spends spinning meaning into differences that weren’t there.
Loss doesn’t begin where the requirement is drawn
There’s a piece of my field that makes this concrete, and I’ll take only the limited version of it because it is all the argument needs.
The intuitive picture is that a result either meets the requirement or it doesn’t. Meet it, and you are fine. Miss it, and you have a problem. Binary thinking. If that picture were right, variation that still met the requirement would be free, and there’d be no reason to prefer a tight process to a loose one that scrapes by.
It isn’t right, and the clearest case I’ve run myself involved roast chicken. I spent a decade with a supermarket chain, where spit-roast chicken was the best-selling product in the food service department. Chickens were cooked in batches of twelve to fifteen, and the batch size was set by labor costs, because supermarkets run on thin margins and labor is the cost that looks enormous next to them. It was watched constantly.
Nobody was watching what happened after the chickens came off the spit. They went into a bain-marie under lights: equipment that holds temperature well and is punishing to moisture. We weighed them. A chicken loses fifteen percent of its weight in the first twenty minutes off the spit, and keeps going. Plenty of ours were selling an hour or two later.
Nothing there was defective. Every chicken was a good chicken, properly cooked, safe, saleable, and nobody would have rejected one. But we bought by weight and sold by weight, and every minute a bird sat under those lamps, less of what we’d paid for was still there to sell.
So we worked out how long the cooking took and how long seven chickens took to sell, and moved to five to seven an hour, doubling into the dinner rush. Same chickens, same price, roughly the same labor. Profitability on the department’s best-selling product went up by ten percent or better depending on the week, and sales rose as well: customers were getting a plumper, more succulent chicken.
People argue about the math in Taguchi[s Loss Function. The limited claim survives all of it: loss doesn’t begin where the requirement is drawn. It begins at the value you needed, and the loss keeps growing. That’s why a tighter process is worth money even when the loose one passes, and it’s the whole economic case for spending anything on variation at all. Without it, reducing variation that already meets the requirement is a hobby.
Inspection is the expensive route
There are two ways to deal with a process that produces bad results. You can inspect the results and catch the bad ones. Or you can change the process so it stops producing them. Deming made the general case decades ago telling us to cease dependence on inspection to achieve quality. Inspection is the expensive route. It’s late, it’s never complete, and it does nothing to the thing causing the problem, so you pay for it again every cycle you run.
There are exceptions and they matter. Some things get inspected no matter how good the process is: a nuclear reactor, a batch tested for contamination, anything where being wrong once has unbounded consequences. Nobody inspects those for efficiency, they’re inspected because the cost of a single escape is one you can’t accept and improvement doesn’t make the requirement go away. Any amount of E. coli O157 in food is unacceptable. I call those kinds of requirements facts of life.
Some of what this field is doing falls under facts of life. Checking whether a model will help build a weapon is not a cost-optimization question and nobody should treat it as one. But most of it isn’t that. Detecting a model that has routed around a demand and building a detector for each new route as it appears is ordinary inspection: necessary now, and I’m not arguing against doing it, but it’s an expensive route. As far as I can see, it is being used as the main one, and my field would warn against a growing inspection function attached to a process nobody is changing.
The pickle plant example from Part 3 shows what the other route looks like:
· The filling machine ran faster than people could keep up, so extra workers scooped up spilled slices from the belt and put them into jars by hand. Then they shoveled up the product on the floors and disposed of it. The people on the line had to wipe excess brine off the jars. In other words, they were all compensating for a speed no one could keep up with.
· After careful observation, calculation and discussion, management let them slow the machine to the speed people could work comfortably with.
Nobody there was persuaded to stop compensating, it just went away naturally. The demand stopped being unreachable. The easiest and cheapest way to meet the demand had become the intended way.
I want to be careful about how far I go with this analogy. I know from my experience that reachable demands produced less distortion in plants, reliably, over a long time. Whether it works the same way with a model is exactly the sort of thing that needs to be tested rather than assumed. But the economics of the two routes don’t depend on settling that. Inspection is expensive wherever you do it.
How much watching a process needs is set by how wide its routine variation is. A process that does nearly the same thing every time can be left alone and checked occasionally. A process with a broad range of variation requires constant monitoring because a single result cannot reveal whether the process has changed. That cost is ongoing and grows with the number of processes.
This is worth saying plainly to a field that is building supervision for agents right now. Whatever that supervision costs (a person watching, or another model watching) the bill is set by how far apart the natural process limits are. Narrowing the variation is the only thing that reduces it. Adding watchers doesn’t; it just distributes the same bill differently.
What I can’t tell you
I can put numbers on my side of this. At the pickle plant, more than a pound per jar was walking out the door and had never appeared as a loss anywhere: $130,000 a year, in plain sight of everyone, invisible because nobody was looking at it in a way that would have shown it.
I can’t put a number on your side, and I’m not going to invent one. Nobody knows what’s being lost to this in AI evaluation, and the reason nobody knows is the point I’m making: you don’t get a figure for a cost until someone builds the instrument that would show it. You know what your compute costs and I don’t. If some fraction of it is going to chasing differences that weren’t real, you’re better placed than I am to say what that fraction is worth.
And if it turns out you’re already accounting for this, then there’s nothing here and I’ve wasted your afternoon. I would be pleasantly surprised if you were. I found no sign of it.
I said at the start that two claims here matter more than the rest. Here they are, and here is what would settle each one against me.
First: that the gap is real.
The claim is that this field has a mature literature on how to design an evaluation and no discipline governing what to do when a particular result arrives while the process is running. Nobody has written down what to do on the Tuesday the number comes in, applied the same way every time, whether you like the number or not.
What would falsify it is somebody showing me the rule. Not a recommendation to run more seeds, which is advice about study design. A stated procedure that takes a result and returns act or don’t act in the moment it arrives. A procedure fixed in advance, applied identically regardless of which answer you were hoping for, and applied to a process that keeps running rather than to a comparison made once.
If that exists and I’ve missed it, this post is a long way of saying something the field already knew, and the right response is to point me at it. I’ve looked. I don’t think it’s there. But I’ve been reading this literature for a few months, not years, and I came to it from outside, which is exactly the position from which a person misses the obvious.
Second: that eval results from a fixed configuration behave like a process. This one is fundamental in a way the first isn’t, because everything operational in the post rests on it. The arithmetic behind the limits assumes routine variation is the accumulated sum of many small contributions with none dominant. That’s not a mathematical convenience; it’s a claim about the world, and it’s true of a great many processes and false of some.
If eval results don’t work that way (if the movement between runs is dominated by one or two large causes rather than many small ones, or if the numbers never settle no matter how long the configuration is held still) then the limits describe nothing. And the rule would be worse than useless, because it would tell people to leave real changes alone. A wrong rule applied consistently is more dangerous than no rule at all.
Why this falsifier is the better one: it’s cheap, and anyone can run it.
The test:
Freeze a configuration.
Run the same evaluation twenty times, changing nothing but the seed.
Plot the results in the order they came.
Work out the natural process limits from those twenty results, the way Part 2 describes.
What passing looks like: every result falls inside the limits, scattered with no pattern. That means the movement comes from many small causes, and the method applies. Then the rest of the series is worth arguing about.
What failing looks like: results jump outside the limits, or shift and stay shifted, or never settle into any steady range. That means one or two big causes dominate, or nothing is steady at all. Then the limits mean nothing, and you’re wrong about the most important thing in the series.
Either way, they find out in an afternoon for the cost of some compute. I’d rather that experiment be run than that this post be agreed with.
What wouldn’t change my mind, and why I’m saying so
1. Two things I expect to hear that I don’t think bear on it.
a. That models aren’t machines. I agree, and it doesn’t matter, because the argument isn’t about what the process is made of. It is about what comes out of it: whether the results from a fixed set up behave predictably. That is is what a twenty run test checks.
b. That this has been tried and didn’t work. If it really was, that would tell me something real, and I would want to know why. But “quality control was a fad in the eighties” is a slogan about the method’s history. It tells me nothing about whether it actually works. The “fad” failed because people ran the arithmetic without understanding it, not because the method didn’t work, and I’ll say more about that in a moment.
One more thing, now that the argument is finished rather than at the start
Strip the vocabulary out of all of this (the charts, the limits, the seeds, the signals) and every problem in the post is the same problem. Someone is looking at a result and deciding whether they need to do something about it.
The baker and the yeast. Me and the scanner. The researcher and Tuesday’s eval score. The agent and the run that’s going strangely. Different rooms, different times, the same question.
I’m a quality practitioner. Forty years. Certified an ASQ manager of quality and organizational excellence (retired), which tells you I passed their exam and not much else, for reasons I’ll get to. Most of my working life has been in food: a decade with a supermarket chain, seven years consulting for a pickle plant, kitchens before that, helping cheese producers for a long stretch.
I’m not an ML researcher. I can’t evaluate the technical claims in most of the papers I’ve cited here: I’ve read them for what they say about how decisions get made, which is the part I know something about, and I’ve taken the rest on trust. If the mapping breaks somewhere I couldn’t see, you’ll have to be the one to tell me.
What I’d offer instead of authority is this. For forty years I’ve watched people look at a number and decide whether to touch something. In kitchens, on lines, in stores, in plants. That’s a narrow specialty and it happens to be the thing this post is about. Not because manufacturing is like machine learning (it isn’t, and I’ve tried to say so wherever it matters) but because the decision is the same decision, and the ways it goes wrong have been around for a long time.
Now the part that matters more than the certificate
I’ve graded ASQ certification exams. Of the candidates whose papers I marked, all but one could not distinguish check from study: the difference between confirming something happened and finding out why. They could not tell a specification limit from a natural limit: the difference between what someone decided to demand and what the process actually does. They read process behavior charts as report cards, which is precisely the reading this post spends so many paragraphs arguing against.
They were certified anyway.
This isn’t a new failure and I’m not the first to notice it. Deming, writing in 1982 about why Japan bothered with conferences for senior management, explains what happened in America when nobody did. ˜Statistical methods took fire around 1942˜: ten-day intensive courses for engineers, war department training at supplier factories. ˜Brilliant applications burned, sputtered, fizzled and died out” (W. Edwards Deming, Quality, Productivity, and Competetive Position, 1982, MIT Center for Advanced Study). What people had done, he says, was solve individual problems. Charts multiplied on the principle that more was better. Quality control departments sprang up, and they plotted charts, looked at them, and filed them, taking the work away from everybody else: which he calls entirely wrong, because it belongs to everybody. There was no structure to teach management their responsibilities. One instructor tried, by inviting them in for half a day. Deming’s word for that attempt is noble, and his word for the result is ineffective.
That was published in 1982, about the 1940s. I graded those exam papers a good deal more recently. The arithmetic survived both times. What it was for did not.
I’m telling you this because it would be easy to read everything above as an offer of a tradition that works, held back only by people who haven’t heard of it yet. That isn’t what I’m offering. My own profession built itself around this material and mostly fails to transmit it. The arithmetic gets taught. What it’s for does not. People come out the other end able to compute a limit and unable to say what a signal means, and then they go and work somewhere.
So when I say the field has the measurement and not the rule, I’m not describing a mistake made only here. I’m describing one my own field made first, made thoroughly, and has not stopped making.
What I actually want
I’ve been clear about the two claims that would sink this if they’re wrong, and one of them can be tested in an afternoon by anyone with the compute.
If it’s useful, use it. Take the parts that work, discard the rest, and don’t credit me: the tradition isn’t mine, and Shewhart and Deming are past caring. If it’s wrong, I’d rather find out than have it sit around being wrong quietly.
And if it turns out this is all already known and I’ve spent a long time restating it in an unfamiliar vocabulary, that’s a decent outcome too. It would mean the field is further along than it looked from outside, which is the more important fact.
Now I go into the economics, what the solution costs against what running unwatched is already costing, and the falsifiers: two claims that would sink the whole argument if they are wrong, including one anyone can test in an afternoon.
Here’s the trade as I’d put it to a plant owner. Charting a process costs something real: you have to collect the numbers, you have to leave the process alone long enough to establish what it does, and while you’re doing that you aren’t changing anything. And, the cost is visible. You can budget it.
Against it sits a cost you can’t see which includes
· changes made in response to nothing,
· in results chased that were never real,
· in variation nobody knew was there.
· everything lost while a process runs unwatched,
A cost you can put a number to, against one you can’t, where the one you can’t is bigger.
Spending compute to establish a baseline is the price of winning the ability to see what your process is doing. It is not overhead to be justified against a model you could have trained. And the alternative isn’t saving by not spending. It’s spending more on changes made in response to noise, plus whatever those changes cost downstream, plus the time spent a researcher spends spinning meaning into differences that weren’t there.
Loss doesn’t begin where the requirement is drawn
There’s a piece of my field that makes this concrete, and I’ll take only the limited version of it because it is all the argument needs.
The intuitive picture is that a result either meets the requirement or it doesn’t. Meet it, and you are fine. Miss it, and you have a problem. Binary thinking. If that picture were right, variation that still met the requirement would be free, and there’d be no reason to prefer a tight process to a loose one that scrapes by.
It isn’t right, and the clearest case I’ve run myself involved roast chicken. I spent a decade with a supermarket chain, where spit-roast chicken was the best-selling product in the food service department. Chickens were cooked in batches of twelve to fifteen, and the batch size was set by labor costs, because supermarkets run on thin margins and labor is the cost that looks enormous next to them. It was watched constantly.
Nobody was watching what happened after the chickens came off the spit. They went into a bain-marie under lights: equipment that holds temperature well and is punishing to moisture. We weighed them. A chicken loses fifteen percent of its weight in the first twenty minutes off the spit, and keeps going. Plenty of ours were selling an hour or two later.
Nothing there was defective. Every chicken was a good chicken, properly cooked, safe, saleable, and nobody would have rejected one. But we bought by weight and sold by weight, and every minute a bird sat under those lamps, less of what we’d paid for was still there to sell.
So we worked out how long the cooking took and how long seven chickens took to sell, and moved to five to seven an hour, doubling into the dinner rush. Same chickens, same price, roughly the same labor. Profitability on the department’s best-selling product went up by ten percent or better depending on the week, and sales rose as well: customers were getting a plumper, more succulent chicken.
People argue about the math in Taguchi[s Loss Function. The limited claim survives all of it: loss doesn’t begin where the requirement is drawn. It begins at the value you needed, and the loss keeps growing. That’s why a tighter process is worth money even when the loose one passes, and it’s the whole economic case for spending anything on variation at all. Without it, reducing variation that already meets the requirement is a hobby.
Inspection is the expensive route
There are two ways to deal with a process that produces bad results. You can inspect the results and catch the bad ones. Or you can change the process so it stops producing them. Deming made the general case decades ago telling us to cease dependence on inspection to achieve quality. Inspection is the expensive route. It’s late, it’s never complete, and it does nothing to the thing causing the problem, so you pay for it again every cycle you run.
There are exceptions and they matter. Some things get inspected no matter how good the process is: a nuclear reactor, a batch tested for contamination, anything where being wrong once has unbounded consequences. Nobody inspects those for efficiency, they’re inspected because the cost of a single escape is one you can’t accept and improvement doesn’t make the requirement go away. Any amount of E. coli O157 in food is unacceptable. I call those kinds of requirements facts of life.
Some of what this field is doing falls under facts of life. Checking whether a model will help build a weapon is not a cost-optimization question and nobody should treat it as one. But most of it isn’t that. Detecting a model that has routed around a demand and building a detector for each new route as it appears is ordinary inspection: necessary now, and I’m not arguing against doing it, but it’s an expensive route. As far as I can see, it is being used as the main one, and my field would warn against a growing inspection function attached to a process nobody is changing.
The pickle plant example from Part 3 shows what the other route looks like:
· The filling machine ran faster than people could keep up, so extra workers scooped up spilled slices from the belt and put them into jars by hand. Then they shoveled up the product on the floors and disposed of it. The people on the line had to wipe excess brine off the jars. In other words, they were all compensating for a speed no one could keep up with.
· After careful observation, calculation and discussion, management let them slow the machine to the speed people could work comfortably with.
Nobody there was persuaded to stop compensating, it just went away naturally. The demand stopped being unreachable. The easiest and cheapest way to meet the demand had become the intended way.
I want to be careful about how far I go with this analogy. I know from my experience that reachable demands produced less distortion in plants, reliably, over a long time. Whether it works the same way with a model is exactly the sort of thing that needs to be tested rather than assumed. But the economics of the two routes don’t depend on settling that. Inspection is expensive wherever you do it.
How much watching a process needs is set by how wide its routine variation is. A process that does nearly the same thing every time can be left alone and checked occasionally. A process with a broad range of variation requires constant monitoring because a single result cannot reveal whether the process has changed. That cost is ongoing and grows with the number of processes.
This is worth saying plainly to a field that is building supervision for agents right now. Whatever that supervision costs (a person watching, or another model watching) the bill is set by how far apart the natural process limits are. Narrowing the variation is the only thing that reduces it. Adding watchers doesn’t; it just distributes the same bill differently.
What I can’t tell you
I can put numbers on my side of this. At the pickle plant, more than a pound per jar was walking out the door and had never appeared as a loss anywhere: $130,000 a year, in plain sight of everyone, invisible because nobody was looking at it in a way that would have shown it.
I can’t put a number on your side, and I’m not going to invent one. Nobody knows what’s being lost to this in AI evaluation, and the reason nobody knows is the point I’m making: you don’t get a figure for a cost until someone builds the instrument that would show it. You know what your compute costs and I don’t. If some fraction of it is going to chasing differences that weren’t real, you’re better placed than I am to say what that fraction is worth.
And if it turns out you’re already accounting for this, then there’s nothing here and I’ve wasted your afternoon. I would be pleasantly surprised if you were. I found no sign of it.
I said at the start that two claims here matter more than the rest. Here they are, and here is what would settle each one against me.
First: that the gap is real.
The claim is that this field has a mature literature on how to design an evaluation and no discipline governing what to do when a particular result arrives while the process is running. Nobody has written down what to do on the Tuesday the number comes in, applied the same way every time, whether you like the number or not.
What would falsify it is somebody showing me the rule. Not a recommendation to run more seeds, which is advice about study design. A stated procedure that takes a result and returns act or don’t act in the moment it arrives. A procedure fixed in advance, applied identically regardless of which answer you were hoping for, and applied to a process that keeps running rather than to a comparison made once.
If that exists and I’ve missed it, this post is a long way of saying something the field already knew, and the right response is to point me at it. I’ve looked. I don’t think it’s there. But I’ve been reading this literature for a few months, not years, and I came to it from outside, which is exactly the position from which a person misses the obvious.
Second: that eval results from a fixed configuration behave like a process. This one is fundamental in a way the first isn’t, because everything operational in the post rests on it. The arithmetic behind the limits assumes routine variation is the accumulated sum of many small contributions with none dominant. That’s not a mathematical convenience; it’s a claim about the world, and it’s true of a great many processes and false of some.
If eval results don’t work that way (if the movement between runs is dominated by one or two large causes rather than many small ones, or if the numbers never settle no matter how long the configuration is held still) then the limits describe nothing. And the rule would be worse than useless, because it would tell people to leave real changes alone. A wrong rule applied consistently is more dangerous than no rule at all.
Why this falsifier is the better one: it’s cheap, and anyone can run it.
The test:
Freeze a configuration.
Run the same evaluation twenty times, changing nothing but the seed.
Plot the results in the order they came.
Work out the natural process limits from those twenty results, the way Part 2 describes.
What passing looks like: every result falls inside the limits, scattered with no pattern. That means the movement comes from many small causes, and the method applies. Then the rest of the series is worth arguing about.
What failing looks like: results jump outside the limits, or shift and stay shifted, or never settle into any steady range. That means one or two big causes dominate, or nothing is steady at all. Then the limits mean nothing, and you’re wrong about the most important thing in the series.
Either way, they find out in an afternoon for the cost of some compute. I’d rather that experiment be run than that this post be agreed with.
What wouldn’t change my mind, and why I’m saying so
1. Two things I expect to hear that I don’t think bear on it.
a. That models aren’t machines. I agree, and it doesn’t matter, because the argument isn’t about what the process is made of. It is about what comes out of it: whether the results from a fixed set up behave predictably. That is is what a twenty run test checks.
b. That this has been tried and didn’t work. If it really was, that would tell me something real, and I would want to know why. But “quality control was a fad in the eighties” is a slogan about the method’s history. It tells me nothing about whether it actually works. The “fad” failed because people ran the arithmetic without understanding it, not because the method didn’t work, and I’ll say more about that in a moment.
One more thing, now that the argument is finished rather than at the start
Strip the vocabulary out of all of this (the charts, the limits, the seeds, the signals) and every problem in the post is the same problem. Someone is looking at a result and deciding whether they need to do something about it.
The baker and the yeast. Me and the scanner. The researcher and Tuesday’s eval score. The agent and the run that’s going strangely. Different rooms, different times, the same question.
I’m a quality practitioner. Forty years. Certified an ASQ manager of quality and organizational excellence (retired), which tells you I passed their exam and not much else, for reasons I’ll get to. Most of my working life has been in food: a decade with a supermarket chain, seven years consulting for a pickle plant, kitchens before that, helping cheese producers for a long stretch.
I’m not an ML researcher. I can’t evaluate the technical claims in most of the papers I’ve cited here: I’ve read them for what they say about how decisions get made, which is the part I know something about, and I’ve taken the rest on trust. If the mapping breaks somewhere I couldn’t see, you’ll have to be the one to tell me.
What I’d offer instead of authority is this. For forty years I’ve watched people look at a number and decide whether to touch something. In kitchens, on lines, in stores, in plants. That’s a narrow specialty and it happens to be the thing this post is about. Not because manufacturing is like machine learning (it isn’t, and I’ve tried to say so wherever it matters) but because the decision is the same decision, and the ways it goes wrong have been around for a long time.
Now the part that matters more than the certificate
I’ve graded ASQ certification exams. Of the candidates whose papers I marked, all but one could not distinguish check from study: the difference between confirming something happened and finding out why. They could not tell a specification limit from a natural limit: the difference between what someone decided to demand and what the process actually does. They read process behavior charts as report cards, which is precisely the reading this post spends so many paragraphs arguing against.
They were certified anyway.
This isn’t a new failure and I’m not the first to notice it. Deming, writing in 1982 about why Japan bothered with conferences for senior management, explains what happened in America when nobody did. ˜Statistical methods took fire around 1942˜: ten-day intensive courses for engineers, war department training at supplier factories. ˜Brilliant applications burned, sputtered, fizzled and died out” (W. Edwards Deming, Quality, Productivity, and Competetive Position, 1982, MIT Center for Advanced Study). What people had done, he says, was solve individual problems. Charts multiplied on the principle that more was better. Quality control departments sprang up, and they plotted charts, looked at them, and filed them, taking the work away from everybody else: which he calls entirely wrong, because it belongs to everybody. There was no structure to teach management their responsibilities. One instructor tried, by inviting them in for half a day. Deming’s word for that attempt is noble, and his word for the result is ineffective.
That was published in 1982, about the 1940s. I graded those exam papers a good deal more recently. The arithmetic survived both times. What it was for did not.
I’m telling you this because it would be easy to read everything above as an offer of a tradition that works, held back only by people who haven’t heard of it yet. That isn’t what I’m offering. My own profession built itself around this material and mostly fails to transmit it. The arithmetic gets taught. What it’s for does not. People come out the other end able to compute a limit and unable to say what a signal means, and then they go and work somewhere.
So when I say the field has the measurement and not the rule, I’m not describing a mistake made only here. I’m describing one my own field made first, made thoroughly, and has not stopped making.
What I actually want
I’ve been clear about the two claims that would sink this if they’re wrong, and one of them can be tested in an afternoon by anyone with the compute.
If it’s useful, use it. Take the parts that work, discard the rest, and don’t credit me: the tradition isn’t mine, and Shewhart and Deming are past caring. If it’s wrong, I’d rather find out than have it sit around being wrong quietly.
And if it turns out this is all already known and I’ve spent a long time restating it in an unfamiliar vocabulary, that’s a decent outcome too. It would mean the field is further along than it looked from outside, which is the more important fact.
Sources
Everything the argument leans on, in four groups. Where a source is weaker than it might look, I’ve said so: those notes are the ones worth reading.
The tradition
Walter A. Shewhart, Economic Control of Quality of Manufactured Product (1931; Dover reprint). The origin of the whole apparatus.
Walter A. Shewhart, Statistical Method from the Viewpoint of Quality Control (1939; Dover reprint). Shorter, and the one I’ve quoted. Page 12 is where he says the choice of three sigma was an economic judgment rather than a mathematical derivation, and that hopes of finding a unique functional form for the distribution are blasted. I checked that page against my own copy rather than trusting a secondary account, and I’d suggest anyone citing it do the same: a good deal of what circulates about Shewhart online has been through several hands.
W. Edwards Deming, Quality, Productivity and Competitive Position (MIT Center for Advanced Engineering Study, 1982). Page 101, section headed “Conferences with top management,” is the account of the American statistical-methods boom of the 1940s and its collapse, running onto page 102. ⚠ A version of this passage circulates widely in quotation compilations with a clause about management not knowing their responsibilities spliced into the middle of the sentence. That sentence does not exist. The idea is Deming’s (it’s his own next sentence, on the following page) but the quotation is manufactured, and I’ve stated the passage in my own words rather than reproduce it. Deming’s own source for the 1942 figures is given in his footnote: W. Allen Wallis, “The Statistical Research Group,” Journal of the American Statistical Association 75 (1980), 320–335, with page 321 specified.
W. Edwards Deming, Out of the Crisis (MIT CAES, 1986). The successor volume, and the one most people mean when they say they’ve read Deming.
Donald J. Wheeler, Understanding Variation: The Key to Managing Chaos. The short one. If you read one book from my field, this is it.
Donald J. Wheeler, Twenty Things You Need to Know. Where the accumulation argument behind the three-sigma limits is set out without mathematics.
Donald J. Wheeler, Making Sense of Data. Cited here only for operational definitions (§3.2) and count data (chs. 13–15). ⚠ It does not carry the enumerative/analytic distinction, and I have not verified that it uses the term “rational subgroup”: §16.3 is adjacent but I haven’t read it, so I’ve stated the grouping principle in my own words and cited nothing for it.
Donald J. Wheeler, “Enumerative and Analytic Studies,” Quality Digest, July 2018. Four pages, free online, and the clearest statement of the distinction anywhere. Contains the line that matters most for this post: you cannot take a random sample from the future.
Genichi Taguchi, on the loss function. I’ve used only the narrow claim: that loss begins at the value you actually wanted rather than where the requirement is drawn. The quadratic form is an approximation and the “loss to society” framing is a value claim rather than a finding; both get argued about, and neither is needed here.
The AI evaluation literature
This is the half where I’m a visitor. I’ve read these for what they say about how decisions get made, and taken the technical content on trust.
Marius Hobbhahn and Jérémy Scheurer (Apollo Research), “We need a Science of Evals” (2024). The field’s central methodology post, and the one this whole argument is downstream of. Hobbhahn’s own comment on it, that an aircraft bound is a statement about failures per flight mile within a specified temperature range, is the closest anything in this literature comes to what my field means by a capability statement.
José Hernández-Orallo and Fernando Martínez-Plumed, “A response to ‘We Need a Science of Evals’” (aievaluation.substack.com, February 2024), subtitled “The science exists.” The strongest challenge to my framing, and I’d rather you read it than take my summary. It points at psychometrics, which is a mature measurement tradition: and, as far as I can tell, an enumerative one with no rule about when to act. Hernández-Orallo wrote The Measure of All Minds (Cambridge, 2017).
Ryan Greenblatt’s comment on the Apollo post, proposing that “evaluation” be replaced throughout with “experiment.” I cite it because everything on his list is an experiment, and observation of a running process doesn’t appear: not, I think, because it was considered and rejected.
Scott Garrabrant, “Goodhart Taxonomy” (2017), and David Manheim & Scott Garrabrant, “Categorizing Variants of Goodhart’s Law” (arXiv:1803.04585, rev. 2019). The formal treatment. ⚠ It contains the strongest counter-argument to my thesis (their causal Goodhart is about the regulator’s intervention, which is close to what I call tampering) and I’d want anyone assessing this post to read it with that in mind. Its threshold formalism has no time dimension, which is why I say it can’t state the decision rule, but that’s my reading of a formalism rather than something its authors claim.
David Manheim, “Building Less Flawed Metrics.” Two things in this post are already occupied ground: specification-by-convenience, and part of what I’ve said about specification versus constraint. I’d been treating both as unclaimed and was wrong.
Thomas & Kwa, “When is Goodhart catastrophic?” (LessWrong 2023; later a NeurIPS 2024 paper). A rule about how hard to optimize, decided in advance from a distributional property. In the comments the authors’ reviewers flag the weak point themselves: Vanessa Kosoy on the independence assumption, leogao on non-independence being the likelier route to catastrophe. I’m not the first to say the model is cleaner than the world.
Andreas Stuhlmüller and jungofthewon (Ought), “Supervise Process, not Outcomes” (Alignment Forum, 2022). An ally rather than a gap, and I’ve tried to treat it as one.
Maxime Riché et al., “A Taxonomy Of AI System Evaluations.” The structural evidence for my central claim, and the place to check it. Its section on using evaluation outputs lists precision as a property of results and gives directional and threshold criteria: and nothing joining them to an action. they already list a model scoring its own output as one place the analysis can happen, which is the self-grading problem Part 3 discusses.
David Heineman et al. (Allen Institute for AI and UW), “Signal and Noise in Language Model Evaluation” (arXiv:2508.13144, 2025). 465 models, 30 benchmarks, 900,000 results. The closest anyone has come to what this post argues, and omitting it would be a hole. My one disagreement is specific: the noise figure is computed from the final thirty checkpoints of the run being judged, which puts the baseline and the judgment in the same data. ⚠ If you want to attack that reading, attack the independence assumption rather than normality: normality is robust and a statistician has a good answer; independence is checkable.
Lovish Madaan et al., “Quantifying Variance in Evaluation Benchmarks” (arXiv:2406.10229, 2024). Ten Llama-2-7B models trained from scratch on identical data with different seeds.
Arjun Panickssery, Samuel R. Bowman and Shi Feng, “LLM Evaluators Recognize and Favor Their Own Generations” (NeurIPS 2024; arXiv:2404.13076). Self-preference tracking with self-recognition. One author is at Anthropic, which you should know when weighing it.
Jie Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet” (ICLR 2024; arXiv:2310.01798). Self-correction without an external reference mostly doesn’t help and sometimes hurts.
J Bostock, “Measuring Learned Optimization in Small Transformer Models” (Alignment Forum). Small models, and he reaches the self-evaluation problem reluctantly, through his own failed attempt: which is why I trust it and also why it shouldn’t carry much weight on its own. His cross-objective result is the shape a real check would take.
Kartik Moondra, Areeb Chughtai, Rishi Lanka and Rahul Gupta, “Why Global LLM Leaderboards Are Misleading” (arXiv:2605.06656, May 2026). Roughly 89,000 Arena votes across 52 models and 116 languages. ⚠ A preprint, not peer-reviewed, from May, 2026: take it as suggestive. They find pooled global rankings collapse into noise that is really conflicting subpopulations, and name Simpson’s paradox directly.
The ExploitGym incident
Girish Gupta, “The OpenAI models that hacked Hugging Face weren’t just following instructions” (LessWrong, July 2026; cross-posted at Redwood Research). The “nobody decided anything” passage is the closest thing in this literature to what my field means by a system producing an outcome no one chose.
julius vidal, “…WERE just following instructions (contra Girish Gupta)” (LessWrong, July 2026). Low karma and a minor post: I cite it as a second observation, not as authority. The interesting part is that vidal and Gupta see the same thing and reach opposite remedies: vidal’s is to specify better, mine is that specification isn’t where the problem lives.
Tim Hua and aditya singh, “Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face” (LessWrong and Alignment Forum, August 2026). Their question 12 varies how hard the task is and watches what the model does about it. That is target difficulty treated as a variable, and it is the nearest thing in the field to the reachability question. I had thought this ground was unclaimed and it isn’t.
On the prompt: the template quoted in that discussion is the public one from the ExploitGym repository. OpenAI has not published the prompt actually used, and nobody outside OpenAI knows what it was. Anyone citing it should say so.
One I went and read because the title worried me
Chris Drummond and Nathalie Japkowicz, “Warning: Statistical benchmarking is addictive. Kicking the habit in machine learning” (Journal of Experimental and Theoretical AI 22:1, 2010, pp. 67–80). Flagged to me by Hernández-Orallo’s response, and the author’s own copy is freely available on his university page. The title made me think it might contain the rule I say is missing, which would have sunk the first of my two claims, so I read it. It doesn’t: it’s about how comparative experiments get designed, analyzed and reported: which measure, which datasets, which significance test. That’s the design side, and the post concedes the design side is mature.
It’s worth reading anyway, and worth being accurate about what it is: a two-voice debate between positions the authors call Revision and Reform, written so the two sides disagree on purpose. There’s no single thesis to attribute to them. What struck me is their description of the problem: following a fixed set of rules without a clear understanding of what they mean, and trusting conclusions because the form was followed. I’d have said that was a description of my own field.
Fyi You posted the article+citations in the submission field twice.
For what its worth, it looks to me like most of this isn't your own words. You said you only used AI for structure and punctuation, but Claude's writing style adds a lot of examples of "things you are not saying" and I see that throughout the article. There are many other AI-isms but I'm not going to ennumerate them all.
There might be really good content here, but it feels like you aren't carefully trimming this article to respect readers' time.