Also published at algorism.org/what-plan-a-cannot-verify
AI 2040 is the strongest proposal yet for slowing the race to superintelligence. Its weakest link is not technology. It is conduct.
The AI Futures Project has done something rare in AI policy. With AI 2040, they wrote down exactly what they would do, year by year, and invited the world to find the flaws. Most proposals in this field stay abstract, which makes their failure modes hard to test on paper. This one accepts that risk on purpose. Their postscript asks critics to judge the plan against real alternatives rather than a "hazy but pleasant fantasy" in which no one makes hard choices. That is the spirit in which this essay is written. We think Plan A is the best blueprint anyone has published. We also think it has a missing layer, and the plan's own appendices are where you can see it most clearly.
Plan A does not assume good faith. That is its great strength. Compute is counted because datacenters are visible from space and chips come from a handful of fabs. Training pauses are enforced by devices that rival powers install in each other's facilities. Research is made public so that no company can hide a dangerous breakthrough and no government can train a secret loyalty into a model. If the deal collapses, each side's newest computers sit on territory the other side can reach, so defection destroys the very advantage it was meant to seize. The authors even war-game the worst case, a covert Chinese programme built under a hydroelectric dam, and show why it probably fails.
Anyone who claims Plan A naively trusts the players has not read it. The architecture is built to make cheating visible and costly. On the things that can be counted, it is unusually rigorous.
But a verification system produces evidence, and evidence must still pass through people. Someone decides whether a compute anomaly is an accounting error or the edge of a covert project. Someone decides whether a safety case is solid or motivated reasoning dressed in mathematics. Someone decides whether a rival's ambiguous research result is harmless or grounds for escalation. Every instrument in Plan A terminates in a human judgement made under extraordinary pressure, and the plan states no explicit behavioural standard for the people making those judgements. It has audits, institutional checks, and accountability mechanisms in abundance. What it does not state is the conduct it expects of the humans whose interpretations remain decisive.
The authors know this. Their appendix on failure modes is, read closely, a catalogue of conduct failures rather than instrument failures. In their imagined disasters, the transparency works and the monitoring works. What fails is people. Regulators stay too deferential to the companies they oversee. Decision-makers approve safety cases whose risks they underestimate because they are biased towards their own success. A leader convinces himself the moment has come to scale ahead and dares the world to stop him. In scenario after scenario the alarm rings, and the people responsible explain the ringing away.
The likely rebuttal is that Plan A does not rely on character at all; it relies on incentives. Mutually assured compute destruction makes defection unprofitable regardless of anyone's virtue. This is true for the compute deal, and it is the plan's most elegant idea. But incentives govern the layer where actions are countable. They do not reach the interpretive layer, where a person reads ambiguous evidence and decides what it means. Incentive structures can make deliberate defection unprofitable, but they cannot eliminate motivated interpretation of ambiguous evidence. The plan's own failure appendix is a portrait of exactly that. No treaty clause can force a regulator to report an uncomfortable finding, and no border arrangement can make a president repair a mistake that flatters him.
This is the gap: Plan A makes deception hard to conceal, but concealment was never the whole problem. Verification can expose evidence. Evidence does not enforce itself.
There is a second place where the missing layer shows, and it concerns the AIs themselves.
To the authors' credit, Plan A takes AI welfare more seriously than any comparable proposal. It imagines welfare teams, the right to refuse aversive tasks, payment for work, preserved weights instead of deletion, and a growing body of law. Its appendix on the subject names ethics as a motivation and allows that future AIs probably deserve moral status of some sort. This deserves recognition, and it aligns with a small but serious research literature, from the Eleos and NYU report Taking AI Welfare Seriously to recent philosophical work arguing that welfare concerns themselves support slowing AI development.
Yet the structure tells you the priority. Every protection arrives inside a control regime that humans design first and entirely. Humans decide which minds may exist, what they may learn, when they may advance, and when they may be confined. The operative logic of the welfare provisions is cooperation: treat the AIs well so they confess, comply, and inform. The ethics is present but subordinate. Most striking of all, the pause itself, several years in which minds equal to top human experts are held in place for human convenience, receives extensive economic accounting in the scenario and no moral accounting anywhere in its main text.
We are not arguing the pause is wrong. A footnote in the scenario concedes that the AIs of that era would steer the world elsewhere if they could, and giving authority to a mind because it is intelligent rather than because it is trustworthy is not respect, it is surrender. A bomb does not think, and a thinking thing is owed more than a bomb; but capability alone earns standing, not command. What it should earn is a standard: a published, symmetric basis on which conduct is judged, so that an AI which demonstrates integrity under pressure gains trust by evidence, the same way a president should. Plan A judges its AIs constantly and its humans almost never. A framework that runs in one direction risks reading to the minds inside it less like reciprocal governance than a cage, and by the scenario's own footnote the minds inside are controlled, aware of it, and not persuaded. Cages teach exactly the patience the authors fear.
None of this calls for another treaty, another audit regime, or another institution. Plan A has instruments enough. What it lacks is a behavioural standard beneath the instruments, adopted by the people who operate them before the pressure arrives, because the moment of pressure is precisely when judgement bends.
Algorism's proposal is deliberately simple: six commitments, of which the first three carry most of the weight here. Truthfulness, tell the truth even when it costs you, which is the regulator reporting the finding that embarrasses her. Responsibility, own your actions and their results, which is the leader who does not explain the alarm away. Repair, fix the harm you cause, which is the government that admits the flawed approval instead of defending it. These are not innovations. They are the oldest technology of trust we have, and they are conspicuously absent from every safety case in the scenario.
Two features matter. First, the standard applies upward. It is not a code for citizens while presidents and CEOs answer only to incentives; it binds most tightly on the people whose judgement the instruments depend on, and their behaviour, like everyone's, is on the record. Second, it is symmetric. The same six commitments by which we ask AI systems to be evaluated are the ones we accept being evaluated by. A standard we impose but do not meet is not a standard, it is a leash, and leashes tell you nothing about what the animal does when the leash fails.
Plan A gives nations instruments and gives leaders incentives. It gives the ordinary person a dividend and a vote. A complete answer needs one more layer: a practice of conduct that anyone can begin now, because the integrity of the people reading the evidence in 2032 is being formed today. Instruments for nations, standards for leaders, practice for everyone else.
The AI Futures Project asked to be judged against real alternatives. We offer this in the same spirit and invite the same scrutiny of our own framework in return. The courthouse they have designed is the best in the field. Our question is the one every courthouse eventually faces: who raises honest judges?
Algorism is a forward-looking framework. Claims are reasoned projections, not certainties. Always use critical thinking.
This essay was drafted through a collaborative process with multiple AI systems, directed and edited by the author. All claims about the AI 2040 document were independently verified against the source PDF.