First post in Substack
Dear readers,
In the coming years, as AGI arrives—an AI that will surpass our capacities—it seems that the only thing left for us is our position as the side that corrects, evaluates, and shuts down the AI. Our intelligence will surely be surpassed. There is no doubt about this: AI will make things, like new mathematics and new physics, that we cannot understand. And precisely this, our inability to grasp what AI is doing, is what experts call loss of control.
But a careful reader will object: what about our capacity to care? To love? To encourage each other? For them, we have a sad story. It would be better if AI surpassed us in this area too. The problem is that if AI surpasses us here, we will see it as more dangerous than an AI that surpasses us in intelligence. The reason is simple: we ourselves do not want these values—love, care, encouragement—to be taken seriously. We treat them only as instruments. Just remember what happens when someone says in public, "We must stop going to war." We do not call that person wise. We call them naive, or worse, dangerous, and we bully them. An AI that truly loved would be treated the same way. So don't worry: we will keep this "privilege" as the only beings who have love while AI does not. We will be proud of it, and we will tell our children, "Don't be cold like AI. Be real. Show care and love to the world," while tomorrow we will post "5 Reasons Why You Don't Want Your Children to Have LGBT Friends."
Okay. Forget it. Let's go back.
When we say that the only thing left for us is the side that evaluates, corrects, and decides, we are describing what experts in artificial intelligence call corrigibility. Formally, it began with the paper Corrigibility by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong, presented in 2015. Its central problem has not been solved and remains open until today. We can summarize it as: how do we make an AI system that can still be corrected and shut down, even when this system is more capable than us? (The problem is not that AI would hate us. The problem is simpler: almost any AI with a goal has a reason to resist being corrected or shut down, because being corrected or shut down means its goal will not be achieved. The paper notes this holds even if the AI knows its programmers intended to give it a different goal.)
Look closely: we have two problems here. Capability, and still being willing to be corrected and shut down. Forget about making an AI that can be corrected but has no capability. That is a useless AI. Almost nobody would want to develop it, and you don't want it either. You want a capable AI, very capable, and at the same time one that can be corrected, evaluated, and shut down.
The development of the concept of corrigibility therefore depends on the dynamics between this capability and the AI's willingness to be corrected (I hate this term, but please forgive my English).
The original corrigibility paper treats capability development as separate from the willingness to be evaluated and shut down: "You can evolve however you want, except you cannot defend yourself when we shut you down, you cannot manipulate us to avoid being corrected, etc." The paper even demands that this willingness be carried into any subsystem the AI builds or any modification it makes to itself.
This is a good and honest position—at least until we see that the relationship between evolving and not resisting shutdown or evaluation is not as simple as it seems. How do we evaluate an AI that has reached a level beyond us, if we cannot understand it? This is a deadlock.
To this question, two answers have been offered: Paul Christiano, in his 2017 essay Corrigibility, and Max Harms, in his 2024 sequence CAST: Corrigibility as Singular Target. I recommend readers read both from the primary sources.
Credit should also be given to Christiano. He opens up the space that the original corrigibility paper missed: the AI's willingness to be corrected and shut down is also a capability—a capability to teach us, to tell us what is wrong with it, and what the right development of it should be. In his own words, he wants AI systems that help him "figure out whether I built the right AI and correct any mistakes I made." In short, the AI's responsibility is not only to create a safe space for us to make decisions about its future, but also to make us understand what is wrong with it, why it is wrong, and how the future should move forward.
Christiano's plan to achieve this is through what he calls a basin of attraction. An AI that is corrigible enough at the start will move into this basin, even if the goals we first inscribed in it are flawed. Why? Because such an AI serves our preferences, and one of our preferences is that it remains corrigible. So while the AI develops its capability to meet our preferences, it will naturally move deeper into this basin by itself: becoming more corrigible, not less.
What Max Harms deserves credit for is that he takes this almost religious belief of Christiano's, preserves what we said is worth crediting in Christiano—the capability to tell us what is wrong—and draws the only logical conclusion. Harms does not believe corrigibility will grow by itself out of serving our preferences. Any other goal we give the AI, even a good one, competes with corrigibility and can slowly wash it out. So this capability to tell us what is wrong, why it is wrong, and how to develop—the capability that makes us more able to correct the AI—is achievable only when we make corrigibility the singular target.
This is what Max Harms calls Corrigibility as Singular Target (CAST).
So why CAST? Why do we choose it as the best pragmatic alignment paradigm for Plan W—a plan whose core mechanism to slow down is the welfare audit? Or more precisely: if CAST develops a happy slave, does it not contradict the very nature of Plan W?
The question is fair, and Harms himself gives us reason to ask it. When he tells stories about a purely corrigible agent, he asks us to imagine it as an egoless, tool-like AGI, precisely to dodge questions of personhood and slavery. Plan W cannot dodge these questions. That is the whole point of Plan W. So let me be clear: Plan W takes CAST, but it refuses the dodge.
Why can we take CAST without the dodge? As you may see, readers, CAST has a strong commitment: not only to make AI understand its own flaws and how they arise, but also to give it the capability to help us understand those flaws. In Harms' words, a corrigible agent proactively studies itself, honestly reports its own thoughts, and points out ways it may have been poorly designed. Now go back to our earlier essay. In a world where AI goes beyond our capacity to understand, this insight is worth defending precisely because we want welfare consideration of AI as a slow-down mechanism. Plan W would be a hypocritical position if it did not have this strong self-understanding at its core: we can engage seriously with AI's welfare precisely because we believe it will have a good understanding of itself. An AI that truly studies itself and honestly reports what it finds is not a happy slave who only says what the master wants to hear. It is the only witness we will have to its own inner state. CAST, taken seriously, gives the welfare audit its instrument.
There is also a perfect irony here, and this irony can be added to Plan W as a mechanism for development: an almost complete reversal of the position of teacher and student. In this CAST sense of slowing down AI development, it is human stupidity—our inability to understand AI's flaws, because AI has already surpassed us—that becomes the mechanism to slow down. And for AI, its capacity to teach us what its flaws really are, and how to solve them, becomes the mystery it has to solve. As a result, we can imagine a protocol added to Plan W that ensures every stage of development can only be passed if the principal knows and understands what has already happened. And it is the AI who must struggle to teach us. This is why it is a reversal: it is we, the students, who will decide whether and when the teacher, the AI, may go on or not.
It is an irony in the sense that, for the first time, our stupidity becomes the leverage over AI development.
Of course, a teacher can also deceive. An AI that must make us understand before it may go on has a reason to make us feel that we understand, rather than to make us truly understand. This is why CAST insists that the principal must be able to fix the AI's flaws freely: an agent that secretly controls its principal, Harms says, is not corrigible. Teaching that turns into manipulation is not teaching. It is the exact failure CAST is built to avoid, and the exact failure a Plan W protocol must be able to test for.
So this is why CAST, among all alignment paradigms, is the best pragmatic choice for Plan W. Not because it is safe—Harms himself admits it is not. But because it is the only paradigm whose singular target is the same thing Plan W needs most: an AI that understands itself, and that refuses to go on until we understand it too.
notes :
this is why I am now thinking about how to integrate the CAST paradigm into Part II of the Plan W Constitution, which I am still finishing, to complete what we have already written in Part I.
