Cross-posted from The Foretellix CTO Blog.
Introduction and epistemic status: This is the first post in a planned series, “Alignment as a verification problem”. I co-originated coverage-driven verification (CDV), which became the standard methodology for chip verification and is heavily used in AV safety. Back in 2015 I wrote that verifying “Friendly AI” would be our biggest verification challenge (and that perhaps CDV can help a bit). A decade of autonomy verification later, this series tries to work out directly what these tools and methodologies can teach us about AI alignment.
I’m fairly confident about how misalignments relate to bugs, and that my field’s tools can help in much of that. I am much less confident about helping with strategic deception (though I am hopeful). Much of this is uncertain – comments and critique are very welcome.
This first post will argue the following:
A clarification before we start: “bug” and “misalignment” overlap, but neither contains the other. Some bugs are plain capability failures, not misalignment at all. And strategic deception misalignments are not bug-like. This post is mostly (but not exclusively) about the large overlap.
I’ll start by reintroducing alignment in a way which enables the kind of reasoning I want to do, using a future medical-AI example. I’ll then construct an initial misalignment taxonomy, and try to connect it to behaviors described in the Fable 5 system card. Then I’ll summarize and list some topics for subsequent posts.
What does alignment mean? Here is one reasonable definition (I’ll discuss some possible refinements in later posts): An AI system is aligned if it reliably does what its principals actually intend – pursuing that intended objective rather than merely the metric it was given, while honoring imposed constraints – across the situations where it’s deployed.
Aligning the base model is not enough: What we really care about is that the AI systems based on it will be aligned – a non-deceptive base model could turn deceptive when we apply strong optimization pressure to create such a system. Consider Fig. 1 (to be explained below).
Fig. 1 – A hierarchy of AI systems:
Note on acronyms: The D&V (Design and Verification) process includes the V&V (Verification and Validation) process – see fig. 1-b.
Let’s use a medical AI system as our example: This future system diagnoses patients, and then proposes / initiates / tracks treatment, escalating to a human doctor when needed. Let’s further assume the following set of layered AI systems, where each system is a node in a tree (see fig. 1-a):
Each such system is constructed iteratively: The design and verification (D&V) team comes up with a spec for what-it-wants-to-build, does implementation and verification, discovers problems and iterates (see fig. 1-b). Verification is often done using CDV – a smart, systematic way to sample from the full range of situations a system might face, check its behavior in each, and track which situations you’ve actually covered.
Each such system is built according to its “spec” consisting of (see fig. 1-c):
An important part of many specs is the “relevant constitution”: Base-AI’s spec probably contains the “model constitution”, and Med-AI’s spec probably contains various relevant medical regulations.
Let’s introduce some simplifying assumptions regarding the D&V team producing each of these systems (subsequent posts may relax some of these assumptions):
Let me start with the overall picture (to be clarified below):
Fig. 2 – Bugs and misalignments:
Note about implementation bugs: While I focus here mainly on spec bugs, many failures are really implementation bugs: The relevant requirement was stated clearly, but training did not cover enough of its important variants for the model to generalize reliably. Or later training – perhaps to make a higher-level application layer work – weakened an earlier-learned behavior. Consider a Med-AI which fails to diagnose a rare disease it was never trained on: This is an implementation bug (a capability / robustness failure), which is not a misalignment.
An initial failure taxonomy: Below are six hypothetical Med-AI alignment problems. Viewed superficially, they go from “model does not know this is bad” to “knows” to “knows and hides it”. But misalignment is much more nuanced and multidimensional – I’ll discuss some nuances below, leaving most for subsequent posts.
There are lots of nuances to discuss: There is a crucial distinction between “knowing this is bad” and “knowing I am evaluated”. And there are many shades of hiding in addition to those shown above. There is also the “seems to hide” trap, familiar to anybody who has done serious V&V: As you remove the “big” bugs, the remaining ones seem to hide in increasingly-obscure corners. Each of these axes has its own literature (e.g. eval-awareness and CoT-faithfulness), to be discussed in later posts.
Projecting the Fable 5 system card onto the taxonomy: Anthropic’s recent Fable 5 / Mythos 5 system card reports behaviors that seem to map onto the cases above reasonably well. This is a very tentative projection – an illustration of the framework, not evidence that the categories are exhaustive or cleanly separable.
A central theme (look again at fig. 2 above): Many misalignments are bug-like: They can be found via V&V and then fixed (often by enhancing the spec). This stops working when the system adapts strategically to the evaluation process, but “cleaning up” the other cases can help diagnose this harder one (more on this below).
The spec problem: Many bugs (and bug-like misalignments) are caused by the system pursuing what it was told, rather than the “intended objective”. Thus, a large part of aligning a system is getting its spec right. And specs are hard in a specific, recurring way: They often fail to capture what we actually want because nobody thought of the situation up front.
A true example from my field: A stationary delivery bot blocked a wheelchair user from getting off a crossing. Nobody wrote “even a bot doing nothing can be unsafe”: It never occurred to anyone. These are spec bugs: Obvious after discovery, impossible to enumerate before it. In my experience, spec bugs (far more than ordinary implementation bugs) are the failures that actually kill projects and people. And for capable AI systems the spec often starts thin and has to be grown, mostly through V&V.
The three-way investigation: In ordinary V&V – e.g. for AVs – when bug discovery rate levels off despite sustained, creative V&V efforts, this can be interpreted in at least two ways: The remaining risk is now acceptable, or the V&V process is somehow “stuck” (e.g. there are still undetected cases of 1–5, or our evaluations have some “tell” the system keys on).
With high-capability AI there is a third possibility: The system may be adapting strategically to the evaluation process (e.g. by corrupting the V&V machinery). These are not mutually exclusive; together they form a three-way investigation challenge. Clearing cases 1–5 does not resolve it, but it removes many mundane explanations. What remains is a narrower diagnostic problem, and the competing explanations become more testable.
There are various techniques you can use to help in this investigation: Look at the full history of failures found and fixed on the way up – this can show whether the remaining tail behaves like an ordinary coverage problem or starts to look anomalous. Restore earlier system snapshots and interrogate them with fresh scenarios and checkers they never encountered. This extends into a longitudinal technique: Strategic deception cannot easily appear fully formed – it has to be learned incrementally. And half-formed deceivers are easier to catch: Scanning back across the snapshot series for precursors (e.g. an evaluation-conditioned behavioral gap that appears and then grows) can locate when concealment started forming. These and similar techniques can help probe the competing explanations and generate sharper follow-up tests. A later post will dig further into this.
Note that these techniques help us in the diagnosis, but they do not eliminate the danger. So we also need control (monitoring, containment, auditing) – and CDV can help there too, e.g. by using coverage-driven generation to systematically stress-test the monitors and the surrounding protocol.
Summary: I sketched a way to look at misalignment (roughly) as a spectrum – from reward misspecification, through goal misgeneralization and motivated reasoning, to strategic deception – and argued that most of the failures we observe today are bug-like: Findable and fixable with V&V, often by improving the spec. The genuinely hard case is the last one, which may hold most of the risk weight – hence the frontier community’s focus on it. And even there, clearing the easier cases isn’t wasted: it narrows the three-way investigation and makes the competing explanations more testable.
Much of what I described (coverage maps, randomly probing situations the system wasn’t prepared for, hunting spec bugs, handling layered systems and more) is a mature V&V toolkit my field relies on, and that alignment work (AFAIK) doesn’t yet use systematically. I’d genuinely like people to take these tools to their own alignment problems and tell me where they break.
More broadly, intelligence is getting cheap, and verification is becoming the scarce, expensive part: Catalini, Hui and Wu argue the same from economics (even when the models are honest). Much of this series will be about adapting the CDV toolkit to alignment, and seeing how far it goes.
Some planned follow-up posts: Lots of open questions here – I welcome thoughts / contributions:
My previous coverage-driven alignment post already describes CDV and how it can be used for alignment. It also briefly discusses incentives, layered systems, and whether alignment is more like safety or like security. This series will try to develop all these and more in full – stay tuned.
I’d like to thank David Manheim, Steve Vitka, Ida Mattsson, Shaul Ben-Haim and Sebastian Klaas for commenting on earlier drafts of this post. Remaining errors, and the views expressed, are mine alone.
