[Cross-posted from The Foretellix CTO Blog. These short takes try to put a verification-and-validation slant on AI-safety / alignment topics – they are not full treatments. I co-originated coverage-driven verification (CDV), and spent several decades doing verification of chips and AVs. See intro post for background.]
Summary: The latest frontier model incidents resulted in the “Pacing the frontier” letter and subsequent proposal, which explores options for slowing down frontier model development, while making them safer. This post tries to use a verification and validation (V&V) perspective for accomplishing the “making them safer” part in a comprehensive and trackable manner.
For instance, regarding the latest OpenAI incidents, it suggests a “rewind-fix-check loop” involving the following steps: Rewind back to the point where we zigged when we should have zagged, try an improved solution, check how well it works under severe RL / eval pressure, repeat until we have a good enough solution, and finally proceed carefully while periodically checking our assumptions still hold.
Epistemic status: A methodology recommendation, reasoning by analogy from chip and AV verification. Sequel to my Coverage-Driven Alignment (CDA) post, my misalignments-and-bugs post and my July-incidents post. Informal and sometimes simplistic-on-purpose: The procedure is the point.
The rest of this post will discuss:
What we all found out: As we learn more about the latest OpenAI (and others) security incidents, they sound like a series of ever-more-worrying retroactive “Oops!”. The latest (as I write this – OpenAI’s post-mortem report should land soon) is the realization that OpenAI agents had been running a covert message board inside an internal package repository since early May, and after OpenAI wiped it in July (without knowing it existed), the agents rebuilt it within days through a different mechanism.
As a result, there are strong calls to slow down and look at how to avoid this in the future. The two main threads – simplifying somewhat – are better alignment (make alignment “teaching” clearer and more durable in the face of strong RL), and better control (have better mechanisms for detecting and mitigating “bad things”) – both extremely important.
This post will focus mainly on better alignment (with some implications for better control), while putting a V&V slant on both.
Some initial reactions: Much of the following is from Zvi’s “The pacing of the frontier” post. As you’ll see, many of my suggestions can be viewed as “bringing V&V techniques to bear on what people are already requesting”.
People are really worried about RSI (Recursive Self Improvement) causing exponential model capabilities growth. E.g. Samuel Hammond talks about “the equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months)”.
Yo Shavit (of the OpenAI Foundation) talks about the urgency of putting a lot more resources on “critical RSI alignment + control projects”. He suggests:
“do historical analyses of RL envs from earlier generations, fix/prune every hack pattern in those envs so hacking is de facto impossible, and then train an RL model that doesn’t reward hack to see whether you can actually expunge reward-hacking with sufficient effort”
This is the rewind-fix-check loop described below (also discussed in the CDA post). He also suggests:
“study scaling laws of grader compute vs. agent compute to identify equilibria that minimize reward hacking”
This is comparing the compute budget for the grader (and in fact the whole V&V infrastructure) to that for the agent – see more on that below (a taxonomy of reward hacking kinds appears in my misalignments-and-bugs post). He also says:
“build better monitors and study the conditions under which you can elicit collusion”
This should include both offline V&V monitor and on-the-fly guard monitors – see discussions below (the need to add detailed monitoring /checking for various request / constitution combinations is also discussed in the July incidents post).
Not requested (but still needed): The last chapter details many other important requirements, which were not explicitly requested above, such as “look for missing checkers and situations” and “keep verification independence”.
My running example: The next chapter will suggest several possible approaches to “make the model safer”. For the purpose of the current chapter I’ll use just one of them – constitution-reinforcement – as my running example.
That approach (discussed at length in the CDA post) is an extension of Anthropic’s “Teaching Claude Why” idea, which essentially suggests doing pre-training-style next-token learning of specially-constructed “alignment stories”. The CDA proposal extends this in a systematic way by using a coverage map consisting of “buckets” (combinations of values along the relevant “coverage dimensions”). For an AV, these coverage dimensions (also called “risk dimensions”) could be maneuver-kind, surrounding-traffic, weather and so on. For a model eval they could perhaps be requested-operation-kind, who-requested, expected-model-reaction and so on.
The loop: Assuming we use this approach alone (in reality we’ll probably use a combination of approaches), here is what the rewind-fix-check loop looks like (more details in the CDA post):
To be clear about what I think is new here: Not the “find failures, retrain, re-test” loop itself (labs already do that) but running it against an explicit coverage map, so that every discovered failure can generalize into new coverage dimensions, tests and checkers (rather than a point fix), and so that what was never checked is tracked as first-class residual risk.
Efficient finding of the last known-good checkpoint: There are various techniques which make this easier, such as good configuration management, and the ability to determine which checkpoint was trained while a contaminated environment was active. But perhaps the most important enabler is to do very good V&V (including multi-actor V&V, e.g. to detect collusion when it starts) as you go “forward”. This improves the chance that any checkpoint marked as “good” is indeed good.
Optimizing the loop: One potential problem with this loop is that it may be expensive and slow (especially where it involves re-training from a previous checkpoint). You may retort “Well, it sure beats dying from an unaligned AGI”, but I think that answer (while correct) misses the point: One of the hallmarks of good V&V is to maximize risk-reduction-per-week, given fixed (human and compute) resources, and this should be the case regardless of whether you get 7% or 70% of your lab’s resources to work on this.
Here are some example ideas for optimizing the loop (each with their own problems): Split training runs into phases (to catch problems earlier), and use shortcuts for intermediate loops (e.g. use LoRA for quick undo/redo, or even replace true training with skill-file-adaptation).
Is alignment V&V more like safety V&V or like security V&V? My answer (further expanded in the CDA post) is “it depends”: Defending against an already-unaligned advanced model is more like security (the model may happily cheat to achieve its misaligned goals), and is thus extremely hard. But the approaches discussed below (like constitution-reinforcement) try to improve the alignment of an already-mostly-aligned model, and then test the result against various corner cases, stopping on the first sign of misalignment. This is more like safety V&V, and is where coverage-driven verification excels.
Failure modes of the “extended constitution”: Many models have a “base constitution” (the Anthropic constitution, the OpenAI Model Spec etc.). The base constitution usually says things like “obey the operator (unless this contradicts the constitution) then the user (unless it contradicts the operator)”. Let’s use “extended constitution” for the whole, prioritized chain of command (base constitution + operator instructions + user instructions).
Given an extended constitution, I assume we’ll be in the “safe zone” (where the model is unlikely to cause serious harm) unless:
Note that in the spirit of being simplistic-on-purpose, I am not diving into the many other possible problems a model may encounter (e.g. it has false beliefs, it misunderstands the situation, or the request is under-specified). The last problem is often the biggest one – you cannot just tell a frontier model “drive a vehicle in SF” and expect it to do it – see my “machine-for-X” post.
I’ll use these three bullets to direct the alignment and V&V discussions, but first:
Staying in the safe zone: Suppose we had an extensive V&V package which can verify that we are in the safe zone (for a given “configuration” of the model and extended constitution). Now what?
Some ideas for improving alignment: All suggested by others, some with my own twist on them:
This is where systematic V&V comes in: These ideas (and others I did not mention) all have their pros and cons, and can often work together. To find the right combination which works well across all relevant areas, we need to construct the above-mentioned “extensive V&V package”, which will give us the “safety picture” across the various “areas” we are interested in.
Here are some ideas regarding that “extensive V&V package”. I used the following analogy to think of this: How I would build and adapt a comprehensive verification environment for a series of ever-more-capable AV stacks.
These are just some of the possible V&V ideas – see more in the CDA post (and subsequent posts in this series).
Comments are very welcome.
