@Jan Wehner🔸 thanks for writing this.
Please remove your 'Bart.et al' reference and replace with the original by Barrett et al. here: [2502.05791] Assessing confidence in frontier AI safety cases. The former is a plagiarised version we are investigating.
Many thanks.
Hi Ben, sorry about the mistake and thanks for letting me know. I'll update it here and on LessWrong immediately.
Many thanks Jan, really appreciate the rapid resolution.
I'm curious how and why safety cases got so popular in AI safety. There are so many other risk assessment techniques out there, for reference ISO31010 lists 30 of them (see here) and they're far from exhaustive. My instinct is that it's because safety cases are purely text-based, easily understandable, and does not require proper risk management concepts (e.g. hazards, events, consequences) for it to work; so at some point perhaps someone suggested to use it and the rest of the field just went with it.
(also I don't know how common safety cases are in safety engineering, in my decade in the oil and gas industry I have never heard of it before, though it may be more common in other industries)
It's a great question. I see Safety Cases more as a meta-framework in which you can use different kinds of evidence. Other risk management techniques can be used as evidence in a Safety Case (eg this paper uses a delphi method).
Also I think Safety Cases are attractive to people in AI Safety because:
1) They offer flexibility for the kind of evidence and reasoning that is allowed. From skimming it seems to me that many of the other risk management practices you linked are more strict about the kind of arguments or the kind of evidence that can be brought.
2) They strive to comprehensively prove that overall risk is low. I think most of the other techniques don't let you make claims such as "overall risk from a system is <x%" (which AI Safety people want).
3) (I might be wrong here), but it seems to me that many other risk management techniques require you to understand the system and it's environment decently well, whereas this is very difficult for AI Safety.
Overall, you might well be right that other risk management techniques have been overlooked and we shouldn't just focus on Safety Cases.
Yeah 1 and 3 seems right to me, thanks.
On 2, I think there are quite a number of techniques that give you quantitative risk estimates, and it's quite routine in safety engineering and often required (e.g. to demonstrate that you have achieved 1e-4 fatality threshold and any further risk reduction is impractical). I don't fully understand most of the techniques listed in ISO31010, but it seems that a number of them do give quantitative risk estimates as a result the risk evaluation process, e.g. monte carlo, bayesian networks, F/N diagrams, VaR, toxicological risk assessment, etc.
If you haven't already seen this paper on risk modelling, they use FTA and bayesian networks to estimate risks quantitatively.
There are so many other risk assessment techniques out there, for reference ISO31010 lists 30 of them (see here) and they're far from exhaustive.
Almost nothing on the list you've linked is an alternative approach to the same problem safety cases try to solve. E.g. "brainstorming" is obviously not a competitor to safety cases. And safety cases are not even an item in that list!
Of course. Many of the these techniques are specific to certain parts of the risk assessment process. The document is unfortunately paywalled, but risk assessment can be said to have these three parts:
Risk treatment is missing here becacuse it's sort of a separate process outside of risk assessment (but is tied to risk evaluation), and ISO 31010 specifically addresses the risk assessement phase. "Brainstorming" as a class of techniques and can take different shapes and form (in my previous work we used to have structured sessions to generate "what-if scenarios"), and they specifically address mainly 1 and some of 2a above.
But as Jan pointed out in his comment, perhaps safety cases are a meta-framework and not the technique itself, so the quality of a safety case depends on the quality of the evidence put forth alongside the arguments, and this quality may be related to the suitability and implementation of the specific techniques used to generate the evidence.
I recently wrote an Introduction to AI Safety Cases. It left me wondering whether they are actually an impactful intervention that should be prioritized by the AI Safety Community.
Safety Cases are structured arguments, supported by evidence that a system is safe enough in a given context. They sound compelling in theory—structured arguments forcing developers to affirmatively demonstrate safety rather than just "test and hope" (read my previous post for a longer explanation). They force reasoning about safety to be explicit, are flexible to changes in the technology and put the burden on the developer. Anthropic and DeepMind have committed to making Safety Cases in their RSPs.
To figure out how useful they are, I emailed researchers actively working on AI Safety Cases and reviewed the growing literature. This post synthesizes what I learned: first, a brief overview of what Safety Cases are and what work currently exists; then, expert responses to key questions about their feasibility, impact, and limitations. If you have perspectives you'd like to add, I'm happy to incorporate additional expert opinions—feel free to reach out or comment here.
Questions addressed in this post:
Actors such as companies, governments, and academics are building sketches for Safety Cases for Frontier LLMs and thus advancing best practices. Here is a (likely incomplete) list of Sketches for AI Safety Cases:
An incomplete list of Safety Case sketches, including the paper title, institutions involved, threat model, and core strategy.
| Sketches of Safety Cases | Objective | Core Strategy |
|---|---|---|
| An Example Safety Case for Safeguards Against Misuse (Redwood Research, UK AISI and others) | Misuse | Threat Modelling from expert judgment + capability evaluations + safeguard evaluations via red teaming |
| A sketch of an AI control safety case (UK AISI and Redwood Research) | Internally deployed Model Exfiltrating sensitive data | Control Protocol evaluated by Red vs Blue Team Evaluation |
| An alignment safety case sketch based on debate (UK AISI) | Outer Alignment | Hypothetical, theoretical, and empirical evidence that give safety assurances for Debate Protocols |
| Safety case template for frontier AI: A cyber inability argument (GovAI and UK AISI) | Harm from Cyber Offense | Risk Modelling -> Proxy Tasks -> Capability Evaluation |
| Towards evaluations-based safety cases for AI scheming (Apollo and others) | Scheming Risks | Evaluating ability to Scheme, cause Harm, or circumvent safeguards |
| Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework (Meridian Impact) | Ability of internally deployed model to manipulate employees | Multiple sketches of arguments for Inability, Control and Trustworthiness |
| Anthropic's Pilot Sabotage Risk Report (Anthropic) | Sabotage from internal models | Analyzing multiple risk pathways with Inability, Control, and Trustworthiness arguments |
| Three Sketches of ASL-4 Safety Case Components (Anthropic) | Sabotage from internal models | Sketches based on Mechanistic Interpretability, AI Control, and Incentive Analysis |
| Evaluating Frontier Models for Stealth and Situational Awareness (Google) | Scheming Risks | Inability Argument by evaluating proxy tasks |
While some of these sketches focus on proving some Objective (top-down), others are motivated by sketching how a particular method could give safety assurances (bottom-up). Many Safety Cases use CAE or GSN as standard notation.
Further work attempts to gather the kind of evidence that would let us build strong safety cases like Science of Evals, Chain-of-Thought Monitorability, or Red-Blue-Team Control Evaluations. Finally, Barrett et al argue for making Claims in Safety Cases Probabilistic, allowing us to assess the confidence that our Objective is met.
After an initial wave of position papers arguing for safety cases, people are now busy making “sketches” (i.e., outlines of possible safety cases) for different safety properties. This is an attempt to develop a better methodology for Safety Cases. Some Frontier AI Companies are attempting to make initial Safety Cases for real production models.
TLDR: Experts disagree on whether Safety Cases will substantially improve frontier AI safety. As best practice in safety-critical industries, they offer flexibility compared to rigid rules and can coordinate diverse stakeholders. However, skeptics argue their practical impact on company behavior may be small (especially for less responsible actors), and they risk providing false assurance while being costly to produce and enforce.
Epistemic effects on safety researchers cut both ways: Constructing Safety Cases forces researchers to make their reasoning explicit, confronting assumptions and rare failure modes that standard evaluation might miss. However, framing the task as "building an argument for safety" risks incentivizing motivated reasoning over objective risk assessment.
Joshua Clymer: "I expect most of the value of safety cases will come from improving the clarity of researchers' thinking.
I think I have personally developed a better understanding of how to mitigate AI risks as a result of thinking about safety cases. For example, one problem you run into when thinking about safety cases is that AI systems might have problems that are rare enough to not be seen in evaluations, but common enough to occur in deployment. These problems could be really catastrophic. For example, maybe some future version of Claude is very susceptible to adversarial towards humans at a low probability, and once it becomes adversarial towards humans, it repeats the argument that convinced it to all the other instances of Claude, and soon they're all adversarial too. This is the kind of thing that's really hard to rule out just by evaluating Claude's behavior or even looking at its internals.
So, how should developers mitigate this risk? I think they're going to have to set up a diverse society of AI models with de-correlated misalignment issues.
Not many people are thinking about this right now, and I wouldn't be either if I wasn't considering how to make a comprehensive safety case.“
Anon Expert: “Yes”
Ryan Greenblatt: "Trying to write a safety case seems like it would have bad epistemic effects on researchers in practice due to putting researchers in the frame of trying to argue why a model is safe."
Ryan Greenblatt (separate post): “There is still the question of whether making such safety cases with a less conservative threshold would influence AI company behavior... My sense is that all of these effects exist, but are relatively small, especially for relatively less responsible companies.”
Lucas Texeira et al argue Safety Cases will be valuable as "boundary objects" that coordinate collaboration between basic researchers, applied scientists, and policymakers.
Buhl et al.: "Safety cases [...] are seen as more comprehensive and flexible than traditional approaches to assurance. [...] However, safety cases have also been criticized for providing a false sense of assurance when in reality it is very difficult to produce an adequate safety case or review it effectively [...] While there is little empirical evidence for the efficacy of safety cases... and such evidence is difficult to produce given the nature of risk, practitioners mostly consider them effective and they are a recognised best practice in the UK."
They further note that "Safety cases provide a way to integrate different sources of evidence into a single, comprehensive assessment of a system's safety" and are "helpful for checking if risk assessment is comprehensive and valid." Additionally, "Safety cases make otherwise implicit arguments explicit, which [...] can highlight reasoning flaws or assurance gaps [..., ] makes it easier to see how overall system safety is affected if the evidence changes [... and] can help stakeholders communicate more clearly about disagreements"
Buhl et al. argue that, compared to governments setting specific rules, safety cases are advantageous because they are more flexible and thus durable, incentivize innovation in safety, and allocate the responsibility to the developer (who is best placed to figure out reasonable safety procedures). However, rules are easier to enforce, more transparent and cheaper to comply with.
TLDR: Currently, few companies are attempting to publish safety cases, mostly because it's substantial work with unclear value-add. Safety Cases for AI systems are especially difficult because they have many use cases, are constantly advancing in capabilities, and are opaque.
We currently don't have a consensus on good methodology, agreement on when a Safety Case is adequate, and lack basic science to justify assumptions and quantify uncertainty.
There are also many technical challenges to making strong safety cases. These include insufficient cybersecurity, monitors colluding with the AI systems they oversee, and misbehaviors that only appear outside of testing distributions (e.g., sandbagging or alignment faking). Making. stronger safety cases would require advances in scaling human supervision, interpretability, formal verification, and mitigating scheming risks.
Joshua Clymer: "I think we totally can build comprehensive AI safety cases right now. But publishing them is more work than it's worth.
I think the main barrier to establishing safety cases in the AI industry is getting companies to publish them. And the main barrier to getting companies to publish them is that there's a lot of work to write. So, if you want to work on safety cases, I think you should just join an AI company and help them write safety cases. I would start by joining Anthropic since they're already interested in doing things like this.
The next question is how useful this would be in absolute terms, relative to some other way you could spend your time.
I think it would be more useful to do ordinary alignment or control research at an AI company like Anthropic, or if you want to do non-technical work, it's probably more useful to help with comms/awareness building projects if you think you'd be good at that.”
Anon expert: “Frontier AI is a general purpose technology, the range of applications is very wide. Currently, the emphasis is on the AI developers building these safety cases. Due to the wide range of applications, many such safety cases will be required, and building a good one takes a lot of work. Also the AI capabilities are advancing rapidly, which creates a challenge. The top-level claim of safety cases is often given in binary claims (e.g. of the type 'The system is sufficiently safe' - in practice such claims often cannot be made unequivocally - there should generally be a 'level of confidence' associated with such claims - but this level of confidence is hard to determine. The AI is also inscrutable - which can make it hard to make safety claims for the AI itself (may be easier to make claims of the system within which an AI is embedded - but this is a difficult thing for an AI developer to do)”
Ryan Greenblatt argues that companies are unlikely to be able to reach SL-5 level of cybersecurity and to sufficiently mitigate risk from scheming. These would be necessary for building a comprehensive safety case.
“A core challenge with [empirical safety case arguments] is that misaligned behavior may only arise for parts of the input distribution, which may not arise during testing”
Rohin Shah et al point to multiple technical challenges:
Lucas Texeira et al: "progress on constructing substantive safety cases is bottlenecked by the absence of a mature basic science which could, in the future, support system modeling with well founded assumptions and quantifiable uncertainty."
Buhl et al.: One “technical challenge is to develop and build consensus on methodology for frontier AI safety case”. This includes “setting an appropriate bar for what constitutes an adequate safety case”. Another “technical challenge is to develop the safeguards necessary to assure the safety of advanced future systems.”
Goemans et al.: Argue there is a need for more "granular evaluations that correspond to specific risk models", to address "limitations of human uplift experiments" and for "a more systematic approach to addressing defeaters". Additionally, they point out that "the opacity of frontier AI systems in combination with their open-endedness and large action space make it challenging to assert the completeness of any safety analysis."
TLDR: While we can already make rudimentary safety cases for some risks (e.g., against misuse), experts agree that we cannot yet make them for existential risks from more advanced AI systems and likely won't for at least five years—largely because we can't empirically iterate on superintelligent systems. One expert argues that a Safety Case for ASI Safety with >90% confidence could be constructed that justifies the safety of an initial AI system and uses it to recursively validate increasingly capable successors up to superintelligence. Others are skeptical that there will be sufficient progress in cybersecurity and scheming mitigation in time.
Joshua Clymer: "I think it depends on what you mean by "catastrophic risk." OpenAI and Anthropic have both said that their frontier LLMs are close to providing misuse actors meaningful assistance with constructing chemical and biological agents. If that meets your definition of catastrophic risk, then I do think that these companies could provide high assurance of safety. Anthropic showed that their safeguards are robust to thousands of hours of red teaming from fairly high-skilled contractors. So jailbreaks are hard to find. Bad actors might purchase them, but Anthropic can monitor the black markets too. If these don't really exist, or if Anthropic can patch the problems before misuse actors can benefit much from them, then that's a good argument for safety.
Now, if by "catastrophic risk," what you mean is something like extinction-level threats from dramatically superhuman AI, then I don't expect that companies will be able to provide high assurance against this risk for at least the next five years. Currently, there's very little scientific consensus around how you justify the safety of dramatically superhuman AI. And even if there was a scientific consensus, we should have a lot of model-uncertainty. This isn't a domain where we can empirically iterate very much. We can't train takeover-capable superintelligent systems and be wrong about the risks they pose many times. So there's always a chance that we're all kind of confused about how to evaluate these risks, and reality proves us wrong.
However, I do think it's possible to be more than 90% confident that superintelligence is safe - or maybe even more than 95% confident if developers have a couple of years to do careful research. Many people would disagree with me about this, so let me elaborate. The basic argument you'd want to construct looks like this: We started with some AI system that we trust within some scope of tasks. We trust this AI system with those tasks because it generally seemed to behave well on similar tasks, and we don't think this AI system is intelligently throwing off our evaluations. Now, if we can trust this initial AI system to tackle ML research problems within some scope, then perhaps one of those problems is developing another AI system, AI System 2, that will safely tackle harder problems still. And then we can trust AI system 2 to develop an AI system 3 that we can trust with even more difficult tasks. And the process continues like that until there's some AI system N that's superintelligent.
At every step, AI system N credibly assures us that AI system N+1 is safe, so our trust gets passed on in a chain. We don't have to understand on the object level why a superintelligent system is safe so long as we are confident that the initial conditions were set correctly.”
See Appendix A for a more precise sketch of this argument.
Ryan Greenblatt argues that “AI companies are unlikely to make high-assurance safety cases if timelines are short”. He argues it will be very difficult and unlikely that, within 4 years, AI companies can reach sufficient levels of cybersecurity and that they can sufficiently mitigate risks from scheming AIs. Additionally, he finds it unlikely that companies would “unilaterally slow down if they are unable to make high-assurance safety cases”.
Buhl et al.: "It is already feasible to produce rudimentary safety cases based on existing safety frameworks, though significant research breakthroughs will likely be needed to produce safety cases for future systems."
TLDR: Safety Cases can be burdensome and have a reputation for overregulation, making them unattractive to US policymakers. Mandating Safety Cases for Frontier AI would require much more political buy-in into AI Risk than is currently there. Additionally, if governments want to mandate AI Safety Cases, they need to establish bodies with sufficient capacity to review safety cases.
Joshua Clymer: “Safety cases have historically been burdensome. They're often used in industries like the nuclear industry where people write hundreds of pages, and the documents are reviewed in a very slow process. People in the United States don't want to introduce those frictions, and I think for good reason. Safety cases are also most popular in Europe, and the European approach to regulation has a bad rap in U.S. politics right now.
So I think people shouldn't use the word "safety cases" in the United States in order to avoid all the historical baggage, and instead use some other term like "risk assessments"”
Ryan Greenblatt: “It seems somewhat unlikely that in short timelines the government ends up broadly agreeing with my views on AI capabilities and misalignment risks to the extent that would be needed for the relevant government action. I don't expect the government to be fully sleepwalking, but in short timelines, very fast action might be required and the government might need to update very quickly toward aggressive action.”
Buhl et al. argue that there are important institutional challenges to adopt safety cases in regulation like: "Appointing or establishing a body to receive safety cases securely" "Building capacity and expertise to effectively review safety cases" and "Incorporating third parties into safety case review"
TLDR: While one expert argues that Safety Cases are important, others argue that Safety Cases currently don't seem politically feasible, and the AI Safety community should focus more on scenarios in which there is little willingness to pay safety taxes. Instead, researchers should focus on Transparency through Risk Reports and producing legible evidence of danger.
Joshua Clymer: "I think risk reports are a better framing in the near-term (see this post).
Safety cases aren't as politically viable because they are more work to make, and create a much more restrictive gate on AI development.
In the near term, I think it makes sense for policy to focus on transparency. And the best thing for transparency is for companies to report the risks they observe (not to try to convince everyone that AI systems are safe).”
Anon Expert: "Probably yes, given answer to (2) [positive influence of Safety Cases] - AI developer's want to deploy (potentially unsafe) AI - and should therefore be required to make an argument for why they think it is sufficiently safe to deploy the AI (i.e. provide a safety case)."
Ryan Greenblatt: “I'm relatively less excited about work very specifically focused on safety cases that does not transfer to potentially producing legible evidence of danger. In particular, I worry that some people are spending time constructing tests which could be part of a safety case (if they indicated safety), but if the tests indicated danger this wouldn't actually change anyone's actions.”. Instead people should focus on providing evidence of risk and generally focus on a regime where there is less willingness to pay safety taxes. “I think risk analysis is often a better frame, at least when thinking about technical safety (rather than political or messaging implications), in the case where development would likely proceed even if a safety case can't be made.”
Buck Shlegeris: “In general, AI safety researchers focus way too much on scenarios where there's enough political will to adopt safety techniques that are seriously costly and inconvenient. [...] So people focus way more on safety cases and other high-assurance safety strategies than is deserved given how likely they seem.”
Ryan Greenblatt argues Transparency should focus on Risk Reports rather than Safety Cases because
Safety Cases were proposed in recent years as a mechanism for governing Frontier AI. Nowadays, the field is mostly busy building prototypes of arguments that could be made for different safety problems. Anthropic and DeepMind have stated they must provide Safety Cases once their models hit specific risk thresholds and are currently doing test runs to develop the institutional muscle for actually making strong safety cases.
In my ideal world, companies would be required to provide strong Safety Cases before deploying (or building) advanced AI Systems, because I believe they have properties (flexibility, responsibility on the developer, comprehensive) that make them well-suited for governing AI. Additionally, governments would have capable teams to review them, and 3rd parties would be able to scrutinize the arguments made by companies. However, building such safety cases for advanced AI requires levels of political buy-in that are currently not there and solutions to currently unsolved safety problems. This makes me sceptical that the AI Safety Community should do more work on AI Safety Cases on the current margin.
I still believe it is an important thing for companies to work towards being able to make strong safety cases. This will make it more feasible for governments to adopt Safety Cases as a governance paradigm and can sharpen researchers' thinking. However, technical researchers should prioritize work that remains valuable regardless of whether Safety Cases become load-bearing governance mechanisms.
Thanks to Joshua Clymer and the anonymous expert for sharing their perspectives via email, and to the researchers whose published work I've synthesized here.
“
This is the first deference safety case. And subsequent deference safety cases can be made via an inductive argument:
Suppose we want to justify that Agent K can be deferred to with tasks in S_K.
Then, we can argue:
So the third safety case amounts to "each AI system keeps telling us that it's safe to defer to the next one.
Once again, I think this kind of argument could provide >90% assurance of the safety of superintelligence, but couldn't provide the level of assurance that would be warranted in the near-term.”
I think EAs are put way too much effort into thinking about safety cases compared to thinking about reducing risks on the margin in cases where risk is much higher (and willingness-to-pay for safety is much lower), because it seems unlikely that willingness-to-pay will be high enough that we'll have low risk at the relevant point. See e.g. here.