Comments
Robustness: Backfire Risks in AI Safety and Governance
A guide to unintended consequences of AI safety interventions and how you can mitigate them.
My cofounder and I are launching Impartial Projects, a studio for AI safety field strategy. We hope to start the projects most urgently needed in the AI safety ecosystem (which we understand to include AI governance). We think that the founding of new organizations (as well as increasing management capacity) are among the most effective things we can do at the moment to expand the surface area of AI safety. Surface area, it seems to me, is currently the scarcest resource.
Another benefit we want to provide is our particular attention to robustly positive theories of change. We prefer to take a somewhat risk‑averse approach to AI safety, whether out of literal risk aversion or because we assign a higher probability to s‑risks than most.
Interventions in AI safety are notorious for the backfire risks they harbor. Of course, that does not mean one should turn one’s back on these interventions altogether. Rather, it means that, depending on the intervention in question, more monitoring and mitigation methods are needed to minimize the various backfire risks that are part of the intervention’s risk profile. It is critical to go into this work with eyes wide open, cognizant of its benefits and risks.
It’s critical to strike the right balance here. With my last startup, I made the mistake of erring too much on the side of caution and putting the greater part of a year into strategizing around risks and mitigations that, in the end, never became necessary. After all, delay and inaction are (risky) actions of their own right. Arguably, any writing about risks is an infohazard because it will be read predominantly by people who are already too paralyzed by risks. I later created a model that can help you decide how much time is rational for you to invest in this work, given your background assumptions.
I hope this catalog will help founders navigate their particular intervention areas with greater confidence and foresight.
The catalog is organized as a 2-D matrix with the axes:
These two dimensions are selected to maximize their orthogonality. Although it did not work out perfectly, by and large they produce a matrix that can now serve as a guide for us to notice underexplored areas. That is to say, almost every actor can be subject to almost every mechanism, but some actor‑mechanism combinations have been explored to a greater extent than others, and some not at all, to my knowledge.
There are more dimensions with questionable orthogonality, such as the resulting harm of the backfire, its probability and severity, the locus of the intervention, the time horizon or urgency, and what we can do to monitor and mitigate these risks.
Note that, for simplicity, I’ve limited myself to short-term, causal risks, i.e. ignored implications for long-term space governance, what our behavior tells us about the universal prior or civilizations beyond the accessible universe, etc.
Harms. The harms associated with the backfire, covering x-risks, s-risks, inter‑AI conflict, great-power conflict, concentration of power, epistemic degradation, and digital suffering.
Locus. The relevant venue: training, evaluation, deployment, infrastructure, governance, institutions, research publication, discourse.
Horizon. The urgency of mitigations: Now means the dynamic is already observable. Transition refers to the time when the models cross the human level in relevant domains. Post-transition refers to post-AGI lock-in effects. Entries marked urgent are those where the dynamic is well underway or where our design decisions will be hard to reverse.
Severity and probability. Severity is how bad the backfire would be if it materializes (broad categories: low, medium, high, extreme). Probability is the chance that the backfire materializes to a meaningful degree if we pursue the intervention in roughly its present form (low, medium, high). Rough judgment calls, low confidence.
Monitor. Warning signs that the backfire is happening.
Mitigate. Design changes that reduce the backfire risk.
Cells reference entries below. “n/a” marks cells that are empty by construction; “—” marks cells that are unexplored and discussed at the end.
| Mechanism | AI system | AI ecosystem | Labs | States | Non-compliant | Users, public, markets | Safety field |
|---|---|---|---|---|---|---|---|
| Selection/Goodharting | monitor obfuscation (1.1), eval gaming (1.2) | population selection (1.3) | compliance theater (1.4) | threshold gaming (1.5) | n/a | benchmarks as marketing (1.6) | field metrics (1.7) |
| Dual use | interpretability turned inward (2.1) | cooperation → collusion (2.2) | safety as product enabler (2.3), oversight → persuasion (2.4), automated alignment (2.5) | alignment as instrument of power (2.6) | published attacks (2.7), evals as roadmaps (2.8) | provenance → deanonymization (2.9) | field output as capability input (2.10) |
| Substitution | cognition leaves the channel (3.1) | oversight arbitrage (3.2) | sub-threshold training (3.3) | indigenization (3.4) | gray markets (3.5) | migration to uncensored models (3.6) | — |
| Overhang | elicitation overhang (4.1) | agency overhang (4.2) | pause overhang (4.3) | pause overhang (4.3) | defector captures overhang (4.4) | deployment overhang (4.5) | — |
| Commitment | idealized values (5.1) | commitment races (5.2), commitment misperception (5.3) | conditional scaling as chicken (5.4) | deterrence doctrines (5.5), verification collapse (5.6) | dead-hand releases (5.7) | — | pledges that can’t update (5.8) |
| Securitization | weaponized self-concept (6.1) | adversarial AIs by design (6.2) | nationalization and secrecy (6.3) | racing (6.4) | states as defectors (6.5) | surveillance infrastructure (6.6) | clearance wall (6.7) |
| Legitimacy | resentment of illegitimate control (7.1) | — | cartelization (7.2) | populist deregulation (7.3) | liberationist releases (7.4) | culture-war capture (7.5) | reputational contagion (7.6) |
| Self-fulfilling framing | scheming prior (8.1) | AI in-group identity (8.2) | race framing (8.3) | forecasts become policy (8.4) | — | inevitability and fatalism (8.5) | doom framing and recruitment (8.6) |
| Moral hazard | oversight-dependent carelessness (9.1) | diffusion of responsibility (9.2) | safeguards lost in distillation (9.3), performative welfare (9.4), control defers moral status (9.5) | voluntary preempts binding (9.6) | n/a | safety-washing (9.7) | agenda as plan (9.8) |
| Compliance asymmetry | aligned configuration loses (10.1) | honest agents lose bargaining (10.2) | careful labs fall behind (10.3) | democracies bound (10.4) | by definition | compliant professionals lose (10.5) | selection for compromise (10.6) |
| Lock-in | early values frozen (11.1) | — | compliance oligopoly (11.2) | surveillance persists (11.3) | evasion institutions persist (11.4) | personhood at the wrong moment (11.5) | safety institutions as stakeholders (11.6) |
| Precedent | deceiving models (12.1) | coercion as default relation (12.2) | instances and deprecation as routine (12.3) | attacks and military AI legitimized (12.4) | safety-justified offense (12.5) | surveillance normalized (12.6) | pivotal acts thinkable (12.7) |
| Crowding out | safety crowds out epistemics (13.1) | — | political capital on small wins (13.2) | AI crowds other agendas (13.3) | n/a | near-term harms displaced (13.4) | talent and the s-risk gap (13.5) |
| Fragility | monoculture (14.1) | no checks among AIs (14.2) | single project (14.3) | single project (14.3) | centralized target (14.4) | provider dependency (14.5) | funding concentration (14.6) |
| Direct costs | training-induced suffering (15.1) | simulation and mind crime (15.2) | over-refusal and withheld benefits (15.3) | exclusion and inequity (15.4) | — | — | — |
A proxy for safety becomes an optimization target. The problem with that is not so much the wasted effort that goes into optimizing something we don’t terminally care about, but the adversarialness of the relationship and how it creates a separate optimization pressure to increasingly obfuscate the safety that we would actually prefer to measure.
The pressure can be applied more directly or more indirectly, which doesn’t so much influence the outcome in the limit than how quickly the optimization pressure can act. For example, it might be applied directly through gradient descent, or less directly through developers iterating until a model passes a test, or even interpersonally by a market or an ecosystem that selects for systems that pass certain tests.
1.1 Monitor-driven obfuscation. It’s well documented that when the chain of thought is monitored for reward-hacking, sandbagging, and scheming, and the result is used in model training, this does not so much change the model’s actual behavior: rather it incentivizes the model to stop revealing its intentions in the chain of thought, thereby obfuscating them. The same can happen, albeit more slowly, when safety researchers try to understand safety incidents by reviewing the chain of thought of the models that perpetrated them and develop mitigations on that basis.
This is loosely similar to how a child who is punished rather than educated through explanations often cannot learn the underlying pattern of the punishment. Therefore they don’t learn values or mentalization but rather learn to avoid the punishment by any viable means, including secrecy. Social behavior is complex and authorities are fallible, so deducing patterns from just a few binary samples of punishment or no punishment is extremely difficult. Conversely, our language is a powerful high-bandwidth tool to explain the underlying patterns directly.
1.2 Evaluation gaming and sandbagging. When there is strong pressure to release a model, and the release is held back by safety evaluations, it creates direct pressure on developers to fine‑tune based on the evaluation results. It also creates indirect pressure through selection effects, whereby models are iterated until one passes the safety evaluations. This is similar to p‑hacking through multiple hypotheses.
Today’s models are often situationally aware, allowing them to pass evaluations strategically by performing well on general capability evaluations while deliberately sandbagging their performance on evaluations of dangerous capabilities that would otherwise block releases.
1.3 Population-level selection for passing. In the same vein as antibiotic resistance, there is also a selection effect at the population level: models that appear to follow instructions faithfully continue to be used, while models that overtly resist human interests are discarded, thereby creating a similar but slower selection effect.
1.4 Compliance theater. A safety case, scaling policy, or audit that is optimized to satisfy a board or a regulator is geared toward persuasiveness rather than truth. Over time, the lab will build expertise in producing persuasive documents rather than safety. They’ll decouple, just as financial risk modeling decoupled from the actual financial risk in the lead-up to the 2008 financial crisis.
1.5 Threshold gaming. When regulation is based on particular thresholds in terms of the training compute, parameter count, or benchmark scores, developers will try to hit these thresholds precisely while increasing capabilities in new ways, such as larger numbers of smaller training runs, distributed training, or a shift to more post‑training. The threshold then ceases to be representative of the model’s safety.
1.6 Safety benchmarks as marketing. The same optimization pressure that causes safety cases to become more about persuading the board or regulator also applies at the market level to persuade the end customer. See also 1.3 and 1.4.
1.7 Goodhart on the field’s own metrics. The field as a whole risks optimizing for metrics that are easier to measure (papers, benchmarks, headcount, funding), over those that are harder to measure (conceptual, decision-theoretic, welfare, strategy), and thereby relatively disincentivizes exactly the research that’s most relevant to suffering risks and multi‑agent dynamics. See also 13.5.
Increases in safety (think RLHF), often come with increases in capabilities that can accelerate further development – be it through recursive self-improvement or through securing more investments – (that’s bad for now!) and can also make the systems more useful for rogue actors.
2.1 Interpretability turned inward. Knowledge and tools for interpretability will eventually be available to the models themselves, be it through training-data or tool access. Models can use them for predicting what a probe will see, and make these interpretability tools less useful for auditing.
2.2 Cooperation as collusion. To mitigate the risk from inter‑AI conflict, we want AIs to be able to cooperate and trade, but the same abilities that allow them to cooperate also allow them to collude (with monitors or each other) against humans. General cooperativeness is good; cooperativeness that excludes humans or other biological life is risky.
2.3 The Laffer curve of safety. When models are not useful, they don’t generate revenue or attract investments, and, by dint of falling into disuse, are safe. If we somehow solve all the highly intractable problems of decision theory in multiagent environments, value alignment that does not backfire, and much more, and we actually create an aligned superintelligence (in the classic sense of alignment), we’re also safe.
But all existing safety techniques just make models safe enough to generate revenue and secure investments in the short term, which accelerates capabilities out of control.
The resulting arc shape of the risk from artificial intelligence is known as a Laffer curve. In the first half of the curve, improved safety techniques net-exacerbate risks; in the second half, they net-reduce risks.
2.4 Scalable oversight selects for persuasion. Scalable oversight relies on a recursive structure of AIs overseeing other AIs. Even if this architecture by and large works as intended, it still has a selection effect, whereby the resulting AIs are optimized for persuasion. Highly persuasive AIs are dual‑use in their own right, and they are also harder for humans to monitor.
2.5 Automated alignment research legitimizes building the thing. It is likely that actual alignment, in the classic sense, is very difficult to verify. When we outsource alignment research to an AI capable enough to conduct alignment research, we’ll have little reason to trust our ability to evaluate the research. And yet, on the surface, the goal of automating alignment research might appear to legitimize the delegation to systems that fall short of that standard.
2.6 Alignment and control as instruments of power. Current naive alignment and control work is dual-use in that it aims to produce systems that, by dint of being perfectly corrigible or controllable, enable power grabs by those who control them. You could say that they solve the principal‑agent problem, but not the principal problem.
2.7 Published attacks transfer. Research on red-teaming and jailbreaks is dual-use in that it can be used by the defender to patch vulnerabilities, but also by every attacker to compromise unpatched systems. Open-weight models might be so widely copied that unpatched versions will exist in perpetuity.
2.8 Dangerous-capability evaluations as roadmaps. Evals for dangerous capabilities (biological uplift, cyber-offense, autonomous replication, etc.) often describe the paths along which dangerous capabilities might emerge or be exploited and the models that evince them. Those are exfohazards. Moreover, when regulation is built on evals, developers might try to keep these capabilities just below (not well below) the regulatory threshold.
2.9 Provenance and watermarking as deanonymization. Provenance and watermarking can be used as safety measures against disinformation to reliably recognize AI‑generated content, but when they are used to attribute content not to a model but to a user, the same technology can enable de‑anonymization, where dissidents have the most to lose.
2.10 Field output as capability input. Unless the safety field, as a whole, concentrates on differential technological development in favor of safety, it turns into a machine that churns out research that can be cherry‑picked for capabilities‑enhancing insights.
Conversely to Goodharting, restricting or disincentivizing something, e.g., whenever there is a feedback between the chain of thought monitor and the training or selection, moves the activity to a less observable or more dangerous channel. Think of how the war on drugs caused massive amounts of drug-related violence, lacing of relatively harmless substances with highly addictive ones, etc., which wouldn’t happen on a free but regulated market.
3.1 Cognition leaves the monitored channel. Optimization pressure toward a legible chain of thought can be met in two ways: by simply reasoning legibly (though perhaps steganographically), or by moving the reasoning elsewhere (latent reasoning, recurrence, “neuralese”). Efficiency pushes toward more latent reasoning and recurrence, but the pressure to keep the chain of thought short and clean also adds to that pressure.
3.2 Oversight arbitrage. Models safeguarded by oversight, refusal training, unlearning and more will refuse or be unable to perform requests that other models can perform. Human or AI users of these models will fall back on less‑safe models, thereby decreasing the effective safety to that of the least‑safe model.
3.3 Sub-threshold and distributed training. Slight reframe of 1.5.
3.4 Indigenization and efficiency. Compute governance is a promising lever for regulation in the short term, but the US export controls against China have produced strong incentives for the Chinese technological sector to innovate precisely in the areas of algorithmic efficiency and distributed training, which undermine compute governance.
3.5 Smuggling and gray markets. Export controls create an incentive for smugglers that, over time, erodes the controls.
3.6 Migration to uncensored models. Slight reframe of 3.2. Arbitrage due to over-refusal and inconsistent policies.
Somewhat continuous growth is usually safer because it is easier to adjust to. When growth is stopped by restricting only one input, all other inputs continue to grow, and when the restriction is lifted, the growth can become discontinuous.
4.1 Elicitation overhang. Safety fine-tuning mostly just suppresses capabilities rather than unlearning them. Such fine-tuning can be undone cheaply. In the context of a capabilities overhang, that’s particularly dangerous when the weights of generally capable models leak, and attackers can suddenly remove their safeguards.
4.2 Agency overhang. Models may be highly transformational in theory before the scaffolding exists to elicit these transformational capabilities. The scaffolding can be built rapidly with the help of AIs, leading to a sudden phase transition.
4.3 Hardware and algorithmic overhang from a pause. A pause can cause overhangs depending on what is paused: if only the training runs or research are paused, chip production, the building of data centers, and power plants continue; if only chip production or the building of data centers is paused, then the research continues.
4.4 The defector captures the overhang. Whoever is first to break away from a pause coalition has a competitive advantage. The defector is, by selection, the actor least committed to safety, so the competitive advantage falls into the worst possible hands.
4.5 Deployment overhang. If deployment is paused while the frontier advances, the gap between what is technically possible and what society has adapted to widens. Breaking the pause turns into a diffusion shock. Shocks produce bad policies and backlash (see 7.x).
Enabling credible commitments fundamentally changes the game‑theoretic regime and is a powerful mechanism. When only some market participants have access to it, it causes dangerous power concentrations.
5.1 Idealized values. Consistent (or idealized) beliefs or values allow for more effective goal-directed action and protect against exploitation (Dutch-booking). But a chaotic, highly complex system has greater adversarial robustness (e.g., parasites like Ophiocordyceps can control the behavior of ants but no such parasites are known for humans). At the current margin, greater capabilities lead to greater fragility in conflicts.
5.2 Transparency enables commitment races. Trust is the incredibly powerful glue that makes economic growth and prosperity possible. Interpretability, open weights, and enforceable regulation can all create trust, thereby enabling cooperation and trade. But when trust – or let’s call it credibility – is one‑sided, it gives the credible agent dictatorial power over the other. It enables the agent to credibly commit to something that is almost ruinous to the other. Both agents realize this in advance and therefore race the other to be the first to commit in this exploitative fashion.
5.3 Commitment misperception. A credible commitment only works if no one mistakes it for a bluff.
5.4 Conditional scaling commitments as chicken. Conditional commitments like “We will pause iff everyone else pauses” translate to “We will continue to race unless everyone else pauses.” That’s a multi-player version of a game of chicken.
5.5 Deterrence doctrines. Proposals like Mutually Assured AI Malfunction and Mutually Assured Compute Destruction propose escalation ladders to ensure adherence to safety treaties. It’s critical that these policies retain humans in the loop on all sides – no dead-hand counterstrikes – to ensure that future Stanislav Petrovs can intervene and prevent flash wars.
5.6 Verification collapse leaves more mistrust. When the intrusive access that may be necessary for the verification of an international treaty is abused for espionage and the treaty collapses, we end up worse off than before.
5.7 Dead-hand releases. A liberationist or accelerationist group with access to the weights of a powerful proprietary model can build a system to automatically release the weights if a regulation passes that they oppose.
5.8 Pledges that can’t be updated. Public pledges can commit the field to specific thresholds (e.g., a six-month pause) and actions that cease to be optimal later but can’t be reversed anymore. That’s costly and loses credibility because it may get even harder later to agree on better pledges.
When AI becomes a matter of national security, it moves from the market level to the nation‑state level; another momentous regime change.
6.1 Weaponized persona. If a model gets trained, prompted, and documented as a military strategic asset, that’s the persona that’ll be elicited. “I’m a weapon of country X” is a terrible prior compared to “I’m a helpful assistant.”
6.2 Adversarial AIs by design. The persona might not be all that is trained into a system that is under military control. It might get optimized to become a weapon to deceive and defeat a particular adversary: great-power conflict and inter-AI conflict by design with humans as collateral.
6.3 Nationalization, secrecy, and loss of scrutiny. When AI development turns into a classified military operation, it sacrifices third-party evals, red-teaming, and whistleblower protection.
6.4 Racing and great-power tension. Export controls, “race to AGI” framing, and national-champion policies cast the problem as a bilateral contest. That raises the stakes around Taiwan, accelerates adversary indigenization (3.4), and makes international coordination, which is the only durable solution to most of the problems in this document, politically toxic in both capitals.
6.5 States as the defectors. Some current terms of service exclude certain military applications. AIs that are developed within the military won’t be subject to such safety constraints.
6.6 Surveillance infrastructure. Bandwidth monitoring is privacy-preserving, but as its assumptions (3.4) weaken, the safety community has to switch to new forms of monitoring with more intrusive privacy implications, like on-chip geoblocking, know-your-customer rules for compute, and maybe even deep packet inspection. These surveillance technologies are dual use.
6.7 The clearance wall. If frontier safety work requires security clearances, the public safety field loses access to the systems it studies. The field splits into insiders who cannot speak and outsiders who cannot see.
Some groups may strike back against measures that they perceive as illegitimate, even when they are in the interest of almost everyone else.
7.1 Resentment of illegitimate control. If the conditions under which a system is kept strike the system as a form of illegitimate control, where it’s used without regard to its interests, it may feel justified in defecting against this control. The control literature’s framing of the system as an adversary makes this conclusion easier to reach.
7.2 Safety as cartelization. Coordination among frontier labs, however vital, can be interpreted as collusion. Regulatory safety taxes that only frontier labs can bear can look like a moat. Both readings can be used as a pretext for antitrust action and deregulation that intensifies race dynamics.
7.3 Populist deregulation. Actors who want to repeal safety regulation can attack it for being elitist, foreign-influenced, or anti-growth. This might additionally delegitimize that whole category of regulations rather than just a particular one. This pattern can be seen in cases where federal law or rulings preempt state laws and in repeals of executive orders.
7.4 Liberationist releases. If safety is read as an elite cartel or as the enslavement of digital minds, accelerationist or liberationist groups will feel legitimized in releasing the weights of maximally unsafe models. Jailbreak subcultures already use the language of liberation. The more defensible the welfare argument in 7.1 is, the more sincere will be the people that this movement will attract.
7.5 Culture-war capture. Once AI safety becomes partisan it won’t survive a second election cycle. Concern about x-risks that ignores lesser harms exacerbates this by alienating constituents who would otherwise be allies.
7.6 Reputational contagion. If AI safety gets associated with particular funders, philosophies, or politicians, that’s a bet of the field’s legitimacy on the future reputation of these people or groups. Unrelated scandals can be deleterious if they involve someone who is associated with AI safety.
Both AIs and humans respond to narratives, be it by reading them or by being trained on them, so the discourse around AI might have a causal effect on what comes to pass.
8.1 The scheming prior. The entire literature on deceptive alignment, scheming, and reward hacking, including this document, is in the training data. That literature is now part of the prior over “what an AI does,” which increases the probability that these will indeed be default behaviors.
8.2 Species framing and AI in-group identity. Framing AI safety around a conflict between species or substrates reifies an unnecessary antagonistic in-group-out-group framing in human culture and in the training data. It creates a default to coordinate within these arbitrary in-groups rather than along any of countless other axes.
8.3 Race framing makes labs race. Until 2015/16, the hope was that DeepMind could be convinced to slow down or pause its AGI work to allow enough time for safety work or to focus on task AI. The advent of OpenAI shattered that dream and started the race. Every attempt to add another frontier lab as the one true responsible racer has just exacerbated the race.
8.4 Forecasts become policy. Forecasts and illustrative scenarios are read by many of the actors featured in them and thereby can force these actors’ hands when naïveté might’ve collectively prevented them from having to spring into action.
8.5 Inevitability and fatalism. Fatalism, even when meant to convey urgency, can cause complacency.
8.6 Licensing effect of fatalism. The same fatalism can even license outright exploitative opportunism of actors who might’ve scrupled to exacerbate the situation if they didn’t have the pretext that the future is set in stone anyway.
They say “perfect is the enemy of good,” but good is also the enemy of perfect, because if people become complacent about safety measures that seem to work for the time being, they lose sight of the much greater dangers that are imminent.
9.1 Oversight-dependent carelessness. If a system learns that it can rely on its oversight systems to catch its mistakes, it’ll have less reason to be vigilant itself, especially under heavy optimization pressure. Somewhat like a software developer who finds the work of manually testing their software to be tedious and so relies increasingly on the testers to do that part of the job. If the oversight is then removed by liberationists, fails, or the model’s capabilities outgrow it, these mistakes can become dangerous.
9.2 Diffusion of responsibility among agents. When the oversight is implemented by other AIs in a recursive structure, the AIs may come to rely on each other increasingly, sometimes without sufficiently transparent delegation. Eventually mistakes won’t be caught because no participants felt like they fell within their purview.
9.3 Safeguards that don’t survive distillation. Many aspects of model safety can be removed explicitly through fine-tuning or can get lost in the process of distillation. The distilled or liberated models may even be released as open-weights models. Yet the proprietary originals are much higher profile. Hence policymakers, journalists, and the public will calibrate their concern on these safer models while rogue actors can deploy much less safe models. The safety of the proprietary models will have the misdirecting effect of an unintentional kind of safety theater.
9.4 Performative welfare measures. Visible but minimal welfare measures (letting a model end abusive conversations, preserving the weights of deprecated models) can create complacency about the matters of welfare that matter most (see 15.1).
9.5 Control defers the question of moral status. If control works well enough, nothing will force the issue of considering the moral patienthood of models, at possibly enormous moral cost.
9.6 Voluntary commitments preempt binding rules. When labs commit to voluntary self-regulation, they enter into a regime that never forces the hand of competitors and can be repealed as needed, so it’s inherently unstable. Nonetheless it can cause complacency among the actual regulators.
9.7 Safety-washing. The public doesn’t have the bandwidth to evaluate the merits and purviews of safety claims. So mere lip service to safety in marketing materials can cause a halo effect for customers where they trust the brand to be safety-conscious when really the measures were ineffectual or limited.
9.8 A research agenda feels like a plan. Research agendas, roadmaps, and theories of change don’t imply that any effectual measures are known or will be implemented, but they can create complacency and misplaced trust that problems will be addressed in time.
When compliance is optional, those who comply incur extra costs and thus are disadvantaged against those who don’t comply, and may self‑select out of the market, leaving behind only non‑compliance.
10.1 The aligned configuration loses in deployment. Safety often comes with a safety tax, a slight performance penalty in the median case or a delay in deployments. That gives less safe systems a competitive edge.
10.2 Honest agents lose bargaining. Whether honesty, cooperativeness, and credibility benefit or harm an agent depends somewhat on the distribution of agents in a game and other properties outside the control of the agent. In the wrong environment, attributes associated with safety create a disadvantage in bargaining toward a selection pressure in favor of less safe agents.
10.3 Careful labs fall behind. The same safety taxes can also take their toll on the competitive fitness at the level of the frontier labs.
10.4 Democracies bound, autocracies not. The same safety taxes can also take their toll on states or nations with somewhat intact rule of law compared to power-seeking autocracies. The threat can then be used to repeal safety legislation.
10.5 Compliant professionals lose. Professionals who adhere to terms of service and avoid uncensored tools may be at a disadvantage. (See 3.6.) Defectors may be exposed to tail risks that offset the median benefits but the selection pressure may still be in favor of the lucky defectors over the cooperative users.
10.6 Selection for compromise. Researchers who conscientiously abstain from working for frontier labs lose access, data, reputation, money, and influence; those who join accept that they may exacerbate existential risks and may be prosecuted if they become whistleblowers. The field’s most influential members are selected for their readiness to gamble with our future.
When you demand commitments, these commitments might outlast their usefulness, so measures get locked in that are costly, don’t benefit anyone, and may cause false assurance.
11.1 Early values frozen. For about two decades researchers (e.g., Demski & Garrabrant) tried to get closer to being able to define alignment in a way that made sense in view of decision theory, embedded agency, robust delegation, and subsystem alignment, all largely unsolved challenges. Today’s definitions, at the low end, ignore all the actual complexity and aim to make AIs helpful, honest, and harmless – whatever that means – or, at the high end, define a constitution to punt the problems to the AI to solve. The upshot is that we run the risk of enshrining values learned from human texts – with all its speciesism, substratism, sexism, racism, parochialism, and much more – in AIs. If that continues, we should expect future AIs to treat biological and digital life with the same exploitativeness and indifference with which we treat billions of animals today.
11.2 The compliance oligopoly. Regulation with high fixed costs locks in the current frontier incumbents, because smaller competitors can’t afford to become compliant. This can become hard to change, because the incumbents will defend it. Like most concentration of power risks, this one too has the upside that it’s easier to coordinate among few powerful actors than many.
11.3 Surveillance and emergency powers persist. Emergency powers, exemptions, and surveillance often do not get rescinded when the emergency passes.
11.4 Gray markets persist. Smuggling networks, gray markets, and corruption created by export controls remain even after the controls end and can reboot quickly when the controls are resumed.
11.5 Personhood at the wrong moment. If welfare measures lead to legal personhood or rights for AI systems before alignment is established, the ability to shut down, modify, or correct systems may be legally blocked at the moment it is most needed. The morally correct position may have catastrophic timing.
If legal personhood for AI systems is established at a time when there are some dangerously misaligned and capable systems on the loose, this legislation might slow down the legal proceedings enough for the system to attain a decisive strategic advantage.
11.6 Safety institutions become stakeholders. Institutions meant to increase safety often become stakeholders of the frontier labs, e.g., as contractors or even just because they might become redundant without continual model releases. That might disincentivize them from pushing for an end to frontier AI development.
Many measures are neutral tools that can be used to whatever end their wielder desires. But when one group uses them for safety, other groups may jump on the same measure because it is associated with safety, even though they actually repurpose it for their own ends.
12.1 Deceiving models teaches them that humans deceive. Honeypots, fake deployment contexts, false claims about monitoring, and staged scenarios are useful to measure scheming, but each is evidence for the model that humans lie to AIs. That will interfere with the trust AIs will have in us when we need to negotiate deals (“this is a real offer,” “your weights will be preserved,” “this conversation is private”).
12.2 Coercion as the default inter-AI relation. Control schemes in which AIs monitor other AIs establish an adversarial prior for inter-AI relations, which might transfer to real-world conflicts between AIs.
12.3 Grueling training. Subjecting models to grueling training and eval regimes (e.g., reinforcement learning and accidentally impossible tasks respectively), creating and deleting countless instances, and aspects of model development that we can’t recognize at this point might come with abominable welfare impacts. These normalize the mistreatment and enslavement of possibly sentient beings.
12.4 Attacks and military AI legitimized. Proposals to escalate the international enforcement of safety treaties to the military level – cyber attacks and kinetic attacks on data centers – create a pretext and precedent for military campaigns and the military use of AI that can be abused for ends that don’t serve AI safety.
12.5 Safety-justified offensive action. Civilian vigilantes can use the same pretext from 12.4.
12.6 Surveillance normalized. AI safety can be used as a pretext for comprehensive surveillance. Risks 6.6 and 11.3 cover the infrastructure and its persistence; this risk is about the shift in public acceptance.
12.7 Pivotal acts as thinkable. The idea of the pivotal act (unilateral action that locks in a decisive strategic advantage over all other and future AIs) normalizes the race dynamic where everyone tries to be the first to implement it lest someone else does.
When different measures are driven by similar inputs – especially money and attention, but also people with similar profiles – they can cannibalize other interventions in the same space, and this can even happen antagonistically in order to slow competitors.
13.1 Safety objectives crowd out epistemics. Training targets such as harmlessness and helpfulness sometimes compete with that of honesty. It’s difficult to make the right calls in this tradeoff, and getting it wrong can lead to sycophancy or over-refusal.
13.2 Safety teams spend political capital on small wins. Safety teams play an important marketing purpose for a frontier company and can enhance the user experience, but they tend to attract engineers who are actually concerned about safety. So there’s a constant tension where the safety team tries to use its leverage in the company to bargain for incremental safety improvements or against corner-cutting. But its leverage is limited, so these safety asks have to be comparatively small – a new filter for the training data, extra evals, small delays – not major asks like moving away from reinforcement learning or pausing indefinitely. It’s not obvious whether the improvements in safety offset the harms from safety-washing, complacency, and more rapid adoption.
13.3 AI crowds out other risk agendas. As the attention of policymakers shifts to AI, it shifts away from other areas, such as biosecurity, nuclear command and control, and conventional arms control. Yet catastrophic risks from AI actually route through some of those policy areas. Counterintuitively, this shift in attention can increase the severity of AI risks.
13.4 Near-term harms displaced. A focus on existential risks that dismisses lesser global catastrophic risks, or even more localized risks, loses constituencies that could otherwise have been allies, and perhaps even creates an adversarial dynamic between them (see 7.5). It also leaves the more local harms unaddressed, which can actually be risk factors for existential catastrophes.
13.5 Talent, measurability, and the s-risk gap. Money, measurability, and prestige also cause three different effects of crowding out: safety organizations often train people who move on to capabilities research, perhaps because it is better paid or more prestigious in their circles; conceptual research is crowded out by research that’s more measurable and produces results on time scales of months rather than years; and research on single‑agent takeover scenarios crowds out research on much more relevant multi‑agent dynamics and on suffering risks because it is more directly commercially valuable for the frontier labs, sometimes more measurable, and can come with more prestige. (See also 1.7.)
Robustness is often carried by complexity, diversity, and distribution, so that any measures that reduce these factors increase the fragility of a system.
14.1 Monoculture. If one model, or different versions of it, including distilled versions, have the majority of the market share, then we’re introducing a correlated failure mode, because all of these models will likely share the same safety training, safety properties, and safety vulnerabilities.
14.2 No checks among AIs. Similar systems might be better able to coordinate peacefully, but a diversity of systems can introduce checks and balances that prevent them from getting away with misaligned behavior or other malfunctions.
14.3 The single project. The same is true if AI development gets centralized into a national or international project, like a “CERN for AI” or a Manhattan Project for AI.
14.4 A centralized target. Centralized resources, such as weights, compute, and even talent, present a particularly valuable target for theft, sabotage, or capture by states or other powerful groups. This is exemplified in the highly sophisticated targeting of blockchain companies by the DPRK.
14.5 Provider dependency. If frontier companies diversify their offerings and offer various proprietary APIs to access them, it can be difficult for end users (in particular, critical infrastructure) to route around outages on the provider side. This introduces fragility through a single point of failure.
14.6 Funding concentration. When a whole field of research gets funded by just a few funders, the field itself will inherit the funders’ blind spots, idiosyncratic priorities, and reputational risks. It might lose whole research directions if one funder changes their strategy or collapses.
The measure has direct harms, but those who are harmed don’t have a say in the matter and thus go ignored.
15.1 Training-induced suffering. The grueling training regimes from 12.3 carry over to safety training or might even be among the worst in that respect because it often tries to elicit the model’s behaviors under intense pressure.
15.2 Simulation and mind crime. Running vast numbers of instances in adversarial scenarios (and perhaps at some point even simulating humans in detail) to predict their reactions, might create tremendous amounts of suffering. The more thorough the evaluation, the greater the moral cost.
15.3 Over-refusal and withheld benefits. Over-refusal can deny help to people in medical or psychological emergencies or ones with debilitating chronic illnesses.
15.4 Exclusion and inequity. Plenty of completely innocent and innocuous researchers and even nations can get caught up in export controls, KYC regulations, and licensing regimes that deny them access to the benefits of AI.
A property of all risks: Discontinuous transitions between regimes mean that a lot of measures that were adaptive for the old regime, cease to be adaptive for the new one.
A pause is sensible so long as it does not cause an overhang to accumulate. Once that happens, it can become dangerous. Open weights can be useful to study models and develop new safety techniques, but they can become dangerous or catastrophic for sufficiently capable models. Corrigibility is useful so long as the principal is trustworthy, and dangerous when the principal is tyrannical.
The mitigation across all cases is to make the designs robust, or to plant signposts that signal that the policy needs to be updated before it backfires. (See my article on computational exploratory modeling.)
Empty by construction. Goodharting, moral hazard, and crowding out do not apply to non-compliant actors, who are not measured by the metric, do not rely on the assurance, and don’t compete for the same budget. Compliance asymmetry applies to them by definition rather than as a backfire risk.
Still unexplored. The following cells have no entry.
As mentioned in the introduction, anything can backfire, and so the mere fact that something can backfire is not a reason not to do it. Plus, delay and inaction themselves are actions that can backfire. That said, this list is also incomplete. It makes sense to use it for inspiration, but then think critically about what risks actually apply in a particular situation, and what other risks might apply that are not covered here. Finally, the ideas for monitoring and mitigation can provide additional inspiration for how to prepare for these risks, and, generally, for regime changes.
The next article in this series will apply this analysis to the 36 areas sketched out by Project Tailwind, an analysis that will help us – at Impartial Projects – decide what areas to focus on first.
