Can reducing small risks increase large ones? Evidence from other fields and implications for AI

This is a crosspost for a post I created with Claude Opus and Sonnet 5.5.

Bottom line: cutting small risks can raise large risks in the same category, but mostly through physical accumulation (wildfire), behavioural compensation (levees) or masking (proxy metrics) — rarely through lost salience, the channel the AI-slowdown argument relies on. Fields that reduce large risks almost never respond to the salience worry by tolerating small harms; they respond by capturing the signal another way (incident reporting, near-miss analysis, communication). The one major exception, fire management, deliberately allows small fires because small fires physically remove fuel, not because they make fire risk salient.

1. The claim and when it holds

Jordan Arel's 9 reasons against a near-term AI slow-down argues (reason 1): "A near-term slowdown could prevent the warning shots that would make the public take AI risk more seriously." In a reply in the comments: "the risks seem like they will get much worse over time, so an earlier warning shot could be high value if it reduces risk skepticism for the bigger risks."

Reducing small risks affects the large risk through two channels. A simple linear model shows how they combine:

$\begin{array}{cc}\text{large risk}& =b\times \text{small risk}-c\times \text{safety effort}\\ \text{safety effort}& =e\times \text{small risk}\\ \Rightarrow \text{large risk}& =(b-c\times e)\times \text{small risk}\end{array}$

The coefficients b, c and e are non-negative numbers:

• b (direct channel): how much large risk rises per unit of small risk at fixed safety effort. It is positive when small and large risks share causes, so cutting small risks also cuts large ones.
• c: how much large risk falls per unit of safety effort.
• e: how much safety effort rises per unit of small risk, i.e. how strongly small events prompt more effort.
• c × e (salience channel): the large risk avoided through the safety effort that small events prompt.

Cutting small risks raises the large risk only if c × e is greater than b, i.e. only if the salience channel outweighs the direct channel. That requires conditions 1–3 below: a large e (condition 1), a large c (condition 2) and a small b (condition 3). Condition 4 does not affect the sign: it decides only whether tolerating small risks is worth it even when the salience channel dominates (and cheap substitutes, once adopted, reduce e).

1. Small events substantially raise safety effort (salience is large and does not decay).
2. That extra effort is well directed at the large risk (not at the visible small one).
3. Small and large risks share few causes, so cutting small ones does little directly.
4. The salience cannot be bought more cheaply without the harm (reporting, demonstrations, simulations).

Four other mechanisms can also make cutting small risk raise large risk, and they need none of conditions 1–3:

• Accumulation: small events discharge a stock that otherwise feeds big events (fuel, leverage).
• Compensation: feeling safer, people add exposure or drop precautions (levees, Peltzman effect).
• Masking: small harms are suppressed at the symptom level while the cause persists, degrading the signal (BP, AI chain-of-thought training).
• Overcorrection: avoiding the small event pushes behaviour toward a riskier failure mode.

2. Other domains

The effect is real in several domains, but the pure-salience version has the weakest evidence; the strongest cases run through accumulation, compensation or masking.

Risk compensation, the mechanism behind the levee and Peltzman cases, is a general caveat for safety measures. Hedlund 2000 states the theory: "Risk compensation theory hypothesizes that they do, that we "use up" the additional safety though [sic] more risky actions." The paper "surveys risk compensation by reviewing its history, discussing its theoretical foundations, outlining evidence for and against its claims, and providing the author's own views"; its abstract does not say how large the effect is, so I make no claim about that here.

3. Do large-risk reducers worry about this?

Only where small events physically or behaviourally reduce the large risk. Where the link is salience alone, practitioners keep cutting small risks and protect the signal instead.

The revealed consensus: the salience value of small harms is real but almost always cheaper to obtain without the harms. No field I found reduces large risk by tolerating small harms for their salience alone. Fire management, the closest analogue, relies on accumulation, which has no clear AI counterpart. This is from the sources linked above and my knowledge of these fields, not a systematic survey.

4. Implications for AI

For the warning-shot argument:

• Real incidents have produced lasting safety institutions: Three Mile Island led to INPO the same year (Blandford & May).
• AI risk scepticism is high, and concrete harms persuade publics and legislators more than arguments.
• If AI risks grow steeply with capability, an early, survivable incident is worth more than a late one (Arel).
• Accumulation has a possible AI analogue: a slowdown may build a compute or algorithmic overhang, so capabilities later jump faster (speculative, contested).

Against:

• Shared causes: small and large AI harms both stem from weak alignment, weak security and racing; reducing those reduces both, making the direct channel large.
• Salience decays: the panic-and-neglect pattern in pandemics shows attention fading within a few years.
• Near misses often reassure: a harmless AI incident can read as "disaster that did not occur" (Tinsley, Dillon & Cronin).
• Misdirected response: incidents provoke fixes to the visible failure, as in BP's personal-safety focus, not to the large risk.
• Salience has cheaper substitutes: dangerous-capability evals, model-organism demonstrations, mandatory incident reporting.
• A slowdown is not mainly a small-risk reduction: it also buys alignment and governance time, which cuts large risk directly.

Conclusion: the argument is valid in form, but its three sign conditions (section 1) look unlikely to hold jointly for AI. My rough judgement (speculative) is that cutting small AI risks raises large AI risk only in the masking case. That calls for refusing to hide small failures, not refusing to prevent them. Practical upshots:

• Fund incident databases and mandatory disclosure, so small failures that do occur are maximally salient.
• Stage "controlled warning shots": public, harmless demonstrations of dangerous capabilities or deception.
• Avoid training against monitored misbehaviour; track catastrophic-risk indicators separately from everyday harms.
• Judge the overhang concern on its own evidence; it is an accumulation argument, not a salience one.