I think I buy that interventions which reduce either catastrophic or extinction risk by 1% for < $1 trillion exist. I'm less sure as to whether many of these interventions clear the 1,000x bar though, which (naively replacing US VSL = $7 mil with AMF's ~$5k) seems to imply 1% reduction for < $1 billion. (I recall Linch's comment being bullish and comfortable on interventions reducing x-risk ~0.01% at ~$100 mil, which could either be interpreted as ~100x i.e. in the ballpark of GiveDirectly's cash transfers, or as aggregating over a longer timescale than by 2050; the latter is probably the case. The other comments to that post offer a pretty wide range of values.)
That said, I've never actually seen a BOTEC justifying an actual x-risk grant (vs e.g. Open Phil's sample BOTECs for various grants with confidential details redacted), so my remarks above seem mostly immaterial to how x-risk cost-effectiveness estimates inform grant allocations in practice. I'd love to see some real examples.
How much is reducing catastrophic and extinction risk worth, assuming XPT forecasts?
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
This is a post I drafted some months ago, in the course of analysing some XPT data and reading Shulman and Thornley. It’s not very sophisticated, I haven’t checked the workings, and I haven’t polished the language; but I’m posting anyway because that seems better than not posting. Note that it’s a personal take and doesn’t represent FRI’s views.
Thanks to Josh Rosenberg at FRI and Elliot Thornley for help and comments.
BLUF: if you make a bunch of assumptions, then even quite low absolute risk forecasts like the XPT ones imply quite high spending on reducing GCRs, conditional on there being sufficiently cost-effective ways to do so.
In 2022, what has become the Forecasting Research Institute ran the Existential Risk Persuasion Tournament (XPT). Over 200 forecasters, including superforecasters and domain experts, spent 4 months making forecasts on various questions related to existential and catastrophic risk.
You can see the results from the tournament overall here, and a discussion of the XPT AI risk forecasts in particular here.
These are the main XPT forecasts on catastrophic and extinction risk:
If we take these numbers at face value, how much is catastrophic and extinction risk reduction worth?
One approach is to take the XPT forecasts, convert them into deaths in expectation, then assume a value of a statistical life and a discount rate, and estimate how much averting those deaths is ‘worth’. (I’m stealing this method directly from Shulman and Thornley.)
Using the XPT superforecasts and OWID population projections gives us the following deaths in expectation, in millions:
For catastrophic risks, the deaths in expectation should be read as a lower bound, because they assume 10% deaths and the question includes scenarios with >10% deaths.
That’s deaths in expectation worldwide. But the value of a statistical life varies by country: governments have different resources and the cost of interventions in different places varies.
So the most straightforward way to think about the worth of catastrophic and extinction risk reduction is to ask how much this would be worth in a given country. Let’s take the US as an example.
We can then assume a US value for a statistical life, and a discount rate, and use these to estimate how much averting the deaths in expectation is ‘worth’ to the US government.
The primary VSL figure used by the U.S. Department of Transportation for 2021 is $11.8 million, with a range to account for various kinds of uncertainty spanning from about $7 million to $16.5 million.[9]
The Environmental Protection Agency (EPA) uses annual discount rates of 2% and 3%; the Office of Information and Regulatory Affairs (OIRA) instructs agencies to conduct analyses using annual discount rates of 3% and 7%.[10]
It’s a widely held position that 7% is too high a discount rate,[11] so let’s go with 3% as it’s closer to consensus.[12]
Assuming $7m as the value of a statistical life, and a 3% annual discount rate, the value to the US government of reducing total initial risks by 1% (not one percentage point)[13] would be as follows:
Value of a 1% reduction in risk, assuming VSL at $7m and discount rate at 3% (billions of dollars)
There are a few reasons to expect these numbers to underestimate the value of catastrophic and extinction risk reduction:
When it comes to extinction risk, these numbers only concern deaths in expectation, not lives foregone in expectation
These numbers assume that all of the deaths averted are averted at the very end of the period in question (i.e. maximum discounting)
Note that this means that it doesn’t make sense to sum the figures across rows: that would mean double counting deaths averted but at different discount rates
As noted above, the estimated deaths in expectation for catastrophic risks should be read as lower bounds. The corresponding dollar values are therefore also lower bounds
What if I only care about the next few decades?
Suppose I don’t take extinction risk seriously, but I am interested in the XPT forecasts on catastrophic risks. That said, I think that 2030 is so soon that catastrophe seems extremely unlikely, and I don’t care much about things as far out as 2100, partly because I’m sceptical that we can influence things on that timescale, and partly because I only care about current lives. I want to know how much reducing catastrophic risk by 1% by 2050 would be worth, assuming $7m VSL and a 3% discount rate.
That would give me something like this:
Catastrophic risk
Total value of 1% risk reduction by 2050, millions* of dollar
Annual value, millions* of dollars**
AI
$8,400
$299.4
Nuclear
$21,000
$750.6
Non-anthropogenic
$170
$6.2
Total
$44,000
$1,579.2
* Note that this table displays millions of dollars, and the previous tables displayed billions.
**This is just a naive division of the total by 28 (the XPT tournament took place in 2022). Workings here.
How does this compare to current annual spending on these risks? There isn’t good data here, but to give some ballpark ideas:
Ord estimated global spending on reducing AI risk at $10-50m in 2020.
The US spent $5,200m on nuclear threat reduction in 2008 (about $7,170m in 2023 dollars).
NASA spent $150m on its near-earth objects program in 2019.
Note that it’s unclear what level of risk reduction those figures correspond to, so it’s not clear what the direct comparison should be between current total spending and the value of a 1% risk reduction.
What about the value of catastrophic and extinction risk reduction worldwide?
Most of the people potentially affected by catastrophic and extinction risks aren’t US citizens. Can we say anything about how much catastrophic and extinction risk reduction is worth globally, using the VSL method?
Not very accurately, but it might be interesting to have a go anyway.
The problems with extrapolating this method worldwide are:
The figure we used for US VSL was $7m. In many places in the world, the value of a statistical life is much lower: there are fewer resources, and interventions are cheaper.
In the US case, it’s clear to say that using VSL, reducing catastrophic and extinction risk by 1% is worth so much to the US government. It’s not clear which actor this reduction is worth something to in the global case.
That said:
Usually the US government isn’t willing to pay the same amount for the lives of its citizens and those of other countries, but there are exceptions. For example, the US social cost of carbon number counts costs to non-US citizens in the same way as costs to US citizens.
We can ask how much catastrophic and extinction risk reduction would be worth to some hypothetical actor who put the same VSL on all lives that the US government puts on the lives of its citizens.
Value of a 1% reduction in risk, assuming VSL at $7m and discount rate at 3% (billions of dollars)
Given XPT estimates of risk and standard US CBA assumptions ($7m VSL, 3% discount rate):
A 1% reduction in total catastrophic risk by 2050 would be worth up to $44bn to the US government.
A 1% reduction in catastrophic nuclear risk by 2050 would be worth up to $21bn.
A 1% reduction in catastrophic AI risk by 2050 would be worth up to $8bn.
A 1% reduction in total extinction risk by 2050 would be worth up to $35bn to the US government.
To an actor willing to spend US VSL to save the lives of citizens of any country:
A 1% reduction in total catastrophic risk by 2050 would be worth up to $1.1tr.
A 1% reduction in total extinction risk by 2050 would be worth up to $891bn.
Interventions which reduce the relevant catastrophic or extinction risk by 1% by 2050 and cost less than those figures would plausibly be cost-effective for the US government to fund.
This depends on what cost-benefit ratio the US government (or more likely its constituent parts) are aiming for.
It seems pretty likely to me that such interventions exist.
If you make the assumptions above, theneven quite low absolute risk forecasts like the XPT ones imply quite high spending on reducing GCRs, conditional on there being sufficiently cost-effective ways to do so.
This row is the sum of the two following rows (catastrophic risk from engineered and from natural pathogens respectively). We did not directly ask for catastrophic biorisk forecasts.
Because of concerns among our funders about information hazards, we did not include this question in the main tournament, but we did ask about risks from engineered and natural pathogens in a one-shot separate postmortem survey to which most XPT participants responded after the tournament.
Because of concerns among our funders about information hazards, we did not include this question in the main tournament, but we did ask about risks from engineered and natural pathogens in a one-shot separate postmortem survey to which most XPT participants responded after the tournament.
This row is the sum of the two following rows (catastrophic risk from engineered and from natural pathogens respectively). We did not directly ask for catastrophic biorisk forecasts.
Because of concerns among our funders about information hazards, we did not include this question in the main tournament, but we did ask about risks from engineered and natural pathogens in a one-shot separate postmortem survey to which most XPT participants responded after the tournament.
Because of concerns among our funders about information hazards, we did not include this question in the main tournament, but we did ask about risks from engineered and natural pathogens in a one-shot separate postmortem survey to which most XPT participants responded after the tournament.
I think I buy that interventions which reduce either catastrophic or extinction risk by 1% for < $1 trillion exist. I'm less sure as to whether many of these interventions clear the 1,000x bar though, which (naively replacing US VSL = $7 mil with AMF's ~$5k) seems to imply 1% reduction for < $1 billion. (I recall Linch's comment being bullish and comfortable on interventions reducing x-risk ~0.01% at ~$100 mil, which could either be interpreted as ~100x i.e. in the ballpark of GiveDirectly's cash transfers, or as aggregating over a longer timescale than by 2050; the latter is probably the case. The other comments to that post offer a pretty wide range of values.)
That said, I've never actually seen a BOTEC justifying an actual x-risk grant (vs e.g. Open Phil's sample BOTECs for various grants with confidential details redacted), so my remarks above seem mostly immaterial to how x-risk cost-effectiveness estimates inform grant allocations in practice. I'd love to see some real examples.