Is It So Much to Ask for a Nice Reliable Aggregated X-Risk Forecast?

On most questions about the future, I don't hold a strong view. I read the aggregate prediction of forecasters on Metaculus or Manifold Markets and then I pretty much believe whatever it says.

Various attempts have been made to forecast existential risk. I would like to be able to form views based on those forecasts—especially on non-AI x-risks, because I barely know anything about synthetic biology or nuclear winter or catastrophic climate change. Unfortunately, none of the aggregate forecasts look reliable.

First, some general notes about forecasting distant^[1] and low-probability^[2] events:

• According to a literature review by Luke Muehlhauser, we don't have good data on long-range forecasters, and we don't know if people with short-range forecasting skill can make good forecasts over long ranges.
• According to an analysis by niplav, Metaculus predictions become less accurate as the duration gets longer.

So we have good reason to doubt the ability of forecasters to predict existential risk, even when they are known to make accurate forecasts on near-term outcomes such as elections.

Now let's look at what attempts have been made to forecast x-risk, and why I don't find any of them satisfying.

The most rigorous attempt at an aggregate forecast comes from the Existential Risk Persuasion Tournament. The tournament brought in superforecasters and domain experts to make predictions, then had them attempt to persuade each other and make predictions again.

In the end, domain experts forecasted extinction as an order of magnitude more likely than what the superforecasters believed.

And even the domain experts forecasted only a 3% chance of AI extinction. My number is much higher than that, and I notice myself not changing my beliefs after reading this.

Scott Alexander wrote about the tournament:

Confronted with the fact that domain experts/superforecasters had different estimates than they did, superforecasters/domain experts refused to update, and ended an order of magnitude away from each other. That seems like an endorsement of non-updating from superforecasters and domain experts! And who am I to disagree with such luminaries?

Peter McCluskey, who participated in the tournament as a superforecaster, wrote a personal account. His experience aligns with my (biased?) assumption that the people reporting very low P(doom) numbers just don't understand the AI alignment problem.

Okay, the lesson from the X-Risk Persuasion Tournament is that it's not clear whether we can learn anything from it.

What about Metaculus?

Metaculus has several relevant forecasts, but they seem to contradict each other. Some example forecasts:

1. Will humans go extinct before 2100? 0.3% chance. (This is the Metaculus question with the most activity.)
2. Ragnarok question series: Implied 12.16% chance (community prediction) or 3.66% chance (Metaculus prediction)^[3] of a >95% decline in population by 2100.
3. How does the level of existential risk posed by AGI depend on its arrival time? Answers range from 50% to 9.3% depending on date range, which is maybe consistent with question 2 above, but definitely not consistent with question 1.

In a comment, Linch provides some reasons to be suspicious of Metaculus' estimates.

• There's no incentive to do well on those questions.
• The feedback loops are horrible
• Indeed, some people have actually joked betting low on the more existential questions since they won't get a score if we're all dead (at least, I hope they're joking)
• At the object-level, I just think people are really poorly calibrated about x-risk questions
• My comment here arguably changed the community's estimates by ~10%

In 2008, the Future of Humanity Institute ran a Global Catastrophic Risks Survey asking conference participants to give forecasts. The aggregated results look more reasonable than Metaculus or the Existential Risk Persuasion Tournament. But a lot has changed since 2008, so I don't think I can regard them as up-to-date estimates.

For forecasting AI risk, there is a 2023 survey of AI experts (see section 4.3). Survey results suggest the experts aren't thinking carefully—small changes in wording produced vastly different responses.

For example, respondents predicted AI to be able to match humans on all tasks by a median date of 2047, but predicted that AI would not be able to fully automate human labor until 2116.

Or look at the answers to these two questions:

What probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species?

• median: 5%
• mean: 16.2%

What probability do you put on human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species?

• median: 10%
• mean: 19.4%

By my reading, the latter outcome is a strict subset of the former, so the probability must be lower. But instead it's higher.

So we have these various aggregate forecasts, all of which seem suspect, and some of which disagree with each other by more than 10x. I really wish there was a canonical aggregate forecast I could rely on, in the same way that I can rely on Metaculus to predict election outcomes. But I don't think that exists.

At this point, I trust my own x-risk estimates more than any of those aggregate forecasts. My views happen to line up decently well with some of the aggregate forecasts, but only by chance. I feel better about Toby Ord's existential risk estimates than about any of the forecasting platforms or expert surveys.

And just because it feels unfair for me to spend all this time talking about forecasts and then not give any forecasts, here are my (poorly-thought-out, weakly-endorsed) probabilities of existential catastrophe^[4] by 2100: