This post was written for Convergence Analysis, though the opinions expressed are my own.
- Provides a spreadsheet you can use for making your own estimates of existential risks (or of similarly “extreme” outcomes)
- Announces a database of estimates of existential risk (or similarly extreme outcomes), which I hope can be collaboratively expanded and updated
- Discusses why I think this database may be valuable
- Discusses some pros and cons of using or making such estimates
Key links relevant to existential risk estimates
Here’s a spreadsheet listing some key existential-risk-related things people have estimated, without estimates in it.
- The link makes a copy of the spreadsheet so that you can add your own estimates to it.
- I mention this first so that you have the option of providing somewhat independent estimates, before looking at (more?) estimates from others.
- Some discussion of good techniques for forecasting, which may or may not apply to such long-range and extreme-outcome forecasts, can be found here, here, here, here, and here.
Here’s a database of all estimates of existential risks, or similarly extreme outcomes (e.g., reduction in the expected value of the long-term future), which I’m aware of. I intend to add to it over time, and hope readers suggest additions as well.
Here's an EAGx talk I gave that's basically a better structured version of this post (as I've now thought about this topic more), though without the links. So you may wish to watch that, and then just skim this post for links.
The appendix of this article by Beard et al. is where I got many of the estimates from, and it provides more detail on the context and methodologies of those estimates than I do in the database.
Beard et al. also critically discuss the various methodologies by which existential-risk-relevant estimates have been or could be derived. And Baum provides additional excellent commentary.
In this post, I discuss some pros and cons of using or stating explicit probabilities in general.
Why this database may be valuable
I’d bet that the majority of people reading this sentence have, at some point, seen one or more estimates of extinction risk by the year 2100, from one particular source. These estimates may in fact have played a role in major decisions of yours; I believe they played a role in my own career transition. That source is an informal survey of global catastrophic risk researchers, from 2008.
As Millett and Snyder-Beattie note:
The disadvantage [of that survey] is that the estimates were likely highly subjective and unreliable, especially as the survey did not account for response bias, and the respondents were not calibrated beforehand.
Additionally, in any case, it was just one informal survey, and is now 12 years old. So why is it so frequently referenced? And why has it plausibly (in my view) influenced so many people?
I also expect that essentially the same pattern is likely to repeat, perhaps for another dozen years, but now with Toby Ord’s recent existential risk estimates. Why do I expect this?
I originally thought the answer to each of these questions was essentially that we have so little else to go on, and the topic is so important. It seemed to me there had just been so few attempts to actually estimate existential risks or similarly extreme outcomes (e.g., extinction risk, reduction in the expected value of the long-term future).
I’d argue that this causes two problems:
- We have less information to inform decisions such as whether to prioritise longtermism over other cause areas, whether to prioritise existential risk reduction over other longtermist strategies, and especially which existential risks to be most concerned about.
- We may anchor too strongly on the very sparse set of estimates we are aware of, and get caught in information cascades.
Indeed, it seems to me probably not ideal how many strategic decisions people concerned about existential risks (myself included) have made so far without having first collected and critiqued a wide array of such estimates.
This second issue seems all the more concerning given the many reasons we have for skepticism about estimates on these matters, such as:
- The lack of directly relevant data (e.g., prior human extinction events) to inform the estimates
- Estimates often being little more than quick guesses
- It often being hard to interpret what is actually being estimated
- The possibility of response bias
- The estimators typically being people who are especially concerned about existential risks, and thus arguably being essentially “selected for” above average pessimism
- There's limited evidence about how trustworthy long-range forecasts in general are, and some evidence of expert forecasts often being unreliable even for events just a few years out (e.g., from Tetlock)
Thus, we may be anchoring on a small handful of estimates which could in fact warrant little trust.
But what else are we to do? One option would be to each form our own, independent estimates. This may be ideal, which is why I opened by linking to a spreadsheet you can duplicate to do that in.
But it seems plausible that “expert” estimates of these long-range, extreme outcomes would correlate better with what the future actually has in store for us than the estimates most of us could come up with, at least without investing a great deal of time into it. (Though I’m certainly not sure of that, and I don’t know of any directly relevant evidence. Plus, Tetlock’s work may provide some reason to doubt the idea that “experts” can be assumed to make better forecasts in general.)
And in any case, most of us have probably already been exposed to a handful of relevant estimates, making it harder to generate “independent” estimates of our own.
So I decided to collect all quantitative estimates I could find of existential risks or other similarly “extreme outcomes”, as this might:
- Be useful for decision-making, because I do think these estimates are likely better than nothing or than just phrases like “plausible”, plausibly by a very large margin.
- Reduce a relatively “blind” reliance on any particular source of estimates, and perhaps even make it easier for people to produce effectively somewhat independent estimates. My hope is that providing a variety of different estimates will throw anchors in many directions (so to speak) in a way that somewhat “cancels out”, and that it’ll also highlight the occasionally major discrepancies between different estimates. I do not want you to take these estimates as gospel.
Above, I said “I originally thought the answer to each of these questions was essentially that we have so little else to go on, and the topic is so important.” I still think that’s not far off the truth. But since starting this database, I discovered Beard et al.’s appendix, which has a variety of other relevant estimates. And I wondered whether that meant this database wouldn’t be useful.
But then I realised that many or most of the relevant estimates from that appendix weren’t mentioned in Ord’s book. And it’s still the case that I’ve seen the 2008 survey estimates many times, and never seen most of the estimates in Beard et al.’s appendix (see this post for an indication that I’m not alone in that). And Beard et al. don’t mention Ord’s estimates (perhaps because it was released only a couple months before Ord’s book was), nor a variety of other estimates I found (perhaps because they’re from “informal sources” like 80,000 Hours articles).
So I still think collecting all relevant estimates in one place may be useful. And another benefit is that this database can be updated over time, including via suggestions from readers, so it can hopefully become comprehensive and stay up to date.
So please feel free to:
- Make your own estimates
- Take a look through the database as and when you’d find that useful
- Comment in there regarding estimates or details that could be added, corrections if I made any mistakes, etc.
- Comment here regarding whether my rationale for this database makes sense, regarding pros, cons, and best practice for existential risk estimates in general, etc.
This post was sort-of inspired by one sentence in a comment by MichaelStJules, so my thanks to him for that. My thanks also to David Kristoffersson for helpful feedback and suggestions on this post, and to Ofer Givoli, Juan García Martínez, and Michael Dickens for comments or suggestions on the database itself.
See also my thoughts on Toby Ord’s existential risk estimates, which are featured in the database.
For example, it’s referred to in these three 80,000 Hours articles. ↩︎
Additionally, Beard et al.’s appendix says the survey had only 13 participants. The original report of the survey’s results doesn’t mention the number of participants. ↩︎
If I wrote this post again, I would probably include brief discussion somewhere of hedge drift, and perhaps memetic downside risk. I think one downside of this database is that it could induce those things.
Quoting from the first link, hedge drift refers to cases where:
Relatedly and somewhat unfortunately, this database draws more attention to the numbers people gave than to the precise phrasing of their statements and (especially) the context surrounding those statements.
Since I'm now linking to this post decently often, I'll mention a couple more sources that are somewhat relevant to the pros and cons of explicit, quantitative estimates, and best practices for making them. (But note that these sources aren't about existential risk estimates in particular.)
I'll probably add to this list over time. And see also the various sources linked to in this post itself.
Your estimates are presented as numerical values similar to probabilities. Is it actually probabilities and if yes, are they frequentist probabilities or Bayesian? And more generally: How we can define the "probability of end of the world"?
I believe that all the numbers I've shown were probabilities. I'm pretty sure they were always presented in the original source as percentages, decimals, or "1 in [number]".
What was the alternative you had in mind? I've seen some related estimates presented like "This can be expected to happen every x years", or "There's an x year return rate". Is that what you were thinking of? Several such estimates are given in Beard et al.'s appendix. But I don't think any are in the database. That wasn't primarily because they not quite probabilities (or not quite the right type), but rather because they were typically of things like the chance of an asteroid impact of a certain size, rather than direct estimates of the chance of existential catastrophe. (It's possible the asteroid impact wouldn't cause such a catastrophe.)
As for whether the probabilities are frequentist or Bayesian, I think many sources weren't explicit about that. But I generally assumed they were meant as Bayesian, though they might have been based on frequentist probabilities. E.g., Toby Ord's estimates of natural risks seems to be based on the frequency in the past, but then they're explicitly about what'll happen in the next 100 years, and they're modified based on knowledge of whether our other knowledge suggests this period is more or less likely than average to have e.g. an asteroid impact. But to be certain how a given number was meant to be interpreted, one might have to check the original source (which I provide links or references to in the database).
Yes, natural catastrophes probabilities could be presented as frequentist probabilities, but some estimates are based on logical uncertainty of the claims like "AGI is possible".
Also, are these probabilities conditioned on "all possible prevention measures are taken"? If yes, they are final probabilities which can't be made lower.
In the main sheet, the estimates are all unconditional (unless I made mistakes). They're just people's estimates of the probabilities that things will actually occur. There's a separate sheet for conditional estimates.
So presumably people's estimates of the chances these catastrophes occur would be lower conditional on people put in unexpectedly much effort to solve the problems.
Also, here's a relevant quote from The Precipice, which helps contextualise Ord's estimates. He writes that his estimates already:
I don't have time to write a detailed self-review, but I can say that:
You may very well already be aware of this (I didn't look at the post or comments too closely), but Elicit IDE has a "Search Metaforecast database" tool to search forecasts on several sites that may be helpful to your existential risk forecast database project. Here are the first 120 results for "existential risk."
Thanks for sharing that!
From a very quick look, I think that most of the things in that spreadsheet are either already in this database, only somewhat relevant (e.g., general AI capabilities questions), or hardly relevant at all. But I wouldn't be surprised if some fiddling around with that tool could narrow the results down to be more useful (I haven't tried the tool yet), or if it becomes more useful in future.
Thank you for putting this spreadsheet database together! This seemed like a non-trivial amount of work, and it's pretty useful to have it all in one place. Similar to other comments, seeing this spreadsheet really made me want more consistent questions and forecasting formats such that all these people can make comparable predictions, and also to see the breakdowns of how people are thinking about these forecasts (I'm very excited about people structuring their forecasts more into specific assumptions and evidence).
I thought the 2008 GCR questions were really interesting, and plotted the median estimates here. I was surprised by / interested in:
(I can't figure out how to make the image bigger, but you can click the link here to see the full snapshot)
Thanks for your comment!
Yes, I share that desire for more comparability in predictions and more breakdowns of what's informing one's predictions. Though I'd also highlight that the predictions are often not even very clear in what they're about, let alone very comparable or clear in what's informing them. So clarity in what's being predicted might be the issue I'd target first. (Or one could target multiple issues at once.)
And your comments on and graph of the 2008 GCR conference results are interesting. Are the units on the y axis percentage points? E.g., is that indicating something like a 2.5% chance superintelligent AI kills 1 or fewer people? I feel like I wouldn't want to extrapolate that from the predictions the GCR researchers made (which didn't include predictions for 1 or fewer deaths); I'd guess they'd put more probability mass on 0.
The probability of an interval is the area under the graph! Currently, it's set to 0% that any of the disaster scenarios kill < 1 people. I agree this is probably incorrect, but I didn't want to make any other assumptions about points they didn't specify. Here's a version that explicitly states that.
It's not obvious to me how to interpret this without specifying the units on the y axis (percentage points?), and when the x axis is logarithmic and in units of numbers of deaths. E.g., for the probability of superintelligent AI killing between 1 and 10 people, should I multiply ~2.5 (height along x axis) by ~10 (length along y axis) and get 25%? But then I'll often be multiplying the height along the x axis by more than 100 and getting insane probabilities?
So at the moment I can make sense of which events are seen as more likely than other ones, but not the absolute likelihood they're assigned.
I may be making some basic mistake. Also feel free to point me to a pre-written guide to interpreting Elicit graphs.
Also, your comment suggests you already saw my comment quoting Beard et al. on breaking down the thought process behind forecasts. You might also be interested in Baum's great paper replying to and building on Beard et al.'s paper. I also commented on Baum's paper here, and Beard et al. replied to it here.
Oh this looks really interesting, I'll check it out, thanks for linking!
I think this is an excellent initiative, thank you, Michael! (Disclaimer: Michael and I work together on Convergence.)
An assortment of thoughts:
I strongly agree about the value of breaking down the causes of one's estimates, and about estimates building on new sources of info being particularly interesting. And I tentatively agree with your other points.
Two things I'd add:
There is an author who has done a top down analysis using a modification of Gott's estimator . Prospects for Human Survival: Wells, Willard H. It is grounded in numerical methods. He also has an earlier treatment,Apocalypse When?: Calculating How Long the Human Race Will Survive.
His methods are more refined than any I have checked in your table.
An important new aggregate forecast, based on inputs from LW users:
Thanks for sharing this here.
For the benefit of other readers: These estimates are about the "Timeline until human-level AGI". The post Pablo linked to describes the methodology behind the estimates.
I don't think this really falls directly under the scope of the spreadsheet (in that this isn't directly an estimate of the chance of a catastrophe), so I haven't added it to the spreadsheet itself, but it's definitely relevant when considering x-risk/GCR estimates.
Do you know if there's a collection of AI timelines forecasts somewhere? I'd imagine there would be?
Yes, sorry, I should have made that clearer—I posted here because I couldn't think of a more appropriate thread, and I didn't want to create a separate post (though maybe I could have used the "shortform" feature).
AI Impacts has a bunch of posts on this topic. This one discusses what appears to be the largest dataset of such predictions in existence. This one lists several analyses of time to human-level AI. And this one provides an overview of all their writings on AI timelines.
I think sharing that link in a comment here made sense; it's definitely relevant in relation to this topic. I was just flagging that I didn't think it made sense to add it to the spreadsheet itself.
But I think a similar spreadsheet to collect AI timelines estimates seems valuable (though I haven't thought much about information/attention hazards considerations in relation to that). So thanks for linking to that AI Impacts post.
I've now sent AI Impacts a message asking whether they'd considered making it easy for people to comment on that dataset with links to additional predictions. So perhaps they'll do so, or perhaps they'll decide not to for good reasons.
Not in your database, I think:
William Poundstone, The Doomsday Calculation, p. 259
Thanks for mentioning this; it indeed wasn't in the database, and I've now added it.
Though I felt a bit hesitant about doing so, and haven't filled in the "What is their estimate?" column; I left this just as a quote.
We could interpret Tiper as saying "I now think there's a greater than 50% chance humans will be wiped out this century". But when people just say things like "I think X will happen" or "I think X won't happen", I don't feel super confident that they really meant "greater than 50%" or "less than 50%". For example, it seems plausible they just mean something like "I think we should pay [or less] more attention to the chance of X".
Also, when people don't give quantitative estimates, I feel a little less confident that they've thought for more than a few seconds about (a) precisely what they're predicting (e.g., does Tipler really mean extinction, or just the end of civilization as we know it?), and (b) how likely they really think the thing is. I don't feel super confident people have thought about (a) and (b) when they give quantitative predictions either, and indeed it's unclear what many predictions in the database are really about, and many of the predictors explicitly say things like "but this is quite a made up number and I'd probably say something different if you asked me again tomorrow". But I feel like people might tend to take their statements even less seriously when those statements remain fully qualitative.
(I could be wrong about all of that, though.)
Thanks for this database! I'm currently working on a project for the Foresight Centre (a think-tank at the Estonian parliament) about existential risks and the EU’s role in reducing them. I cover risks form AI, engineered pandemics, and climate change. For each risk, I discuss possible scenarios, probabilities, and the EUs role. I've found a couple of sources from your database on some of these risks that I hadn't seen before.
That sounds like a really valuable project! Glad to hear this database has been helpful in that.
Do you mean that those are the only existential risks the Foresight Centre's/Estonian Parliament's work will cover (rather than just that those are the ones you're covering personally, or the ones that are being covered at the moment)? If so, I'd be interested to hear a bit about why? (Only if you're at liberty to discuss that publicly, of course!)
I ask because, while I do think that AI and engineered pandemics are probably the two biggest existential risks, I also think that other risks are noteworthy and probably roughly on par with climate change. I have in mind in particular "unforeseen" risks/technologies, nuclear weapons, nanotechnology, stable and global authoritarianism, and natural pandemics. I'd guess that it'd be hard to get buy-in for discussing nanotechnology, authoritarianism, and maybe unforeseen risks/technologies in the sort of report you're writing, but not too hard to get buy-in for discussing nuclear weapons and natural pandemics.
Existential risks are not something they have worked on before, so my project is a new addition to their portfolio. I didn't mention this but I intend to have a section for other risks depending on space. The reason climate change gets prioritized in the project is that arguably the EU has more of a role to play in climate change initiatives compared to, say, nuclear risks.
I imagine having climate change feature prominently could also help get buy-in for the whole project, since lots of people already care about climate change and it could be highlighted to them that the same basic reasoning arguably suggests they should care about other existential risks as well.
FWIW, I'd guess that the EU could do quite a bit on nuclear risks. But I haven't thought about that question specifically very much yet, and I'd agree that they can tackle a larger fraction of the issue of climate change.
Interesting project! I especially like that there's a sheet for conditional estimates (one can argue that the non-conditional estimates are not directly decision-relevant).
Thanks for the feedback! And for your helpful suggestions in the spreadsheet itself.
Could you expand on what you mean by "one can argue that the non-conditional estimates are not directly decision-relevant"?
It does seem that, holding constant the accuracy of different estimates, the most decision-relevant estimates would be things like pairings of "What's the total x-risk this century, conditional on the EA community doing nothing about it" with "What's the total x-risk this century, conditional on a decent fraction of EAs dedicating their careers to x-risk reduction".
But it still seems like I'd want to make different decisions if I had reason to believe the risks from AI are 100 times larger than those from engineered pandemics, compared to if I believed the inverse. Obviously, that's not the whole story - it ignores at least tractability, neglectedness, and comparative advantage. But it does seem like a major part of the story.
Agreed, but I would argue that in this example, acquiring that belief is consequential because it makes you update towards the estimate: "conditioned on us not making more efforts to mitigate existential risks from AI, the probability of an AI related existential catastrophe is ...".
I might have been nitpicking here, but just to give a sense of why I think this issue might be relevant:
Suppose Alice, who is involved in EA, tells a random person Bob: "there's a 5% chance we'll all die due to X". What is Bob more likely to do next: (1) convince himself that Alice is a crackpot and that her estimate is nonsense; or (2) accept Alice's estimate, i.e. accept that there's a 5% chance that he and everyone he cares about will die due to X, no matter what he and the rest of humanity will do. (Because the 5% estimate already took into account everything that Bob and the rest of humanity will do about X.)
Now suppose instead that Alice tells Bob "If humanity won't take X seriously, there's a 10% chance we'll all die due to X." I suspect that in this scenario Bob is more likely to seriously think about X.
I think I see your reasoning. But I might frame things differently.
In your first scenario, if Alice sees Bob seems to be saying "Oh, well then I've just got to accept that there's a 5% chance", she can clarify "No, hold up! That 5% was my unconditional estimate. That's what I think is going to happen. But I don't know how much effort people are going to put in. And you and I can help decide what the truth is on that - we can contribute to changing how much effort is put in. My estimate conditional on humanity taking this somewhat more seriously than I currently expect is a 1% chance of catastrophe, and conditional on us taking it as seriously as I think we should, the chance is <0.1%. And we can help make that happen."
See also a relevant passage from The Precipice, which I quote here.
Also, I think I see the difference between unconditional and conditional on us doing more than currently expected as more significant than the difference between unconditional and conditional on us doing less than currently expected. For me, the unconditional estimate is relevant mainly because it updates me towards beliefs like "There are probably low-hanging fruit for reducing this risk", or "conditioned on us making more efforts to mitigate existential risks from AI than currently expected, the probability of an AI related existential catastrophe is [notably lower number]."
Sort-of relevant to the above: "If humanity won't take X seriously, there's a 10% chance we'll all die due to X" is consistent with "Even if humanity does take X really really seriously, there still a 10% chance we'll all die due to X." So in either case, you plausibly might need to clarify what you mean and what the alternative scenarios are, depending on how people react.
Maybe a crux here is what fraction of people in the role of Bob would instead convince themselves that the unconditional estimate is nonsense (due to motivated reasoning).
Another thought: If you do make your own existential risk estimates (whether or not you use the template I link to for that), feel free to "comment" a link to them in the main database. And if you do so, perhaps consider noting somewhere how you came up with those estimates, which prior estimates you believe you'd seen already, whether you had those in mind when you made your estimates, etc.
It seems to me that it could be valuable to pool together new estimates from the "general EA public" and track how "independent" they seem likely to be (though I'm unsure about how valuable that'd be).
I think this is basically what Metaculus already does.
(But the post seems good / useful.)
The relevant questions I'm aware of are this and these. So I guess I'd encourage people to answer those as well. Those questions are also linked to in the database already, so answers there would contribute to the pool here. And I believe people could comment there to explain their reasoning, what estimates they'd previously seen, etc.
There are more questions I'd be interested in estimates for which aren't on Metaculus at the moment (e.g., 100% rather than >95%, dystopia rather than other risks). But I imagine they could easily be put up by someone. (I'm not very familiar with the platform yet.)
There's currently an active thread on LessWrong for making forecasts of existential risk across the time period from now to 2120. Activity on that thread is already looking interesting, and I'm hoping more people add their forecasts, their reasoning, and questions/comments to probe and improve other people's reasoning.
I plan to later somehow add to this database those forecasts, an aggregate of them, and/or links.