Welcome, and thanks for posting!

"kills 99% of all humans...In each scenario, humanity almost certainly goes extinct."

I don't think that this is true for your examples, but rather that humanity would almost certainly not be directly extinguished or even prevented from recovering technology by a disaster that killed 99%. 1% of humans surviving is a population of 78 million, more than the Roman Empire, with knowledge of modern agricultural techniques like crop rotation or the moldboard plough, and vast supplies built by our civilization. For a dinosaur-killer asteroid, animals such as crocodiles and our own ancestors survived that, and our technological abilities to survive and recover are immensely greater (we have food reserves, could tap energy from the oceans and dead biomass, can produce food using non-solar energy sources, etc). So not only would human extinction be quite unlikely, but by that point nonhuman extinctions would be very thorough.

For a pandemic, an immune 1% (immunologically or behaviorally immune) rebuild civilization. If we stipulate a disease that killed all humans but generally didn't affect other taxa, then chimpanzees (with whom we have a common ancestor only millions of years ago) are well positioned to take the same course again, as well as more distant relatives if primates perished, so I buy a relatively high credence in intelligence life reemerging there.

Welcome to the forum!

Further development of a mathematical model to realise how important timelines for re-evolution are.

Re-evolution timelines have another interesting effect on overall risk — all else equal, the more confident one is that intelligence will re-evolve, the more confident one should be that we will be able to build AGI,* which should increase one’s estimate of existential risk from AI.

So it seems that AI risk gets a twofold ‘boost’ from evidence for a speedy re-emergence of intelligent life:

• Relative AI risk increases, since risk from most other sources is discounted a bit.
• Absolute AI risk increases, since it pushes towards shorter AGI timelines.

*Shulman & Bostrom 2012 discuss this type of argument, and some complexities in adjusting for observation selection effects

Welcome to the Forum (or to posting)! I found this post interesting, and think it makes important points.

There's been a bit of prior discussion of these sorts of ideas (though I think the sort of model you propose is interesting and hasn't been proposed before). For example:

• Existential risks are not just about humanity
• The expected value of extinction risk reduction is positive (particularly the section "Whether (post-)humans colonizing space is good or bad, space colonization by other agents seems worse")
• How Would Catastrophic Risks Affect Prospects for Compromise? (particularly the section "Might humans be replaced by other species?")
• These comments

I've collected these sources because I'm working on a post/series about "crucial questions for longtermists". One of the questions I have in there is "What's the counterfactual to a human-influenced future?" I break this down into:

• How likely is future evolution of moral agents or patients on Earth, conditional on existential catastrophe? How valuable would that future be?
• How likely is it that our observable universe contains extraterrestrial intelligence (ETI)? How valuable would a future influenced by them rather than us be?

I think your post does a good job highlighting the importance of the question "How likely is future evolution of moral agents or patients on Earth, conditional on existential catastrophe?"

But it seemed like you were implicitly assuming that what other moral agents would ultimately do with the future would be equally valuable in expectation to what humanity would do? This seems a big question to me, and probably depends somewhat on metaethics (e.g., moral realism vs antirealism). From memory, there's some good discussion of this in "The expected value of extinction risk reduction is positive". (This is also related to Azure's comment.)

And I feel like these questions might be best addressed alongside the question about ETI. One reason for that is that discussions of the Fermi Paradox, Drake Equation, and Great Filter (see, e.g., this paper) could perhaps inform our beliefs about the likelihood of both ETI and future evolution of moral agents on Earth.

Am I reading the 0.1% probability for nuclear war right as the probability that nuclear war breaks out at all, or the probability that it breaks out and leads to human extinction? If it's the former, this seems much too low. Consider that twice in history nuclear warfare was likely averted by the actions of a single person (e.g. Stanislav Petrov), and we have had several other close calls ( https://en.wikipedia.org/wiki/List_of_nuclear_close_calls ).

Thank you for this post! Very interesting.

(1) Is this a fair/unfair summary of the argument?

P1 We should be indifferent on anti-speciesist grounds whether humans or some other intelligence life form enjoy a grand future.

P2 The risk of extinction of only humans is strictly lower than the risk of extinction of humans + all future possible (non human) intelligent life form.

C Therefore we should revise downwards the value of avoiding the former/raise the value of the latter.

(2) Is knowledge about current evolutionary trajectories of non-human animals today likely to completely inform us about 're-evolution'? What are the relevant considerations?

Thanks for sharing this!

I happen to have made a not-very-good model a month or so ago to try to get a sense of how much the possibility of future species that care about x-risks impacts x-risk today. It's here, and it has a bunch of issues (like assuming that it will take the same amount of time from now for a new species that it took for humans to evolve since the first neuron, assuming that all of Ord's x-risks don't reduce the possibility of future moral agents evolving etc.), and possibly doesn't even get at the important things mentioned in this post.

But based on the relatively bad assumptions in it, it spat out that if we generally expect moral agents to evolve who reach Ord's 16% 100 year x-risk every 500 million years or so (assuming an existential event happens), and that most the value of the future is beyond the next 0.8 to 1.2B years, then we ought to adjust Ord's figure down to 9.8% to 12%.

I don't think either the figure / approach in that should be taken at all seriously though, as I spent only a couple minutes on it and didn't think at all about better ways to try to do this - just writing this explanation of it has shown me a lot of ways in which it is bad. It just seemed relevant to this post and I wasn't going to do anything else with it :).

We could survive by preserving data about humanity (on the Moon or other places), which will be found by the next civilisation on Earth, and they will recreate humans (based on our DNA) and our culture.

Another crucial consideration may be the timeline of intelligent life re-evolving. In scenario B), intelligent life may re-evolve but it may take 100 million years, as opposed to 1 million years in scenario

This is the biggest argument for me against the consideration. I can easily think that it would take way longer than that for intelligent life to reemerge. It took something like 4.6 billion years for us to evolve and in roughly 0.5 billion years the sun will make life on earth uninhabitable. I guess if other primates survive that is a "good" starting point for evolution but intelligent life doesn't seem to be a necessary step for me for survival.

Something that seems worth noting is that an existential catastrophe (or "human existential catastrophe") need involve human extinction, nor even "killing humans to the extent that human civilisation never achieves a grand future".

It could involve something like locking in a future that's better for all humans than the current world, with no "extra" human death involved (i.e., maybe people still die of old age but not in a sudden catastrophe), but with us now being blocked from ever creating anywhere near as much value as we could've. This might be a "desired dystopia", in Ord's terms. For example, we might forever limit ourselves to the Earth but somehow maintain it far beyond its "natural lifespan", or we might create vast suffering among non-humans.

I mention this here for two reasons:

• Such an existential catastrophe could involve humans still being around, and therefore affecting the probability and nature of future evolution of moral agents on Earth or the evolution and development of intelligent life elsewhere. This might be similar to the scenario you note where a misaligned AGI could prevent future evolution of intelligent life. But this is another pathway to that sort of outcome.
• And in this scenario, humanity might also even increase the chances of such evolution, for some reason, such as if we intentionally try to bring that about out of curiosity.
• I don't really know how this sort of thing would affect analyses, but it seems relevant.
• You write "For probabilities of human extinction from each event I use the probabilities given by Toby Ord in The Precipice." But Ord was estimating existential risk, not extinction risk. (Using Ord's estimates in this way just for illustration purposes seems fine, but it seems worth noting that that's not what he was estimating the odds of.)

Then for question 1, with A), we get a 0.7% chance of a true existential event, and with B) we get a 0.095% chance of a true existential event. So we still care more about A), but now by only ~7 times more.

I believe this should be ~70 times more.