[Draft] The humble cosmologist's P(doom) paradox

titotal

Cosmologist: Well, I’m a little uncomfortable with this, but I’ll give it a shot. I will tentatively say that the odds of doom are higher than 1 in a googol. But I don’t know the order of magnitude of the actual threat. To convey this:
I’ll give a 1% chance it’s between 10^-100 and 10^-99
A 1% chance it’s between 10^-99 and 10^-98
A 1% chance it’s between 10^-98 and 10^-97,
And so on, all the way up to a 1% chance it’s between 1 in 10 and 100%.

I think the root of the problem in this paradox is that this isn't a very defensible humble/uniform prior, and if the cosmologist were to think it through more they could come up with one that gives a lower p(doom) (or at least, doesn't look much like the distribution stated initially).

So, I agree with this as a criticism of pop-Bayes in the sense that people will often come up with a quick uniform-prior-sounding explanation for why some unlikely event has a probability that is around 1%, but I think the problem here is that the prior is wrong^[1] rather than failing to consider the whole distribution, seeing as a distribution over probabilities collapses to a single probability anyway.

Imo the deeper problem is how to generate the correct prior, which can be a problem due to "pop Bayes", but also remains when you try to do the actual Bayesian statistics.

Explanation of why I think this is quite an unnatural estimate in this case

Disclaimer: I too have no particular claim on being great at stats, so take this with a pinch of salt

The cosmologist is supposing a model where the universe as it exists is analogous to the result of a single Bernoulli trial, where the "yes" outcome is that the universe is a simulation that will be shut down. Writing this Bernoulli distribution as ^[2], they are then claiming uncertainty over the value of $γ$ . So far so uncontroversial.

They then propose to take the pdf over $γ$ to be:

p (γ) = {\begin{matrix} 0 & if γ < 10^{- 100} \frac{k}{γ} & for 10^{- 100} \leq γ \leq 1 \end{matrix} (A)

Where $k$ is a normalisation constant. This is the distribution that results in the property that each OOM has an equal probability^[3]. Questions about this:

Is this the appropriate non-informative prior?
Is this a situation where it's appropriate to appeal to a non-informative prior anyway?

Is this the appropriate non-informative prior?

I will tentatively say that the odds of doom are higher than 1 in a googol. But I don’t know the order of magnitude of the actual threat.

The basis on which the cosmologist chooses this model is an appeal to a kind of "total uncertainty"/non-informative-prior style reasoning, but:

They are inserting a concrete value of $x = 10^{- 100}$ as a lower bound
They are supposing the total uncertainty is over the order of magnitude of the probability, which is quite a specific choice

This results in a model where $E (γ) = - 1 / l n (x) = 1 / 100 l n (10) \approx 1 i n 230$ in this case, so the expected probability is very sensitive to this lower bound parameter, which is a red flag for a model that is supposed to represent total uncertainty.

There is apparently a generally accepted way to generate non-informative-priors for parameters in statistical models, which is to use a Jeffreys prior. The Jeffreys prior^[4] for the Bernoulli distribution is:

p (γ) = \frac{1}{\sqrt{γ (1 - γ)}} (B)

This doesn't look much like equation (A) that the cosmologist proposed. There are parameters where the Jeffreys prior is $\propto \frac{1}{x}$ , such as the standard deviation in the normal distribution, but these tend to be scale parameters that can range from 0 to $\infty$ . Using it for a probability does seem quite unnatural when you contrast it with these examples, because a probability has hard bounds at 0 and 1.

Is this a situation where it's appropriate to appeal to a non-informative prior anyway?

Using the recommended non-informative prior (B), we get that the expected probability is 0.5. Which makes sense for the class of problems concerned with something that either happens or doesn't, where we are totally uncertain about this.

I expect the cosmologist would take issue with this as well, and say "ok, I'm not that uncertain". Some reasons he would be right to take issue are:

A general prior that "out of the space of things that could be the case, most are not the case^[5]", this should update the probability towards 0. And in fact massively so, such that in the absence of any other evidence you should think the probability is vanishingly small, as you would for the question of "Is the universe riding on the back of a giant turtle?"
The reason to consider this simulation possibility in the first place, is not just that it is in principle allowed by the known laws of physics, but that there is a specific argument for why it should be the case. This should update the probability away from 0

The real problem the cosmologist has is uncertainty in how to incorporate the evidence of (2) into a probability (distribution). Clearly they think there is enough to the argument to not immediately reject it out of hand, or they would put it in the same category as the turtle-universe, but they are uncertain about how strong the argument actually is and therefore how much it should update their default-low prior.

...

I think this deeper problem gets related to the idea of non-informative priors in Bayesian statistics via a kind of linguistic collision.

Non-informative priors are about having a model which you have not yet updated based on evidence, so you are "maximally uncertain" about the parameters. In the case of having evidence only in the form of a clever argument, you might think "well I'm very uncertain about how to turn this into a probability, and the thing you do when you're very uncertain is use a non-informative prior". You might therefore come up with a model where the parameters have the kind of neat symmetry-based uncertainty that you tend to see in non-informative priors (as the cosmologist did in your example).

I think these cases are quite different though, arguably close to being opposites. In the second (the case of having evidence only in the form of a clever argument), the problem is not a lack of information, but that the information doesn't come in the form of observations of random variables. It's therefore hard to come up with a likelihood function based on this evidence, and so I don't have a good recommendation for what the cosmologist should say instead. But I think the original problem of how they end up with a 1 in 230 probability is due to a failed attempt to avoid this by appealing to an non-informative prior over order of magnitude.

^{^}
There is also a meta problem where the prior will tend to be too high rather than too low, because probabilities can't go below zero, and this leads to people on average being overly spooked by low probability events
^{^}
$γ$ being the "true probability". I'm using $γ$ rather than p because 1) in general parameters of probability distributions don't need to be probabilities themselves, e.g. the mean of a normal distribution, 2) $γ$ is a random variable in this case, so talking about the probability of p taking a certain value could be confusing 3) it's what is used in the linked Wikipedia article on Jeffreys priors
^{^}
$\int_{a}^{10 a} \frac{k}{γ} d γ = k ln (10) = c o n s t a n t$

^{^}

There is some controversy about whether this the right prior to use, but whatever the right one is it would give $E (γ) = 0.5$

^{^}

For some things you can make a mutual exclusivity + uncertainty argument for why the probability should be low. E.g. for the case of the universe riding on the back of the turtle you could consider all the other types of animals it could be riding on the back of, and point out that you have no particular reason to prefer a turtle. For the simulation argument and various other cases it's trickier because they might be consistent with lots of other things, but you can still appeal to Occam's razor and/or viewing this as an empirical fact about the universe

rocks

you are saying that if there are n possibilities and we have no evidence then probability is 1/n rather than 1/2. But for these topics there is an infinite set of discrete possibilities. Since the set is infinite it doesn't make sense to have uniform prior because there is no way for the probabilities to add up to 1.

harfe

So it also makes sense to say “I have a 95% credence that the “true” P(heads) is between X and Y”

I might also say "The 'true' P(heads) is either or $1$ ". If the coin comes up heads, the "'true' P(heads)" is $1$ , otherwise the "'true' P(heads)" is $0$ . Uncertainty exists in the map, not in the territory.

Okay, in in this specific context I can guess what you mean by the statement, but I think this is an inaccurate way to express yourself, and the previous “I have a 95% credence that the bias is between X and Y” is a more accurate way to express the same thing.

To explain, lets formalize things using probability theory. One way to formalize the above would be to use a random variable $B$ which describes the bias of the coin, and then you could say $P (X \leq B \leq Y) = 0.95 and P (heads | B = b) = b for all b \in [0, 1] .$ I think this would be a good way to formalize things. A wrong way to formalize things would be to say $P (X \leq P (heads) \leq Y) = 0.95.$ And people might think of the latter when you say “I have a 95% credence that the “true” P(heads) is between X and Y”. People might mistakenly think that "'true' P(heads)" is a real number, but what you actually mean is that it is a random variable.

So, can you have a probability of probabilities?

If you do things formally, I think you should avoid "probability of probabilities" statements. You can have a probability of events, but probabilities are real numbers, and in almost all useful formalizations, events are not in the domain of real numbers. Making sense of such a statement always requires some kind of interpretation (eg random variables that refer to biased coins, or other context), and I think it is better to avoid such statements. If sufficient context is provided, I can guess what you mean, but otherwise I cannot parse such statements meaningfully.

On a related note, a "median of probabilities" also does not make sense to me.

What about P(doom)?

It does make sense to model your uncertainty about doom by looking at different scenarios. I am not opposed to model P(doom) in more detail rather than just conveing a single number. If you have three scenarios, you can consider a random variable $S$ with values in ${1, 2, 3}$ which describes which scenario will happen. But in the end, the following formula is the correct way to calculate the probability of doom: $P (doom) = P (doom | S = 1) P (S = 1) + P (doom | S = 2) P (S = 2) + P (doom | S = 3) P (S = 3) .$

Talking about a "median $P (doom)$ " does not make sense in my opinion. Two people can have the same beliefs about the world, but if they model their beliefs with different but equivalent models, they can end up with different "median $P (doom)$ " (assuming a certain imo naive calculation of that median).

What about the cosmologist's belief in simulation shutdown?

As you might guess by now, I side with the journalist here. If we were to formalize the expressed beliefs of the cosmologist's using different scenarios (as above in the AI xrisk example), then the resulting $P (doom)$ would be what the journalist reports.

It is fine to say "I don't know" when asked for a probability estimate. But the beliefs of the cosmologist's look incoherent to me as soon as they enter the domain of probability theory.

I suspect it feels like a paradox because he gives way higher estimates for simulation shutdown than he actually believes, in particular the 1% slice where doom is between 10% and 100%.

Owen Cotton-Barratt

I think in this context the natural way to interpret probabilities (or medians) of probabilities is as something like: "what would I think the probability was if I knew a lot more about the situation and got to think about it a lot more, but wasn't literally omniscient". Of course that isn't fully defined (since it's not specified exactly how much more they get to know), but I think it's approximately meaningful, and can capture something quite useful/important about credal resilience.

Relatedly, I think the journalist's presentation is misleading. I think it makes it sound like there's something like an objective 1-in-230 chance of catastrophe. I think a more accurate presentation would be "expert thinks there's a 1% chance we live in a world with a high risk". This changes the appropriate policy implications: if taken at face value they both suggest there's something important to do, but in the latter case the action is less like "work out what to do about simulation shutdown" and more like "start a serious research programme to work out how likely it is we're in the fraction of worlds that was offhand labelled as 1%".

(I also think that you may be right that the cosmologist didn't really believe their numbers. I think that issue alone would be enough reason to be suspicious of the journalist's approach. They might reasonably suspect that such off-the-cuff answers might not withstand scrutiny, so it seems bad to report them in a way that could sound like considered expert opinion.)

Vasco Grilo🔸

Hi titotal,

I think it makes sense to assess the annual risk of simulation shutdown based on the mean annual probability of simulation shutdown. However, I also believe the risk of simulation shutdown is much lower than the one guessed by the humble cosmologist.

The mean of a loguniform distribution ranging from a to 1 is -1/ln(a). If a = 10^-100, the risk is 0.434 % (= -1/ln(10^-100)). However, I assume there is no reason to set the minimum risk to 10^-100, so the cosmologist may actually have been overconfident. Since there is no obvious natural lower bound for the risk, because more or less by definition we do not have evidence about the simulators, I guess the lower bound can be arbitrarily close to 0. In this case, the mean of the loguniform distribution goes to 0 (= -1/(ln(0))), so it looks like the humblest view corresponds to 0 risk of simulation shutdown.

In addition, the probability of surviving an annual risk of simulation shutdown of 0.434 % (= 10^-5.44) over the estimated age of the universe of 13.8 billion years is only 10^-75,072,000,000 (= (10^-5.44)^(13.8*10^9)), which is basically 0. So the universe would have needed to be super super lucky in order to have survived for so long with such high risk. One can try to counter this argument saying there are selection effects. However, it would be super strange to have an annual risk of simulation shutdown of 0.434 % without any partial shutdowns, given that tail risk usually follows something like a power law^[1] without severe jumps in severity.

^{^}
Although I think tail risk often decays faster than suggested by a power law.

SummaryBot

Executive summary: Bayesian probability distributions, rather than single point estimates, should be used when reasoning about highly uncertain risks like existential threats from AI, but this approach has some counterintuitive and potentially problematic implications.

Key points:

In real-world Bayesian reasoning, both priors and likelihood ratios are often uncertain, so probability distributions should be used instead of single numbers.
This approach can be applied to estimating the probability of existential risks like AI doom scenarios, by assigning probabilities to different models of AI development.
However, taking the expected value over a wide probability distribution can lead to counterintuitive results, like a "humble cosmologist" being interpreted as assigning a relatively high probability to simulation shutdown risk.
Naively using expected value in decision making could lead to taking drastic actions based on highly speculative risks.
This approach may privilege unfalsifiable hypotheses and seems to rate speculative risks as more probable than risks grounded in empirical evidence.
The author is uncertain about the full implications and potential solutions, but remains wary of relying on single-number probability estimates for existential risks.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

[Draft] The humble cosmologist's P(doom) paradox

[Draft] The humble cosmologist's P(doom) paradox

Part 1: Bayesian distributions

Part 2: the humble cosmologist paradox

Part 3: the dangers of single numbers:

Conclusion

Explanation of why I think this is quite an unnatural estimate in this case