A model about the effect of total existential risk on career choice

Which existential risk cause should you focus on? The cause where you have the largest impact on decreasing total existential risk. That's not the same as working on the cause where you have the largest impact when seen in isolation.

Model

Suppose there are $K$existential risks, each with its probability $p_k$ of ending the world. For each cause $k$ you can reduce the probability of the world ending from that cause by $d_k$, but only if you spend your whole career doing it.

For instance, suppose the risks are AI, biorisk, and asteroids. They have associated probabilities $p_1=0.9,p_2=0.10$ and $p_3=0.01$.^[1] How much could you decrease the probability of extinction for each cause? You're pretty good at deflecting asteroids and killing viruses escaping from labs, but not that good at making humans lovable for AIs. Your probabilities are, say, $d_3={10}^{-7}$, $d_2={10}^{-7}$, $d_3=5\cdot {10}^{-8}$.

Which career should you choose? It sounds plausible that you should be agnostic between the biorisk and asteroids path. That's where you'll reduce the probability of extinction the most, after all. But we should do a decision-theoretic analysis of the problem to make sure.

Let's use the utility function where the world survives has utility $1$ and the world ceases to exist has utility $0$. Let $a=(a_1,\dots ,a_K)$ be a $0-1$ vector with $a_k=1$ if you choose action $k$ and $0$ otherwise. Then you ought to solve the total utility maximization problem 

 Why? Because you don't care which event causes extinction, only that it doesn't happen. And the total probability of no extinction equals ${\prod }_{k=1}^K(1-p_k+d_k{a}_k)$.

Anyway, we can show that

1. the optimal action, i.e., career path, is the one with the highest $d_k/(1-p_k)$;
2. the multiplicative improvement you're causing by choosing action $k$ is $1+d_k/(1-p_k)$.

Proof

Define 

 The utility when taking action $k$ equals 

 which is clearly maximized in $k$ that maximises $d_k/(1-p_k)$.

Consequences

You need to take both the probability of extinction by cause $k$ and your ability to reduce the probability into account when you choose your career. If, for instance, the probability of AI ending the world ($p_1$) is higher than biorisk ending the world ($p_2$), you need to be at least $\frac{1-p_1}{1-p_2}$ times better at biorisk than AI risk (in terms of reducing the probability) to justify working on biorisk. If the probability of bio extinction is $0.01$ and the probability of AI extinction is $0.90$, you need to be $0.99/0.10=9.9\approx 10$ better at biorisk to justify doing biorisk instead of AI.

We can expand the table above to include the benefit of taking each action:

So, the AI safety career is better than the asteroid career. But not by a lot, as the number $(1+5\cdot {10}^{-7})/(1+{10}^{-7})$ is virtually indistinguishable from $1$. But of course, a higher number is a higher number, and they do add up. If we only care about the part $d_k/(1-p_k)$, which might be reasonable, doing the AI career is $5$ times better than the asteroids career. Which is more impressive.

A model with uncertainty

So, you say you have epistemic uncertainty about the probabilities of extinction from each cause? Perhaps you think your choice of entering a fiend may remove the risk entirely, not reduce it by a small number? (E.g., either you solve AI alignment, or you don't).

That turns out not to matter. For the problem doesn't change much when you allow for uncertainty. Provided $p_j,d_k$, $p_j,p_k$ and $d_j,d_k$ are independent when $j\ne k$ we find that

where $\Pi ={\prod }_{k=1}^{K}E(1-p_k).$ The problem is maximized in the action with the highest $E\left[d_k\right]/(1-E[p_k\left]\right).$

Footnotes

1. ^{^}

   These probabilities sum to more than $1$, but that doesn't matter for our purposes. Think about them as the probabilities of independent events and the event "the world ends" as an event that happens if at least one of them occurs.