Abstract. This article examines the position that cause areas related to existential risk reduction, such as AI safety, should be virtually infinitely preferred to other cause areas such as global poverty. I will explore some arguments for and against this position. My first goal is to raise greater awareness of the crucial importance of a particular assumption concerning the far future, which negates the possibility of long-term exponential growth of our utility function. I will also discuss the classical decision rule based on the maximization of expected values. My second goal is to question this assumption and this decision rule. In particular, I wonder whether exponential growth could be sustained through the exploration of increasingly complex patterns of matter; and whether, when attempting to maximize the expected values of different actions, we might forget to take into account possibly large costs caused by later updates on our beliefs on the far future. While I consider the ideas presented here to be highly speculative, my hope is to elicit a more thorough analysis of the arguments underlying the case for existential risk reduction.
A fictitious conversation on cause priority
The considerations below could be put into complicated-looking mathematical models involving integrals and probability measures. I will not follow this path, and focus instead on a handful of simple model cases. For convenience of exposition, there will be fictitious characters who are supposed to hold each model as their belief on the far future. These models will be very simple. In my opinion, nothing of value is being lost by proceeding in this way.
The characters in the fictitious story have a fantastic opportunity to do good: they are about to spend 0.1% of the world GDP on whatever they want. They will debate what to do in light of their beliefs on the far future. They agree on many things already: they are at least vaguely utilitarian and concerned about the far future; they believe that in exactly 100 years from now (unless they intervene), there will be a 10% chance that all sentient life suddenly goes extinct (and otherwise everything goes on just fine); outside of this event they believe that there will be no such existential risk; finally, they believe that sentient life must come to an end in 100 billion years. Also, we take it that all these beliefs are actually correct.
The characters in the story hesitate between a “growth” intervention, which they estimate would instantaneously raise their utility function by 1%, and an “existential” intervention, which would reduce the probability of extinction they will face in 100 years to 9.9% instead of 10%.
Alice believes that unless extinction occurs, our utility function always grows at a roughly constant rate, until everything stops in 100 billion years. She calculates that in her model, the growth intervention moves the utility function upwards by 1% at any point in the future. In particular, the expectation of the total utility created in the future increases by 1% if she chooses the growth intervention. With the existential intervention, she calculates that (up to a minuscule error) this expectation moves up by 0.1%. Since 1% > 0.1%, she argues for the growth intervention.
Bob’s view on the far future is different. He believes that the growth rate of our utility function will first accelerate, as we discover more and more technologies. In fact, it will accelerate so fast that we will quickly have discovered all discoverable technologies. We will similarly quickly figure out the best arrangement of matter to maximize our utility function locally, and all that will be left to do is colonize space and fill it with this optimal arrangement of matter. However, we are bound by the laws of physics and cannot colonize space faster than the speed of light. This implies that in the long run, our utility function cannot grow faster than t^3 (where t is time). The growth rate of this function decays to zero quickly, like 1/t. So in effect, we may as well suppose that our utility function will spike up quickly, and then plateau at a value that can essentially be regarded as a constant. For the existential intervention, he finds that the expected utility of the future increases by about 0.1%, in agreement with Alice’s assessment. However, he reaches a very different conclusion when evaluating the growth intervention. Indeed, in his model, the growth intervention only improves the fate of the future before the onset of the plateau, and brings this onset a bit closer to the present. In particular, it has essentially no effect on the utility function after the plateau is reached. But this is where the vast majority of the future resides. So the growth intervention will barely budge the total utility of the future. He therefore argues for the existential intervention.
Clara holds a more sophisticated model of the far future than both Alice and Bob. She acknowledges that we cannot be certain about our predictions. She holds that there is a range of different possible scenarios for the far future, to which she assigns certain probabilities. In fact, she puts a weight of 50% on Alice’s model, and a weight of 50% on Bob’s model. Her calculations will depend crucially on comparing the expected value of the total utility of the future under each model. She considers that the growth in utility in Alice’s model is slower than in Bob’s, so much so that the plateau that appears in Bob’s model is never within sight in Alice’s model. She thus concludes that the far future has much greater utility in Bob’s model. Or else, she reasons that Alice must have failed to properly take into account the slowdown appearing in Bob’s model. In any case, she rejoins Bob in arguing for the existential intervention.
In the next sections, we dig deeper into some of the arguments appearing in the preceding discussion.
A closer look at Bob’s arguments (no exponential growth)
As far as I can tell, some version of Bob’s view that our utility function ultimately reaches a plateau (or grows no faster than t^3) is the more typical view among EA people who have thought about the problem. I will focus now on the examination of this point.
This view relies on the assumption that we will quickly discover the essentially optimal way to organize matter in the region of space that we occupy. Once this is done, all that is left to do is to expand in space and reproduce this optimal arrangement of matter over and over again.
It seems extremely unlikely to me that we will come remotely close to discovering the utility-maximizing pattern of matter that can be formed even just here on Earth. There are about 10^50 atoms on Earth. In how many different ways could these atoms be organized in space? To keep things simple, suppose that we just want to delete some of them to form a more harmonious pattern, and otherwise do not move anything. Then there are already 2^(10^50) possible patterns for us to explore.
The number of atoms on our planet is already so large that it is many orders of magnitude beyond our intuitive grasp (in comparison, 100 billion years almost feels like you can touch it). So I’m not sure what to say to give a sense of scale for 2^(10^50); but let me give it a try. We can write down the number of atoms on Earth as a one followed by 50 zeros. If we try to write down 2^(10^50) similarly, then we would basically have to write a one followed by as many zeros as a third of the number of atoms on Earth.
Let me also stress that 2^(10^50) is in fact a very pessimistic lower bound on the number of different patterns that we can explore. Atoms are not all the same. They are made up of smaller parts that we can split up and play with separately. We are not restricted to using only the atoms on Earth, and we can move them further distances away. Also, I do not see why the optimizer of our utility function should be constant in time. In comparison to the potential number of patterns accessible to us, a time scale of 100 billion years is really, really, REALLY ridiculously short.
In order to rescue Bob’s argument, it seems necessary to make the case that, although the space of possible patterns is indeed huge, exploring this space of patterns has only a very limited impact on the growth of our utility function. It find it very difficult to decide whether this is true or not. One has to answer questions such as (1) how rapidly are we capable to explore this space of patterns? (2) Should we expect our speed of exploration to increase over time? If so, by how much? (3) How does our utility function increase as we keep improving the quality of the patterns we discover?
I do not know how to answer these questions. But perhaps it will help to broaden our imagination if I suggest a simple mental image of what it could look like for a civilization to be mostly busy trying to explore the space of patterns available to them. Possibly, our future selves will find that the greatest good will be achieved by preparing for and then realizing a coordinated dance performance of cosmic dimension, spanning a region greater than that of the solar system and lasting millions of years. While they will not completely disregard space colonization, they will find greater value in optimizing over the choreography of their cosmic dance, preparing for the success of their performance, and then realizing it. Generalizing on what I aim to capture with this example, I find it plausible that highly advanced sentient beings will be very interested in extremely refined and intricate doings comparable to art forms, which we cannot even begin to imagine, but which will score particularly highly for their utility function.
A somewhat Bayesian objection to the idea I am defending here could be: if indeed Bob’s view is invalid, then how come the point defended here has not already become more commonplace within the EA community? This is more tangential and speculative, so I will push a tentative answer to this question into a long footnote.
A closer look at Alice’s point of view (exponential growth)
Aside from Bob’s and Clara’s objections, another type of argument that can be raised against Alice’s view is that, somewhat implicitly, it may conflate the utility function with something that at least vaguely looks like the world GDP; and that in truth, if there were a simple relationship between the utility function and the world GDP, it would rather be that our utility function is the logarithm of the world GDP.
This argument would put Alice’s belief that our utility function can grow at a steady rate over long periods of time into very serious doubt. Under a default scenario where GDP growth is constant, it would mean that our utility function only grows by some fixed amount per unit of time.
It is difficult to argue about the relationship between a state of the world and what the value of our utility function should be. I will only point out that the argument is very fragile to the precise functional relation we postulate between our utility function and world GDP. Indeed, if we decide that our utility function is some (possibly small) power of the world GDP, instead of its logarithm, then a steady growth rate of GDP does again imply a steady growth rate of our utility function (as opposed to adding a constant amount per unit of time). If there were a relationship between our utility function and the world GDP, then I do not know how I could go about and decide whether our utility function looks more like log(GDP) or more like, say, (GDP)^0.1. If anything, postulating that our utility function looks like (GDP)^x for some exponent x between 0 and 1 gives us more freedom for adjustment between reality and our model of it. I feel that it would also work better under certain circumstances; for instance, if we duplicate our world and create an identical copy of it, I would find it bizarre if our utility function only increases by a constant amount, and find it more reasonable if it is multiplied by some factor.
Finally, I want to point out that Alice’s view is not at all sitting at the tail end of some spectrum of possible models. Indeed, I see no obstacle in the laws of physics to the idea that the growth rate of our utility function will not only remain positive, but will in fact continue to increase without bound for the next 100 billion years. Indeed, the space of possible patterns of matter we can potentially explore between time t and time 2 t grows like exp(t^4); and the growth rate of this function indeed goes up to infinity. If one takes the position that the growth rate of our utility function can increase without bound, then one would be led to the conclusion that growth interventions are always to be preferred over existential interventions.
A closer look at Clara’s arguments (expectation maximization)
The reasoning that leads Clara to conclude in favor of Bob’s conclusion is, in my opinion, very fragile. Although I do not think that it is necessary to do so, I find it clearest to explain this by supposing that Alice revises her model and says: “In fact I don’t know how our utility function will grow in the far future. The only thing I am certain about is that the growth rate of our utility function will always be at least 3% per year (outside of the possibility of extinction and of the interventions we do).” This is a weaker assumption on the future than her original model, so it should only increase the weight Clara is willing to put on Alice’s model. But with this formulation, whatever Bob’s prediction is for the future, Alice could say that maybe Bob is right up until the moment when he predicts a growth rate below 3%, but then she would insist on being more optimistic and keep the growth rate at (or above) 3%. In this way, Alice’s updated model is guaranteed to yield higher expected utility than Bob’s. Roughly speaking, Clara’s procedure essentially consists in selecting the most optimistic model around.
I suppose that a typical line of defense for the expectation-maximization procedure has something to do with the idea that it is, in some sense, “provably” best; in other words, that there is some mathematical reasoning justifying its superiority. I want to challenge this view here with two counter-arguments.
First, the classical argument for expectation maximization relies on the law of large numbers. This law deals with a series of relatively independent variables which we then sum up. It asserts that, in situations where each term contributes little to the overall sum, the total sum becomes concentrated around the sum of the expected values of each contribution, with comparatively small fluctuations. In such situations, it therefore makes sense to maximize the expected value of each of our actions. But, for all I know, there is only one universe in which we are taking bets on the long-term future. If, say, we never update our bet for the long-term future, then there will be no averaging taking place. In such a circumstance, maximizing expected values seems to me rather arbitrary, and I would see no contradiction if someone decided to optimize for some different quantity.
My most important objection to Clara’s reasoning is that, in my opinion, it fails to take into account certain effects which I will call “switching costs”. Although I explored the opposite hypothesis in the previous paragraph, I find it more likely that we will regularly update our prediction on the far future. And, in view of the scale of the uncertainties, I expect that our beliefs about it will actually mostly look like random noise. Finally, I expect that the standard expectation-maximization prescription will be all-or-nothing: only do growth interventions; or only do existential interventions. It seems to me that Clara’s calculation is too short-sighted, and fails to take into account the cost associated with revising our opinion in the future. To illustrate this, suppose that Alice, Bob and Clara run a charitable organization called ABCPhil which invests 0.1% of world GDP each year to do the maximal amount of good. Imagine if for the next 10 years, ABCPhil was only financing existential interventions; and then it would suddenly switch to only financing growth interventions for the next 10 years; and so on, completely switching every 10 years. Now, compare this with the scenario where ABCPhil finances both equally all the time. While this comparison is not straightforward, I would be inclined to believe that the second scenario is superior. In any case, my point here is that Clara’s reasoning, as stated, simply ignores this question, and this may be an important problem.
Tentative guidelines for concrete decision taking
In view of the discussion in the previous section, and in particular of the possible problem of “switching costs”, I do not believe that there can be a simple recipe that Clara could just apply to discover the optimal decision she should take. (We will take Clara’s point of view here since she is more reasonably acknowledging her uncertainty about her model of the future.) The best that can be done is to indicate a few guidelines for decision taking, which then need to be complemented by some amount of “good judgment”.
I find it most useful to think in terms of “reference points”, a small set of possible decisions that each are the result of a certain type of thinking. Once these reference points are identified, “good judgment” can then weigh in and bend the final decision more or less toward a reference point, depending on rough guesses as to the magnitude of the effects that are not captured well under each perspective.
One such reference point is indeed that resulting from the maximization of expected values (which is what Clara was doing in the original story). A second reference point, which I will call the “hedging” decision rule, is as follows. First, Clara calculates what is the best action under each model of the future; in fact, Alice and Bob have done this calculation for her already. Then, she aggregates the decisions according to the likelihood she places on each model. In other words, she gives money to Alice and Bob in proportion to how much she believes each is right, and then let them do what they think is best.
I want to stress again that I do not claim the hedging decision rule to be always superior to expectation maximization. However, it is also true that, in circumstances in which we believe that the switching costs are large, expectation maximization will lead to conclusions that are inferior to those derived from the hedging decision rule.
The hedging decision rule is designed to be more robust to switching costs. The task of “good judgment” then is to try to evaluate whether these costs (and possibly other considerations) are likely to be significant or not. If not, then one should deviate only very little from expectation maximization. If yes, then one should be more inclined to favor the hedging decision rule.
It is interesting to notice that it is only with Alice’s assumptions that one needs to actually look at the actual efficiency of each intervention, and that one could come up with a concrete rule for comparing them which is not all-or-nothing. While this has limitations, I find it very useful to have a concrete rule of thumb for comparing the efficiency of different interventions. In a final round of adjustment of Clara’s decisions, I believe that this additional information should also be taken into account. The extent of this final adjustment is again left to Clara’s “good judgment”.
How I came to write this
In this section, I want to explain what led me to write the present article. It comes from my attempt to understand the career recommendations given on the 80k website. The advice given there changed recently. In my opinion, the recent version strongly suggests that one should give much higher priority to careers related to existential risk reduction than to careers related to, say, improving health in poor countries.
Before spreading the word, I wanted to make sure that I understood and agreed with it. The present article is a summary of my best effort, and my conclusion is that I still don’t understand it.
In a nutshell, I am worried that switching costs have not been estimated properly. This can be because people at 80k feel more certain than I am about the trajectory of the far future; or because they think that switching costs are not very high. I have already discussed my opinion on the trajectory of the far future at length, so I will now only focus on the sort of switching costs I am worried about.
Suppose that I am very interested in improving health in poor countries; and that I am not all that convinced by relatively convoluted arguments about what will happen to sentient life in billions of years. Even if everyone in the EA community has the best intentions, I would personally find it depressing to be surrounded by people who think that what I intend to do is of negligible importance. I would also feel the pressure to switch to topics such as AI safety, an extremely competitive topic requiring a lot of expertise. I think I would be very likely to simply leave the group.
Imagine now that in 10 years, someone comes up with a great argument which suddenly convinces the EA community that growth interventions are actually vastly superior to existential interventions. If most people interested in growth interventions have left the group, it will be extremely difficult for the EA community to bear the transition. And next, at least some people working on AI safety would consider that the cost of switching is too high, and that working on AI safety still kind of makes sense. As time passes, and supposing that the EA community has managed to transition to growth interventions without just disintegrating, people working on AI safety would grow tired of being reminded that their work is of negligible importance, and would tend to leave the group. Up until the next switch of opinion.
Notice also that in the fictitious scenario outlined above, it will in fact be quite difficult for the “great argument” to emerge from the EA community, and then also very hard for it to be known and acknowledged, since people with different interests no longer interact. And I am not even discussing possible synergistic effects between EA people working on different cause areas, which in my opinion can also be very significant.
Conclusion
This article examined the view that interventions aiming to reduce existential risk are virtually infinitely superior to those that aim to accelerate growth.
In my understanding, this view relies crucially on the assumption that the utility of the future cannot have exponential growth in the long term, and will instead essentially reach a plateau. While I do not intend to rule out this possibility, I tried to explain why I personally find the alternative possibility of a sustained exponential growth at least plausible.
One attempt to aggregate these different predictions on the far future can be to compute the expected value of different interventions, taking our uncertainty on the far future into account. In my opinion, this approach has important limitations, in particular because it ignores certain “switching costs”.
The present article is a summary of my attempt to understand some of the ideas which I consider central to the EA movement. I suppose that the people working full-time on the problems discussed here have a much deeper understanding of the issues at stake, and a much finer position than the one I have outlined here. I hope that this article will signal that it may currently be very difficult to reverse-engineer what this finer position is. If nothing else, this article can thus be taken as a request for clarification.
Acknowledgements. I would like to warmly thank the members of the French EA community, and in particular Laura Green and Lennart Stern, for their support and very useful feedback.
Thank you, I think this is an excellent post!
I also sympathize with your confusion. - FWIW, I think that a fair amount of uncertainty and confusion about the issues you've raised here is the epistemically adequate state to be in. (I'm less sure whether we can reliably reduce our uncertainty and confusion through more 'research'.) I tentatively think that the "received longtermist EA wisdom" is broadly correct - i.e. roughly that the most good we can do usually (for most people in most situations) is by reducing specific existential risks (AI, bio, ...) -, but I think that
One thing in your post I might want to question is that, outside of your more abstract discussion, you phrase the question as whether, e.g., "AI safety should be virtually infinitely preferred to other cause areas such as global poverty". I'm worried that this is somewhat misleading because I think most of your discussion rather concerns the question whether, to improve the long-term future, it's more valuable to (a) speed up growth or to (b) reduce the risk of growth stopping. I think AI safety is a good example of a type-(b) intervention, but that most global poverty interventions likely aren't a good example of a type-(a) intervention. This is because I would find it surprising if an intervention that has been selected to maximize some measure of short-term impact also turned out to be optimal for speeding up growth in the long-run. (Of course, this is a defeatable consideration, and I acknowledge that there might be economic arguments that suggest that accelerating growth in currently poor countries might be particularly promising to increase overall growth.) In other words, I think that the optimal "growth intervention" Alice would want to consider probably isn't, say, donating to distribute bednets; I don't have a considered view on what it would be instead, but I think it might be something like: doing research in a particularly dynamic field that might drive technological advances; or advocating changes in R&D or macroeconomic policy. (For some related back-of-the-envelope calculations, see Paul Christiano's post on What is the return to giving?; they suggest "that good traditional philanthropic opportunities have a return of around 10 and the best available opportunities probably have returns of 100-1000, with most of the heavy hitters being research projects that contribute to long term tech progress and possibly political advocacy", but of course there is a lot of room for error here. See also this post for how maximally increasing technological progress might look like.)
Lastly, here are some resources on the "increase growth vs. reduce risk" question, which you might be interested in if you haven't seen them:
I might follow up in additional comments with some pointers to issues related to the one you discuss in the OP.
I have two comments concerning your arguments against accelerating growth in poor countries. One is more "inside view", the other is more "outside view".
The "inside view" point is that Christiano's estimate only takes into account the "price of a life saved". But in truth GiveWell's recommendations for bednets or deworming are to a large measure driven by their belief, backed by some empirical evidence, that children who grow up free of worms or malaria become adults who can lead more productive lives. This may lead to better returns than what his calculations suggest. (Micronutrient supplementation may also be quite efficient in this respect.)
The "outside view" point is that I find our epistemology really shaky and worrisome. Let me transpose the question into AI safety to illustrate that the point is not related to growth interventions. If I want to make progress on AI safety, maybe I can try directly to "solve AI alignment". Let's say that I hesitate between this and trying to improve the reliability of current-day AI algorithms. I feel that, at least in casual conversations (perhaps especially from people who are not actually working in the area), people would be all too willing to jump to "of course the first option is much better because this is the real problem, if it succeeds we win". But in truth there is a tradeoff with being able to make any progress at all, it is not just better to turn your attention to the most maximally long-term thing you can think of. And, I think it is extremely useful to have some feedback loop that allows you to track what you are doing, and by necessity this feedback loop will be somewhat short term. To summarize, I believe that there is a "sweet spot" where you choose to focus on things that seem to point in the right direction and also allow you at least some modicum of feedback over shorter time scales.
Now, consider the argument "this intervention cannot be optimal in the long run because it has been optimized for the short term". This argument essentially allows you to reject any intervention that has shown great promise based on the observations we can gather. So, effective altruism started as being "evidence based" etc., and now we reached a situation where we have built a theoretical construct that, not only allows to place certain interventions above all others without us having to give any empirical evidence backing this, but moreover, if another intervention is proposed that comes with good empirical backing, we can use this fact as an argument against the intervention!
I may be pushing the argument a bit too far. This still makes me feel very uncomfortable.
Regarding your "outside view" point: I agree with what you say here, but think it cannot directly undermine my original "outside view" argument. These clarifications may explain why:
Ok, I understand your point better now, and find that it makes sense. To summarize, I believe that the art of good planning to a distant goal is to find a series of intermediate targets that we can focus on, one after the other. I was worried that your argument could be used against any such strategy. But in fact your point is that as it stands, health interventions have not been selected by a "planner" who was actually thinking about the long-term goals, so it is unlikely that the selected interventions are the best we can find. That sounds reasonable to me. I would really like to see more research into what optimizing for long-term growth could look like (and what kind of "intermediate targets" this would select). (There is some of this in Christiano's post, but there is clearly room for more in-depth analysis in my opinion.)
.
I think this is a fair point. Specifically, I agree that GiveWell's recommendations are only partly (in the case of bednets) or not at all (in the case of deworming) based on literally averting deaths. I haven't looked at Paul Christiano's post in sufficient detail to say for sure, but I agree it's plausible that this way of using "price of a life saved" calculations might effectively ignore other benefits, thus underestimating the benefits of bednet-like interventions compared to GiveWell's analysis.
I would need to think about this more to form a considered view, but my guess is this wouldn't change my mind on my tentative belief that global health interventions selected for their short-term (say, anything within the next 20 years) benefits aren't optimal growth interventions. This is largely because I think the dialectical situation looks roughly like this: