61 Reactions

I agree with the framing.

Quantitatively, the willingness to pay to avoid extinction even just from the United States is truly enormous. The value of a statistical life in the US — used by the US government to estimate how much US citizens are willing to pay to reduce their risk of death — is around $10 million. The willingness to pay, therefore, from the US as a whole, to avoid a 0.1 percentage point of a catastrophe that would kill everyone in the US, is over $1 trillion. I don’t expect these amounts to be spent on global catastrophic risk reduction, but they show how much latent desire there is to reduce global catastrophic risk, which I’d expect to become progressively mobilised with increasing indications that various global catastrophic risks, such as biorisks, are real. [I think my predictions around this are pretty different than some others, who expect the world to be almost totally blindsided. Timelines and gradualness of AI takeoff is of course relevant here.]

In contrast, many areas of better futures work are likely to remain extraordinarily neglected. The amount of even latent interest in, for example, ensuring that resources outside of our solar system are put to their best use, or that misaligned AI produces a somewhat-better future than it would otherwise have done even if it kills us all, is tiny, and I don’t expect society to mobilise massive resources towards these issues even if there were indications that those issues were pressing.

In some cases, what people want will be actively opposed to what is in fact best, if what's best involves self-sacrifice on the part of those alive today, or with power today.

And then I think the neglectedness consideration beats the tractability consideration. Here are some pretty general reasons for optimism on expected tractability:

In general, tractability doesn’t vary by as much as importance and neglectedness.
In cause areas where very little work has been done, it’s hard for expected tractability to be very low. Because of how little we know about tractability in unexplored cause areas, we should often put significant credence on the idea that the cause will turn out to be fairly tractable; this is enough to warrant some investment into the cause area — at least enough to find out how tractable the area is.
There are many distinct sub-areas within better futures work. It seems unlikely to me that tractability in all of them is very low, and unlikely that their tractability is very highly correlated.
There’s a reasonable track record of early-stage areas with seemingly low tractability turning out to be surprisingly tractable. A decade ago, work on risks from AI takeover and engineered pathogens seemed very intractable; there was very little that one could fund, and very little in the way of promising career paths. But this changed over time, in significant part because of (i) research work improving our strategic understanding, and shedding light on what interventions were most promising; (ii) scientific developments (e.g. progress in machine learning) making it clearer what interventions might be promising; (ii) the creation of organisations that could absorb funding and talent. All these same factors could well be true for better futures work, too.

Of these considerations, it’s the last that personally moves me the most. It doesn't feel long ago that work on AI takeover risk felt extraordinarily speculative and low-tractability, where there was almost nowhere one could work for or donate to outside of the Future of Humanity Institute or Machine Intelligence Research Institute. In the early days, I was personally very sceptical about the tractability of the area. But I’ve been proved wrong. Via years of foundational work — both research work figuring out what the most promising paths forward are, and via founding new organisations that are actually squarely focused on the goal of reducing takeover risk or biorisk, rather than on a similar but tangential goal — the area has become tractable, and now there are dozens of great organisations that one can work for or donate to.

William_MacAskill9h4

Discussion topic: People vary a lot in the extent to which, and how likely it is, that post-AGI, different people will converge on the same moral views. I feel fairly sceptical about having a high likelihood of convergence; I certainly don't think we should bank on it.

[See my response to Andreas below. Here I meant "convergence" as shorthand to refer to "fully accurate, motivational convergence".]

Andreas_Mogensen9h3

Could you clarify what you mean by 'converge'? One thing that seems somewhat tricky to square is believing that convergence is unlikely, but that value lock-in is likely. Should we understand convergence as involving agreement in views facilitated by broadly rational processes, or something along those lines, to be contrasted with general agreement in values that might be facilitated by irrational or arational forces, of the kind that might ensure uniformity of views following a lock-in scenario?

Yeah, thanks for pushing me to be clearer: I meant "convergence" as shorthand to refer to "fully accurate, motivational convergence". So I mean a scenario where people have the correct moral views, on everything that matters significantly, and are motivated to act on those moral views. I'll try to say FAM-convergence from now on.

Toby Tremlett🔹9h2

I'm wondering whether we should expect worlds which converge on moral views to converge on bad moral views.

From the space of world religions - we've seen a trend where we converge over time (at least from a high level of abstraction where we can refer to "Christianity" and "Islam" rather than "mega-church Christians" or whatever). Is this because the religions that succeed are exclusive and expansionary? Of all religions that have existed, I know that many of them don't much care if you also worship other gods. My empirical (ish) question is whether we should expect world in which a sizable fraction of the population follows the same religion to be one where the religion they follow is exclusive (you can't follow others) and expansionary (other people should also follow this religion). PS- I know that not all Christians or Muslims are exclusionary about other religions, this is over-simplified.

This is relevant because, if this is a mechanism, we might expect the same thing of morality or political organisation - beliefs which demand you don't follow others, and that others follow the same beliefs as you, rather than tolerant beliefs. Perhaps this would make it more likely that futures which converge have converged on something extreme and closed, rather than exploratory and open.

This is pretty vague - just wondering if others a) know more than me about the religion question and can speak to that or b) have had similar thoughts, or c) think that the existence of exclusive and expansionary (and wrong) ideologies might make convergence more likely.

Yes, this is yet another reason for a moratorium on further-AGI development imo. If everyone has a genie with unlimited wishes, and are all pushing the world in different directions, the result will be chaos. Yampolskiy's solution to this is everyone having their own private solipsistic universe simulations...

MichaelStJules9h2

I'm also pretty skeptical of convergence, largely because I'm a moral anti-realist. I don't see why we would converge to any pretty precise view in particular, except by coincidence or pressures that don't track stance-independent moral truths (because there are none). Barring value lock-in, I suspect there could be convergence towards the recognition that unnecessary suffering is bad and worth preventing (when cheap enough), but I'd guess there still be disagreement on some of these:

population ethics
hedonism vs preference views vs others
whether non-sentient/non-conscious things matter terminally, e.g. preserving nature
whether groups of moral patients have special status beyond their aggregates, e.g. ethnic groups, species
deontology vs consequentialism vs virtue ethics
what counts as conscious/sentient (I think this is partly normative, not just empirical)
decision theory, attitudes towards risk and ambiguity, fanaticism

Hopefully they do not just converge on the same moral views, but also good ones!

MichaelStJules9h3

Starting my own discussion thread.

My biggest doubt for the value of extinction risk reduction is my (asymmetric) person-affecting intuitions: I don't think it makes things better to ensure future people (or other moral patients) come to exist for their own sake or the sake of the value within their own lives. But if future people will exist, I want to make sure things go well for them. This is summarized by the slogan "Make people happy, not make happy people".

If this holds, then extinction risk reduction saves the lives of people who would otherwise die in an extinction event, which is presumably good for them, but this is only billions of humans.^[1] If we don't go extinct, then the number of our descendant moral patients could be astronomical. It therefore seems better to prioritize our descendant moral patients conditional on our survival because there are far far more of them.

Aliens (including alien artificial intelligence) complicate the picture. We (our descendants, whether human, AI or otherwise) could

use the resources aliens would have otherwise for our purposes instead of theirs, i.e. replace them,
help them, or
harm them or be harmed by them, e.g. through conflict.

I'm interested in others' takes on this.

^{^}

And it's not clear we want to save other animals, in case their lives are bad on average. It can also make a difference whether we're talking about human extinction only or all animal extinction.

How asymmetric do you think things are? I tend to deprioritise s-risks (both accidental and intentional) because it seems like accidental suffering and intentional suffering will be a very small portion of the things that our descendants choose to do with energy. In everyday cases I don't feel a pull to putting a lot of weight on suffering. But I feel more confused when we get to tail cases. Maximising pleasure intuitively feels meh to me, but maximising suffering sounds pretty awful. So I worry that (1) all of the value is in the tails, as per Power Laws of Value and (2) on my intuitive moral tastes the good tails are not that great and the bad tails are really bad.

William_MacAskill9h3

(Crossposted from a quicktake I just did).

Clarifying "Extinction"

I expect this debate week to get tripped up a lot by the term “extinction”. So here I’m going to distinguish:

Human extinction — the population of Homo sapiens, or members of the human lineage (including descendant species, post-humans, and human uploads), goes to 0.
Total extinction — the population of Earth-originating intelligent life goes to 0.

Human extinction doesn’t entail total extinction. Human extinction is compatible with: (i) AI taking over and creating a civilisation for as long as it can; (ii) non-human biological life evolving higher intelligence and building a (say) Gorilla sapiens civilisation.

The debate week prompt refers to total extinction. I think this is conceptually cleanest. But it’ll trip people up as it means that most work on AI safety and alignment is about “increasing the value of futures where we survive” and not about “reducing the chance of our extinction” — which is very different than how AI takeover risk has been traditionally presented. I.e. you could be strongly in favour of "increasing value of futures in which we survive" and by that mean that the most important thing is to prevent the extinction of Homo sapiens at the hands of superintelligence. In fact, because most work on AI safety and alignment is about “increasing the value of futures where we survive”, I expect there won’t be that many people who properly understand the prompt and vote “yes”.

So I think we might want to make things more fine-grained. Here are four different activities you could do (not exhaustive):

Ensure there’s a future for Earth-originating intelligent life at all.
Make human-controlled futures better.
Make AI-controlled futures better.
Make human-controlled futures more likely.

For short, I’ll call these activities:

Future at all.
Better human futures.
Better AI futures.
More human futures.

I expect a lot more interesting disagreement over which of (1)-(4) is highest-priority than about whether (1) is higher-priority than (2)-(4). So, when we get into debates, it might be worth saying which of (1)-(4) you think is highest-priority, rather than just “better futures vs extinction”.

So in the debate week statement (footnote 2) it says "earth-originating intelligent life". What if you disagree that AI counts as "life"? I expect that a singleton ASI will take over and will not be sentient or conscious, or value anything that humans value (i.e. the classic Yudkowskian scenario).

Why so confident that:
- It'll be a singleton AI that takes over
- That it will not be conscious?

I'm at 80% or more that there will be a lot of conscious AIs, if AI takes over.

Interesting. What makes you confident about AI consciousness?

OscarD🔸9h2

If the true/best/my subjective axiology is linear in resources (e.g. total utilitarianism), lots of 'good' futures will probably capture a very small fraction of how good the optimal future could have been. Conversely, if axiology is not linear in resources (e.g. intuitive morality, average utilitarianism), good futures seem more likely to be nearly optimal. Therefore whether axiology is linear in resources is one of the cruxes for the debate week question.

Discuss.

William_MacAskill9h3

The easiest way, in my view, to make a near-optimal future very likely, conditional on non-extinction, is if value is bounded above.

There's an argument that this is the common sense view. E.g. consider:

Common-sense Eutopia: In the future, there is a very large population with very high well-being; those people are able to do almost anything they want as long as they don’t harm others. They have complete scientific and technological understanding. War and conflict are things of the past. Environmental destruction has been wholly reversed; Earth is now a natural paradise. However, society is limited only to the solar system, and will come to an end once the Sun has exited its red giant phase, in about five billion years.

Does this seem to capture less than one 10^22th of all possible value? (Because there are ~10^22 affectable stars, so civilisation could be over 10^22 times as big). On my common-sense moral intuitions, no.

Making this argument stronger: Normally, quantities of value are defined are in terms of the value of risky gambles. So what it means to say that Common-sense Eutopia is less than one 10^22th of all possible value is that a gamble with a one in 10^22 chance of producing an ideal-society-across-all-the-stars, and a (1 - 1/10^22) chance of near-term extinction, is better than producing Common-sense Eutopia for certain.

But that seems wild. Of all the issues facing classical utilitarianism, this seems the most problematic to me.

The easiest way, in my view, to make a near-optimal future very likely, conditional on non-extinction, is if value is bounded above.

Yes.

Of all the issues facing classical utilitarianism, this seems the most problematic to me.

So you doubt fanaticism, the view that a tiny chance of an astronomically good outcome can be more valuable than a certainty of a decent outcome. What about in the case of certainty? Do you doubt the utilitarian's objection to Common-sense Eutopia? This kind of aggregation seems important for the case for longtermism. (See the pages of paper dolls in What We Owe the Future.)

Average utilitarianism is approx linear in resources as long as at least one possible individual's wellbeing is linear in resources.

I.e. we create Mr Utility Monster, who has wellbeing that is linear in resources, and give all resources to benefiting Mr Monster. Total value is the same as it would be under total utilitarianism, just divided by a constant (namely, the number of people who've ever lived).

Andreas_Mogensen9h1

I wasn't sure if it's really useful to think about value being linear in resources on some views. If you have a fixed population and imagine increasing the resources they have available, I assume that the value of the outcome is a strictly concave function of the resource base. Doubling the population might double the value of the outcome, although it's not clear that this constitutes a doubling of resources. And why should it matter if the relationship between value and resources is strictly concave? Isn't the key question something like whether there are potentially realizable futures that are many orders of magnitude more valuable than the default or where we are now? Answering yes seems compatible with thinking that the function relating resources to value is strictly concave and asymptotes, so long as it asymptotes somewhere suitably high up on the scale of value.

Unpacking this: on linear-in-resources (LIR) views, we could lose out on most value if we (i) capture only a small fraction of resources that we could have done, and/or (ii) use resources in a less efficient way than we could have done. (Where on a LIR view, there is some use of resources that has the highest value/unit of resources, and everything should be used in that way.)

Plausibly at least, only a tiny % of possible ways of using resources are close to the value produced by the highest value/unit of resources use. So, the thinking goes, merely getting non-extinction isn't yet getting you close to a near-best future - instead you really need to get from a non-extinction future to that optimally-used-resources future, and if you don't then you lose out on almost all value.

There are funky axiologies where value is superlinear in resources — basically any moral worldview that embraces holism. If you think that the whole, arranged a particular way, is more valuable than the parts, then you will be even more precious about how precisely the world should be arranged than the total utilitarian.

Question: what level of extinction risk are people personally willing to accept in order to realise higher expected value in the futures where we survive? How much would the extinction coming in the next 5 years effect this? Or the next 1 year? How is this reflected in terms of what you are working on / spending resources on?

My position is that Timelines are short, p(doom) is high: a global stop to frontier AI development until x-safety consensus is our only reasonable hope (this post needs updating, to factor in things like inference time compute scaling, but my conclusions remain the same).

The problem is that no one has even established whether aligning or controlling ASI is theoretically, let alone practically, possible. Everything else (whether there is a human future at all past the next few years) is downstream of that.

Suppose p(doom) is 90%. Then preventing extinction multiplies the value of the world by 10 in expectation.

But suppose that the best attainable futures are 1000 times better than the default non-extinction scenario. Then ensuring we are on track to get the best possible future multiplies the value of the world by 100 in expectation, even after factoring in the 90% chance of extinction.

In this toy model, you should only allocate your resources to reducing extinction if it is 10 times more tractable than ensuring we are on track to get the best possible future, at the current margin.

You might think that we can just defer this to the future. But I've assumed in the set-up that the default future is 1/1000th as good as the best future. So apparently our descendants are not going to be very good at optimising the future, and we can't trust them with this decision.

Where do you think this goes wrong?

>suppose that the best attainable futures are 1000 times better than the default non-extinction scenario

This seems rather arbitrary. Why woiuld preventing extinction now guarantee that we (forever) lose that 1000x potential?

>In this toy model, you should only allocate your resources to reducing extinction if it is 10 times more tractable than ensuring we are on track to get the best possible future, at the current margin.

I think it is. Gaining the best possible future requires aligning an ASI, which has not been proven to be even theoretically possible afaik.

JoA🔸10h1

I have a question, and then a consideration that motivates it, which is also framed as a question that you can answer if you like.

If an existential catastrophe occurs, how likely is it to wipe out all animal sentience on earth?

I've already asked that question here (and also, to some acquaintances working in AI Safety, but the answers have very much differed - it seems we're quite far from a consensus on this, so it would be interesting to see perspectives from the varied voices taking part in this symposium.

Less important question, but that may clarify what motivates me to ask my main question: if you believe that a substantial part of X-risk scenarios entail animal sentience being left behind, do you then think that estimating the current and possible future welfare of wild animals is an important factor in evaluating the value of both existential risk reduction and interventions aimed at influencing the future? A few days ago, I was planning on making a post on invertebrate sentience being a possible crucial consideration when evaluating the value and disvalue of X-risk scenarios, but then thought that if this factor was rarely brought up, it could be that I was personally uninformed on the reasons why the experiences of invertebrates (if they are sentient) might not actually matter that much in future trajectories (aside from the possibility that they will all go extinct soon, which is why this question hinges on the prior belief that it is likely that sentient animals will continue existing on earth for a long time). There are probably different reasons to agree (or disagree) with this, and I'd be happy to hear yours in short, though it's not as important to me as my first question. Thank you for doing this!

Position statement: I chose 36% disagreement. AMA!

My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value. This still makes extinction risk astronomically valuable (it's equivalent to optimising a millionth of the whole cosmos!), but if we could increase the chance of optimising 1% of the universe by 1%, this would be 100x more valuable than avoiding extinction. (You're not going to get an extremely well grounded explanation of these numbers from me, but I hope they make my position clearer.)

My view is that if Earth-originating civilisation becomes spacefaring, over the long term we will settle the entire universe and use nearly all of its energy. However, we could use this energy for many different things. I expect that by default we will mostly use this energy to create copies of ourselves (human brains) or digital slaves that allow us to learn and achieve new things, since humans love nothing more than humans and Earthly things. But these are inefficient media for realising moral value. We could instead use this energy to realise vastly more value, as I argue in Power Laws of Value. But our descendants basically won't care at all about doing this. So I expect that we will miss out on nearly all value, despite using most of the universe's energy.

I think that there will be significant path dependency in what Earth-originating civilisation chooses to make with our cosmic endowment. In particular, I expect that artificial intelligence will drive most growth and most choices after the 21st century. Several factors, such as the number of distinct agents, their values, and the processes by which their values evolve over time will therefore make a decisive difference to what our descendants choose to do with the cosmos.

So our choices about AI governance and architecture today are likely to make a significant and, at least on some paths we can choose, predictable difference to what our descendants do with all of the energy in the universe. If we do this well, it could make the difference between attaining almost no value and nearly all value.

Given that I expect us to miss out on almost all value by default, I view the value of avoiding extinction as smaller than those who think we will achieve almost all value by default.

How do these considerations affect what you are doing / spending resources on? Does it change the calculus if extinction is likely to happen sooner? (see also comment here).

Jordan Arel1d7

Will MacAskill stated in a recent 80,000 hours podcast that he believes marginal work on trajectory change toward a best possible future rather than a mediocre future seems likely significantly more valuable than marginal work on extinction risk.

Could you explain what the key crucial considerations are for this claim to be true, and a basic argument for why think each of the crucial considerations resolves in favor of this claim?

Would also love to hear if others have any other crucial considerations they think weigh in one direction or the other.

Davidmanheim1d5

How much of the argument for working towards positive futures rather than existential security rests on conditional value, as opposed to expected value?

One could argue for conditional value, that in worlds where strong AI is easy and AI safety is hard, we are doomed regardless of effort, so we should concentrate on worlds where we could plausibly have good outcomes.

Alternatively, one could be confident that the probability of safety is relatively high, and make the argument that we should spend more time focused on positive futures because it's likely already - either due to efforts towards superintelligence safety are likely to work, (and if so, which ones?) or because alignment by default seems likely.

(Or, I guess, lastly, one could assume, or argue, that no superintelligence is possible, or it is unlikely.)