Summary: Why many different worldviews should prioritise reducing existential risk. Also an exhaustive list of people who can ignore this argument. (Writeup of an old argument I can't find a source for.)
Confidence: 70%.
Crossposted from gleech.org.
---
Imagine someone who thought that art was the only thing that made life worth living. [1] What should they do? Binge on galleries?
Work to increase the amount of art and artistic experience, by going into finance to fund artists? Or by becoming an activist for government funding for the arts? Maybe. But there’s a case that they should pay attention to ways the world might end: after all, you can’t enjoy art if we’re all dead.
1. Aesthetic experience is good in itself: it’s a terminal goal.
2. The extinction of life would destroy all aesthetic experience & prevent future experiences.
3. So reducing existential risk is good, if only to protect the conditions for aesthetic experience.
And this generalises to a huge range of values:
1. [good] is good in itself: it’s a terminal goal.
2. The extinction of life would destroy [good], and prevent future [good].
3. So reducing existential risk is good, if only to protect the conditions for [good].
Caspar Oesterheld gives a few examples of what people can plug into those brackets:
Abundance, achievement, adventure, affiliation, altruism, apatheia, art, asceticism, austerity, autarky, authority, autonomy, beauty, benevolence, bodily integrity, challenge, collective property, commemoration, communism, community, compassion, competence, competition, competitiveness, complexity, comradery, conscientiousness, consciousness, contentment, cooperation, courage, [crab-mentality], creativity, crime, critical thinking, curiosity, democracy, determination, dignity, diligence, discipline, diversity, duties, education, emotion, envy, equality, equanimity, excellence, excitement, experience, fairness, faithfulness, family, fortitude, frankness, free will, freedom, friendship, frugality, fulfillment, fun, good intentions, greed, happiness, harmony, health, honesty, honor, humility, idealism, idolatry, imagination, improvement, incorruptibility, individuality, industriousness, intelligence, justice, knowledge, law abidance, life, love, loyalty, modesty, monogamy, mutual affection, nature, novelty, obedience, openness, optimism, order, organization, pain, parsimony, peace, peace of mind, pity, play, population size, preference fulfillment, privacy, progress, promises, property, prosperity, punctuality, punishment, purity, racism, rationality, reliability, religion, respect, restraint, rights, sadness, safety, sanctity, security, self-control, self-denial, self-determination, self-expression, self-pity, simplicity, sincerity, social parasitism, society, spirituality, stability, straightforwardness, strength, striving, subordination, suffering, surprise, technology, temperance, thought, tolerance, toughness, truth, tradition, transparency, valor, variety, veracity, wealth, welfare, wisdom.
So "from a huge variety of viewpoints, the end of the world is bad"? What a revelation!
: the above is only interesting if we get from “it’s good to reduce x-risk” to “it’s the most important thing to do” for these values. This would be the case if extinction was both 1) relatively likely relatively soon, and 2) we could do something about it. We can't be that confident of either of these things, but there are good reasons to both worry and plan.
(If you think that we can only be radically uncertain about the future, note that this implies you should devote more attention to the worst scenarios, not less: ‘high uncertainty’ is not the same as ‘low probability’.)
It's hard to say at what precise level of confidence and discount rate this argument overrides direct promotion of [good]; I'm claiming that it's implausible that your one lifetime of direct promotion would outweigh all future instances, if you're a consequentialist and place reasonable weigh on future lives.
When I first wrote this, I thought the argument had more force for people with high moral uncertainty - i.e. the more of Oesterheld's list you think are plausibly actually terminal goods, the more you'd focus on x-risk. But I don't think that follows, and anyway there are much stronger kinds of uncertainty, involving not just which terminal values you credit, but whether there are moral properties at all, whether maximisation is imperative, whether promotion or honouring counts as good. The above argument is about goal-independence (within consequentialism), and says nothing about framework-independence. So:
Who doesn’t have to work on reducing x-risk?
* People with incredibly high confidence that nothing can be done to affect extinction (that is, well above 99% confidence).
* Avowed egoists. (Though Scheffler argues that even they have to care here.)
* 'Parochialists': People who think that the responsibility to help those you’re close to outweighs your responsibility to any number of distant others.
* People with values that don’t depend on the world:
* Nihilists, or other people who think there are no moral properties.
* People with an ‘honouring’ kind of ethics - like Kantians, Aristotelians, or some religions. Philip Pettit makes a helpful distinction: when you act, you can either 'honor' a value (directly instantiate it) or 'promote' it (make more opportunities for it, make it more likely in future). This is a key difference between consequentialism and two of the other big moral theories (deontology and virtue ethics): the latter two only value honouring. This could get them off the logical hook because, unless "preventing extinction" was a duty or virtue itself, or fit easily into another duty or virtue, there's no moral force against it. (You could try to construe reducing x-risk as "care for others" or "generosity".) [2]
* People that disvalue life:
* Absolute negative utilitarians or antinatalists: people who think that life is generally negative in itself.
* People who think that human life has, and will continue to have, net-negative effects. Of course, a deep ecologist who sided with extinction would be hoping for a horrendously narrow event, between ‘one which ends all human life’ and ‘one which ends all life’. They’d still have to work against the latter, which covers the artificial x-risks.
* Ordinary utilitarians might also be committed to this view, in certain terrible contingencies (e.g. if we inexorably increased the number of suffering beings via colonisation or simulation).
* The end of the world is not the worst scenario: you might instead have a world with unimaginable amounts of suffering lasting a very long time, an ‘S-risk’. You might work on those instead. This strikes me as admirable and important, it just doesn’t have the complete value-independence that impressed me about the argument at the start of this piece.
* People who don’t think that probability estimates or expected value should be used for moral decisions. ('Intuitionists'.)
* You might be ‘satisficing’ - you might view the Good as a piecewise function, where having some amount of the good is vitally important, but any more than that has no moral significance. This seems more implausible than maximisation.
Uncertainties
* We really don’t know how tractable these risks are: we haven’t acted, as a species, on unprecedented century-long projects with literally only one chance for success. (But again, this uncertainty doesn’t licence inactivity, because the downside is so large.)
* I previously had the following exempted:
People with incredibly high confidence that extinction will not happen (that is, well above 99% confidence). This is much higher confidence than most people who have looked hard at the matter.
But Ord argues that these people actually should prioritise x-risk, since extinction being very hard implies a long future, and so much greater future expected value. It's not clear what assumptions his model makes, besides low discount rate and at least minimal returns to x-risk reduction. (h/t makaea)
* There is some chance that our future will be negative - especially if we spread normal ecosystems to other planets, or if hyper-detailed simulations of people turn out to have moral weight. If the risk increased (if the moral circle stopped expanding, if research into phenomenal consciousness and moral weight stagnated), these could ‘flip the sign’ on extinction, for me.
* I was going to add ‘person-affecting' people to the exemption list. But actually if the probability of extinction in the next 80 years (one lifetime) is high enough (1% ?) then they probably have to act too, even despite ignoring future generations.
* Most people are neither technical researchers nor willing to go into government. So, if x-risk organisation ran out of “room for more funding” then most people would be off the hook (back to maximising their terminal goal directly), until they had some.
* We don’t really know how common real deontologists are. (That one study is n=1000 about Sweden, probably an unusually consequentialist place.) As value-honourers, they can maybe duck most of the force of the argument.
* Convergence, for instance the above argument, is often suspicious, when humans are persuading themselves or others.
---
[1]: For example, Nietzsche said 'Without music, life would be a mistake.' (Though strictly this is bluster: he certainly valued many other things.)
[2]: Pummer claims that all "minimally plausible" versions of the honouring ethics must include some promotion. But I don't see how they can, without being just rule-utilitarians in disguise.
EDIT 8/12/18: Formatting. Also added Ord's hazard rate argument, h/t makaea.
Strong upvote. This is a fantastic post, and I wish that people who downvoted it had explained their reasoning, because I don't see any big flaws.
I don't necessarily agree with everything written here, and I don't think the argument would suffice to convince people outside of EA, but we need more content like this, which:
A couple of things I wish had been different:
Aside from the honor/promote distinction, I think the most common objection to this from someone outside of EA might be something like "extinction is less than 1% likely, not because the world isn't dangerous but because I implicitly trust other people to handle that sort of thing, and prefer to focus on local issues that are especially important to myself, my family, and my community".
[1] Like this.
Good question. To be honest, it was just me intuiting the chance that all of the premises and exemptions are true, which maybe cashes out to your first option. I'm happy to use a conventional measure, if there's a convention on here.
Would also invite people who disagree to comment.
Interesting. This neatly sidesteps Ord's argument (about low extinction probability implying proportionally higher expected value) which I just added, above.
Another objection I missed, which I think is the clincher inside EA, is a kind of defensive empiricism, e.g. Jeff Kaufman:
I take this very seriously; it's why I focus on the ML branch of AI safety. If there is a response to this (excellent) philosophy, it might be that it's equivalent to risk aversion (the bad kind) somehow. Not sure.
You can get similar value-independence in favour of extinction by using "bads" instead of "goods". Many of the values in Oesterheld's list have opposites which could reasonably be interpreted as "bads", and some of them are already "bads", e.g. suffering, pain and racism.
True - but how many people hold these inverses to be their primary value? (That is, I think the argument above is useful because almost everyone has something in the Goods set.)
I think even more people have things in the bads set, and there will be more agreement on these values, too, e.g. suffering, cruelty and injustice. The question is then a matter of weight.
Most people (and probably most EAs) aren't antinatalists, so you would expect, for them, the total good to outweigh the total bad. Or, they haven't actually thought about it enough.
Ah I see. Agreed - thanks for clarifying.
Besides the person-affecting views and disvalue of life covered here, if an individual has an Epicurean view of life and death (another kind of person-affecting view), i.e. death is not bad, then improving wellbeing should probably take priority. And while Epicureanism assigns 0 disvalue to death (ignoring effects on others), one could assign values arbitrarily close to 0.
There are also issues with dealing with infinities that make utilitarianism non-action guiding (it doesn't tell us what to do in most practical cases); you could probably throw these in with nihilism. E.g. if the universe is unbounded ("infinite") in space or time, then we can't change the total sum of utility, and that number is not even well-defined (not even +infinity or -infinity) with the usual definitions of convergence in the real numbers. If you assign any nonzero probability to an infinite universe, you end up with the same problem, but it's actually pretty likely that the universe is spatially unbounded. There are several attempts at solutions, but all of them have pretty major flaws, AFAIK.
Some person-affecting views can help, i.e. using a Pareto principle, but then it's not clear how to deal with individuals whose exact identities depend on your decisions (or maybe we just ignore them; many won't like that solution), and there are still many cases that can't be handled. There's discussion in this podcast, with some links for more reading (ctrl-F "Pareto" after expanding the transcript): https://80000hours.org/podcast/episodes/amanda-askell-moral-empathy/
Rounding sufficiently small probabilities to 0 and considering only parts of the universe we're extremely confident we can affect can help, too. This proposed solution and a few others are discussed here: https://nickbostrom.com/ethics/infinite.pdf
You could also have a bounded vNM utility function, but this means assigning decreasing marginal value to saving lives, and how you divide decisions/events matters, e.g. "saving 1 life and then saving 1 life" > "saving 2 lives and then saving 0 lives".
For the unbounded time case (assuming we can handle or avoid issues with unbounded space, and people might prefer not to treat time and space differently): https://forum.effectivealtruism.org/posts/9D6zKRPfaALiBhnnN/problems-and-solutions-in-infinite-ethics
I was surprised to see person-affecting views weren't on your list of exception, then I saw it was in the uncertainties section. FWIW, taking Gregory Lewis' model at face value - I raised some concerns in a comment replying to that post - he concludes it's $100,000 per life saved. If AMF is $3,500 per life saved then X-risk is a relatively poor buy (although perhaps tempting as a sort of 'hedge'). That would only speak to your use of money: a person-affector could still conclude they'd do more good with a career focused on X-risk than elsewhere.
For AI, one estimate is $3-$1000 per life saved in the present generation (near bottom of model). For alternate foods for global agricultural catastrophes, one estimate is $0.20-$400 globally and $1-$20,000 in the US, both in the present generation.
Good point, thanks. It's definitely not a knock-down argument.
I've written a blogpost on whether Christians should share the emphasis that many EAs put on the long term, including extinction risks. Since this fits nicely with your aim in this blogpost -- i.e. whether *many* worldviews should prioritise existential risks -- I thought I'd mention it here: https://eachdiscussion.wordpress.com/2019/04/06/how-much-should-christian-eas-care-about-the-far-future-part-i/
Thanks for this. I'm not very familiar with the context, but let me see if I understand. (In a first for me, I'm not sure whether to ask you to cite more scripture or add more formal argument.) Let's assume a Christian god, and call a rational consequence-counting believer an Optimising Christian.
Your overall point is that there are (or might be) two disjoint ethics, one for us and one for God, and that ours has a smaller scope, falling short of long-termism, for obvious reasons. Is this an orthodox view?
1. "The Bible says not to worry, since you can trust God to make things right. Planning is not worrying though. This puts a cap on the intensity of our longterm concern."
2. "Humans are obviously not as good at longtermism as God, so we can leave it to Him."
3. "Classical theism: at least parts of the future are fixed, and God promised us no (more) existential catastrophes. (Via flooding.)"
4. "Optimising Christians don't need to bring (maximally many) people into existence: it's supererogatory." But large parts of Christianity take population increase very seriously as an obligation (based on e.g. Genesis 1:28 or Psalm 127). Do you know of doctrine that Christian universalism stops at present people?
5. "Optimising Christians only need to 'satisfice' their fellows, raising them out of subsistence. Positive consequentialism is for God." This idea has a similar structure to negative utilitarianism, a moral system with an unusual number of philosophical difficulties. Why do bliss or happiness have no / insufficient moral weight? And, theologically: does orthodoxy say we don't need to make others (very) happy?
If I understand you, in your points (1) through (4) you appeal to a notion of God's agency outside of human action or natural laws. (So miracles only?) But a better theology of causation wouldn't rely on miracles, instead viewing the whole causal history of the universe as constituting God's agency. That interpretation, which at least doesn't contradict physics, would keep optimising Christians on the hook for x-risk.
Many of your points are appropriately hedged - e.g. "it might also be God’s job" - but this makes it difficult to read off actions from the claims. (You also appeal to a qualitative kind of Bayesian belief updating, e.g. "significant but not conclusive reason".) Are you familiar with the parliamentary model of ethics? It helps us act even while holding nuanced/confused views - e.g. for the causation question I raised above, each agent could place their own subjective probabilities on occasionalism, fatalism, hands-off theology and so on, and then work out what the decision should be. This kind of analysis could move your post from food-for-thought into a tool for moving through ancient debates and imponderables.
Thanks for this! Very interesting.
And really sorry for replying only now -- I somehow missed this and only saw it now.
--- On population increase: yes, many Christians work towards population increase but it's equally true that many Christians don't. An interesting side remark is that the influential passage Genesis 1,28 on which pro-natalism is often based calls for *filling* the earth. Arguably, humanity can claim to have unlocked this achievement. We can tick it off our To-Do-List. (Also, in terms of background information, my view that determining the optimal population size might be God's task rather than a human task started with this blogpost: https://greenfutureethics.wordpress.com/2012/12/03/god-as-a-solution-for-population-paradoxes)
--- On miracles: One thing is that I find it a bit hard to exclude miracles from classical theism. But even if we exclude them (or understand them to be compatible with natural laws) and even if we understand God to act within the causal history of the universe, one thing we surely can't exclude in classical theism is that God acts in addition to human agency (including acts which might be surprising). To the extent that this is true, Christian concern with x-risks should continue to be somewhat mitigated relative to the atheist's concern?
--- And thanks for the helpful observation that the blogpost unhelpfully avoids clear upshots (and thus also avoids responsibility for actions that might follow from it). The thing is: I find it genuinely extremely hard to think about the right approach to long-termism from a Christian perspective and this actually was *merely* a start. The parliamentary model etc would indeed be needed to derive actionable conclusions. (And, just to say, I do agree with many EAs that the far future should definitely receive more consideration than it typically does).
As someone with an interest in government and relatively new to the concept of x-risk, I have a semi-urgent question: who should I support for President? I will probably have to get involved with a campaign in some way or another in the next few months to maximize my odds of getting a decent appointment after the election. There's plenty of interest group ratings, position statements etc. out there on environmental issues but I can't find much that would be of practical use on the other types, which seem to be more serious at least in aggregate and perhaps individually too. I could try compiling my own ratings but I know far less than a lot of the people in this community, so if someone has already figured out or is in the process of figuring out where the candidates stand on the risks they have expertise in, I would greatly appreciate it. Doesn't have to be like standard interest group ratings and maybe shouldn't be. E.g. the fact that someone has a hawkish temperament toward China and that would make them more prone to starting an arms race is probably more important to AI safety than the specifics of any technology-related votes they've taken.
No idea, sorry. I know CSER have held at least one workshop about Trump and populism, so maybe try Julius Weitzdoerfer:
For deep ecologists, I use the argument that without people, the animals and plants will generally go extinct in 500-1000 million years because the increasing brightness of the sun will cause runaway global warming. Humans could delay this by putting reflective material between the Earth and the sun or other means. Or humans could perpetuate species on other planets. So in the long run, it is not a good idea for other species to have humans go extinct.
I haven't read much deep ecology, but I model them as strict anti-interventionists rather than nature maximisers (or satisficers): isn't it that they value whatever 'the course of things without us' would be?
(They certainly don't mind particular deaths, or particular species extinctions.)
But even if I'm right about that, you're surely right that some would bite the bullet when universal extinction was threatened. Do you know any people who accept that maintaining a 'garden world' is implied by valuing nature in itself?
I haven't read much deep ecology either. Seth Baum has written that some people think there is intrinsic value in functioning ecosystems - presumably these people would want the ecosystems to continue as a garden world. Other people value biodiversity (number of species). But you're right that some just want whatever would have happened naturally.
I agree that it covers AI, but I'm not sure about the other artificial x-risks. Nuclear winter severe enough to eventually kill all humans would definitely kill all large animals, but some smaller forms of life would survive. And while bio-risk could vary a lot in how many species were susceptible to it, I don't think anyone could construct a pathogen that affects everything.
You're right. I think I had in mind 'AI and nanotech' when I said that.
By “doesn’t have to work on reducing x-risk”, do you mean that they don’t want to?
I’d expect that negative utilitarians (NUs) do want to reduce x-risk, because
(1) x-risk is rarely an either/or risk of 100% extinction; instead, more x-risk probably correlates with more risk of extreme suffering (from non-total pandemics/disasters/wars/etc., and all of their after-effects)
(2) even facing a 100% human extinction, we’d want to account for our epistemic uncertainty of the conditions from which suffering can evolve (re-evolve on Earth, or be found elsewhere within the reach of our descendants)
NUs don’t necessarily jump to suicide as a solution, because helping others is an infinite game to live for, especially after accounting for the epistemic uncertainty of all possible forms of suffering and their evolution. There is further basic research on suffering to be done before turning off the lights and hoping that all the billions of exoplanets would have their own guardians.
It is a straw man argument that NUs don’t value life or positive states, because NUs value them instrumentally, which may translate into substantial practical efforts (compared even with someone who claims to be terminally motivated by them).
I mean that the end of the world isn't a bad outcome to someone who only values the absence of suffering, and who is perfectly indifferent between all 'positive' states. (This is Ord's definition of absolute NU, so I don't think I'm straw-manning that kind.) And if something isn't bad (and doesn't prevent any good), a utilitarian 'doesn't have to work on it' in the sense that there's no moral imperative to.
(1) That makes sense. But there's an escalation problem: worse risk is better to ANU (see below).
(2) One dreadful idea is that self-replicators would do the anti-suffering work, obviating the need for sentient guardians, but I see what you're saying. Again though, this uncertainty about moral patients licences ANU work on x-risks to humans... but only while moving the degenerate 'solution' upward, to valuing risks that destroy more classes of candidate moral patients. At the limit, the end of the entire universe is indisputably optimal to an ANU. So you're right about Earth x-risks (which is mostly all people talk about) but not for really farout scifi ones, which ANU seems to value.
Actually this degenerate motion might change matters practically: it seems improbable that it'd be harder to remove suffering with biotechnology than to destroy everything. Up to you if you're willing to bite the bullet on the remaining theoretical repugnance.
(To clarify, I think basically no negative utilitarian wants this, including those who identify with absolute NU. But that suggests that their utility function is more complex than they let on. You hint at this when you mention valuing an 'infinite game' of suffering alleviation. This doesn't make sense on the ANU account, because each iteration can only break even (not increase suffering) or lose (increase suffering).)
Most ethical views have degenerate points in them, but valuing the greatest destruction equal to the greatest hedonic triumph is unusually repugnant, even among repugnant conclusions.
I don't think instrumentally valuing positive states helps with the x-risk question, because they get trumped by a sufficiently large amount of terminal value, again e.g. the end of all things.
(I'm not making claims about other kinds of NU.)