Which World Gets Saved

I'd guess that we don't have to think much about which world we're saving. My reasoning would be that the expected value of the world in the long-run is mostly predictable from macrostrategic -- even philosophical -- considerations. e.g. agents more often seek things that make them happier. The overall level of preference fulfilment that is physically possible might be very large. There's not much reason to think that pain is easier to create than pleasure (see [1] for an exploration of the question), and we'd expect the sum of recreation and positive incentivization to be more than the amount of disincentivization (e.g. retribution or torture).

I think a proper analysis of this would vindicate the existential risk view as a simplification of maximize utility (modulo problems with infinite ethics (!)). But I agree that all of this needs to be argued for.

1. http://reflectivedisequilibrium.blogspot.com/2012/03/are-pain-and-pleasure-equally-energy.html

I thought this post was particularly cool because it seems to be applicable to lots of things, at least in theory (I have some concerns in practice). I'm curious about further reviews of this post.

I find myself using the reasoning described in the post in a bunch of places related to the prioritization of longtermist interventions. At the same time,  I'm not sure I ever get any useful conclusions out of it. This might be because the area of application (predicting the impact of new technologies in the medium-term future) is particularly challenging. (One could argue that there's a "Which world gets saved" argument against "Which world gets saved" arguments: In worlds where the consideration is useful, we start out with wide uncertainty. Having wide uncertainty is bad for having lots of impact.)

I wonder if there's something a bit fishy about imagining only/exactly the worlds where one's contribution makes a pivotal difference (the post hints at this with the discussion of evidential decision procedures). 

I wonder if the reasoning in this post can be used to strengthen particular combinations of interventions. E.g., "AI alignment research is particularly impactful in worlds where alignment is neither too hard nor too easy. In those worlds, there will be people who would build misaligned AIs if it weren't for our inputs. Accordingly, AI alignment research becomes particularly important if we combine it with raising awareness of safety mindset and building leading coalitions of safety-minded research teams."

 

Very interesting and well written, thank you for this important idea!

Playing around with toy examples, I noticed another version of this consideration. The success of some actions might imply that there is a higher likelihood of a dangerous world (for which there might be better alternative actions).

For example, consider allocating resources for handling bio-terrorism. One important question is how likely is it that there will be individuals that want to cause massive harm and are able to. If this resource allocation does indeed stop a global extinction, this means that we should update our estimated likelihood for such an individual. In this world, there may better alternatives than focusing on something specific to current biological techniques and instead to put effort into methods which can help against any individual that wants to cause massive harm by any technology.

It is an interesting idea. If I remember correctly, something slightly similar was explored in the context of GCR by Seth Baum. The case there: if the recovery time after a global catastrophic event is relatively short (compared to background extinction rate), a violent civilization destroying itself before reaching technology allowing it to make whole species go extinct may be a better outcome.

As a quick guess, I'd predict careful analysis of this style of considerations will lead to more emphasis on risks from AI and less on nuclear. Within AI safety it would likely lead to less emphasis on approaches which run the risk of creating "almost aligned" AI - the adjustments would be somewhat similar to what the negative utilitarian camp says, just much less extreme.

I'm slightly skeptical studying this consideration in more depth than a few days is marginally effective. The reason is the study will run into questions like "should we expect a society recovering after near-extinction due to bio- attack to be in better or worse position to develop grand futures?" or doing some sort of estimates on how future where AI risks were just narrowly avoided could look like.

Great points, Trammell! Thank you for this post.

Your example comparing the peaceful-psychology hypothesis and the violent-psychology hypothesis is effective, and it stands on its own. However, I don't think it's the best way to represent Steven Pinker's argument, and I think representing that argument more accurately leads in some interesting new directions.

As I understand him, Pinker does not argue humans have a peaceful psychology. Rather, he acknowledges that there are many aspects of our psychology that predispose us to violence, and he attributes the historical decline in violence primarily to social developments, e.g., establishing trade networks, accumulating social norms, elevating women's power in society, etc. These changes in society have built stronger and stronger defenses against our violent instincts, which have remained relatively unchanged over the course of human history.

Stating the hypothesis this way raises the question of how far we expect this trend to continue. We would be much more interested in saving a world ("World 1") where society continues to grow increasingly less violent and more compassionate, and substantially less interested in saving a world ("World 2") where that trend stops, or where it doesn't go far enough fast enough to prevent huge new sources of suffering.

Moral circle expansion is a strategy to make our world more like the first one and less like the second one. Unlike the strategies discussed in this post, it doesn't deal with affecting the likelihood of extinction scenarios. Rather, it tries to directly influence the speed and direction of the trends that determine the expected value of the long-term future (a.k.a. it tries to shift us to a world more worth saving). For what it's worth, I think Sentience Institute is doing some really valuable research on this topic.

But we might oppose efforts like the Nuclear Threat Initiative, which disproportionately save violent-psychology worlds.

Does it? It seems a lot of risk comes from accidental catastrophes rather than intentional ones. Accidental catastrophes to me don't seem like proof of the future being violent.

Also, I think that we should treat our efforts to reduce the risk of intentional catastrophe or inflicted suffering as evidence. Why wouldn't the fact that we choose to reduce the impact of malicious actors be proof that malicious actors' impact will be curtailed by other actors in the future?

Closely related, and also important, is the question of "which world gets precluded". Different possibilities include:

1. By reducing extinction risk from a (hypothetical) scenario in which Earth explodes and falls into pieces, we preclude a world in which there's no life (and therefore no powerful agent) on what previously was Earth.
2. By reducing extinction risk from pandemics, we preclude a world in which there's no human on Earth, but possibly other intelligent species that have evolved to fill the niche previously occupied by humans.
3. By reducing extinction risk from unaligned AGI, we preclude a world in which there's no human on Earth, but powerful unaligned AGIs, who may then start its conquest to the broader universe.
4. By reducing non-extinction existential risks (e.g. disappointing futures), we preclude a world in which humans continue to exist, but fail to realize their potential in doing good.

What will the rankings be like, if we sort the four precluded worlds in decreasing order of badness?  I'm highly unsure either, but I would guess something like 4>2>3>1 (larger means worse).

Very interesting post.

But it seems to me that this argument assumes a relatively stable, universal, and fixed "human nature", and that that's a quite questionable assumption.

For example, the fact that a person was going to start a nuclear war that would've wiped out humanity may not give much evidence about how people tend to behave if in reality behaviours are quite influenced by situations. Nor would it give much evidence about how people in general tend to behave if behaviours vary substantially between different people. Even if behavioural patterns are quite stable and universal, if they're at least quite manipulable then the fact that person would've started that war only gives strong evidence about current behavioural tendencies, not what we're stuck with in the long term. (I believe this is somewhat similar to Cameron_Meyer_Shorb's point.)

Under any of those conditions, the fact that person would've started that war provides little evidence about typical human behavioural patterns in the long term, and thus little evidence about the potential value of the long-term.

I suspect that there's at least some substantial stability and universality to human behaviours. But on the other hand there's certainly evidence that situational factors often important and that different people vary substantially (https://www.ncbi.nlm.nih.gov/pubmed/20550733).

Personally, I suspect the most important factor is how manipulable human behavioural patterns are. The article cited above seems to show a huge degree to which "cultural" factors influence many behavioural patterns, even things we might assume are extremely basic or biologically determined like susceptibility to optical illusions. And such cultural factors typically aren't even purposeful interventions, let alone scientific ones.

It's of course true that a lot of scientific efforts to change behaviours fail, and even when they succeed they typically don't succeed for everyone. But some things have worked on average. And the social sciences working on behavioural change are very young in the scheme of things, and their methods and theories are continually improving (especially after the replication crisis).

Thus, it seems very plausible to me that even within decade we could develop very successful methods of tempering violent inclinations, and that in centuries far more could be done. And that's all just focusing on our "software" - efforts focusing on our biology itself could conceivably accomplish far more radical changes. That, of course, if we don't wipe ourselves out before this can be done.

I recently heard someone on the 80,000 Hours podcast (can't remember who or which episode, sorry) discussing the idea that we may not yet be ready, in terms of our "maturity" or wisdom, for some of the technologies that seem to be around the corner. They gave the analogy that we might trust a child to have scissors but not an assault rifle. (That's a rough paraphrasing.)

So I think there's something to your argument, but I'd also worry that weighting it too heavily would be somewhat akin to letting the child keep the gun based on the logic that, if something goes wrong, that shows the child would've always been reckless anyway.

How are the mentioned first and second objections distinct?

"Should we incorporate the fact of our own choice to pursue x-risk reduction itself into our estimate of the expected value of the future, as recommended by evidential decision theory, or should we exclude it, as recommended by causal?"

I fail to get the meaning. Could anybody reword this for me?

"The consideration is that, even if we think the value of the future is positive and large, the value of the future conditional on the fact that we marginally averted a given x-risk may not be."

Not sure I get this. Is a civilisation stalling irrevocably into chaos after narrowly surviving a pandemic a central example of this?

Dear all,

For example: There are things we currently do not know about human psychology, some of which bear on how inclined we are toward peace and cooperation. Perhaps Steven Pinker is right, and violence will continue its steady decline, until one evening sees the world's last bar fight and humanity is at peace forever after.

• My belief is that peace and cooperation occurs when the marginally benefit of doing so outweighs the benefit of not doing so. The incremental benefit of an individual to decide to join cooperate in a group would essential be based on the decision of whether the marginal utility of doing so outweighs the decision to not do so. A fundamental aspect of the human condition is that it generates marginal utility to even sense a physical forms of what bears similarities to you (as we are going to the basest of human conditions) and hence, at heart of the human condition is a underlying base for peace and cooperation. And hence of course, the next important question to ask is then, to what extent of our physical conditions of the world and ourselves bearing upon these base conditions would trigger a non-cooperation decision.

Once we start thinking along these lines, we open various cans of worms. If our x-risk reduction effort starts far "upstream", e.g. with an effort to make people more cooperative and peace-loving in general, to what extent should we take the success of the intermediate steps (which must succeed for the x-risk reduction effort to succeed) as evidence that the saved world would go on to a great future? Should we incorporate the fact of our own choice to pursue x-risk reduction itself into our estimate of the expected value of the future, as recommended by evidential decision theory, or should we exclude it, as recommended by causal? How should we generate all these conditional expected values, anyway?

My beliefs is that teachings and information dissemination can only go so far to make people more cooperative and peace loving. The final limitations we face are the fundamental human conditions that we are born with, and of course, if if we set out to improve and enhance the base fundamentals attributable to our biological conditions based on a concept of positive value creation benchmarked to the fundamental human conditions since the Big Bang, i believe that these should tend to magnitudes of infinity and a near zero concept of our physical vessel will be ideal for humanity.

The way i would think of the dilemma between EDT and Casual theory would be that, we have as humans, thrive since the origin as a result of cooperation and more importantly, a love for ourselves and those around us, and that i think people generally don't die, even from having too much of a good thing.. but too much of a bad thing, on a balance of probability kills more than the former.

Thank you, i will be happy to hear all of your comments on this

Kind regards,

Wei Lun