The idea that the future might contain astronomical amounts of suffering, and that we should work to prevent such worst-case outcomes, has lately attracted some attention. I've written this FAQ to help clarify the concept and to clear up potential misconceptions.

[Crossposted from my website on s-risks.]

General questions

What are s-risks?

In the essay Reducing Risks of Astronomical Suffering: A Neglected Priority, s-risks (also called suffering risks or risks of astronomical suffering) are defined as “events that would bring about suffering on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far”.

If you’re not yet familiar with the idea, you can find out more by watching Max Daniel's EAG Boston talk or by reading the introduction to s-risks.

Can you give an example of what s-risks could look like?

In the future, it may become possible to run such complex simulations that the (artificial) individuals inside these simulations are sentient. Nick Bostrom coined the term mindcrime for the idea that the thought processes of a superintelligent AI might cause intrinsic moral harm if they contain (suffering) simulated persons. Since there are instrumental reasons to run many such simulations, this could lead to vast amounts of suffering. For example, an AI might use simulations to improve its knowledge of human psychology or to predict what humans would do in a conflict situation.

Other common examples include suffering subroutines and spreading wild animal suffering to other planets.

Isn’t all that rather far-fetched?

At first glance, one could get the impression that s-risks are just unfounded speculation. But to dismiss s-risks as unimportant (in expectation), one would have to be highly confident that their probability is negligible, which is hard to justify upon reflection. The introduction to s-risks gives several arguments why the probability is not negligible after all:

First, s-risks are disjunctive. They can materialize in any number of unrelated ways. Generally speaking, it’s hard to predict the future and the range of scenarios that we can imagine is limited. It is therefore plausible that unforeseen scenarios – known as black swans – make up a significant fraction of s-risks. So even if any particular dystopian scenarios we can conceive of is highly unlikely, the probability of some s-risk may still be non-negligible.

Second, while s-risks may seem speculative at first, all the underlying assumptions are plausible. [...]

Third, historical precedents do exist. Factory farming, for instance, is structurally similar to (incidental) s-risks, albeit smaller in scale. In general, humanity has a mixed track record regarding responsible use of new technologies, so we can hardly be certain that future technological risks will be handled with appropriate care and consideration.

Which value systems should care about reducing s-risks?

Virtually everyone would agree that (involuntary) suffering should, all else equal, be avoided. In other words, ensuring that the future does not contain astronomical amounts of suffering is a common denominator of almost all (plausible) value systems.

Work on reducing s-risks is, therefore, a good candidate for compromise between different value systems. Instead of narrowly pursuing our own ethical views in potential conflict with others, we should work towards a future deemed favourable by many value systems.

The future

Aren't future generations in a much better position to do something about this?

Future generations will probably have more information about s-risks in general, including which ones are the most serious, which does give them the upper hand in finding effective interventions. One might, therefore, argue that later work has a significantly higher marginal impact. However, there are also arguments for working on s-risks now.

First, thinking about s-risks only as they start to materialize does not suffice because it might be too late to do anything about it. Without sufficient foresight and caution, society may already be “locked in” to a trajectory that ultimately leads to a bad outcome.

Second, one main reason why future generations are in a better position is that they can draw on previous work. Earlier work – especially research or conceptual progress – can be effective in that it allows future generations to more effectively reduce s-risk.

Third, even if future generations are able to prevent s-risks, it’s not clear whether they will care enough to do so. We can work to ensure this by growing a movement of people who want to reduce s-risks. In this regard, we should expect earlier growth to be more valuable than later growth.

Fourth, if there’s a sufficient probability that smarter-than-human AI will be built in this century, it's possible that we already are in a unique position to influence the future. If it’s possible to work productively on AI safety now, then it should also be possible to reduce s-risks now.

Toby Ord’s essay The timing of labour aimed at reducing existential risk addresses the same question for efforts to reduce x-risks. He gives two additional reasons in favor of earlier work: namely, the possibility of changing course (which is more valuable if done early on) and the potential for self-improvement.

Seeing as humans are (at least somewhat) benevolent and will have advanced technological solutions at their disposal, isn’t it likely that the future will be good anyway?

If you are (very) optimistic about the future, you might think that s-risks are unlikely for this reason (which is different from the objection that s-risks seem far-fetched). A common argument is that avoiding suffering will become easier with more advanced technology; since humans care at least a little bit about reducing suffering, there will be less suffering in the future.

While this argument has some merit, it’s not airtight. By default, when we humans encounter a problem in need of solving, we tend to implement the most economically efficient solution, often irrespective of whether it involves large amounts of suffering. Factory farming provides a good example of such a mismatch; faced with the problem of producing meat for millions of people as efficiently as possible, a solution was implemented which happened to involve an immense amount of nonhuman suffering.

Also, the future will likely contain vastly larger populations, especially if humans colonize space at some point. All else being equal, such an increase in population may also imply (vastly) more suffering. Even if the fraction of suffering decreases, it's not clear whether the absolute amount will be higher or lower.

If your primary goal is to reduce suffering, then your actions matter less if the future will 'automatically' be good (because the future contains little or no suffering anyway). Given sufficient uncertainty, this is reason to focus on the possibility of bad outcomes anyway for precautionary reasons. In a world where s-risks are likely, we can have more impact.

Does it only make sense to work on s-risks if one is very pessimistic about the future?

Although the degree to which we are optimistic or pessimistic about the future is clearly relevant to how concerned we are about s-risks, one would need to be unusually optimistic about the future to rule out s-risks entirely.

From the introduction to s-risks:

Working on s-risks does not require a particularly pessimistic view of technological progress and the future trajectory of humanity. To be concerned about s-risks, it is sufficient to believe that the probability of a bad outcome is not negligible, which is consistent with believing that a utopian future free of suffering is also quite possible.

In other words, being concerned about s-risks does not require unusual beliefs about the future.

S-risks and x-risks

How do s-risks relate to existential risks (x-risks)? Are s-risks a subclass of x-risks?

First, recall Nick Bostrom’s definition of x-risks:

Existential risk – One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.

S-risks are defined as follows:

S-risks are events that would bring about suffering on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far.

According to these definitions, both x-risks and s-risks relate to shaping the long-term future, but reducing x-risks is about actualizing humanity’s potential, while reducing s-risks is about preventing bad outcomes.

There are two possible views on the question of whether s-risks are a subclass of x-risks.

According to one possible view, it’s conceivable to have astronomical amounts of suffering that do not lead to extinction or curtail humanity’s potential. We could even imagine that some forms of suffering (such as suffering subroutines) are instrumentally useful to human civilization. Hence, not all s-risks are also x-risks. In other words, some possible futures are both an x-risk and an s-risk (e.g. uncontrolled AI), some would be an x-risk but not an s-risk (e.g. an empty universe), some would be an s-risk but not an x-risk (e.g. suffering subroutines), and some are neither.

		S-risk?
		Yes	No
X-risk?	Yes	Uncontrolled AI	Empty universe
	No	Suffering subroutines	Utopian future

The second view is that the meaning of “potential” depends on your values. For example, you might think that a cosmic future is only valuable if it does not contain (severe) suffering. If “potential” refers to the potential of a utopian future without suffering, then every s-risk is (by definition) an x-risk, too.

How do I decide whether reducing extinction risks or reducing s-risks is more important?

This depends on each of us making difficult ethical judgment calls. The answer depends on how much you care about reducing suffering versus increasing happiness, and how you would make tradeoffs between the two. (This also raises fundamental questions about how happiness and suffering can be measured and compared.)

Proponents of suffering-focused ethics argue that the reduction of suffering is of primary moral importance, and that additional happiness cannot easily counterbalance (severe) suffering. According to this perspective, preventing s-risks is morally most urgent.

Other value systems, such as classical utilitarianism or fun theory, emphasize the creation of happiness or other forms of positive value, and assert that the vast possibilities of a utopian future can outweigh s-risks. Although preventing s-risks is still valuable in this view, it is nevertheless considered even more important to ensure that humanity has a cosmic future at all by reducing extinction risks.

In addition to normative issues, the answer also depends on the empirical question of how much happiness and suffering the future will contain. David Althaus suggests that we consider both the normative suffering-to-happiness trade ratio (NSR), which measures how we would trade off suffering and happiness in theory, and the expected suffering-to-happiness ratio (ESR), which measures the (relative) amounts of suffering and happiness we expect in the future.

In this framework, those who emphasize happiness (low NSR) or are optimistic about the future (low ESR) will tend to focus on extinction risk reduction. If the product of NSR and ESR is high – either because of a normative emphasis on suffering (high NSR) or pessimistic views about the future (high ESR) – it’s more plausible to focus on s-risk-reduction instead.

Miscellaneous

Is the concept of s-risks tied to the possibility of AGI and artificial sentience?

Many s-risks, such as suffering subroutines or mindcrime, have to do with artificial minds or smarter-than-human AI. But the concept of s-risks is not conceptually dependent on the possibility of AI scenarios. For example, spreading wild animal suffering to other planets does not require artificial sentience or AI.

Examples often involve artificial sentience, however, due to the vast number of artificial beings that could be created if artificial sentience becomes feasible at any time in the future. Combined with humanity’s track record of insufficient moral concern for “voiceless” beings at our command, this might pose a particularly serious s-risk. (More details here.)

Why would we think that artificial sentience is possible in the first place?

This question has been discussed extensively in the philosophy of mind. Many popular theories of consciousness, such as Global workspace theory, higher-order theories, or Integrated information theory, agree that artificial sentience is possible in principle. Philosopher Daniel Dennett puts it like this:

I’ve been arguing for years that, yes, in principle it’s possible for human consciousness to be realised in a machine. After all, that’s what we are. We’re robots made of robots made of robots. We’re incredibly complex, trillions of moving parts. But they’re all non-miraculous robotic parts.

As an example of the sort of reasoning involved, consider this intuitive thought experiment: if you were to take a sentient biological brain, and replace one neuron after another with a functionally equivalent computer chip, would it somehow make the brain less sentient? Would the brain still be sentient once all of its biological neurons have been replaced? If not, at what point would it cease to be sentient?

The debate is not settled yet, but it seems at least plausible that artificial sentience is possible in principle. Also, we don’t need to be certain to justify moral concern. It’s sufficient that we can't rule it out.

Ok, I’m sold. What can I personally do to help reduce s-risks?

A simple first step is to join the discussion, e.g. in this Facebook group. If more people think and write about the topic (either independently or at EA organizations), we’ll make progress on the crucial question of how to best reduce s-risks. At the same time, it helps build a community that, in turn, can get even more people involved.

If you’re interested in doing serious research on s-risks right away, you could have a look at this list of open questions to find a suitable research topic. Work in AI policy and strategy is another interesting option, as progress in this area allows us to shape AI in a more fine-grained way, making it easier to identify and implement safety measures against s-risks.

Another possibility is to donate to organizations working on s-risks reduction. Currently, the Foundational Research Institute is the only group with an explicit focus on s-risks, but other groups also contribute to solving issues that are relevant for s-risk reduction. For example, the Machine Intelligence Research Institute aims to ensure that smarter-than-human artificial intelligence is aligned with human values, which probably also reduces s-risks. Charities that promote broad societal improvements such as better international cooperation or beneficial values may also contribute to s-risk reduction, albeit in a less targeted way.

[Disclaimer: I'm in close contact with the Foundational Research Institute, but I am not employed there and don't receive any financial compensation.]