Comments
Inevitably, utilitarians would bite the bullet here, since ex hypothesi, there is more utility in the world in which all beings are replaced with beings with higher utility.
I think the question is whether this implication renders utilitarianism implausible. I have several observations.
(1) The assumption here of the thought experiment is that the correct way to assess moral theories is to test them against intuitions about lots of particular cases. And utilitarianism has plenty of counterintuive implications about particular cases: eg the one in the main post, the repugnant conclusion, counting sadistic pleasure, and so on ad infinitum. The problem is that I don't think this is the correct way to assess moral theories.
Many of the moral intuitions people have are best explained by the fact that those intuitions would be useful to have in the ancestral environment, rather than that they apprehend moral reality. eg incest taboos are strong across all cultures, as are beliefs that wrongdoers simply deserve punishment regardless of the benefits of punishment. These would be evolutionarily useful, which makes it hard for us to shake these beliefs. I don't think the belief that subjective wellbeing is intrinsically good is debunkable in the same way, though discussing that is beyond the scope of this post.
Analogy: the current state of moral philosophy is similar to maths if mathematicians judged mathematical proofs and theories on the basis of how intuitive they are. Thus, people's intuition against the Monty Hall Problem was thought to be a good reason to try to build an alternative theory of probability. This form of maths wouldn't get very far. By the same token, moral philosophy doesn't get far in producing agreement because it uses a predictably bad moral epistemology that overwhelmingly focuses on intuitions about particular cases.
(2) Rough outline argument:
a. Subjective experience is all that matters about the world. (Imagine a world without subjective experience - why would it matter? Imagine a world in which people complete their plans but feel nothing - why would it be good?)
b. Personal identity doesn't matter. (See Parfit. or Imagine if you were vaporised in your sleep and then a perfect clone appeared a millisecond afterwards. Why would this be bad?)
From a and b, with some plausible additional premises, you eventually end up with utilitarianism. This means you have to bite the bullet mentioned in the text, and you also find the bullet plausible because you accept a and b and the other premises.
Related to (1), I think a response to utilitarianism that started in the right way would attack these basic premises a and b, along with the other premises. eg It would try and show that something aside from subjective experience matters fundamentally.
I don't identify as a utilitarian, but I am more sympathetic to consequentialism than the vast majority of people, and reject such thought experiments (even in the unlikely event they weren't practically self-defeating: utilitarians should want to modify their ideology and self-bind so that they won't do things that screw over the rest of society/other moral views, so that they can reap the larger rewards of positive-sum trades rather than negative-sum conflict). The contractarian (and commonsense and pluralism, but the theory I would most invoke for theoretical understanding is contractarian) objection to such things greatly outweighs the utilitarian case.
For the tiny population of Earth today (which is astronomically small compared to potential future populations) the idea becomes even more absurd. I would agree with Bostrom in Superintelligence (page 219) that failing to leave one galaxy, let alone one solar system for existing beings out of billions of galaxies would be ludicrously monomaniacal and overconfident (and ex ante something that 100% convinced consequentialists would have very much wanted to commit to abstain from).
If we are concerned with how vulnerable moral theories such as traditional total act-utilitarianism and various other forms of consequentialism are to replacement arguments, I think much more needs to be said. Here are some examples.
1. Suppose the agent is very powerful, say, the leader of a totalitarian society on Earth that can dominate the other people on Earth. This person has access to technology that could kill and replace either everyone on Earth or perhaps everyone except a cluster of the leader’s close, like-minded allies. Roughly, this person (or the group of like-minded people the leader belongs to) is so powerful that the wishes of others on Earth who disagree can essentially be ignored from a tactical perspective. Would it be optimal for this agent to kill and replace either everyone or, for example, at least everyone in other societies who might otherwise get in the way of the maximization of the sum of well-being ?
2. You talk about modifying one’s ideology, self-bind and commit, but there are questions about whether humans can do that. For example, if some agent in the future would be about to be able to kill and replace everyone, can you guarantee that this agent will be able to change ideology, self-bind and commit to not killing? It would not be sufficient that some or most humans could change ideology, self-bind and commit.
3. Would it be optimal for every agent in every relevant future situation to change ideology, self-bind or commit to not kill and replace everyone or billions of individuals? Again, we can consider a powerful, ruthless dictator or totalitarian leader. Assume this person has so far neither modified nor commited to non-violence. This agent is then in a situation in which the agent could kill and replace everyone. Would it at that time be optimal for the leader to change ideology, self-bind or commit to not killing and replacing everyone?
The first words of my comment were "I don't identify as a utilitarian" (among other reasons because I reject the idea of things like feeding all existing beings to utility monsters for a trivial proportional gains to the latter, even absent all the pragmatic reasons not to; even if I thought such things more plausible it would require extreme certainty or non-pluralism to get such fanatical behavior).
I don't think a 100% utilitarian dictator with local charge of a society on Earth removes pragmatic considerations, e.g. what if they are actually a computer simulation designed to provide data about and respond to other civilizations, or the principle of their action provides evidence about what other locally dominant dictators on other planets will do including for other ideologies, or if they contact alien life?
But you could elaborate on the scenario to stipulate such things not existing in the hypothetical, and get a situation where your character would commit atrocities, and measures to prevent the situation hadn't been taken when the risk was foreseeable.
That's reason for everyone else to prevent and deter such a person or ideology from gaining the power to commit such atrocities while we can, such as in our current situation. That would go even more strongly for negative utilitarianism, since it doesn't treat any life or part of life as being intrinsically good, regardless of the being in question valuing it, and is therefore even more misaligned with the rest of the world (in valuation of the lives of everyone else, and in the lives of their descendants). And such responses give reason even for utilitarian extremists to take actions that reduce such conflicts.
Insofar as purely psychological self-binding is hard, there are still externally available actions, such as visibly refraining from pursuit of unaccountable power to harm others, and taking actions to make it more difficult to do so, such as transferring power to those with less radical ideologies, or ensuring transparency and accountability to them.
Carl, you write that you are “more sympathetic to consequentialism than the vast majority of people.” The original post by Richard is about utilitarianism and replacement thought experiments but I guess he is also interested in other forms of consequentialism since the kind of objection he talks about can be made against other forms of consequentialism too.
The following you write seems relevant to both utilitarianism and other forms of consequentialism:
Even if these other pragmatic considerations you mention would not be removed by having control of Earth, the question remains whether they (together with other considerations) are sufficient to make it suboptimal to kill and replace everyone. What if the likelihood that they are in a simulation is not high enough? What if new scientific discoveries about the universe or multiverse indicate that taking into account agents far away from Earth is not so important?
You say,
I don’t mean that the only way to object to the form of consequentialism under consideration is to stipulate away such things and assume they do not exist. One can also object that what perhaps make it suboptimal to kill and replace everyone are complicated and speculative considerations about living in a simulation or what beings on other planets will do. Maybe your reasoning about such things is flawed somewhere or maybe new scientific discoveries will speak against such considerations. In which case (as I understand you) it may become optimal for the leader we are talking about to kill and replace everyone.
You bring up negative utilitarianism. As I write in my paper, I don’t think negative utilitarianism is worse off than traditional utilitarianism when it comes to these scenarios that involve killing everyone. The same goes for negative vs. traditional consequentialism or the comparison negative vs. traditional consequentialist-leaning morality. I would be happy to discuss that more, but I guess it would be too off-topic given the original post. Perhaps a new separate thread would be appropriate for that.
You write,
In that case the ideology (I would say morality) is not restricted to forms of utilitarianism but also include many forms of consequentialism and views that are consequentialist-leaning. It may also include views that are non-consequentialist but open to that killing is sometimes right if it is done to accomplish a greater goal, and that, for example, place huge importance on the far future so that far future concerns make what happens to the few billion humans on Earth a minor consideration. My point is that I think it’s a mistake to merely talk about utilitarianism or consequentialism here. The range of views about which one can reasonable ask ‘would it be right to kill everyone in this situation, according to this theory?’ is much wider.
Another variant along these lines:
Suppose almost all humans adopt utilitarianism as their moral philosophy and fully colonize the universe, and then someone invents the technology to kill humans and replace them with beings of greater well-being.
Utilitarianism seems to imply that humans who are utilitarians should commit mass suicide in order to bring the new beings into existence.EDIT: Utilitarianism endorses humans voluntarily replacing themselves with these new beings. (New wording suggested by Richard Ngo.)Well, that is their own belief and they aren't (necessarily) harming others, so I don't think that's nearly as repugnant.
I think if you propose a hypothetical where everyone agrees with what should be done and everyone accepts the consequences, then disagreeing with the outcome seems worse to me than accepting it. In effect, you want your views to trump everyone else's.
Yea, one can formulate many variants. I can't recall seening yours before. The following is one thing that might seem like nitpicking, but which I think it is quite important: In academia, it seems standard to formulate utilitarianism and other consequentialist theories so that they apply to everyone. For example,
Traditional total act-utilitarianism: An act is right if and only if it results in a sum of well-being (positive well-being minus negative well-being) that is at least as great as that resulting from any other available act.
These theories are not formulated as 'traditional utilitarians ought to ...'. I can't recall ever seeing a version of utilitarianism or consequentialism formluated as 'utilitarians/consequentialists ought to ...'.
So when you write "Utilitarianism seems to imply that humans who are utilitarians should" I would rephrase as 'Utilitarianism seems to imply that humans should' since utilitarianism applies to all agents not only utilitarians. But perhaps you mean 'utilitarianism seems to imply that humans, including those who are utilitarians, should...' which would make sense.
Why does my nitpicking matter? One reason is when thinking about scenarios or thought experiments. For example, I don't think one can reply to world destruction or replacement arguments by saying 'a consequentialist ought to not kill everyone because ...'. We can picture a dictator who has never heard of consequentialism, and who is just about to act out of hatred. And we can ask, 'According to the traditional total act-utilitarian criterion of rightness (i.e. an act is right if and only if ...), would the dictator taking action X (say, killing everyone) be right?
Another reason the nitpicking matters is when thinking about the plausibility of the theories. A theory might sound nicer and more appealing if it merely says 'Those who endorse this theory act in way X' rather than as they are usually roughly formulated 'Everyone act in way X, regardless of whether you endorse this theory or not'.
You seem to be assuming moral realism, so that "Utilitarianism seems to imply ..." gets interpreted as "If utilitarianism is the true moral theory for everyone, then everyone should ..." whereas I'm uncertain about that. In particular if moral relativism, subjectivism, or anti-realism is true, then "Utilitarianism seems to imply ..." has to be interpreted as "If utilitarianism is the right moral theory for someone or represents their moral preferences, then that person should ..." So I think given my meta-ethical uncertainty, the way I phrased my statement actually does make sense. (Maybe it's skewed a bit towards anti-realism by implicature, but is at least correct in a literal sense even if realism is true.)
I think that "utilitarianism seems to imply that humans who are utilitarians should..." is a type error regardless of whether you're a realist or an anti-realist, in the same way as "the ZFC axioms imply that humans who accept those axioms should believe 1+1=2". That's not what the ZFC axioms imply - actually, they just imply that 1+1 = 2, and it's our meta-theory of mathematics which determines how you respond to this fact. Similarly, utilitarianism is a theory which, given some actions (or maybe states of the world, or maybe policies) returns a metric for how "right" or "good" they are. And then how we relate to that theory depends on our meta-ethics.
Given how confusing talking about morality is, I think it's important to be able to separate the object-level moral theories from meta-ethical theories in this way. (For more along these lines, see my post here).
I'd have to think more about this, but my initial reaction is that this makes sense. How would you suggest changing my sentence in light of this? The following is the best I can come up with. Do you see anything still wrong with it, or any further improvements that can be made?
"It seems that (at least) the humans who are utilitarians should commit mass suicide in order to bring the new beings into existence, because that's what utilitarianism implies is the right action in that situation."
My first objection is that you're using a different form of "should" than what is standard. My preferred interpretation of "X should do Y" is that it's equivalent to "I endorse some moral theory T and T endorses X doing Y". (Or "according to utilitarianism, X should do Y" is more simply equivalent to "utilitarianism endorses X doing Y"). In this case, "should" feels like it's saying something morally normative.
Whereas you seem to be using "should" as in "a person who has a preference X should act on X". In this case, should feels like it's saying something epistemically normative. You may think these are the same thing, but I don't, and either way it's confusing to build that assumption into our language. I'd prefer to replace this latter meaning of "should" with "it is rational to". So then we get:
"it is rational for humans who are utilitarians to commit mass suicide in order to bring the new beings into existence, because that's what utilitarianism implies is the right action."
My second objection is that this is only the case if "being a utilitarian" is equivalent to "having only one preference, which is to follow utilitarianism". In practice people have both moral preferences and also personal preferences. I'd still count someone as being a utilitarian if they follow their personal preferences instead of their moral preferences some (or even most) of the time. So then it's not clear whether it's rational for a human who is a utilitarian to commit suicide in this case; it depends on the contents of their personal preferences.
I think we avoid all of this mess just by saying "Utilitarianism endorses replacing existing humans with these new beings." This is, as I mentioned earlier, a similar claim to "ZFC implies that 1 + 1 = 2", and it allows people to have fruitful discussions without agreeing on whether they should endorse utilitarianism. I'd also be happy with Simon's version above: "Utilitarianism seems to imply that humans should...", although I think it's slightly less precise than mine, because it introduces an unnecessary "should" that some people might take to be a meta-level claim rather than merely a claim about the content of the theory of utilitarianism (this is a minor quibble though. Analogously: "ZFC implies that 1 + 1 = 2 is true").
Anyway, we have pretty different meta-ethical views, and I'm not sure how much we're going to converge, but I will say that from my perspective, your conflation of epistemic and moral normativity (as I described earlier) is a key component of why your position seems confusing to me.
I originally wrote a different response to Wei's comment, but it wasn't direct enough. I'm copying the first part here since it may be helpful in explaining what I mean by "moral preferences" vs "personal preferences":
Each person has a range of preferences, which it's often convenient to break down into "moral preferences" and "personal preferences". This isn't always a clear distinction, but the main differences:
1. Moral preferences are much more universalisable and less person-specific (e.g. "I prefer that people aren't killed" vs "I prefer that I'm not killed").
2. Moral preferences are associated with a meta-preference that everyone has the same moral preferences. This is why we feel so strongly that we need to find a shared moral "truth". Fortunately, most people are in agreement in our societies on the most basic moral questions.
3. Moral preferences are associated with a meta-preference that they are consistent, simple, and actionable. This is why we feel so strongly that we need to find coherent moral theories rather than just following our intuitions.
4. Moral preferences are usually phrased as "X is right/wrong" and "people should do right and not do wrong" rather than "I prefer X". This often misleads people into thinking that their moral preferences are just pointers to some aspect of reality, the "objective moral truth", which is what people "objectively should do".
When we reflect on our moral preferences and try to make them more consistent and actionable, we often end up condensing our initial moral preferences (aka moral intuitions) into moral theories like utilitarianism. Note that we could do this for other preferences as well (e.g. "my theory of food is that I prefer things which have more salt than sugar") but because I don't have strong meta-preferences about my food preferences, I don't bother doing so.
The relationship between moral preferences and personal preferences can be quite complicated. People act on both, but often have a meta-preference to pay more attention to their moral preferences than they currently do. I'd count someone as a utilitarian if they have moral preferences that favour utilitarianism, and these are a non-negligible component of their overall preferences.
I found this very helpful.
As I mentioned earlier, I am uncertain about meta-ethics, so I was trying to craft a sentence that would be true under a number of different meta-ethical theories. I wrote "should" instead of "it is rational to" because under moral realism that "should" could be interpreted as a "moral should" while under anti-realism it could be interpreted as an "epistemic should". (I also do think there may be something in common between moral and epistemic normativity but that's not my main motivation.) Your suggestion “Utilitarianism endorses replacing existing humans with these new beings.” would avoid this issue, but the main reason I wrote my original comment was to create a thought experiment where concerns about moral uncertainty and contractarianism clearly do not apply, and “Utilitarianism endorses replacing existing humans with these new beings.” doesn't really convey that since you could say that even in scenarios where moral uncertainty and contractarianism do apply.
Using those two different types of "should" makes your proposed sentence ("It seems that (at least) the humans who are utilitarians should commit mass suicide in order to bring the new beings into existence, because that's what utilitarianism implies is the right action in that situation.") unnecessarily confusing, for a couple of reasons.
1. Most moral anti-realists don't use "epistemic should" when talking about morality. Instead, I claim, they use my definition of moral should: "X should do Y means that I endorse/prefer some moral theory T and T endorses X doing Y". (We can test this by asking anti-realists who don't subscribe to negative utilitarianism whether a negative utilitarian should destroy the universe - I predict they will either say "no" or argue that the question is ambiguous.) And so introducing "epistemic should" makes moral talk more difficult.
2. Moral realists who are utilitarians and use "moral should" would agree with your proposed sentence, and moral anti-realists who aren't utilitarians and use "epistemic should" would also agree with your sentence, but for two totally different reasons. This makes follow-up discussions much more difficult.
How about "Utilitarianism endorses humans voluntarily replacing themselves with these new beings." That gets rid of (most of) the contractarianism. I don't think there's any clean, elegant phrasing which then rules out the moral uncertainty in a way that's satisfactory to both realists and anti-realists, unfortunately - because realists and anti-realists disagree on whether, if you prefer/endorse a theory, that makes it rational for you to act on that theory. (In other words, I don't know whether moral realists have terminology which distinguishes between people who act on false theories that they currently endorse, versus people who act on false theories they currently don't endorse).
Very interesting :) I don’t mean to be assuming moral realism, and I don’t think of myself as a realist. Suppose I am an antirealist and I state some consequentialist criterion of rightness: ‘An act is right if and only if…’. When stating that, I do not mean or claim that it is true in a realist sense. I may be expressing my feelings, I may encourage others to act according to the criterion of rightness, or whatever. At least I would not merely be talking about how I prefer to act. I would mean or express roughly ‘everyone, your actions and mine are right if and only if …’. But regardless of whether I would be speaking about myself or everyone, we can still talk about what the criterion of rightness (the theory) implies in the sense that one can check which actions satisfy the criteria. So we can say: according to the theory formulated as ‘an act is right if and only if…’ this act X would be right (simply because it satisfies the criteria). A simpler example is if we understand the principle ‘lying is wrong’ from an antirealist perspective. Assuming we specify what counts as lying, we can still talk about whether an act is a case of lying and hence wrong, according to this principle. And then one can discuss whether the theory or principle is appealing, given which acts it classifies as right and wrong. If repugnant action X is classified as right or if something obviously admirable act is classified as wrong, we may want to reject the theory/criterion, regardless of realism or antirealism.
Maybe all I’m saying is obvious and compatible with what you are saying.
I think there is at least one plausible meta-ethical position under which when I say "I think utilitarianism is right." I just mean something like "I think that after I reach reflective equilibrium my preferences will be well-described by utilitarianism." and it is not intended to mean that I think utilitarianism is right for anyone else or applies to anyone else or should apply to anyone else (except insofar as they are sufficiently similar to myself in the relevant ways and therefore are likely to reach a similar reflective equilibrium). Do you agree this is a plausible meta-ethical position? If yes, does my sentence (or the new version that I gave in the parallel thread) make more sense in light of this? In either case, how would you suggest that I rephrase my sentence to make it better?
Sure. I’ll use traditional total act-utilitarianism defined as follows as the example here so that it’s clear what we are talking about:
Traditional total act-utilitarianism: An act is right if and only if it results in a sum of well-being (positive well-being minus negative well-being) that is at least as great as that resulting from any other available act.
I gather the metaethical position you describe is something like one of the following three:
(1) When I say ‘I think utilitarianism is right’ I mean ‘I think that after I reach reflective equilibrium I will think that any act I perform is right if and only if it results in a sum of well-being (positive well-being minus negative well-being) that is at least as great as that resulting from any other available act.’
This (1) was about which of your actions will be right. Alternatively, the metaethical position could be as follows:
(2) When I say ‘I think utilitarianism is right’ I mean ‘I think that after I reach reflective equilibrium I will think that any act anyone performs is right if and only if it results in a sum of well-being (positive well-being minus negative well-being) that is at least as great as that resulting from any other available act.’
Or perhaps formulating it in terms of want or preference instead of rightness, like the following, better describes your metaethical position (using utilitarianism as just an example):
(3) When I say ‘I think utilitarianism is right’ I mean ‘I think that after I reach reflective equilibrium I will want or have a preference for that everyone act in a way that results in a sum of well-being (positive well-being minus negative well-being) that is at least as great as that resulting from any other available act.’
My impression is that in the academic literature, metaethical theories/positions are usually, always or almost always formulated as general claims about what, for example, statements such as ‘one ought to be honest’ means; the metaethical theories/position do not have the form ‘when I say “one ought to be honest” I mean …’ But, sure, talking, as you do, about what you mean when you say ‘I think utilitarianism is right’ sounds fine.
The new version of your thought experiment sounds fine, which I gather would go something like the following:
Suppose almost all humans adopt utilitarianism as their moral philosophy and fully colonize the universe, and then someone invents the technology to kill humans and replace humans with beings of greater well-being. (Assume it would be optimal, all things considered, to kill and replace humans.) Utilitarianism seems to imply that at least humans who are utilitarians should commit mass suicide (or accept being killed) in order to bring the new beings into existence, because that's what utilitarianism implies is the optimal and hence morally right action in that situation.
It is worth noting that this is not, as it stands, a reply available to a pure traditional utilitarian.
But a relevant question here is whether that also holds true given a purely utilitarian view, as opposed to, say, from a perspective that relies on various theories in some notional moral parliament.
It is, of course, perfectly fine to respond to the question "how do most utilitarians feel about X?" by saying "I'm not a utilitarian, but I am sympathetic to it, and here is how someone sympathetic to utilitarianism can reply by relying on other moral frameworks". But then it's worth being clear that the reply is not a defense of pure traditional utilitarianism — quite the contrary.
That's why the very first words of my comment were "I don't identify as a utilitarian."
I think I, and maybe others, are still confused about the point of your top-level comment. Simon Knutsson's argument is against utilitarianism, and I think Richard Ngo wanted to see if there was a good counter-argument against it from a utilitarian perspective, or if a utilitarian just has to "bite the bullet". It seems like the motivation for both people were to try to figure out whether utilitarianism is the right moral philosophy / correct normative ethics.
Your reply doesn't seem to address their motivation, which is why I'm confused. (If utilitarianism is the right moral philosophy then it would give the right action guidance even if one was 100% sure of it and other considerations such as contractarianism didn't apply, so it seems beside the point to talk about contractarianism and overconfidence.) Is the point that utilitarianism probably isn't right, but some other form of consequentialism is? If so, what do you have in mind?
I appreciate that, and as I noted, I think this is fine. :-)
I just wanted to flag this because it took me some time to clarify whether you were replying based on 1) moral uncertainty/other frameworks, or 2) instrumental considerations relative to pure utilitarianism. I first assumed you were replying based on 2) (as Brian suggested), and I believe many others reading your answer might draw the same conclusion. But a closer reading made it clear to me you were primarily replying based on 1).
I think the idea is that even a pure utilitarian should care about contractarian-style thinking for almost any practical scenario, even if there are some thought experiments where that's not the case.
I think that is basically true in practice, but I am also saying that even absent those pragmatic considerations constraining utilitarianism, I still would hold these other non-utilitarian normative views and reject things like not leaving some space for existing beings for a tiny proportional increase in resources for utility monsters.
Okay, thanks. So I guess the thing I'm curious about now is: what heuristics do you have for deciding when to prioritise contractarian intuitions over consequentialist intuitions, or vice versa? In extreme cases where one side feels very strongly about it (like this one) that's relatively easy, but any thoughts on how to extend those to more nuanced dilemmas?