Looking into utilitarian aggregation and how it might handle identical copies and subjective continuity. Focused on a potential solution I call the Markov View (or Markov utilitarianism).
Adding to Tristan's 'large enough multiverse' point, and re section iii ('We Might Get Terminated Soon'). There seem to be a few ways to think about personal continuity here. To describe two:
Continuity is causal, and termination of simulations contributes to some probability of "seeing nothing" next equal to the probability of being a simulated copy.
Continuity is content-based, and termination of simulations just means you continue to whatever remains and is sufficiently coherent to be your next moment, e.g. copies in base reality (related to Wei Dai's 'identical copy immortality').
I kinda lean toward 2 > 1. If there’s nothing special about causality that links you from one moment to the next, then what links them might just be that the later moment’s contents coherently follow on from the earlier one’s.
Continuity is ontologically tricky, but in practice we act as if there’s a fact about what we’ll experience next. Overall, I expect to see stuff even after simulations end.
I recently proposed a view I call Markov utilitarianism. To me, it feels like a natural way to think about ethics. I came to it after thinking about how I care about my future, and how I really care about how I feel right now, from the first-person perspective.
I care about my actual subjective futures, weighted by how likely I am to find myself in each, rather than the physical futures someone looking from a third-person perspective implicitly treats as relevant to me. (Physical futures and my subjective futures might come apart because of copying and merging.)
I care about how I feel right now because of the contents of my experience, rather than any elusive "number of subjectively identical copies" someone looking from a third-person perspective would associate with me.
This is prudence, in the philosophical sense of rational concern for one's own wellbeing. And it has exactly the same form as the equation reinforcement learning uses for an agent's value of being in a state, the Bellman equation (hence 'Markov'):
V(t) = r(t) + γ Σ P(t → t') V(t') over types of experience t.
To make it an ethics that considers everybody, we universalize it and sum impartially across all unique perspectives, mine and everyone else's:
U = Σ V(t)
In slogan form: "Prudence is caring about how you feel right now, and about each possible next experience in proportion to how likely you are to find yourself there. Ethics is doing that for every qualitatively unique perspective, counting each once".
Or more precisely: "Count each qualitatively unique perspective once, and care about it as it would care about how it feels now and what it will experience next, if fully informed about its futures".
Markov utilitarianism is closely related to total utilitarianism, but it structurally prefers lives that get better over time, and once minds can be copied and merged, it avoids conclusions like it being permissible to merge people into a being fated for extreme suffering.
So it feels like a natural extension of how you already care about yourself, extended to every perspective. Starting an ethics from this first-person point of view seems better to me because it avoids the third-person assumptions hidden in total utilitarianism.
What if exact copies don't matter but future experiences still do?
I think an ethics resulting from the following two premises is worth exploring:
P1: The total moral value of subjectively indistinguishable observer-moments doesn't scale with the number of copies, i.e. they sum to a constant value.
This is because subjectively indistinguishable observer-moments, in some sense, happen to the same "someone". One could think of themselves as all their copies. N observer-moments that are subjectively indistinguishable aren't felt N times by that same "someone". Only felt "once".
E.g. Even if there were multiple exact copies of me, of my current present experience, I might not really care personally because it doesn't seem to change anything for me. "I" don't notice anything.
We can say that multiple "tokens" of subjectively indistinguishable observer-moments belong to the same "type" of observer-moment.
P2: What matters, and can be changed, for each type of observer-moment is the distribution of its subjectively "future" experiences.
You can't "un-exist" the type of observer-moment itself, but you can change what it continues into.
A higher fraction of "future" observer-moments experiencing suffering is dispreferable.
A higher fraction of "future" observer-moments experiencing happiness/tranquility is preferable, even if only instrumentally to minimize subjectively "future" suffering.
This is related to thinking about "anthropic immortality" / "quantum immortality" / "multiverse immortality" in which distributions of post-death observer-moments are of substantial concern.
Using these lenses, the moral situation of the world looks like:
Consider a graph with types of observer-moments as nodes, such that one node is one type.
Each type of observer-moment (each node) is associated with some value/disvalue attributed to its current experience.
Importantly, the value/disvalue of the current experience is not something we can change in order to help the observer-moment type. This is already fixed.
Each type of observer-moment (each node) is also associated with any number of subjectively indistinguishable token observer-moments belonging to that specific type.
The absolute number of token observer-moments appears to only matter instrumentally insofar as it affects the frequency of expectations of other types of observer-moments.
Edges represent subjective continuation. An edge from type A to type B exists if an observer-moment of type A can possibly expect to continue as an observer-moment of type B in the next moment.
The weight of an edge from type A to type B is the frequency at which an observer-moment of type A should expect to continue as an observer-moment of type B.
These weights are things that we might be able to change in order to help the observer-moment type.
Perhaps the "future" welfare of each type / node could be considered with equal weight in the aggregate to keep things impartial.
In conclusion... well, I'm still thinking about what the objective function might look like.
There's some scale invariance here. E.g. suppose you duplicated the world. That duplication would multiply all token observer-moments by the same factor. This leaves you with the same types and the same distributions for future observer-moments.
In a small enough world, you can create or prevent the existence of new types of observer-moment. I think it's unlikely (30%) that we live in a small enough world.
On the other hand, in a large enough world, all possible types of observer-moments exist. The only thing you can change is adjusting the distributions for future observer-moments for each type of observer-moment. Try to lead types of observer-moments down paths of non-suffering etc. So, this seems like an ethics of flow (rather than stock).
Follow-up: Someone sent me leads, which brought me to a working paper by philosopher Tim Campbell at the Institute for Futures Studies, with a thought experiment where 1,000 people are merged into one being who suffers. So it turns out a similar thought experiment has been written up after all.
If we count by moments, the suffering counts once.
If we count by persons, and suppose the fusion ends the 1,000 and creates one new person, it similarly counts once.
If we count by persons, and suppose all 1,000 survive sharing that one mind, it counts 1,000 times.
He thinks this case is a problem for the third method, and that his other cases in the paper are a problem for the first two. He argues that none of these work. I agree, partly for different reasons, and I think there’s another way.
In any case, it’s good to see the case taken seriously. Campbell and I seem to agree that cases like this pose a challenge for how utilitarians add up welfare. More generally, I think these questions deserve more moral reflection.
In other places people have asked me "what does it mean to merge very different minds together"?
I think I'm claiming something simpler here.
If there is any way at all for two observer-moments to continue as just one observer-moment later on - let's say it's two near-copies with a difference between them erased, as in Yudkowsky's Anthropic Trilemma - then it seems like there's a way to make two observer-moments' futures worse instead of one, while only having one observer-moment to be that worse future. This alone makes utilitarian aggregation odd to me (for near-copies).
If you can merge near-copies, and if additionally you can gradually turn very different minds into near-copies, then you can presumably eventually merge them. Then, it also seems to make utilitarian aggregation odd to me (for more distinct minds).
In the post I never actually specified how close the minds of 'everybody' in the thought experiment are. Some people assumed I was talking about merging very different minds. Whereas I was comfortable working from the more abstract assumption that merging minds is possible at all, to argue that if you can merge minds, utilitarian aggregation has a problem. Perhaps I should have made that clearer.
(The setup has other idealizations too, like everyone’s lives ending at the right time for the merge. I was treating the possibility of merging the same way.)
So yes, perhaps very different minds can’t be merged. Or perhaps they can be. But the simple conditional of "if minds can merge, utilitarian aggregation has a problem" is all I’m arguing for - and even if only near-copies can merge, aggregation still has a problem.
Agreed that counting paths at a full multiple would seem silly.
My proposed fix is a bit different. It cares about immediate welfare plus the welfare of future types of observer-moments, where futures are weighted by how likely we are to continue into each one, so each time step adds up to a weight of one. In your example, splitting shares the future between the copies and merging adds the shares back up, so the merged mind’s future isn’t multiplied by 2^1000.
Each brief copy does count as its own observer-moment with its own future, so the splitting adds a little. But that grows with the number of experiences that actually happened, not exponentially.
I don’t go into the fix in this post. Post was mainly to raise the question.
I agree with you that B is much worse. But I disagree that standard utilitarianism, as it is formulated now, reaches our conclusion that B is worse than A.
In the thought experiment, merging into one mind doesn’t add any negative experiences. The hell mind’s experiences are exactly the same in both options. It's still just one mind suffering in hell, and that seems to be all standard utilitarianism counts.
Rather than just counting immediate experience, I think it makes more sense to also account for subjective continuity, i.e. that multiple minds lead into the one mind in hell.
I think that analogy doesn’t quite fit. The road doesn’t have a point of view, but people do. Two people headed to hell seems worse than one.
My question is really “if merging were possible, would we merge everybody into hell?” Copying minds didn’t seem possible a decade ago, but assuming digital sentience is a thing, it now seems quite routine. If we wouldn’t merge everybody into hell, then there are some unresolved issues in utilitarianism.
Posting this subsection from the main post because it got asked about in the comments:
"What about near-identical copies?"
Exact copies count once but the smallest difference in feeling between them makes them near-copies, which would count twice. This is a jump, which some might consider unacceptable.
It doesn't seem to me that any view that counts observer-moments fully can avoid a jump. For instance, Bostrom's insulator can be gradually inserted, which means that at some point the token view has to say that one experience became two, and that this is morally relevant, which is over a physical change nobody feels. Bostrom himself ends up allowing fractional numbers of experiences, which doesn't seem to reflect how experience works. Where the physics separates is a judgment call, whereas where the contents of experience differ isn't, which seems like the better place for a jump.
A view could avoid a jump by discounting smoothly with similarity. For instance, the Saturation View of MacAskill and Tarsney (2026) does something like this, discounting near-duplicate lives toward a bound. However, this needs a similarity metric that we've decided is important. Any similarity metric seems to be something we choose from the outside. And I think we should recognize that "near identical" assumes a lot - and might be appealing to the sense that they can't be that different. Yet, you can win the lottery by a difference of a single digit.
Discounting for similarity would also fail the test that we should "restate every value claim in terms of what some being feels or will feel" (Bar, 2026). This is because neither copy feels how similar it is to the other. The same argument doesn't apply to exact copies, since counting one's feelings would count everything about the other's feelings. So, replication neutrality agrees with the Saturation View that filling the world with exact replicas of one existence would not, by itself, be the best use of resources. However, it gets to this position without valuing variety (which is something we judge from the outside) and unlike the Saturation View, it counts near-duplicates fully (on the basis of their experience).
More objections and responses are in the full post.
This is the near-identical copies objection. I think the post makes the case better than I can in comments, so I'll point you to this section, under the heading "What about near-identical copies?".
Adding to Tristan's 'large enough multiverse' point, and re section iii ('We Might Get Terminated Soon'). There seem to be a few ways to think about personal continuity here. To describe two:
I kinda lean toward 2 > 1. If there’s nothing special about causality that links you from one moment to the next, then what links them might just be that the later moment’s contents coherently follow on from the earlier one’s.
Continuity is ontologically tricky, but in practice we act as if there’s a fact about what we’ll experience next. Overall, I expect to see stuff even after simulations end.
Markov utilitarianism as universalized prudence
I recently proposed a view I call Markov utilitarianism. To me, it feels like a natural way to think about ethics. I came to it after thinking about how I care about my future, and how I really care about how I feel right now, from the first-person perspective.
This is prudence, in the philosophical sense of rational concern for one's own wellbeing. And it has exactly the same form as the equation reinforcement learning uses for an agent's value of being in a state, the Bellman equation (hence 'Markov'):
V(t) = r(t) + γ Σ P(t → t') V(t') over types of experience t.
To make it an ethics that considers everybody, we universalize it and sum impartially across all unique perspectives, mine and everyone else's:
U = Σ V(t)
In slogan form: "Prudence is caring about how you feel right now, and about each possible next experience in proportion to how likely you are to find yourself there. Ethics is doing that for every qualitatively unique perspective, counting each once".
Or more precisely: "Count each qualitatively unique perspective once, and care about it as it would care about how it feels now and what it will experience next, if fully informed about its futures".
Markov utilitarianism is closely related to total utilitarianism, but it structurally prefers lives that get better over time, and once minds can be copied and merged, it avoids conclusions like it being permissible to merge people into a being fated for extreme suffering.
So it feels like a natural extension of how you already care about yourself, extended to every perspective. Starting an ethics from this first-person point of view seems better to me because it avoids the third-person assumptions hidden in total utilitarianism.
What if exact copies don't matter but future experiences still do?
I think an ethics resulting from the following two premises is worth exploring:
P1: The total moral value of subjectively indistinguishable observer-moments doesn't scale with the number of copies, i.e. they sum to a constant value.
P2: What matters, and can be changed, for each type of observer-moment is the distribution of its subjectively "future" experiences.
Using these lenses, the moral situation of the world looks like:
In conclusion... well, I'm still thinking about what the objective function might look like.
There's some scale invariance here. E.g. suppose you duplicated the world. That duplication would multiply all token observer-moments by the same factor. This leaves you with the same types and the same distributions for future observer-moments.
In a small enough world, you can create or prevent the existence of new types of observer-moment. I think it's unlikely (30%) that we live in a small enough world.
On the other hand, in a large enough world, all possible types of observer-moments exist. The only thing you can change is adjusting the distributions for future observer-moments for each type of observer-moment. Try to lead types of observer-moments down paths of non-suffering etc. So, this seems like an ethics of flow (rather than stock).
Follow-up: Someone sent me leads, which brought me to a working paper by philosopher Tim Campbell at the Institute for Futures Studies, with a thought experiment where 1,000 people are merged into one being who suffers. So it turns out a similar thought experiment has been written up after all.
He thinks this case is a problem for the third method, and that his other cases in the paper are a problem for the first two. He argues that none of these work. I agree, partly for different reasons, and I think there’s another way.
In any case, it’s good to see the case taken seriously. Campbell and I seem to agree that cases like this pose a challenge for how utilitarians add up welfare. More generally, I think these questions deserve more moral reflection.
In other places people have asked me "what does it mean to merge very different minds together"?
I think I'm claiming something simpler here.
If there is any way at all for two observer-moments to continue as just one observer-moment later on - let's say it's two near-copies with a difference between them erased, as in Yudkowsky's Anthropic Trilemma - then it seems like there's a way to make two observer-moments' futures worse instead of one, while only having one observer-moment to be that worse future. This alone makes utilitarian aggregation odd to me (for near-copies).
If you can merge near-copies, and if additionally you can gradually turn very different minds into near-copies, then you can presumably eventually merge them. Then, it also seems to make utilitarian aggregation odd to me (for more distinct minds).
In the post I never actually specified how close the minds of 'everybody' in the thought experiment are. Some people assumed I was talking about merging very different minds. Whereas I was comfortable working from the more abstract assumption that merging minds is possible at all, to argue that if you can merge minds, utilitarian aggregation has a problem. Perhaps I should have made that clearer.
(The setup has other idealizations too, like everyone’s lives ending at the right time for the merge. I was treating the possibility of merging the same way.)
So yes, perhaps very different minds can’t be merged. Or perhaps they can be. But the simple conditional of "if minds can merge, utilitarian aggregation has a problem" is all I’m arguing for - and even if only near-copies can merge, aggregation still has a problem.
Agreed that counting paths at a full multiple would seem silly.
My proposed fix is a bit different. It cares about immediate welfare plus the welfare of future types of observer-moments, where futures are weighted by how likely we are to continue into each one, so each time step adds up to a weight of one. In your example, splitting shares the future between the copies and merging adds the shares back up, so the merged mind’s future isn’t multiplied by 2^1000.
Each brief copy does count as its own observer-moment with its own future, so the splitting adds a little. But that grows with the number of experiences that actually happened, not exponentially.
I don’t go into the fix in this post. Post was mainly to raise the question.
I agree with you that B is much worse. But I disagree that standard utilitarianism, as it is formulated now, reaches our conclusion that B is worse than A.
In the thought experiment, merging into one mind doesn’t add any negative experiences. The hell mind’s experiences are exactly the same in both options. It's still just one mind suffering in hell, and that seems to be all standard utilitarianism counts.
Rather than just counting immediate experience, I think it makes more sense to also account for subjective continuity, i.e. that multiple minds lead into the one mind in hell.
I think that analogy doesn’t quite fit. The road doesn’t have a point of view, but people do. Two people headed to hell seems worse than one.
My question is really “if merging were possible, would we merge everybody into hell?” Copying minds didn’t seem possible a decade ago, but assuming digital sentience is a thing, it now seems quite routine. If we wouldn’t merge everybody into hell, then there are some unresolved issues in utilitarianism.
Posting this subsection from the main post because it got asked about in the comments:
More objections and responses are in the full post.
This is the near-identical copies objection. I think the post makes the case better than I can in comments, so I'll point you to this section, under the heading "What about near-identical copies?".