What if exact copies don't matter but future experiences still do?

I think an ethics resulting from the following two premises is worth exploring:

P1: The total moral value of subjectively indistinguishable observer-moments doesn't scale with the number of copies, i.e. they sum to a constant value.

• This is because subjectively indistinguishable observer-moments, in some sense, happen to the same "someone". One could think of themselves as all their copies. N observer-moments that are subjectively indistinguishable aren't felt N times by that same "someone". Only felt "once".
  • E.g. Even if there were multiple exact copies of me, of my current present experience, I might not really care personally because it doesn't seem to change anything for me. "I" don't notice anything.
  • See Wei Dai's "The Moral Status of Independent Identical Copies" for the tension this creates with standard utilitarianism.
• We can say that multiple "tokens" of subjectively indistinguishable observer-moments belong to the same "type" of observer-moment.

P2: What matters, and can be changed, for each type of observer-moment is the distribution of its subjectively "future" experiences.

• You can't "un-exist" the type of observer-moment itself, but you can change what it continues into.
  • A higher fraction of "future" observer-moments experiencing suffering is dispreferable.
  • A higher fraction of "future" observer-moments experiencing happiness/tranquility is preferable, even if only instrumentally to minimize subjectively "future" suffering.
• This is related to thinking about "anthropic immortality" / "quantum immortality" / "multiverse immortality" in which distributions of post-death observer-moments are of substantial concern.

Using these lenses, the moral situation of the world looks like:

• Consider a graph with types of observer-moments as nodes, such that one node is one type.
• Each type of observer-moment (each node) is associated with some value/disvalue attributed to its current experience.
  • Importantly, the value/disvalue of the current experience is not something we can change in order to help the observer-moment type. This is already fixed.
• Each type of observer-moment (each node) is also associated with any number of subjectively indistinguishable token observer-moments belonging to that specific type.
  • The absolute number of token observer-moments appears to only matter instrumentally insofar as it affects the frequency of expectations of other types of observer-moments.
• Edges represent subjective continuation. An edge from type A to type B exists if an observer-moment of type A can possibly expect to continue as an observer-moment of type B in the next moment.
  • The weight of an edge from type A to type B is the frequency at which an observer-moment of type A should expect to continue as an observer-moment of type B.
  • These weights are things that we might be able to change in order to help the observer-moment type.
• Perhaps the "future" welfare of each type / node could be considered with equal weight in the aggregate to keep things impartial.

In conclusion... well, I'm still thinking about what the objective function might look like.

There's some scale invariance here. E.g. suppose you duplicated the world. That duplication would multiply all token observer-moments by the same factor. This leaves you with the same types and the same distributions for future observer-moments.

In a small enough world, you can create or prevent the existence of new types of observer-moment. I think it's unlikely (30%) that we live in a small enough world.

On the other hand, in a large enough world, all possible types of observer-moments exist. The only thing you can change is adjusting the distributions for future observer-moments for each type of observer-moment. Try to lead types of observer-moments down paths of non-suffering etc. So, this seems like an ethics of flow (rather than stock).

FYI, if people want to look into what aliens might value, an interesting direction might be to think about convergent evolution. One (the only?) existing book on the topic: The Zoologist's Guide to the Galaxy/ Quanta magazine article. Geoffrey Miller mentioned related work in a comment a few months ago.

Is this pointless speculation? I suspect knowing about what aliens might value would be useful in understanding how to better implement Evidential Cooperation in Large Worlds (ECL) (right now, right here, on Earth) although some people may disagree with me on that.

I've also encountered thinking that this could help avoid/reduce conflicts with aliens (which may motivate work on it from various longtermist perspectives).

I guess this kind of stuff would be particularly suited for people with an evolutionary biology/related field background but it also seems like people can pick these things up quickly/use AI assistants to help out.

I was brought up in a very religious environment. After reading this comment I'm reflecting on what I'm finding off-putting about that upbringing:

• The idea that there is a clear divide between good and evil.
• The idea that there are unforgivable sins/heresies.
• The idea that sexual things are bad, or are particularly bad.
• Laying claim to humility and being the underdog even though one's group has a lot of power.
• The idea that arguing against sacred beliefs is bad.
• Shaming those who have sinned and demanding that they repent.
• The idea that everything considered evil must/will be punished severely.
• ... and more.

I find myself agreeing with much of the comparison that the comment makes.

I find it odd that many people's ideas about other minds don't involve, or even contradict, the existence of some non-arbitrary function that maps a (finite) number of discrete fundamental physical entities (assuming physics is discrete) in a system to a corresponding number of minds (or some potentially quantifiable property of minds) in that same system.

I have intuitions (which could be incorrect) that "physics is all there is" and that "minds are ultimately physical," and it feels possible, in principle, to unify them somehow and relate "the amount of stuff" in both the physical and mental domains through such a function.

To me, this solution ("count all subsets of all elements within systems") proposed by Brian Tomasik appears to be among plausible non-arbitrary options, and it could also be especially ethically relevant. Solutions such as these that suggest the existence of a very large number of minds imply moral wagers, e.g. to minimize possible suffering in the kinds of minds that are implied to be most numerous (in this case, those that comprise ~half of everything in the universe), which might make them worth investigating further.

Even if physics is continuous rather than discrete, it still seems possible that there could be a mapping from continuous physics to discrete minds. (disclaimer: I don't know much physics, and I haven't thought much about how it relates to the philosophy of mind.)

This is all speculative and counterintuitive. On the other hand, common-sense intuitions developed through evolution might not accurately represent the first-person experiences, or lack thereof, of other systems. They seem to have instead evolved because they helped model complicated systems relevant to fitness by picturing them as similar to one's own mind. Common-sense intuitions aren't necessarily reliable, and counterintuitive conclusions could potentially be true.

I'm skeptical about the value of slowing down leading AI labs primarily because it likely reduces the influence of the values of EAs in shaping the deployment of AGI/ASI. Anthropic is the best example of a lab with people who share these values, but I'd imagine that EAs also have more overlap with the staff at OpenAI and DeepMind than actors who would catch up because of a slowdown. And for what it's worth, the labs were founded with the stated goal of benefiting humanity before it became far more apparent that current paradigms have a high chance of resulting in AGI with the potential of granting profit/power to their human operators and investors.

As others have noted, people and powerful groups outside of this community and surrounding communities don't seem to be interested in consequentialist, impartial, altruistic priorities like creating a positive long-term future for humanity, but are instead more self-interested. Personally I'm more downside-focused, but I think it's relevant to most EAs that other parties wouldn't be as willing to dedicate a large amount of resources towards creating large amounts of happiness for others, and because of that, the reduction of influence of the values of EAs will result in a considerable loss of expected future value.

EDIT (2024-05-19): When I wrote this I had in mind Anthropic > OpenAI > DeepMind but Anthropic > DeepMind > OpenAI seems more sensible now. Unclear where to insert various governments/militaries/politicians/CEOs into this ranking.

A while back I wrote that I agreed with the observation that some of (new wave) EA’s norms seem similar to those of the religion imposed on me and others as children. My current thinking is that there may actually be a link connecting the culture in parts of Protestantism and some of the (progressive) norms EA adopts, along with an atypical origin that probably deserves more scrutiny. The "link" part might be more apparent to people who've noticed a "god-shaped hole" in the West that makes some secular movements resemble certain parts of religions. The "origin" part might be less apparent but it's been discussed by Scott Alexander before. So, this theory isn't all that original.

Essentially: Puritans, as one of four major cultures originating from the UK, exert huge founder effects on America, which both influences parts of itself as well as other countries for better or worse → Protestant culture gradually changed to be more socially judgmental in some ways etc. → More recently, people increasingly reject the existence of a God but keep elements of the culture of that religion → EA now draws heavily from nth generation ex-Protestants/Protestant-adjacents who also tend to be more active in trying to change society (other people's actions) and approach it with some of the same inherited attitudes

That is one causal chain but a tree might show more causes and effects. For example, the Puritan founder effects probably also influenced modern academia (in part, spearheaded by a few institutions in New England) which again, EA heavily draws from. Other secular institutions might also be influenced by osmosis, and produce downstream effects.

It seems difficult to believe these attitudes just disappeared without affecting other movements, culture, and society. The Puritan legacy also seems to have a track record of being quite influential.