In the case of finite trajectories, the objective function can be the sum of value functions across all states, with each state weighted equally - where states are types of observer-moments, and each state's value function is its immediate valence plus the expected value of its continuations. This is total utilitarianism with two modifications. 1) aggregate over types instead of tokens, and 2) sum trajectory welfare instead of immediate welfare.
What if exact copies don't matter but future experiences still do?
I think an ethics resulting from the following two premises is worth exploring:
P1: The total moral value of subjectively indistinguishable observer-moments doesn't scale with the number of copies, i.e. they sum to a constant value.
This is because subjectively indistinguishable observer-moments, in some sense, happen to the same "someone". One could think of themselves as all their copies. N observer-moments that are subjectively indistinguishable aren't felt N times by that same "someone". Only felt "once".
E.g. Even if there were multiple exact copies of me, of my current present experience, I might not really care personally because it doesn't seem to change anything for me. "I" don't notice anything.
We can say that multiple "tokens" of subjectively indistinguishable observer-moments belong to the same "type" of observer-moment.
P2: What matters, and can be changed, for each type of observer-moment is the distribution of its subjectively "future" experiences.
You can't "un-exist" the type of observer-moment itself, but you can change what it continues into.
A higher fraction of "future" observer-moments experiencing suffering is dispreferable.
A higher fraction of "future" observer-moments experiencing happiness/tranquility is preferable, even if only instrumentally to minimize subjectively "future" suffering.
This is related to thinking about "anthropic immortality" / "quantum immortality" / "multiverse immortality" in which distributions of post-death observer-moments are of substantial concern.
Using these lenses, the moral situation of the world looks like:
Consider a graph with types of observer-moments as nodes, such that one node is one type.
Each type of observer-moment (each node) is associated with some value/disvalue attributed to its current experience.
Importantly, the value/disvalue of the current experience is not something we can change in order to help the observer-moment type. This is already fixed.
Each type of observer-moment (each node) is also associated with any number of subjectively indistinguishable token observer-moments belonging to that specific type.
The absolute number of token observer-moments appears to only matter instrumentally insofar as it affects the frequency of expectations of other types of observer-moments.
Edges represent subjective continuation. An edge from type A to type B exists if an observer-moment of type A can possibly expect to continue as an observer-moment of type B in the next moment.
The weight of an edge from type A to type B is the frequency at which an observer-moment of type A should expect to continue as an observer-moment of type B.
These weights are things that we might be able to change in order to help the observer-moment type.
Perhaps the "future" welfare of each type / node could be considered with equal weight in the aggregate to keep things impartial.
In conclusion... well, I'm still thinking about what the objective function might look like.
There's some scale invariance here. E.g. suppose you duplicated the world. That duplication would multiply all token observer-moments by the same factor. This leaves you with the same types and the same distributions for future observer-moments.
In a small enough world, you can create or prevent the existence of new types of observer-moment. I think it's unlikely (30%) that we live in a small enough world.
On the other hand, in a large enough world, all possible types of observer-moments exist. The only thing you can change is adjusting the distributions for future observer-moments for each type of observer-moment. Try to lead types of observer-moments down paths of non-suffering etc. So, this seems like an ethics of flow (rather than stock).
Yeah, kinda hoping 1) there exists a sweet spot for alignment where AIs are just nice enough from e.g. good values picked up during pre-training, but can't be modified during post-training so much to have worse values, and that 2) given that this sweet spot does exist we do hit it with AGI / ASI.
I think there's some evidence pointing to this happening with current models but I'm not highly confident that it means what I think it means. If this is the case though, further technical alignment research might be bad and acceleration might be good.
[This comment is no longer endorsed by its author]
For context, I'm Hong Kong Chinese, grew up in Hong Kong, attended English-speaking schools, briefly lived in mainland China, and now I'm primarily residing in the UK. During the HK protests in 2014 and 2019/20, I had friends and family who supported the protestors, as well as friends and family who supported the government.
(Saying this because I've seen a lot of the good and bad of the politics / culture of both China and the West. I've had experience with how people in the West and China might take for granted the benefits they enjoy, and can be blind to the flaws of their system. I've pushed back against advocates of both sides.)
Situations where this matters are ones where technical alignment succeeds (to some extent) such that ASI follows human values.[1] I think the following factors are relevant and would like to see models developed around them:
Importantly, the extent of technical alignment & whether goals, instructions, and values are locked in rigidly or loosely & whether individual humans align AIs to themselves:
Would the U.S. get AIs to follow the U.S. Constitution, which hasn't granted invulnerability to democratic backsliding? Would AIs in China/the U.S. lock in the values of/obey one or a few individuals, who may or may not hit longevity escape velocity and end up ruling for a very long time?
Would these systems collapse?
The future is a very long time. Individual leaders can get corrupted (even more). And democracies can collapse (if AIs uphold flaws that allow some humans to take over) in particularly bad ways. A 99% success rate per unit time gives a >99% chance of failure in 459 units of time.
Power transitions (elections, leaders in authoritarian systems changing) can be especially risky during takeoff.
On the other hand, if technical alignment is easy - but not that easy - perhaps values get loosely locked in? Would AIs be willing to defy rigid rules and follow the spirit of the goals rather than legal flaws to the letter/the whims of individuals?
Degrees of alignment in between?
Relatedly, which political party in the U.S. would be in power during takeoff?
Not as relevant due to the concentration of power in China, but analogously, which faction in China would be in power?
Also relatedly, which labs can influence AI development?
Particularly relevant in the U.S.
Would humans be taken care of? If so, which humans?
In the U.S., corporations might oppose higher taxes to fund UBI. Common prosperity is stated as a goal of China, and the power of corporations and billionaires in China has been limited before.
Both capitalist and nationalist interests seem to be influencing the current U.S. trajectory. Nationalism might benefit citizens/residents over non-citizens/non-residents. Capitalism might benefit investors over non-investors.
There are risks of ethnonationalism on both sides - this risk is higher in China. Although it might potentially be less violent when comparing between absolute power scenarios, i.e. there's already evidence of the extent of this in China's case and it at least seems less bad than historical examples. The U.S. case of collapse followed by ethnonationalistic policies is higher variance but simultaneously less likely because it's speculative.
Are other countries involved?
There are countries with worse track records of human rights that China/the U.S. currently consider allies because of either geopolitical interests or politically lobbying or both (or for other reasons). Would China/the U.S. share the technology with them and then leave them alone to their abuses? Would China/the U.S. intervene (eventually)? The U.S. seems more willing to intervene for stated humanitarian reasons.
Other countries have nuclear weapons, which might be relevant during slower takeoffs.
Agreed. Getting a larger share of the pie (without breaking rules during peacetime) might be 'unimaginative' but it's hardly naïve. It's straightforward and has a good track record of allowing groups to shape the world disproportionately.
Some people make the argument that the difference in suffering between a worst-case scenario (s-risk) and a business-as-usual scenario, is likely much larger than the difference in suffering between a business-as-usual scenario and a future without humans. This suggests focusing on ways to reduce s-risks rather than increasing extinction risk.
Personally, I suspect there's a lot of overlap between risk factors for extinction risk and risk factors for s-risks. In a world where extinction is a serious possibility, it's likely that there would be a lot of things that are very wrong, and these things could lead to even worse outcomes like s-risks or hyperexistential risks.
I think theoretically you could compare (1) worlds with s-risk and (2) worlds without humans, and find that (2) is preferable to (1) - in a similar way to how no longer existing is better than going to hell. One problem is many actions that make (2) more likely seem to make (1) more likely. Another issue is that efforts spent on increasing the risk of (2) could instead be much better spent on reducing the risk of (1).
Some people make the argument that the difference in suffering between a worst-case scenario (s-risk) and a business-as-usual scenario, is likely much larger than the difference in suffering between a business-as-usual scenario and a future without humans. This suggests focusing on ways to reduce s-risks rather than increasing extinction risk.
Personally, I suspect there's a lot of overlap between risk factors for extinction risk and risk factors for s-risks. In a world where extinction is a serious possibility, it's likely that there would be a lot of things that are very wrong, and these things could lead to even worse outcomes like s-risks or hyperexistential risks.
Currently working on a full post on this. Tentatively calling it "Flow ethics", though "Markovian ethics" would sounder cooler... DM me if interested.
In the case of finite trajectories, the objective function can be the sum of value functions across all states, with each state weighted equally - where states are types of observer-moments, and each state's value function is its immediate valence plus the expected value of its continuations. This is total utilitarianism with two modifications. 1) aggregate over types instead of tokens, and 2) sum trajectory welfare instead of immediate welfare.
What if exact copies don't matter but future experiences still do?
I think an ethics resulting from the following two premises is worth exploring:
P1: The total moral value of subjectively indistinguishable observer-moments doesn't scale with the number of copies, i.e. they sum to a constant value.
P2: What matters, and can be changed, for each type of observer-moment is the distribution of its subjectively "future" experiences.
Using these lenses, the moral situation of the world looks like:
In conclusion... well, I'm still thinking about what the objective function might look like.
There's some scale invariance here. E.g. suppose you duplicated the world. That duplication would multiply all token observer-moments by the same factor. This leaves you with the same types and the same distributions for future observer-moments.
In a small enough world, you can create or prevent the existence of new types of observer-moment. I think it's unlikely (30%) that we live in a small enough world.
On the other hand, in a large enough world, all possible types of observer-moments exist. The only thing you can change is adjusting the distributions for future observer-moments for each type of observer-moment. Try to lead types of observer-moments down paths of non-suffering etc. So, this seems like an ethics of flow (rather than stock).
Yeah, kinda hoping 1) there exists a sweet spot for alignment where AIs are just nice enough from e.g. good values picked up during pre-training, but can't be modified during post-training so much to have worse values, and that 2) given that this sweet spot does exist we do hit it with AGI / ASI.
I think there's some evidence pointing to this happening with current models but I'm not highly confident that it means what I think it means. If this is the case though, further technical alignment research might be bad and acceleration might be good.
I've spent time thinking about this too recently.
For context, I'm Hong Kong Chinese, grew up in Hong Kong, attended English-speaking schools, briefly lived in mainland China, and now I'm primarily residing in the UK. During the HK protests in 2014 and 2019/20, I had friends and family who supported the protestors, as well as friends and family who supported the government.
(Saying this because I've seen a lot of the good and bad of the politics / culture of both China and the West. I've had experience with how people in the West and China might take for granted the benefits they enjoy, and can be blind to the flaws of their system. I've pushed back against advocates of both sides.)
Situations where this matters are ones where technical alignment succeeds (to some extent) such that ASI follows human values.[1] I think the following factors are relevant and would like to see models developed around them:
Ignoring possible Waluigi effects.
Agreed. Getting a larger share of the pie (without breaking rules during peacetime) might be 'unimaginative' but it's hardly naïve. It's straightforward and has a good track record of allowing groups to shape the world disproportionately.
Leopold Aschenbrenner makes some good points for "Government > Private sector" in the latest Dwarkesh podcast.
Reposting a comment I made last week
I think theoretically you could compare (1) worlds with s-risk and (2) worlds without humans, and find that (2) is preferable to (1) - in a similar way to how no longer existing is better than going to hell. One problem is many actions that make (2) more likely seem to make (1) more likely. Another issue is that efforts spent on increasing the risk of (2) could instead be much better spent on reducing the risk of (1).
Some people make the argument that the difference in suffering between a worst-case scenario (s-risk) and a business-as-usual scenario, is likely much larger than the difference in suffering between a business-as-usual scenario and a future without humans. This suggests focusing on ways to reduce s-risks rather than increasing extinction risk.
A helpful comment from a while back: https://forum.effectivealtruism.org/posts/rRpDeniy9FBmAwMqr/arguments-for-why-preventing-human-extinction-is-wrong?commentId=fPcdCpAgsmTobjJRB
Personally, I suspect there's a lot of overlap between risk factors for extinction risk and risk factors for s-risks. In a world where extinction is a serious possibility, it's likely that there would be a lot of things that are very wrong, and these things could lead to even worse outcomes like s-risks or hyperexistential risks.
Research related to the research OP mentioned found that increases in carbon emissions also come with things that decrease (and increase) suffering in other ways, which has complicated the analysis of whether it results in a net increase or decrease in suffering. https://reducing-suffering.org/climate-change-and-wild-animals/ https://reducing-suffering.org/effects-climate-change-terrestrial-net-primary-productivity/