Many longtermists seem hopeful that our successors (or any advanced civilization/superintelligence) will eventually act in accordance with some moral truth.^[1] While I’m sympathetic to some forms of moral realism, I believe that such a scenario is fairly unlikely for any civilization and even more so for the most advanced/expansionist ones. This post briefly explains why.

To be clear, my case does under no circumstances imply that we should not act according to what we think might be a moral truth. I simply argue that we can't assume that our successors -- or any powerful civilization -- will "do the (objectively) right thing". And this matters for longtermist cause prioritization.

Epistemic status: Since I believe the ideas in this post to be less important than those in future ones within this sequence, I wrote it quickly and didn’t ask anyone for thorough feedback before posting, which makes me think I’m more likely than usual to have missed important considerations. Let me know what you think!

Update April 10th: When I first posted this, the title was "It Doesn't Matter what the Moral Truth might be". I realized this was misleading. It was making it look like I was making a strong normative claim regarding what matters while my goal was actually to predict what might happen, so I changed it.

Rare are those who will eventually act in accordance with some moral truth

For agents to do what might objectively be the best thing to do, you need all these conditions to be met:

There is a moral truth.
It is possible to “find it” and recognize it as such.
They find something they recognize as a moral truth.
They (unconditionally) accept it, even if it is highly counterintuitive.
The thing they found is actually the moral truth. No normative mistake.
They succeed at acting in accordance with it. No practical mistake.
They stick to this forever. No value drift.

I think these seven conditions are generally quite unlikely to be all met at the same time, mainly for the following reasons:

...

What values will control the Future?