This is an essay I’ve drafted but haven’t completed — I’m posting it in the spirit of Draft Amnesty and because there’s a Debate Week on the topic; expect mistakes, cut-off sentences, missing context, etc., and feel free to ask follow-up questions here or in this thread!

This is a draft as part of a bigger essay series I'm working on, on “Better Futures”.

I define moral error as an outcome where most future beings endorse the civilisation that they inhabit, but where there is an enormous loss of value because those future beings’ preferences are for the wrong things.

In this post I list some examples of axes for moral error: decisions where mistakes by future decision-makers could result in the loss of most possible value. I do not try to be exhaustive; my goal is to illustrate the variety and number of different decisions that could lead to moral error.

This exercise increased my confidence in the idea that, if future decision-makers act on the wrong moral view, then that would very likely result in the loss of most value. This is true even if the view they have is reasonable (and it’s especially likely if decision-makers go all-in on optimising for one view).

I define a mistopia as a society that is only a small fraction as good as it could have been. A society being mistopian doesn’t mean that the society is worse than nothing or that there’s more suffering than happiness in that society. A mistopian civilisation society could still be, on balance, better than a barren world — it could even be very good compared to today, just much less good than would be possible. “Moral error” appears to me to be the likeliest variety of mistopia.

Moral error appears to be the likeliest variety of mistopia (others; "catastrophe", "evolutionary outcome", "enforced dystopia/coordination failure")

Where should we look for potential sources of moral error?

One way to identify potential sources of moral error is to think of moral atrocities of the past, and imagine them returning to widespread acceptance. So, for example, the future could see the return of widely accepted slavery, racial hierarchy, torture of one’s enemies, or totalitarianism. But that gives us only a very limited set of possibilities; of all possible moral catastrophes, we have probably only witnessed a tiny handful in history. Restricting ourselves in this way might make moral error seem much less likely than it really is.

A different method — and the one I’m focusing on here — is to consider various positions in mo...

Best of: Existential Choices Week