A few years ago, I made a outline of Evan G. Williams' excellent philosophy paper, for a local discussion group. It slowly got circulated on the EA internet. Somebody recently recommended that I make the summary more widely known, so here it is.

The paper is readable and not behind a paywall, so I'd highly recommend reading the original paper if you have the time.


I. Core claim

  1. Assuming moral objectivism (or a close approximation), we are probably unknowingly guilty of serious, large-scale wrong-doing (“ongoing moral catastrophe”).

II. Definition: What is a moral catastrophe? Three criteria:

  1. Must be a serious wrong-doing (closer to wrongful death or slavery than mild insults or inconveniences).
  2. Must be large-scale (instead of a single wrongful execution, or a single man tortured)
  3. Broad swathes of society are responsible through action or inaction (can’t be unilateral unavoidable actions by a single dictator).

III. Why we probably have unknown moral catastrophes. Two core arguments:

  1. The Inductive Argument
    1. Assumption: It’s possible to engage in great moral wrongdoings even while acting in accordance to your own morals, and that of your society.
      1. Basic motivation: an honest, sincere Nazi still seems to be acting wrongly in important ways.
      2. It’s not relevant whether this wrongdoing is due to mistaken empirical beliefs (All Jews are part of a major worldwide conspiracy) or wrong values (Jews are subhuman and have no moral value).
    2. Given that assumption in mind, pretty much every major society in history has acted catastrophically wrongly.
      1. Consider conquistadores, crusaders, caliphates, Aztecs etc. who conquered in the name of God(s), who they called good and just.
      2. It’s unlikely that all of these people in history only professed such a belief, and that all of them were liars instead of true believers.
      3. Existence proof: People can (and in fact do) do great evil without being aware of this.
    3. Us having ongoing moral catastrophes isn’t just a possibility, but probable.
      1. We are not that different from past generations: Literally hundreds of generations have thought that they actually were right and had figured out the One True Morality
      2. As recent as our parents’ generation, it was a common belief that some people have more rights than others because of race, sexuality etc.
      3. We live in a time of moral upheaval, where our morality is very different from our grandparents’.
      4. Even if some generation would eventually figure out All of Morality, the generation that gets everything right is probably a generation whose parents gets almost everything right.
  2. The Disjunctive Argument
    1. Activists are not exempt. Even if all your pet causes come to fruition, this doesn’t mean our society is good, because there are still unknown moral catastrophes.
    2. There are so many different ways that a society could get things very wrong, that it’s almost impossible to get literally everything right.
      1. This isn’t just a minor concern, we could be wrong in ways that are a sizable proportion of how bad the Holocaust is.
    3. There are many different kinds of ways that society could be wrong.
      1. We could be wrong about who has moral standing.(eg. fetuses, animals)
      2. We could be empirically wrong about what harms or hurts people who morally matter (eg. religious indoctrination of children)
      3. We could be right about some obligations but not others.
        1. We can act immorally in paying too much attention and using resources on false moral obligations (a la crusaders)
      4. We could be right about what’s wrong and should be fixed, but wrong at how to prioritize different fixes.
      5. We could be right about what’s wrong, but wrong about what is and is not our responsibility to fix. (eg. poverty, borders)
      6. We could be wrong about the far future (natalism, existential risk)
    4. Within each category, there are multiple ways to go wrong.
      1. Further, some are mutually exclusive. Eg. Pro-lifers could be right and abortion is a great sin, or fetuses don’t matter and it’s greatly immoral to deprive women of their freedom in eg. third trimester abortions.
      2. Unlikely that we’re currently at the golden mean for all of these trade-offs.
    5. Disjunction comes into play.
      1. Even if you believe that we’re 95% right at each major issue, and there are maybe 15 of them, the total probability that we are right is maybe ~.95^15~=46% (LZ: Assumes independence)
      2. In practice, 95% sure we’re right at each major issue seems way too confident, and 15 items too low.

IV. What should we do about it? 

  1. Discarded possibility: hedging. If you’re not sure, play it “safe”, morally speaking.
  2. Eg. even if you think farmed animals probably aren’t sentient, or sentience doesn’t morally matter, you can go vegetarian “just in case”
  3. This does NOT generally work well enough because it’s not robust: as noted, too many things can go wrong, some in contradictory directions.
  4. Recognition of Wrongdoing
    1. Actively try to figure out which catastrophic wrongs we’re committing
      1. Research more into practical fields (eg. animal consciousness) where we can be critically wrong
      2. Research more into moral philosophy
        1. Critical: bad to have increased technological knowledge w/o increased moral wisdom
        2. imagine Genghis Khan w/nuclear weapons
      3. These fields must interact
        1. Not enough for philosophers to say that animals are important if they are conscious and for scientists to say that dolphins are conscious but don’t know if this is important...our society must be able to integrate this.
    2. Need marketplace of ideas where true ideas win out
    3. Rapid intellectual progress is critical.
      1. If it’s worth fighting literal wars to defeat the Nazis or end slavery, it’s worth substantial material investment and societal loss to figure out what we’re currently doing wrong.
  5. Implementation of improved values
    1. Once we figure out what great moral wrongs we’ve committed, we want to be able to make moral reparations for past harms, or at least stop doing future harms in that direction as quickly as possible.
    2. To do this, we want to maximize flexibility in material conditions
      1. Extremely poor/war-torn societies would be unable to make rapid moral changes as needed
      2. LZ example: Complex systems built along specific designs are less resilient to shocks, and also harder to change, cf. Antifragile.
      3. In the same way we stock up resources for war preparation, we might want to save up resources for future moral emergencies, so we can eg. pay reparations, or at least quickly make the relevant changes.
        1. LZ: Unsure how this is actually possible in practice. Eg, individuals usually save by investing, and governments save by buying other government’s debt or by investing in the private sector, but it’s unclear how the world “saves” as a whole.
    3. We want to maximize flexibility in social conditions
      1. Even if it’s materially possible to make large changes, society might make such changes very difficult, because inertia and conservatism bias.
      2. Constitutional amendments, for example, are suspect.

V. Conclusion/Other remarks

  1. Counterconsideration One: Building a society that can correct moral catastrophes isn’t the same as actually correcting moral catastrophes.
  2. Counterconsideration Two : Many of the measures suggested above to prepare for correcting moral catastrophes may themselves be evil
    1. e.g. money spent on moral research could have instead been spent on global poverty, building a maximally flexible society might involve draconian restrictions on current people’s rights
  3. However, this is still worth doing in the short term.


This work is licensed under a Creative Commons Attribution 4.0 International License.


Sorted by Click to highlight new comments since: Today at 4:51 AM

I considered Evan Williams' paper one of the most important papers in cause prioritization at the time, and I think I still broadly buy this. As I mention in this answer, there are at least 4 points his paper brought up that are nontrivial, interesting, and hard to refute.

If I were to write this summary again, I think I'd be noticeably more opinionated. In particular, a key disagreement I have with him (which I remember having at the time I was making the summary, but this never making it into my notes) is on the importance of the speed of moral progress vs the sustainability of continued moral progress. In "implementation of improved values", the paper focuses a lot on the flexibility of setting up society to be able  to make moral progress quickly, but naively I feel about as worried or more worried that society can make anti-progress and do horrifyingly dumb and new things in the name of good. So I'd be really worried about trajectory changes for the worse, especially longer-lasting ones ("lock-in" is a phrase that's in vogue these days).

I've also updated significantly on both the moral cost and the emprical probability of near-term extinction risks, and of course extinction is the archetypal existential risk that will dramatically curtail the value of the far future.

It feels weird getting my outline into the EA decade review, instead of the original paper, though I wouldn't be very surprised if at this point more EAs have read my outline than the paper itself.

I vaguely feel like Williams should get a lot more credit than he has received for this paper. Like EA should give him a prize or something, maybe help him identify more impactful research areas, etc.

You should sit in with some friends of Bill W. Tell them you're visitor, then just sit there and listen. They'll teach you about a pure altruism. It's pretty damn awesome.

I'm entering philosophy grad school now, but in a few years I'm going to have to start thinking about designing courses, and I'm thinking of designing an intro course around this paper. Would it be alright if I used your summary as course material?


Sure! In general you can assume that anything I write publicly is freely available for academic purposes. I'd also be interested in seeing the syllabus if/when you end up designing it.

Definitely, I'll send it along when I design it. Since intro ethics at my institution is usually taught as applied ethics, the basic concept would be to start by introducing the students to the moral catastrophes paper/concept, then go through at least some of the moral issues Williams brings up in the disjunctive portion of the argument to examine how likely they are to be moral catastrophes. I haven't picked particular readings yet though as I don't know the literatures yet. Other possible topics: a unit on historical moral catastrophes (e.g. slavery in the South, the Holocaust); a unit on biases related to moral catastrophes; a unit on the psychology of evil (e.g. Baumeister's work on the subject, which I haven't read yet); a unit on moral uncertainty; a unit on whether antirealism can escape or accommodate the possibility of moral catastrophes.

Assignment ideas:

  1. pick one of the potential moral catastophes Williams mentions, which you think is least likely to actually be a moral catastrophe. Now, imagine that you are yourself five years from now and you’ve been completely convinced that it is in fact a moral catastrophe. What convinced you? Write a paper trying to convince your current self that it is a moral catastrophe after all.
  2. Come up with a potential moral catastrophe that Williams didn’t mention, and write a brief (maybe 1-2 pages?) argument for why it is or isn’t one (whatever you actually believe). Further possibility: Once these are collected, I observe how many people argued that the one they picked was not a moral catastrophe, and if it’s far over 50%, discuss with the class where that bias might come from (e.g. status quo bias, etc.).

This is all still in the brainstorming stage at the moment, but feel free to use any of this if you're ever designing a course/discussion group for this paper.


For #2, Ideological Turing Tests could be cool too.


You may also like our discussion sheets for this topic:


Thanks for the summary. I have two takeaways:

1. EA is (in part) claiming that there are several ongoing moral catastrophes caused by inaction against global poverty, animal suffering, x-risk,... (some of them are definitely caused by action, but that does not matter as much on consequentialist grounds). Unknown ongoing moral catastrophes are cause-X.

2. The possibility of working to increase our capability to handle undiscovered ongoing moral catastrophe in the future as a major goal. The idea I saw here was to reserve resources, which is a very interesting argument to invest in economic growth.

Strong upvoting because I want to incentivize people to write and share more summaries.

Summaries are awesome and allow me to understand the high level of papers that I would not have read otherwise. This summary in particular is well written and well-formatted.

Thanks for writing it and sharing it!

We absolutely welcome summaries! People getting more ways to access good material is one of the reasons the Forum exists.

That said, did you consider copying the summary into a Forum post, rather than linking it? That's definitely more work, but my impression is that it usually leads to more discussion when people don't have to click away into another page. I don't have strong evidence to back that up, though.

Also: because the title is long and long titles are cut short in some views of the Forum, I'd recommend that summaries of pieces be something like "The Possibility of an Ongoing Moral Catastrophe (Summary)".

We absolutely welcome summaries! People getting more ways to access good material is one of the reasons the Forum exists.


did you consider copying the summary into a Forum post, rather than linking it?

Yes. I did a lot of non-standard formatting tricks in Google Docs when I first wrote it (because I wasn't expecting to ever need to port it over to a different format). So when I first tried to copy it over, the whole thing looked disastrously unreadable.

Changed the title. :)