This is not a contest submission; I don’t think it’d be appropriate for me to enter this contest given my position as a CEA funder. This also wasn’t really inspired by the contest - I’ve been thinking about writing something like this for a little while - but I thought the contest provided a nice time to put it out.
This piece generally errs on the concise side, gesturing at intuitions rather than trying to nail down my case thoroughly. As a result, there’s probably some nuance I’m failing to capture, and hence more ways than usual in which I would revise my statements upon further discussion.
For most of the past few years, I’ve had the following view on EA criticism:
- Most EA criticism is - and should be - about the community as it exists today, rather than about the “core ideas.”
- The core ideas are just solid. Do the most good possible - should we really be arguing about that?
Recently, though, I’ve been thinking more about this and realized I’ve changed my mind. I think “do the most good possible” is an intriguing idea, a powerful idea, and an important idea - but it’s also a perilous idea if taken too far. My basic case for this is that:
- If you’re maximizing X, you’re asking for trouble by default. You risk breaking/downplaying/shortchanging lots of things that aren’t X, which may be important in ways you’re not seeing. Maximizing X conceptually means putting everything else aside for X - a terrible idea unless you’re really sure you have the right X. (This idea vaguely echoes some concerns about AI alignment, e.g., powerfully maximizing not-exactly-the-right-thing is something of a worst-case event.)
- EA is about maximizing how much good we do. What does that mean? None of us really knows. EA is about maximizing a property of the world that we’re conceptually confused about, can’t reliably define or measure, and have massive disagreements about even within EA. By default, that seems like a recipe for trouble.
The upshot is that I think the core ideas of EA present constant temptations to create problems. Fortunately, I think EA mostly resists these temptations - but that’s due to the good judgment and general anti-radicalism of the human beings involved, not because the ideas/themes/memes themselves offer enough guidance on how to avoid the pitfalls. As EA grows, this could be a fragile situation.
I think it’s a bad idea to embrace the core ideas of EA without limits or reservations; we as EAs need to constantly inject pluralism and moderation. That’s a deep challenge for a community to have - a constant current that we need to swim against.
How things would go if we were maximally “hard-core”
The general conceptual points behind my critique - “maximization is perilous unless you’re sure you have the right maximand” and “EA is centrally about maximizing something that we can’t define or measure and have massive disagreements about” - are hopefully reasonably clear and sufficiently explained above.
To make this more concrete, I’ll list just some examples of things I think would be major problems if being “EA” meant embracing the core ideas of EA without limits or reservations.
We’d have a bitterly divided community, with clusters having diametrically opposed goals.
For example:
- Many EAs think that “do the most good possible” ends up roughly meaning “Focus on the implications of your actions for the long-run future.”
- Within this set, some EAs essentially endorse: “The more persons there are in the long-run future, the better it is” while others endorse something close to the opposite: “The more persons there are in the long-run future, the worse it is.”1
In practice, it seems that people in these two camps try hard to find common ground and cooperate. But it’s easy to envision a version of EA so splintered by this sort of debate that learning someone is an EA most-often tells you that they are dedicated to maximizing something other than what you’re dedicated to maximizing, and that you should take a fundamentally adversarial and low-trust stance toward each other.
We’d have a community full of low-integrity people, and “bad people” as most people define it.
A lot of EAs’ best guess at the right maximand is along the lines of utilitarianism.
Does utilitarianism recommend that we communicate honestly, even when this would make our arguments less persuasive and cause fewer people to take action based on them? Or does utilitarianism recommend that we “say whatever it takes” to e.g. get people to donate to the charities we estimate to be best?
Does utilitarianism recommend that we stick to promises we made? Or does utilitarianism recommend that we go ahead and break them when this would free us up to pursue our current best-guess actions?
It seems that the answers to these questions are, at best, unclear, and different people have very different takes on them. In general, it seems to be extremely uncertain and debatable what utilitarianism says about a given decision, especially from a longtermist point of view.
Even if, say, 80% of utilitarian EAs thought that utilitarianism supported honesty and integrity, while 20% thought it did not, I think the result would be a noticeably and unusually low-integrity community, full of deception and “bad actors” by the standards of its reference class. I also think that high-integrity behavior works better in a setting where it’s common; 20% of EAs behaving in noticeably low-integrity ways might change the calculus for the other 80%, making things worse still.
My view is that - for the most part - people who identify as EAs tend to have unusually high integrity. But my guess is that this is more despite utilitarianism than because of it. (One hypothesis I’ve heard is that people who care a lot about holding themselves to a high ethical standard, and achieving consistency between their views, statements and actions, are drawn to both utilitarianism and high-integrity behavior for this reason.)
We’d probably have other issues that should just generally give us pause.
I’d expect someone who is determined to derive all their actions from utilitarianism - or from any other explicit maximization of some best-guess maximand - to be at high risk of things like being a bad friend (e.g., refusing to do inconvenient or difficult things when a friend is in need), bad partner (same), narrow thinker (not taking an interest in topics that don’t have clear relevance to the maximand), etc. This is because I doubt there is a maximand and calculation method available that reliably replicates all of the many heuristics people use to be “virtuous” on a variety of dimensions.
Can we avoid these pitfalls by “just maximizing correctly?”
You could argue that for nearly any maximand, it’s a good idea to be the sort of person other people trust and like; to keep lots of life options open; and generally to avoid the sorts of behaviors I worry about above, unless you’re quite confident that you have a calculation pointing that way.
You could make a case for this either from an instrumental point of view (“Doing these things will ultimately make you better at maximizing your maximand”) or using various “nonstandard decision theories” that some EAs are fond of (including myself to some degree).
But I doubt you can make a case that’s robustly compelling and is widely agreed upon, enough to prevent the dynamics I worry about above. Especially if you accept my claim that a significant minority of people behaving badly can be extremely bad. To the extent that the EA community is avoiding these pitfalls, I don’t think this is enough to explain it.
(I do in fact think - though not with overwhelming confidence - that the “pitfalls” I describe would be bad for most plausible EA maximands. How can I simultaneously think this, while also fearing that non-tempered acceptance of EA would result in these “pitfalls?” I address this in a brief appendix.)
Avoiding the pitfalls
I think the EA community does mostly avoid these pitfalls - not in the sense that the dynamics I worry about are absent, but in the sense that they don’t seem more common among EAs than in other analogous communities.
I think a major reason for this is simply that most EAs are reasonable, non-fanatical human beings, with a broad and mixed set of values like other human beings, who apply a broad sense of pluralism and moderation to much of what they do.
My sense is that many EAs’ writings and statements are much more one-dimensional and “maximizy” than their actions. Most EAs seem to take action by following a formula something like: “Take a job at an organization with an unusually high-impact mission, which it pursues using broadly accepted-by-society means even if its goals are unusual; donate an unusual amount to carefully chosen charities; maybe have some other relatively benign EA-connected habits like veganism or reducetarianism; otherwise, behave largely as I would if I weren’t an EA.”
I’m glad things are this way, and with things as they stand, I am happy to identify as an EA. But I don’t want to lose sight of the fact that EA likely works best with a strong dose of moderation. The core ideas on their own seem perilous, and that’s an ongoing challenge.
And I’m nervous about what I perceive as dynamics in some circles where people seem to “show off” how little moderation they accept - how self-sacrificing, “weird,” extreme, etc. they’re willing to be in the pursuit of EA goals. I think this dynamic is positive at times and fine in moderation, but I do think it risks spiraling into a problem.
Brief appendix: spreading the goal of maximizing X can be bad for the goal of maximizing X
There’s a potentially confusing interplay of arguments here. To some degree, I’m calling certain potential dynamics "pitfalls" because I think they would (in fact) be bad for most plausible EA maximands. You might think something like: “Either these dynamics would be bad for the right maximand, in which case you can’t complain that a maximizing mindset is the problem (since proper maximizing would avoid the pitfalls) … or they wouldn’t be bad for the right maximand, and maybe that means they’re just good.” I have a couple of responses to this:
- First, I think the “pitfalls” above are just broadly bad and should give us pause. The fact that a low-trust, bitterly divided EA community would probably be less effective is part of why I think it would be a bad thing, but only part of it. I think honesty is good partly because it seems usually instrumentally valuable, but I also think it’s just good, and would have some trouble being totally comfortable with any anti-honesty conclusion even if the reasoning seemed good.
- Second, I think you can simultaneously believe “X would be bad for the maximand we care about” and “Broadly promoting and accepting a goal of maximizing that maximand would cause X.” EA isn’t just a theoretical principle, it’s a set of ideas and messages that are intended to be broadly spread and shared. It’s not contradictory to believe that spreading a goal could be bad for the goal, and it seems like a live risk here.
Notes
-
I’m thinking of classical utilitarianism for the former, suffering-focused ethics for the latter. Some additional assumptions are needed to reach the positions I list. In particular, some assumption (which I find very plausible) along the lines of: “If we condition on the long-run future having lots of persons in it, most of those lives are probably at least reasonably good [the persons would prefer to exist vs. not exist], but there’s a significant risk that at least some are very bad.” ↩
Thank you for writing this. For a while, I have been thinking of writing a post with many similar themes and maybe I still will at some point. But this post fills a large hole.
As is obligatory for me, I must mention Derek Parfit, who tends to have already well-described many ideas that resurface later.
In Reasons and Persons, Part 1 (especially Chapter 17), Derek Parfit argues that good utilitiarians should self-efface their utilitarianism. This is because people tend to have motivated reasoning, and tend to be wrong. Under utilitarianism, it is possible to justify nearly anything, provided your epistemics are reasonably bad (your epistemics would have to be very bad to justify murder under deontological theories that prohibit murder; you would have to claim that something was not in fact murder at all). Parfit suggests adopting whatever moral system seems to be most likely to produce the highest utility for that person in the long run (perhaps some theory somewhat like virtue ethics). This wasn't an original idea, and Mill said similar things.
One way to self-efface your utilitiarianism would be to say "yeah, I know, it makes sense under utilitarianism for me to keep my promises" (or whatever it may be). Parfit suggests that may not be enough, because deep down you still believe in utilitarianism; it will come creeping through (if not in you, in some proportion of people who self-efface this way). He says that you may instead need to forget that you ever believed in utilitarianism, even if you think it's correct. You need to believe a lie, and perhaps even convince everyone else of this lie.
He also draws an interesting caveat: what if the generally agreed upon virtues or rules are no longer those with the highest expected utility? If nobody believed in utilitarianism, why would they ever be changed? He responds:
This wasn't an original idea either; Parfit here is making a reference to Sidgwick's "Government House utilitarianism," which seemed to suggest only people in power should believe utilitarianism but not spread it. Parfit passingly suggests the utilitarians don't need to be the most powerful ones (and indeed Sidgwick's assertion may have been motivated by his own high position).
Sometimes I think that this is the purpose of EA. To attempt to be the "few people" to believe consequentialism in a world where commonsense morality really does need to change due to a rapidly changing world. But we should help shift commonsense morality in a better direction, not spread utilitarianism.
Maybe utilitarianism is an info hazard not worth spreading. If something is worth spreading, I suspect it's virtues.
Which virtues? Some have suggestions.
"utilitiarians should self-efface their utilitarianism" "Parfit suggests adopting whatever moral system seems to be most likely to produce the highest utility" "you may instead need to forget that you ever believed in utilitarianism"
This sounds plausible: you orient yourself towards the good and backpropagate over time how things play out and then learn through it which system and policies are reliable and truly produce good results (in the context and world you find yourself). This is also exactly what has played out in my own development, by orienting to... (read more)