This is the first out of two posts attempting to make EA strategy discussions more productive. The second post examines EA movement course corrections and where you might disagree.
- Following an influx of funding, media attention, and influence, the EA movement is speeding along an exciting, yet perilous, trajectory.
- A lot of the EA community’s future impact rests on this uncertain growth going well (and thereby avoiding movement collapse scenarios).
- Yet, discussions or critiques of EA’s trajectory rarely feel action-guiding. Even when critiques propose course corrections that are tempting to agree with (e.g., EA should be bigger!), proposed course corrections to make EA more like X often don’t rigorously engage with the downsides of being more like X, or the opportunity cost of not being like Y. Proposals to make EA more like X also often leave me with only a vague understanding of what X looks like and how we get from here to X.
- In hopes of making discussions of the EA community’s trajectory more productive (and to clarify my own thinking on the matter), I will lay out a series of posts that provide an overview of:
- (1) Important ways in which the EA movement could fail
(2) “Domains” in which EA could make course corrections (e.g., more cause-area tailored outreach, new professional networks and events, etc.)
(3) Key considerations that inform course corrections (i.e., places to disagree about course corrections)
(4) Next steps to help guide the EA movement through exciting and perilous times
- (1) Important ways in which the EA movement could fail
- This is the first post in this series: Ways in which EA could fail. Consider it an attempt at bounding the later discussions of strategy updates.
Ways in which EA could fail
The EA movement could collapse. Many movements before us have, and we’re not that special. But other movements, like abolitionism, have left lasting social change. In hopes of being the type of movement that does so well it doesn’t need to exist anymore, this section outlines many of the ways EA could fail.
In this post, I’ll define “failure” as the EA/EA-adjacent ecosystem achieving substantially less impact (let's say, 30%) than it could have along some other trajectory. Note that this is a pretty broad definition. Depending on your worldview and person-affecting ethical views, failure could look more like millions or billions of people alive in the near-future suffering in ways that could have been prevented – or failure could look more like an existential catastrophe that permanently stops sentient life from achieving its full potential.
Implicit in this definition of failure is a statement about the influence EA already has, or could grow to have: I think our ideas and resources are powerful enough to seriously influence how many beings suffer today and how many beings live beautiful lives in the future, such that there’s a massive difference between the most good a flourishing EA ecosystem could achieve (i.e., upper bound) and the good, or possibly even harm, a collapsed EA ecosystem leaves us with (i.e., lower bound).
In order to land closer to the heights of positive impact, let’s think concretely about the worst cases that we must collectively avoid. In what ways might the EA movement fail?
I identify four clusters of failures:
- Reputation failures: failures that result in EA losing a substantial amount of possible impact because the EA community develops a negative reputation.
- Resource failures: failures that result in EA losing a substantial amount of possible impact because the community becomes preventably constrained by financial, human, or infrastructure resources.
- Rigor failures: failures that result in EA losing a substantial amount of possible impact because EA enthusiasts aren’t thinking clearly about prioritizing problems and their solutions.
- Reservation failures: failures that result in EA missing a substantial amount of possible impact because we are too risk averse, or just not ambitious enough
But note a few caveats before I elaborate on these different failure clusters:
- This taxonomy is imperfect. Some failures could fit into multiple categories, or don’t fit cleanly into any. I discuss other ways to group failures in the footnotes.
- Causes of failures are likely to chain into each other across different clusters of failures. For example, diluted epistemic norms (which I categorize as rigor failure) could lead to risky unilateral moves that result in a PR scandal and media firestorm (which I categorize as a reputation failure). In turn, this could lead to a loss of people and funding support (which I categorize as a resource failure). Failures are a messy business – don’t let a nice 4R alliteration taxonomy fool you.
- There are subtleties within failure modes. Some of the failures I discuss below like, for example, EA becoming politicized, could further be broken down into different consequences that plausibly have a different impact (e.g., the consequences of EA becoming disliked by left or right US political parties are probably very different). Similarly, internal disenchantment and negative press are already happening at some level, so these causes of ‘failure’ are clearly a matter of degree.
- Not all failure modes below are equally bad or equally likely. Considering both the badness and likelihood of different failures is an important next step in prioritizing which failures to course-correct away from.
- Forgive me for vaguely using “EA.” For the sake of simplicity – and because I really am gesturing at the whole movement – I often invoke the mysterious, definitely-not-an-agent “EA” in reference to some action. But try to avoid doing this!
- EA could still be on its optimal trajectory despite some evidence of ‘failure’. For example, I expect at least some parts of the optimal version of the EA movement just aren’t everyone's vibe and some people will feel disenchanted by them. While I classify disenchantment as a failure mode below, the fact that we might still expect to see disenchantment in the optimal version of EA means evidence of the beginning of a failure doesn’t necessarily mean EA needs to course-correct away from that failure.
Reputation failures result in EA losing a substantial amount of possible impact because the community develops a negative reputation, either internally, externally, or both.
|Causes of reputation failure||Examples|
|Media firestorm||Some version of the meme that “EA is just a bunch of white billionaires claiming to do good so they can feel good about themselves” catches on and lots of media outlets jump on the bandwagon. Now typical college students think EA is lame and politicians don’t want to touch EA.|
|Internal disenchantment ||Engaged EAs start feeling disenchanted or misrepresented by the overall aesthetic (e.g., elitist) or actions (e.g., free-spending) of the EA movement and distance themselves from the community. Now many cool people are missing out on valuable coordination and communication, and this process may be reinforced since those that remain in EA are the most hardcore.|
|Politicization (of EA itself or core ideas)||Wild animal welfare or speciesism become seen as just another woke idea and honest intellectual discussion around the topic becomes increasingly difficult.|
|Infighting||Disagreements about funding and relative attention between global health and wellbeing vs. longtermist works boils over into a messy, public fracture of EA. Or this leads to internal disenchantment.|
|Risky unilateral project||Someone carelessly takes on a risky project, say middle school outreach, that has clear downside risks that have caused others to steer clear of it. The project blows up and EA’s reputation is worse for it.|
|Too demanding or totalizing||EA develops a public reputation as too hardcore and demanding, and people, like politicians, who could have a great deal of impact feel like they can’t be ‘kinda EA.’ Related to concerns about EA as an identity.|
|Scandal||An instance (or multiple instances) of sexual harassment in EA social circles cripples EAs reputation.|
Resource failures result in EA losing a substantial amount of possible impact because the community becomes constrained by financial and/or human resources.
|Causes of resource failure||Examples|
|Running out of money|
(1) Sam Bankman Fried’s and other top EA donors’ net worth plummets because of cryptocurrency or US stock market crashes
|No new talent (or evaporating old talent)||For whatever reason (probably reputation collapse or infrastructure constraints), EA can’t attract – or keep – the quantity and quality of talented individuals it urgently needs (above and beyond the typical talent constraint we see today).|
|Inadequate infrastructure||EA fails to build the scalable infrastructure you need to coordinate many people pushing in a coherent direction.|
Rigor failures result in EA losing a substantial amount of possible impact because EA enthusiasts aren’t thinking clearly about prioritizing problems and their solutions.
|Causes of rigor failures||Examples|
|Dilution of truth-seeking and maximizing norms||(1) EA doubles in size multiple years in a row, and now more than half the conversations people have at EAGx-style events are with people brand new to EA norms. Gradually rigorous truth-seeking norms lose their footing.|
(2) EA splits along cause areas or philosophical differences, and the community starts to focus on naively optimizing within certain cause areas and misses new opportunities
|Funding and status-seeking muddle intellectual rigor||EAs implicitly follow incentives to do what gets them in-group clout and larger paychecks (e.g., embrace longtermism) in a way that weakens general quality of thought.|
|Excessive deference culture||New EAs start assuming that old wise EAs before them already have answers, and we build our epistemic foundations and prioritization schemes on shaky foundations.|
|Bad actors||The meme that EA has a lot of money to give if you know the right things to say spreads outside the community, and bad actors degrade trust networks that underlie existing coordination.|
|Excessive nepotism||Infiltrating key EA decision-making spaces becomes too much a function of who you know, in a way that selects for qualities other than those of the best decision-makers.|
|Insufficient intellectual diversity||Many people throughout history have landed at different answers for what good means, yet EA uncritically concludes it has found the right answer and pushes towards it without maintaining option value. And then we realize that we missed the mark (or, more likely, we never realize because moral philosophy literally has the worst feedback loops).|
|Echo chambers||EAs become siloed in their own worldviews and shared models and underweight the importance of existing power structures or things like systemic cascading risks.|
|Goodharting||EA decision-makers begin to optimize for legible metrics that don’t actually track our overarching goals|
|Poor feedback loops||Longtermist work focuses too much on preparing for future success without getting feedback from the real world about whether we’re preparing in the right ways.|
|Out-of-touch leadership||The EAs closest to leadership become isolated from the rest of the community. They lose a source of outside feedback and a check on their epistemics.|
Reservation failures result in EA missing a substantial amount of possible impact because we are too risk-averse, or just not ambitious enough. While reservation failures are less likely to lead to movement collapse, they are arguably more likely to increase the likelihood of an existential catastrophe – or just the preventable suffering of (m?b??t???)illions.
|Causes of reservation failures||Examples|
|Not enough focus on scalability||Too many EAs twiddle around with small-scale projects that lack the entrepreneurial mindset to deploy all the resources at our disposal (or acquire much more).|
|Unwillingness to trade reputation for impact||EAs don’t take an action such as global talent search because it might harm our reputation (in this case elitism critiques), even if these are the types of projects we need to solve the most pressing problems.|
|Overlooking serious possibility of short transformative AI timelines||EAs don’t take the non-negligible chance of transformative AI arriving around 2030 seriously and we’re caught without a robust emergency plan when things start getting crazy and dangerous.|
|Spending-averse||EAs are overly wary of deploying large amounts of money because it’s in tension with the movement's frugal foundations.|
|Not joining the adult table||EAs cautiously hold off on bringing their ideas to spaces like US politics, intergovernmental organizations (e.g., UN), or other powerful institutions because we think “we’re not ready yet.”|
|Bureaucracy||As EA institutions mature, they start looking more like traditional orgs and become bogged down by red tape, an overemphasis on legibility (e.g., in grants), and/or the “bureaucrat’s curse” where everyone needs to sign off on everything.|
Next steps re: failure modes
- Analyze relative badness and the likelihood of failure modes to decide which to prioritize.
- Analyze how different failure modes could play out. I picture many “failures” of EA still leaving behind a core group of people committed to maximal impartial altruism. But the damage to branding, coordination, and resources might well vary depending on what caused the failures, and I expect this to have a big influence on how much any group could accomplish.
- Identify what warning signs we might expect (or are already seeing) for each failure mode and some action threshold. I expect this action threshold to vary depending on people’s idealized vision for EAs trajectory, which will be the focus of the next posts.
- Identify more EA failure modes or debate the ones I listed: What did I miss?
- Improve the taxonomy: Would refactoring the failure modes make them more action-guiding?
- Proposals to make cause-areas more robust to EA brand collapse. Depending on how concerned one is about the reputation of the EA brand, there may be arguments to preemptively make work in different areas (e.g., AI safety) not too coupled to EA.
UPDATE: The second post on EA movement course corrections and where you might disagree is live.
The next post, or posts, will identify:
- “Domains” in which EA could make course corrections (e.g., more cause-area tailored outreach, new professional networks and events, etc.).
- Key considerations that inform course corrections (i.e., places to disagree about course corrections).
- More next steps proposals to help guide the EA movement through this exciting and perilous period of growth.
I hope to have these next sections out within the next two weeks.
Examples of ways EA could have a net-negative impact is by directing finite altruistic talent in the wrong direction, permanently tainting ideas like doing good with an emphasis on rationality, or by discouraging others to tackle X important problem because we give the impression we're on it when we're really not.
Other failure mode taxonomies include:
(1) Sequestration, Attrition, Dilution, and Distraction
(2) Failures from pushing too hard, not pushing hard enough, or pushing in the wrong direction.
(3) Failures where the recognizable EA collapses before an existential catastrophe vs. failures where EA is still intact but fails to prevent an existential catastrophe
(4) Failures of Commission vs. omission: causing harm vs. squandering an opportunity.
You can imagine that this makes it really difficult for CEA and other EA thought leaders to know when to course-correct upon criticism, especially when you factor in an uncertain switching cost one incurs trying to coordinate and implement a trajectory change.
See Leaning into EA Disillusionment for more discussion
For more discussion, see EA considerations regarding increasing political polarization.
Linch accurately notes in comments that the consequences of infighting are more complicated than a mere reputation failure.
See also: Will MacAskill’s writing on resentment and the Book How Change Happens by Leslie Crutchfield, which notes effective management of infighting as a consistent attribute of the most effective movements in recent US history (Chapter 5).
See also: Will MacAskill’s writing on the current funding situation harming quality of thought
See the epistemic deference forum tag for more discussion
Idea borrowed from this post on movement collapse scenarios
For what it’s worth, I think reservation failures are the most overlooked cluster of failures. I think it’s just much easier to criticize ostensibly bad things EA does than all the invisible impact EA loses by being overly cautious or doing some things that turn people off. However, reservation failures are a delicate matter, because they are often – although not always – associated with downside risks that require serious consideration.
See Will MacAskill’s writing on “Risks of Omission” for more discussion of scalability:
It seems to me to be more likely that we’ll fail by not being ambitious enough; by failing to take advantage of the situation we’re in, and simply not being able to use the resources we have for good ends.
Credit to Nick Beckstead for "Bureaucrat’s curse"