Luke Kemp and I just published a paper which criticises existential risk for lacking a rigorous and safe methodology:

It could be a promising sign for epistemic health that the critiques of leading voices come from early career researchers within the community. Unfortunately, the creation of this paper has not signalled epistemic health. It has been the most emotionally draining paper we have ever written.

We lost sleep, time, friends, collaborators, and mentors because we disagreed on: whether this work should be published, whether potential EA funders would decide against funding us and the institutions we're affiliated with, and whether the authors whose work we critique would be upset.

We believe that critique is vital to academic progress. Academics should never have to worry about future career prospects just because they might disagree with funders. We take the prominent authors whose work we discuss here to be adults interested in truth and positive impact. Those who believe that this paper is meant as an attack against those scholars have fundamentally misunderstood what this paper is about and what is at stake. The responsibility of finding the right approach to existential risk is overwhelming. This is not a game. Fucking it up could end really badly.

What you see here is version 28. We have had approximately 20 + reviewers, around half of which we sought out as scholars who would be sceptical of our arguments. We believe it is time to accept that many people will disagree with several points we make, regardless of how these are phrased or nuanced. We hope you will voice your disagreement based on the arguments, not the perceived tone of this paper.

We always saw this paper as a reference point and platform to encourage greater diversity, debate, and innovation. However, the burden of proof placed on our claims was unbelievably high in comparison to papers which were considered less “political” or simply closer to orthodox views. Making the case for democracy was heavily contested, despite reams of supporting empirical and theoretical evidence. In contrast, the idea of differential technological development, or the NTI framework, have been wholesale adopted despite almost no underpinning peer-review research. I wonder how much of the ideas we critique here would have seen the light of day, if the same suspicious scrutiny was applied to more orthodox views and their authors.

We wrote this critique to help progress the field. We do not hate longtermism, utilitarianism or transhumanism,. In fact, we personally agree with some facets of each. But our personal views should barely matter. We ask of you what we have assumed to be true for all the authors that we cite in this paper: that the author is not equivalent to the arguments they present, that arguments will change, and that it doesn’t matter who said it, but instead that it was said.

The EA community prides itself on being able to invite and process criticism. However, warm welcome of criticism was certainly not our experience in writing this paper.

Many EAs we showed this paper to exemplified the ideal. They assessed the paper’s merits on the basis of its arguments rather than group membership, engaged in dialogue, disagreed respectfully, and improved our arguments with care and attention. We thank them for their support and meeting the challenge of reasoning in the midst of emotional discomfort. By others we were accused of lacking academic rigour and harbouring bad intentions. 

We were told by some that our critique is invalid because the community is already very cognitively diverse and in fact welcomes criticism. They also told us that there is no TUA, and if the approach does exist then it certainly isn’t dominant.  It was these same people that then tried to prevent this paper from being published. They did so largely out of fear that publishing might offend key funders who are aligned with the TUA. 

These individuals—often senior scholars within the field—told us in private that they were concerned that any critique of central figures in EA would result in an inability to secure funding from EA sources, such as OpenPhilanthropy. We don't know if these concerns are warranted. Nonetheless, any field that operates under such a chilling effect is neither free nor fair. Having a handful of wealthy donors and their advisors dictate the evolution of an entire field is bad epistemics at best and corruption at worst. 

The greatest predictor of how negatively a reviewer would react to the paper was their personal identification with EA. Writing a critical piece should not incur negative consequences on one’s career options, personal life, and social connections in a community that is supposedly great at inviting and accepting criticism.

Many EAs have privately thanked us for "standing in the firing line" because they found the paper valuable to read but would not dare to write it. Some tell us they have independently thought of and agreed with our arguments but would like us not to repeat their name in connection with them. This is not a good sign for any community, never mind one with such a focus on epistemics. If you believe EA is epistemically healthy, you must ask yourself why your fellow members are unwilling to express criticism publicly. We too considered publishing this anonymously. Ultimately, we decided to support a vision of a curious community in which authors should not have to fear their name being associated with a piece that disagrees with current orthodoxy. It is a risk worth taking for all of us. 

The state of EA is what it is due to structural reasons and norms (see this article). Design choices have made it so, and they can be reversed and amended. EA fails not because the individuals in it are not well intentioned, good intentions just only get you so far.

EA needs to diversify funding sources by breaking up big funding bodies and by reducing each orgs’ reliance on EA funding and tech billionaire funding, it needs to produce academically credible work, set up whistle-blower protection, actively fund critical work, allow for bottom-up control over how funding is distributed, diversify academic fields represented in EA, make the leaders' forum and funding decisions transparent, stop glorifying individual thought-leaders, stop classifying everything as info hazards…amongst other structural changes. I now believe EA needs to make such structural adjustments in order to stay on the right side of history. 

Sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Hi - Thanks so much for writing this. I'm on holiday at the moment so have only have been able to  quickly  skim your post and paper.  But, having got the gist, I just wanted to say:
(i) It really pains me to hear that you lost time and energy as a result of people discouraging you from publishing the paper, or that you had to worry over funding on the basis of this. I'm sorry you had to go through that. 
(ii) Personally, I'm  excited to fund or otherwise encourage engaged and in-depth "red team" critical work on either (a) the ideas of EA, longtermism or strong longtermism, or (b) what practical implications have been taken to follow from EA, longtermism, or strong longtermism.  If anyone reading this comment  would like funding (or other ways of making their life easier) to do (a) or (b)-type work, or if you know of people in that position, please let me know at  I'll try to consider any suggestions, or put the suggestions in front of others to consider, by the end of  January. 

I just want to say that I think this is a beautifully accepting response to criticism. Not defensive. Says hey yes maybe there is a problem here. Concretely offers time and money and a plan to look into things more. Really lovely, thank you Will. 

Thanks for stating this publically  here Will! 

Hi Carla and Luke, I was sad to hear that you and others were concerned that funders would be angry with you or your institutions for publishing this paper. For what it's worth, raising these criticisms wouldn't count as a black mark against you or your institutions in any funding decisions that I make. I'm saying this here publicly in case it makes others feel less concerned that funders would retaliate against people raising similar critiques. I disagree with the idea that publishing critiques like this is dangerous / should be discouraged.

+1 to everything Nick said, especially the last sentence. I'm glad this paper was published; I think it makes some valid points (which doesn't mean I agree with everything), and I don't see the case that it presents any risks or harms that should have made the authors consider withholding it. Furthermore, I think it's good for EA to be publicly examined and critiqued, so I think there are substantial potential harms from discouraging this general sort of work.

Whoever told you that funders would be upset by your publishing this piece, they didn't speak for Open Philanthropy. If there's an easy way to ensure they see this comment (and Nick's), it might be helpful to do so.

+1, EA Funds (which I run) is interested in funding critiques of popular EA-relevant ideas.

Thanks for saying this publically too Nick, this is helpful for anyone who might worry about funding. 

I thought the paper itself was poorly argued, largely as a function of biting off too much at once. Several times the case against the TUA was not actually argued, merely asserted to exist along with one or two citations for which it is hard to evaluate if they represent a consensus. Then, while I thought the original description of TUA was accurate, the TUA response to criticisms was entirely ignored. Statements like "it is unclear why a precise slowing and speeding up of different technologies...across the world is more feasible or effective than the simpler approach of outright bans and moratoriums" were egregious, and made it seem like you did not do your research. You spoke to 20+ reviewers, half of which were sought out to disagree with you, and not a single one could provide a case for differential technology? Not a single mention of the difficulty of incorporating future generations into the democratic process?

Ultimately, I think the paper would have been better served by focusing on a single section, leaving the rest to future work. The style of assertions rather than argument and skipping over potential responses comes across as more polemical than evidence-seeking. I bel... (read more)


I would agree that the article is too wide-ranging. There's a whole host of content ranging from criticisms of expected value theory, arguments for degrowth, arguments for democracy, and then criticisms of specific risk estimates. I agreed with some parts of the paper, but it is hard to engage with such a wide range of topics. 

Where? The paper doesn't mention economic growth at all.

The paper doesn't explicitly mention economic growth, but it does discuss technological progress, and at points seems to argue or insinuate against it.

"For others who value virtue, freedom, or equality, it is unclear why a long-term future without industrialisation is abhorrent: it all depends on one’s notion of potential." Personally, I consider a long-term future with a 48.6% child and infant mortality rate  abhorrent and opposed to human potential, but the authors don't seem bothered by this. But they have little enough space to explain how their implied society would handle the issue, and I will not critique it excessively.

There is also a repeated implication that halting technological progress is, at a minimum, possible and possibly desirable.
"Since halting the technological juggernaut is considered impossible, an approach of differential technological development is advocated"
"The TUA rarely examines the drivers of risk generation. Instead, key texts contend that regulating or stopping technological progress is either deeply difficult, undesirable, or outright impossible"
"regressing, relinquishing, or stopping the development of many technologies is often disregarded as ... (read more)

"For others who value virtue, freedom, or equality, it is unclear why a long-term future without industrialisation is abhorrent: it all depends on one’s notion of potential."

Point taken. Thank you for pointing this out.

"The TUA rarely examines the drivers of risk generation. Instead, key texts contend that regulating or stopping technological progress is either deeply difficult, undesirable, or outright impossible"
"regressing, relinquishing, or stopping the development of many technologies is often disregarded as a feasible option" implies to me that one of those three options is a feasible option, or is at least worth investigating.

I think this is more about stopping the development of specific technologies - for example, they suggest that stopping AGI from being developed is an option. Stopping the development of certain technologies isn't necessarily related to degrowth - for example, many jurisdictions now ban government use of facial recognition technology, and there have been calls to abolish its use, but these are motivated by civil liberties concerns.

I think this conflates the criticism of the idea of unitary and unstoppable technological progress with opposition to any and all technological progress.

Suggesting that a future without industrialization is morally tolerable does not imply opposition to "any and all" technological progress, but the amount of space left is very small. I don't think they're taking an opinion on the value of better fishhooks.

It is morally tenable under some moral codes but not others. That's the point.

Several times the case against the TUA was not actually argued

I think that they didn't try to oppose the TUA in the paper, or make the argument against it themselves. To quote: "We focus on the techno-utopian approach to existential risk for three reasons. First, it serves as an example of how moral values are embedded in the analysis of risks. Second, a critical perspective towards the techno-utopian approach allows us to trace how this meshing of moral values and scientific analysis in ERS can lead to conclusions, which, from a different perspective, look like they in fact increase catastrophic risk. Third, it is the original and by far most influential approach within the field."

I also think that they don't need to prove that others are wrong to show that the lack of diversity has harms - as you agreed.

If the wrong arguments being made would cause harm when believed, it is not only the right but the responsibility of funders to reduce their reach. 

That puts a huge and dictatorial responsibility on funders in ways that are exactly what the paper argued are inappropriate.

The responsibility of the researcher is to make their case as bulletproofed as possible, and designed to c

... (read more)

That puts a huge and dictatorial responsibility on funders in ways that are exactly what the paper argued are inappropriate.

If not the funders,  do you believe anyone should be responsible for ensuring harmful and wrong ideas are not widely circulated? I can certainly see the case that even wrong, harmful ideas should only be addressed by counterargument. However, I'm not saying that resources should be spent censoring wrong ideas harmful to EA, just that resources should not be spent actively promoting them. Funding is a privilege, consistently making bad arguments should eventually lead to the withdrawal of funding, and if on top of that those bad arguments are harmful to EA causes that should expedite the decision.

To be clear, that is absolutely not to say that publishing Democratizing Risk is/was justification for firing or cut funding, I am still very much talking abstractly.

Guy Raveh
Guy Raveh
I mostly agree with your comments, but I think we need to stop referring to specific people as leaders of the movement. Will MacAskill's opinion is not really more important than anyone else's.

I disagree pragmatically and conceptually. First, people pay more attention to Will than to me about this, and that's good, since he's spent more time thinking about it, is smarter, and has more insight into what is happening. Second, in fact, movements have leaders, and egalitarianism is great for rights, but direct democracy a really bad solution to running anything which wants to get anything done. (Which seems to be a major thing I disagree with the authors of the article on.)

Saying thepaper is poorly argued is not particularly helpful or convincing. Could you highlight where and why Rubi? Breadth does not de-facto mean poorly argued.  If that was the case then most of the key texts in  x-risk would all be poorly argued. Importantly, breadth was necessary to make a critique. There are simply many interrelated matters that are worth critical analysis.  Several times the case against the TUA was not actually argued, merely asserted to exist along with one or two citations for which it is hard to evaluate if they represent a consensus. As  David highlights in his response: we do not argue against the TUA, but point out the unanswered questions we observed. We do not argue against the TUA , but highlight assumptions that may be incorrect or smuggle in values. Interestingly, it's hard to find  how you believe the piece is both polemic but also not directly critiquing the TUA sufficiently.  Those two criticisms are in tension.  If you check our references, you will see that we cite many published papers that treat common criticisms and open questions of the TUA (mostly by advancing the research).  You spoke to 20+ reviewers, half of which were sought out to disagree with you, and not a single one could provide a case for differential technology? Of course there are arguments for it, some of which are discussed in the forum. Our argument is that there is a lack of peer-review evidence to support differential technological development as a cornerstone of a policy approach to x-risk. Asking that we articulate and address every hypothetical counterargument is an incredibly high-bar, and one that is not applied to any other literature in the field (certainly not the key articles of the TUA we focus on). It would also make the paper far longer and broader. Again,  your points are in tension.  I think the paper would have been better served by focusing on a single section, leaving the rest to future work. The style of assertions rather than ar

Hi Carla,

Thanks for taking the time to engage with my reply. I'd like to engage with a few of the points you made.

First of all, my point prefaced with 'speaking abstractly' was genuinely that. I thought your paper was poorly argued, but certainly within acceptable limits that it should not result in withdrawn funding. On a sufficient timeframe, everybody will put out some duds, and your organizations certainly have a track record of producing excellent work. My point was about avoiding an overcorrection, where consistently low quality work is guaranteed some share of scarce funding merely out of fear that withdrawing such funding would be seen as censorship. It's a sign of healthy epistemics (in a dimension orthogonal to the criticisms of your post) for a community to be able to jump from a specific discussion to the general case, but I'm sorry you saw my abstraction as a personal attack.

You saw "we do not argue against the TUA, but point out the unanswered questions we observed. .. but highlight assumptions that may be incorrect or smuggle in values".  Pointing out unanswered questions and incorrect assumptions is how you argue against something! What makes your paper polemic... (read more)


I share the surprise and dismay other commenters have expressed about the experience you report around drafting this preprint. While I disagree with many claims and framings in the paper, I agreed with others, and the paper as a whole certainly doesn't seem beyond the usual pale of academic dissent. I'm not sure what those who advised you not to publish were thinking.

In this comment, I'd like to turn attention to the recommendations you make at the end of this Forum post (rather than the preprint itself). Since these recommendations are somewhat more radical than those you make in the preprint, I think it would be worth discussing them specifically, particularly since in many cases it's not clear to me exactly what is being proposed.

Having written what follows, I realise it's quite repetitive, so if in responding you wanted to pick a few points that are particularly important to you and focus on those, I wouldn't blame you!

You claim that EA needs to...

diversify funding sources by breaking up big funding bodies 

Can you explain what this would mean in practice? Most of the big funding bodies are private foundations. How would we go about breaking these up? Who even is "we" in th... (read more)

While I appreciate that we're all busy people with many other things to do than reply to Forum comments, I do think I would need clarification (and per-item argumentation) of the kind I outline above in order to take a long list of sweeping changes like this seriously, or to support attempts at their implementation.

Especially given the claim that "EA needs to make such structural adjustments in order to stay on the right side of history".


The discussion of Bostrom's Vulnerable World Hypothesis seems very uncharitable. Bostrom argues that on the assumption that technological development makes the devastation of civilisation extremely likely, extreme policing and surveillance would be one of the few ways out. You give the impression that he is arguing for this now in our world ("There is little evidence that the push for more intrusive and draconian policies to stop existential risk is either necessary or effective"). But this is obviously not what he is proposing - the vulnerable world hypothesis is put forward as a hypothesis and he says he is not sure whether it is true. 

Moreover, in the paper, Bostrom discusses at length the obvious risks associated with increasing surveillance and policing:

"It goes without saying that a mechanism that enables unprecedentedly intense forms of surveillance, or a global governance institution capable of imposing its will on any nation, could also have bad consequences. Improved capabilities for social control could help despotic regimes protect themselves from rebellion. Ubiquitous surveillance could enable a hegemonic ideology or an intolerant majority view to impose itself on... (read more)

That was my reading of VWH too - as a pro tanto argument for extreme surveillance and centralized global governance, provided that the VWH is true. However, many of its proponents seem to believe that the VWH is likely to be true. I do agree that the authors ought to have interpreted the paper more carefully, though.

* It doesn't matter whether Nick Bostrom speculates or wants to implement surveillance globally. In respect to what we talk about (justification of extreme actions) what matters is how readers perceive his work and who the readers are. * There’s some hedging in the article but… * He published in a policy journal, with an opening ‘policy implication’ box * He published an outreach article about in Aeon, which also ends with the sentence: ”If you find yourself in a position to influence the macroparameters of preventive policing or global governance, you should consider that fundamental changes in those domains might be the only way to stabilise our civilisation against emerging technological vulnerabilities.” * In public facing interviews such as with Sam Harris and on TED, the idea of ‘turnkey totalitarianism’ was made the centrepiece. This was not framed as one hypothetical, possible, future solution for a philosophical thought experiment. * The VWH was also published as a German book (why I don’t know…)

It still seems like you have mischaracterised his view. You say "Take for example Bostrom’s “Vulnerable World Hypothesis”17, which argues for the need for extreme, ubiquitous surveillance and policing systems to mitigate existential threats, and which would run the risk of being co-opted by an authoritarian state." This is misleading imo. Wouldn't it have been better to note the clearly important hedging and nuance and then say that he is insufficiently cognisant of the risks of his solutions (which he discusses at length)?

Thanks for laying this out so clearly. One frustrating aspect of having a community comprised of so many analytic philosophy students (myself included!) is a common insistence on interpreting statements, including highly troubling ones, exactly as they may have been intended by the author, at the exclusion of anything further that readers might add, such as historical context or ways that the statement could be misunderstood or exploited for ill purposes. Another example of this is the discussion around Beckstead's (in my opinion, deeply objectionable) quote regarding the (hypothetical, ceteris-paribus, etc., to be clear) relative value of saving rich versus poor lives.[1] 

I do understand the value of hypothetical inquiry as part analytic philosophy and appreciate its contributions to the study of morality and decision-making. However, for a community that is so intensely engaged in affecting the real world, it often feels like a frustrating motte-and-bailey, where the bailey is the efforts to influence policy and philanthropy on the direct basis of philosophical writings, and the motte is the insistence that those writings are merely hypothetical.

In my opinion, it's insuffici... (read more)

Emile Torres (formerly Phil) just admitted on their Twitter that they were a co-author of a penultimate version of this paper. It is extremely deceptive not to disclose their contribution this in the paper or in the Forum post. At the point this paper was written, Torres had been banned from the EA Forum and multiple people in the community had accused Torres of harassing them. Do you think that that might have contributed to the (alleged) reception to your paper? 

The post in which I speak about EAs being uncomfortable about us publishing the article only talks about interactions with people who did not have any information about initial drafting with Torres. At that stage, the paper was completely different and a paper between Kemp and I. None of the critiques about it or the conversations about it involved concerns about Torres, co-authoring with Torres or arguments by Torres, except in so far as they might have taken Torres an example of the closing doors that can follow a critique. The paper was in such a totally different state and it would have been misplaced to call it a collaboration with Torres. 

There was a very early draft of Torres and Kemp which I was invited to look at (in December 2020) and collaborate on. While the arguments seemed promising to me, I thought it needed major re-writing of both tone and content. No one instructed me (maybe someone instructed Luke?) that one could not co-author with Torres. I also don't recall that we were forced to take Torres off the collaboration (I’m not sure who know about the conversations about collaborations we had): we decided to part because we wanted to move the content and tone i... (read more)


Here's a Q&A which answers some of the questions by  reviewers of early drafts. (I planned to post it quickly, but your comments came in so soon! Some of the comments hopefully find a reply here)

"Do you not think we should work on x-risk?"

  • Of course we should work on x-risk


"Do you think the authors you critique have prevented alternative frameworks from being applied to Xrisk?"

  • No. It’s not really them we’re criticising if at all. Everyone should be allowed to put out their ideas. 
  • But not everyone currently gets to do that. We really should have a discussion about what authors and what ideas get funding.


"Do you hate longtermism?"

  • No. We are both longtermists (probs just not the techno utopian kind).


"You’re being unfair to Nick Bostrom. In the vulnerable world hypothesis, Bostrom merely speculates that such a surveillance system may, in a hypothetical world in which VWH is true, be the only option"

  • It doesn't matter whether Nick Bostrom speculates or wants to implement surveillance globally. In respect to what we talk about (justification of extreme actions) what matters is how readers perceive his work and who the readers are. 
  • There’s some hedging i
... (read more)

It’s been interesting to re-read the discussion of this post in light of new knowledge that Emile P Torres was originally a co-author. For example, Cremer instructs reviewers to ask why they might have felt like the paper was a hostile attack. Well, I’d certainly see why readers could have had this perception if they read it after Emile had already started publicly insinuating that various longtermists are sympathetic to white supremacy, are plagiarists.

Cremer also says some reviewers asked, “Do you hate longtermism?"

The answer she gives above is “No. We are both longtermists (probs just not the techno utopian kind)”, but it seems like the answer would have in fact been “Two of us do not, but one the authors does hate longtermism and has publicly called it incredibly dangerous”

Just noting that I strongly endorse both this format for responding to questions, and the specific responses.

With regard to harshness, I think part of the reason you get different responses is because you're writing in the genre of the academic paper. Since authors have to write in a particular formal style, it's ambiguous whether they intend a value judgment. Often authors do want readers to come away with a particular view, so it's not crazy to read their judgments into the text, but different readers will draw different conclusions about what  you want them to feel or believe. For example: As with many points in your paper, this is literally true, and I appreciate you raising awareness of it! In a different context, I might read this as basically a value-neutral call to arms. Given the context, it's easy to read into it some amount of value judgment around longtermism and longtermists. 

The linked article seems to overstate the extent to which EAs support totalitarian policies. While it is true that EAs are generally left-wing and have more frequently proposed increases in the size & scope of government than reductions, Bostrom did commission an entire chapter of his book on the dangers of a global totalitarian government from one of the world's leading libertarian/anarchists, and longtermists have also often been supportive of things that tend to reduce central control, like charter cities, cryptocurrency and decentralised pandemic control.

Indeed, I find it hard to square the article's support for ending technological development with its opposition to global governance. Given the social, economic and military advantages that technological advancement brings, it seems hard to believe that the US, China, Russia etc. would all forgo scientific development, absent global coordination/governance. It is precisely people's skepticism about global government that makes them treat AI progress as inevitable, and hence seek other solutions. 

I think it's important to distinguish "anarcho"-capitalist thought (which still needs a state to enforce private property and capital rights and generally doesn't acknowledge the problems of monopolies, existing power imbalances etc.) and actual anarchist/anti-totalitarian policies.   All the things you mentioned except the last decentralise control from a democratic state to a moneyed elite, not to a more democratic state, confederation, anarchist commune or whatever.

I really dislike it when left-anarchist-leaning folks put scare quotes around "anarcho" in anarcho-capitalist. In my experience it's  a strong indicator that someone isn't arguing in good faith.

I'm not an ancap (or a left-anarchist), but David Friedman and his ilk is very clearly trying to formulate a way for a capitalist society to exist without a state. You might think their plans are unfeasible or undesirable (and I do), but that doesn't mean they're secretly not anarchists.

From the perspective of every other lineage of anarchists, private property is one of the things that enforces injust hierarchies. Using that label is like calling yourself "vegano-carnivore" because you want to reduce the suffering of eating animals as much as possible while still eating them. Even if you can come up with a justification on it by presenting clearly realizable ways to implement this (e.g. lab grown meat), it is adopting a label from a community that does not want them to do so. Indeed, there was already a ready-made label "laisez-faire", but that one has sufficiently negative historical associations that I guess it is to be avoided. Regarding Friedman,  I would challenge the statement that he provides ways to organize it without a state, given that he romantizices medieval iceland and the western frontier and I am highly skeptical that the law enforcement/military aspect required for enforcing capital right would not lead to the tyranny of the robber barons in their company towns again, but I would have to revisit it for detailed rebuttal.

I don't think most people outside left-anarchism would equate "state" with the existence of any unjust hierarchies. Indeed, defining a state in that way seems to be begging the question with regard to anarchy's desirability and feasibility.

Whether or not Friedman provides ways to organise society without a state, he is clearly trying to do so, at least by any definition of "state" that a non-(left-anarchist) would recognise (e.g. an entity with a monopoly on legitimate violence).

Michael Große
  I don't see where anonea2021 has made that claim. Did you mean to write "property" instead of "state" in this paragraph? (genuine question) Either way, I'm having trouble following what you want to say with this paragraph. What anonea2021 states: I can confirm that this indeed the view of every other lineage of anarchists that I'm aware of.  The anarchist's goal is to minimize unjust hierarchies. And given that private property (esp. of the means of production) is seen as one of the main causes of unjust hierarchies in today's world, it is plausible that a movement that tries create a society which structures itself completely along the lines of private property, is seen as utterly missing the point of anarchism. Thus "anarcho-"capitalism.
Will Bradshaw
Yes, it seems like there's some crossed wires here. I claimed that ancaps are "clearly trying to formulate a way for a capitalist society to exist without a state". The intended implicature was that since anarchy = the absence of a state (according to common understanding, the dictionary definition, and etymology) it was therefore proper to call them anarchists. anonea2021 responded with "From the perspective of every other lineage of anarchists, private property is one of the things that enforces injust hierarchies." I was confused about this, since it didn't seem like a direct response to my claims. I wasn't sure whether to read it as (a) a claim that unjust hierarchies = a state (which seemed like a bad definition of "state"), or (b) a claim that anarchism wasn't actually about the absence of a state but instead about abolishing unjust hierarchies in general (which seemed like a bad, question-begging definition of "anarchism", given that ~everyone wants to minimise unjust hierarchies). I tried to respond to the superposition of these two interpretations, which probably led to my phrasing being more confusing than it needed to be.  As before, this begs the question. Everyone wants to minimise unjust hierarchies, so that's not a useful description of anarchism. People who disagree about which hierarchies are unjust, what interventions are effective for reducing them, and what the costs of those interventions are, will end up advocating for radically different systems of government. Some of those will end up advocating for a society without a state, and it's useful to refer to that subset of positions as "anarchist" even if they are very different from each other. Anarcho-capitalism is really quite different from other forms of capitalist social organisation, and its distinctive feature is the absence of a coercive state. "Anarcho-capitalism" is thus a completely appropriate name for it – indeed, it's hard to see what other name would fit better. Also, it's wha

First of all, I'm sorry to hear you found the paper so emotionally draining. Having rigorous debate on foundational issues in EA is clearly of the utmost importance. For what it's worth when I'm making grant recommendations I'd view criticizing orthodoxy (in EA or other fields) as a strong positive so long as it's well argued. While I do not wholly agree with your paper, it's clearly an important contribution, and has made me question a few implicit assumptions I was carrying around.

The most important updates I got from the paper:

  1. Put less weight on technological determinism. In particular, defining existential risk in terms of a society reaching "technological maturity" without falling prey to some catastrophe frames technological development as being largely inevitable. But I'd argue even under the "techno-utopian" view, many technological developments are not needed for "technological maturity", or at least not for a very long time. While I still tend to view development of things like advanced AI systems as hard to stop (lots of economic pressures, geographically dispersed R&D, no expert consensus on whether it's good to slow down/accelerate), I'd certainly like to see mor
... (read more)
I think you switched the two by accident Otherwise an excellent comment even if I disagree with most of it, have an updoot
Thanks, fixed.

The section on expected value theory seemed unfairly unsympathetic to TUA proponents 

  • The question of what we should do with pascal's mugging-type situations just seems like a really hard under-researched problem where there are not yet any satisfying solutions.
  • EA research institutes like GPI have put a hugely disproportionate amount of research into this question, relative to the field of decision theorists. Proponents of TUA, like Bostrom were the first to highlight these problems in the academic literature. 
  •  Alternatives to expected value have received far less attention in the literature and also have many problems
  • eg The solution you propose of having some probability threshold below which we can ignore more speculative risks also has many issues. For instance, this would seem to invalidate many arguments for the rationality of voting or for political advocacy, such as canvassing for Corbyn or Sanders: the expected value of such activities is high even though the payoff is often very low (eg <1 in 10 million in most US states). Advocating for degrowth also seems extremely unlikely to succeed given the aims of governments across the world and the preferences of ordinary voters. 

So, I think framing it as "here is this gaping hole in this worldview" is a bit unfair. Proponents of TUA pointed out the hole and are the main people trying to resolve the problem, and any alternatives also seem to have dire problems.

You seem to assume that  voting / engaging in political advocacy are all obviously important things to do and that any argument that says don't bother doing them falls prey to a reductio ad absurdum,  but it's not clear to me why you think that. If all of these actions do in fact have incredibly low probability of positive payoff such that one feels they are in a Pascal's Mugging when doing them, then one might rationally decide not to do them. Or perhaps you are imagining a world in which loads of people stop voting such that democracy falls apart. At some point in this world though I'd imagine voting would stop being a Pascal's Mugging action and would be associated with a reasonably high probability of having a positive payoff.
One reason it might be a reductio ad absurdum is that it suggests that in an election in which supporters of one side were rational (and thus would not vote, since each of their votes would have a minuscule chance of mattering) and the others irrational (and would vote, undeterred by the small chance of their vote mattering), the irrational side would prevail. If this is the claim that John G. Halstead is referring to, I regard it as a throwaway remark (it's only one sentence plus a citation):
Which  alternatives to EV have what problems for what uses in what contexts? Why do those problems make them worse than EV, a tool that requires the use of numerical probabilities for poorly-defined events often with no precedent or useful data? What makes all alternatives to EV less preferable to the way in which EV is usually used in existential risk scholarship today, where subjectively-generated probabilities are asserted by "thought leaders" with no methodology and no justification, about events that are not rigorously defined nor separable, which are then fed into idealized economic models, policy documents, and press packs?
Will Bradshaw
Why is writing a sequence of snarky rhetorical questions preferable to just making counter-arguments?

In light of recent events, I came back to take another look at this paper. It’s a shame that so much of the discussion ended up focusing on the community’s reaction rather than the content itself. I think the paranoid response you describe in the post was both unjust and an overreaction. None of the paper’s conclusions seems hugely damaging or unfair to me.

That said, like other commenters, I’m not wholly convinced by your arguments. You’ve asked people to be more specific about this, and I can give two specific examples.

On technological determinism

You write that people in the EA community “disregard controlling technology on the grounds of a perceived lack of tractability” (p. 17). But you think this is probably the wrong approach, since technological determinism is “derided and dismissed by scholars of science and technology studies” and “unduly curtails the available mitigation options” (p. 18).

I’m a bit skeptical of this because I know many AI safety and biosecurity workers who would be stoked to learn that it’s possible to stop the development of powerful technologies. You write that “we have historical evidence for collective action and coordination on technological progress a... (read more)

See Arepo for further discussion and ideas for terminology.
Stephen Clare
Thanks, I hadn't seen those!

Everything written in the post above strongly resonates with my own experiences, in particular the following lines:

the creation of this paper has not signalled epistemic health. It has been the most emotionally draining paper we have ever written.

the burden of proof placed on our claims was unbelievably high in comparison to papers which were considered less “political” or simply closer to orthodox views.

The EA community prides itself on being able to invite and process criticism. However, warm welcome of criticism was certainly not our experience in writing this paper.

I think criticism of EA orthodoxy is routinely dismissed. I would like to share a few more stories of being publicly critical of EA in the hope that doing so adds some useful evidence to the discussion:

  • Consider systemic change. "Some critics of effective altruism allege that its proponents have failed to engage with systemic change" (source). I have always found the responses (eg here and here) to this critique to be dismissive and miss the point. Why can we not just say: yes we are a new community this area feels difficult and we are not there yet? Why do we have to pretend EA is perfect and does systemic change stu
... (read more)

i do think there is a difference between this article and stuff from people like Torres, in terms of good faith

I agree with this, and would add that the appropriate response to arguments made in bad faith is not to "steelman" them (or to add them to a syllabus, or to keep disseminating a cherry-picked quote from a doctoral dissertation), but to expose them for what they are or ignore them altogether. Intellectually dishonesty is the epistemic equivalent of defection in the cooperative enterprise of truth-seeking; to cooperate with defectors is not a sign of virtue, but quite the opposite.

I've seen "in bad faith" used in two ways:

  1. This person's argument is based on a lie.
  2. This person doesn't believe their own argument, but they aren't lying within the argument itself.

While it's obvious that we should point out lies where we see them, I think we should distinguish between (1) and (2). An argument's original promoter not believing it isn't a reason for no one to believe it, and shouldn't stop us from engaging with arguments that aren't obviously false.

(See this comment for more.)

I agree that there is a relevant difference, and I appreciate your pointing it out. However, I also think that knowledge of the origins of a claim or an argument is sometimes relevant for deciding whether one should engage seriously with it, or engage with it at all, even if the person presenting it is not himself/herself acting in bad faith. For example, if I know that the oil or the tobacco industries funded studies seeking to show that global warming is not anthropogenic or that smoking doesn't cause cancer, I think it's reasonable to be skeptical  even if the claims or arguments contained in those studies are presented by a person unaffiliated with those industries. One reason is that the studies may consist of filtered evidence—that is, evidence selected to demonstrate a particular conclusion, rather than to find the truth. Another reason is that by treating arguments skeptically when they originate in a non-truth-seeking process, one disincentivizes that kind of intellectually dishonest and socially harmful behavior.

In the case at hand, I think what's going on is pretty clear. A person who became deeply hostile to longtermism (for reasons that look prima facie mostly unr... (read more)

 One reason is that the studies may consist of filtered evidence—that is, evidence selected to demonstrate a particular conclusion, rather than to find the truth. Another reason is that by treating arguments skeptically when they originate in a non-truth-seeking process, one disincentivizes that kind of intellectually dishonest and socially harmful behavior.

The "incentives" point is reasonable, and it's part of the reason I'd want to deprioritize checking into claims with dishonest origins. 

However, I'll note that establishing a rule like "we won't look at claims seriously if the person making them has a personal vendetta against us" could lead to people trying to argue against examining someone's claims by arguing that they have a personal vendetta, which gets weird and messy. ("This person told me they were sad after org X rejected their job application, so I'm not going to take their argument against org X's work very seriously.")

Of course, there are many levels to what a "personal vendetta" might entail, and there are real trade-offs to whatever policy you establish. But I'm wary of taking the most extreme approach in any direction ("let's just ignore Phil entirely").... (read more)

Agree with this.

Yes I think that is fair.

At the time (before he wrote his public critique) I had not yet realised that Phil Torres was acting in bad faith.

Just to clarify (since I now realize my comment was written in a way that may have suggested otherwise): I wasn't alluding to your attempt to steelman his criticism. I agree that at the time the evidence was much less clear, and that steelmanning probably made sense back then (though I don't recall the details well).

Strong upvote from me - you’ve articulated my main criticisms of EA.

I think it’s particularly surprising that EA still doesn’t pay much attention to mental health and happiness as a cause area, especially when we discuss pleasure and suffering all the time, Yew Kwang Ng focused so much on happiness, and Michael Plant has collaborated with Peter Singer.

In your view, what would it look like for EA to pay sufficient attention to mental health?

To me, it looks like there's a fair amount of engagement on this:

  • Peter Singer obviously cares about the issue, and he's a major force in EA by himself.
  • Michael Plant's last post got a positive writeup in Future Perfect and serious engagement from a lot of people on the Forum and on Twitter (including Alexander Berger, who probably has more influence over neartermist EA funding than any other person); Alex was somewhat negative on the post, but at least he read it.
  • Forum posts with the "mental health" tag generally seem to be well-received.
  • Will MacAskill invited three very prominent figures to run an EA Forum AMA on psychedelics as a promising mental health intervention.
  • Founders Pledge released a detailed cause area report on mental health, which makes me think that a lot of their members are trying to fund this area.
  • EA Global has featured several talks on mental health.

I can't easily find engagement with mental health from Open Phil or GiveWell, but this doesn't seem like an obvious sign of neglect, given the variety of other health interventions they haven't closely engaged with.

I'm limited her... (read more)

I've only just seen this and thought I should chime in. Before I describe my experience, I should note that I will respond to Luke’s specific concerns about subjective wellbeing separately in a reply to his comment.

TL;DR Although GiveWell (and Open Phil) have started to take an interest in subjective wellbeing and mental health in the last 12 months, I have felt considerable disappointment and frustration with their level of engagement over the previous six years.

I raised the "SWB and mental health might really matter" concerns in meetings with GiveWell staff about once a year since 2015. Before 2021, my experience was that they more or less dismissed my concerns, even though they didn't seem familiar with the relevant literature. When I asked what their specific doubts were, these were vague and seemed to change each time ("we're not sure you can measure feelings", "we're worried about experimenter demand effect", etc.). I'd typically point out their concerns had already been addressed in the literature, but that still didn't seem to make them more interested. (I don't recall anyone ever mentioning 'item response theory', which Luke raises as his objection.) In the end, I got the ... (read more)

Really sad to hear about this, thanks for sharing. And thank you for keeping at it despite the frustrations. I think you and the team at HLI are doing good and important work.

To me (as someone who has funded the Happier Lives institute) I just think it should not have taken founding an institute and 6 years and of repeating this message (and feeling largely ignored and dismissed by existing EA orgs) to reach the point we are at now.

I think expecting orgs and donors to change direction is certainly a very high bar. But I don’t think we should pride ourselves on being a community that pivots and changes direction when new data (e.g. on subjective wellbeing) is made available to us.

FWIW, one of my first projects at Open Phil, starting in 2015, was to investigate subjective well-being interventions as a potential focus area. We never published a page on it, but we did publish some conversation notes. We didn't pursue it further because my initial findings were that there were major problems with the empirical literature, including weakly validated measures, unconvincing intervention studies, one entire literature using the wrong statistical test for decades, etc. I concluded that there might be cost-effective interventions in this space, perhaps especially after better measure validation studies and intervention studies are conducted, but my initial investigation suggested it would take a lot of work for us to get there, so I moved on to other topics.

At least for me, I don't think this is a case of an EA funder repeatedly ignoring work by e.g. Michael Plant — I think it's a case of me following the debate over the years and disagreeing on the substance after having familiarized myself with the literature.

That said, I still think some happiness interventions might be cost-effective upon further investigation, and I think our Global Health & Well-Being team... (read more)

Hello Luke, thanks for this, which was illuminating. I'll make an initial clarifying comment and then go on to the substantive issues of disagreement.

At least for me, I don't think this is a case of an EA funder repeatedly ignoring work by e.g. Michael Plant — I think it's a case of me following the debate over the years and disagreeing on the substance after having familiarized myself with the literature.

I'm not sure what you mean here. Are you saying GiveWell didn't repeatedly ignore the work? That Open Phil didn't? Something else? As I set out in another comment, my experience with GiveWell staff was of being ignored by people who weren't at that familiar with the relevant literature - FWIW, I don't recall the concerns you raise in your notes being raised with me. I've not had interactions with Open Phil staff prior to 2021 - for those reading, Luke and I have never spoken - so I'm not able to comment regarding that.

Onto the substantive issues. Would you be prepared to more precisely state what your concerns are, and what sort of evidence would chance your mind? Reading your comments and your notes, I'm not sure exactly what your objections are and, in so far as I do, they ... (read more)

Hi Michael,

I don't have much time to engage on this, but here are some quick replies:

  • I don't know anything about your interactions with GiveWell. My comment about ignoring vs. not-ignoring arguments about happiness interventions was about me / Open Phil, since I looked into the literature in 2015 and have read various things by you since then. I wouldn't say I ignored those posts and arguments, I just had different views than you about likely cost-effectiveness etc.
  • On "weakly validated measures," I'm talking in part about lack of IRT validation studies for SWB measures used in adults (NIH funded such studies for SWB measures in kids but not adults, IIRC), but also about other things. The published conversation notes only discuss a small fraction of my findings/thoughts on the topic.
  • On "unconvincing intervention studies" I mean interventions from the SWB literature, e.g. gratitude journals and the like. Personally, I'm more optimistic about health and anti-poverty interventions for the purpose of improving happiness.
  • On "wrong statistical test," I'm referring to the section called "Older studies used inappropriate statistical methods" in the linked conversation notes with Joel H
... (read more)
Hello Luke, Thanks for this too. I appreciate you've since moved on to other things, so this isn't really your topic to engage on anymore. However, I'll make two comments. First, you said you read various things in the area, including by me, since 2015. It would have been really helpful (to me) if, given you had different views, you had engaged at the time and set out where you disagreed and what sort of evidence would have changed your mind. Second, and similarly, I would really appreciate it if the current team at Open Philanthropy could more precisely set out their perspective on all this.  I did have a few interactions with various Open Phil staff in 2021,  but I wouldn't say I've got anything like canonical answers on what their reservations are about 1. measuring outcomes in terms of SWB  - Alex Berger's recent technical update didn't comment on this - and 2.  doing more research or grantmaking into the things that, from the SWB perspective, seem overlooked.

This is an interesting conversation. It’s veering off into a separate topic. I wish there was a way to “rebase” these spin-off discussions into a different place. For better organisation.

Thank you Luke – super helpful to hear!!

Do you feel that existing data on subjective wellbeing is so compelling that it's an indictment on EA for GiveWell/OpenPhil not to have funded more work in that area? (Founder's Pledge released their report in early 2019 and was presumably working on it much earlier, so they wouldn't seem to be blameworthy.)

I can't say much more here without knowing the details of how Michael/others' work was received when they presented it to funders. The situation I've outlined seems to be compatible both with "this work wasn't taken seriously enough" and "this work was taken seriously, but seen as a weaker thing to fund than the things that were actually funded" (which is, in turn, compatible with "funders were correct in their assessment" and "funders were incorrect in their assessment"). 

That Michael felt dismissed is moderate evidence for "not taken seriously enough". That his work (and other work like it) got a bunch of engagement on the Forum is weak evidence for "taken seriously" (what the Forum cares about =/= what funders care about, but the correlation isn't 0). I'm left feeling uncertain about this example, but it's certainly reasonable to argue that mental health and/or SWB hasn't gotten enough attention.

(Personally, I find the case for additional work on SWB more compelling than the case for additional work on mental health specifically, and I don't know the extent to which HLI was trying to get funding for one vs. the other.)

Do you feel that existing data on subjective wellbeing is so compelling that it's an indictment on EA for GiveWell/OpenPhil not to have funded more work in that area?

Tl;dr. Hard to judge. Maybe: Yes for GW. No for Open Phil. Mixed for EA community as a whole.


I think I will slightly dodge the question and answer the separate question – are these orgs doing enough exploratory type research. (I think this is a more pertinent question, and although I think subjective wellbeing is worth looking into as an example it is not clear it is at the very top of the list of things to look into more that might change how we think about doing good). 

Firstly to give a massive caveat that I do not know for sure. It is hard to judge and knowing exactly how seriously various orgs have looked into topics is very hard to do from the outside. So take the below with a pinch of salt. That said:

  • OpenPhil – AOK.
    • OpenPhil (neartermists) generally seem good at exploring new areas and experimenting (and as Luke highlights, did look into this).
  • GiveWell – hmmm could do better.
    • GiveWell seem to have a pattern of saying they will do more exploratory research (e.g. into policy) and then not doing it (mention
... (read more)
Strong upvote for this if nothing else. (the rest is also brilliant though, thank you so much for speaking up!)

Thank you for writing and sharing. I think I agree with most of the core claims in the paper, even if I disagree with the framing and some minor details.

One thing must be said: I am sorry you seem to have had a bad experience while writing criticism of the field. I agree that this is worrisome, and makes me more skeptical of the apparent matters of consensus in the community. I do think in this community we can and must do better to vet our own views.

Some highlights:

I am big on the proposal to have more scientific inquiry . Most of the work today on existential risk is married to a particular school of Ethics, and I agree it need not be. 

On the science side I would be enthusiastic about seeing more work on eg models of catastrophic biorisk infection, macroeconomic analysis on ways artificial intelligence  might affect society and expansions of IPCC models that include permafrost methane release feedback loops.

On the humanities side I would want to see for example more work on historical, psychological and anthropological evidence for long term effects and successes and failures in managing global problems like the hole in the ozone layer. I would also like to see surveys ... (read more)

"I have regularly seen proposals in the community to stop and regulate AI development" - Are there any public ones you can signpost to or are these all private proposals?

A few thoughts on the democracy criticism. Don't a lot of the criticisms here apply to the IPCC? "A homogenous group of experts attempting to directly influence powerful decision-makers is not a fair or safe way of traversing the precipice."  IPCC contributors are disproportionately white very well-educated males in the West who are much more environmentalist than the global median voter, i.e. "unrepresentative of humanity at large and variably homogenous in respect to income, class, ideology, age, ethnicity, gender, nationality, religion, and professional background." So, would you propose replacing the IPCC with something like a citizen's assembly of people with no expertise in climate science or climate economics, that is representative wrt some of the demographic features you mention? 

You say that decisions about which risks to take should be made democratically. The implication of this seems to be that everyone, and not just EAs, who is aiming to do good with their resources should donate only to their own government. Their govt could then decide how to spend the money democratically. Is that implication embraced? This would eg include all climate philanthropy, which is now at $5-9bn per year.


You seem to assume that we should be especially suspicious of a view if it is not held by a majority of the global population. Over history, the views of the global majority seem to me to have been an extremely poor guide to accurate moral beliefs. For example, a few hundred years ago, most people had abhorrent views about  animals, women and people of other races. By the arguments here, do you think that people like Benjamin Lay, Bentham and Mill should not have advocated for change in these areas, including advocating for changes in policy? 

As I said in a different but related context earlier this week, "If a small, non-representative group disagrees with the majority of humans, we should wonder why, and given base rates and the outside view, worry about failure modes that have affected similar small groups in the past."


I do think we should worry about failure modes and being wrong. But I think the main reason to do that is that people are often wrong, they are bad at reasoning, and subject to a host of biases. The fact that we are in a minority of the global population is an extremely weak indicator of being wrong. The majority has been gravely wrong on many moral and empirical questions in the past and today. It's not at all clear that the base rate of being wrong for 'minority view' vs 'majority view' is higher or not, and that question is extremely difficult to answer because there are lots of ways of slicing up the minority you are referring to.

I feel like there's just a crazy number of minority views (in the limit a bunch of psychoses held by just one individual), most of which must be wrong. We're more likely to hear about minority views which later turn out to be correct, but it seems very implausible that the base rate of correctness is higher for minority views than majority views.

On the other hand I think there's some distinction to be drawn between "minority view disagrees with strongly held majority view" and "minority view concerns something that majority mostly ignores / doesn't have a view on".


that is a fair point. departures from global majority opinion still seems like a pretty weak 'fire alarm' for being wrong. Taking a position that is eg contrary to most experts on a topic would be a much greater warning sign. 

I see how this could be misread. I'll reformulate the statement; 
"If our small, non-representative group comes to a conclusion, we should wonder, given base rates about correctness in general and the outside view, about which failure modes have affected similar small groups in the past, and consider if they apply, and how we might be wrong or misguided."

So yes, errors are common to all groups, and being a minority isn't a indicator of truth, which I mistakenly implied. But the way in which groups are wrong is influenced by group-level reasoning fallacies and biases, which are a product of both individual fallacies and characteristics of the group. That's why I think that investigating how previous similar groups failed seems like a particularly useful way to identify relevant failure modes.

yes I agree with that. 
I think it's simplistic to reduce the critique to "minority opinion bad". At the very least, you need to reduce it to "minority opinion which happens to reinforce existing power relations and is mainly advocated by billionaires and those funded by it bad". Bentham argued for diminishing his own privilege over others, to give other people MORE choice, irrespective of their power and wealth and with no benefit to him. There is a difference imo

My argument here is about whether we should be more suspicious of a view if it is held by the majority or the minority. Whether that is true seems to me to be mainly dependent on the object-level quality of the belief and not whether it is held by the majority or not - that is a very weak indicator, as the examples of slavery, women, racism, homosexuality etc illustrate. 

I don't think your piece argues that TUA reinforces existing power relations. The main things that proponents of TUA have diverted resources to are: engineered pandemics, AI alignment, nuclear war and to a lesser extent climate change. How does any of this entrench existing power relations?

nitpick, but it is also not true that the view you criticise is mainly advocated for by billionaires. Obviously, a tiny minority of billionaires are longtermists and a tiny minority of longtermists are billionaires. 

Guy Raveh
This is moving money to mostly wealthy, Western organisations and researchers, that would've otherwise gone to the global poor. So the counterfactual impact is of entrenching wealth disparity.

I think it is very unclear whether it is true that diverting money to these organisations would entrench wealth disparity. Examining the demographics of the organisations funded is  a faulty way to assess the overall effect on global wealth inequality - the main effect these organisations will have is via the actions they take rather than the take home pay of their staff.  

Consider pandemic risk. Open Phil has been the main funder in this space for several years and if they had their way, the world would have been much better prepared for covid. Covid has been a complete disaster for low and middle-income countries, and has driven millions into extreme poverty. I don't think the net effect of pandemic preparedness funding is bad for the global poor. Similarly, with AI safety, if you actually believe that transformative AI will arrive in 20 years, then ensuring the development of transformative AI goes well is extremely consequential for people in low and middle-income countries. 

I did not mean the demographic composition of organisations to be the main contributor to their impact. Rather, what I'm saying is that that is the only impact we can be completely sure of. Any further impact depends on your beliefs regarding the value of the kind of work done.

I personally will probably go to the EA Long Term Future Fund for funding in the not so distant future. My preferred career is in beneficial AI. So obviously I believe the work in the area has value that makes it worth putting money into.

But looking at it as an outsider, it's obvious that I (Guy) have an incentive to evaluate that work as important, seeing as I may personally profit from that view. Rather, if you think AI risk - or even existential risk as a whole - is some orders of magnitude less important than it's laid out to be in EA - then the only straightforward impact of supporting X-risk research is in who gets the money and who does not. If you think any AI research is actually harmful, then the expected value of funding this is even worse.

Do you mean that most people advocating for techno-positive longtermist concern for x-risk are billionaires, or that most billionaires so advocate? I don't think either claim is true (or even close to true).

It's also not the claim being made:

...minority opinion which happens to reinforce existing power relations and is mainly advocated by billionaires and those funded by [them]...

You're right, my mistake.
James Özden
I had the same reaction as this, in that the dominant worldview today views extreme levels of animal suffering as acceptable but most of us would agree it's not, and believe we should do our utmost to change it. I think the difference between the examples you've mentioned and the parallel to existential risk is with the qualifier Luke and Carla provided in the text (emphasis mine): Where the key difference is that the study of existential risk is tied to the fate of humanity in ways that animal welfare, misogyny and racism aren't (arguably the latter two examples might influence the direction of humanity significantly but probably not whether humanity ceases to exist).  I'm not necessarily convinced that existential risk studies is so different to the examples you've mentioned that we need to approach it in a much more democratic way but I do think the qualifiers given by the authors mean the analogies you've drawn aren't that water-tight.
David Mathers
Most whites had abhorent views on race at certain points in the past (probably not before 1500 though, unless Medieval antisemitism counts) but that is weak evidence that most people did, since whites were always a minority. I'm not sure many of us know what if any racial views people held in Nigeria, Iran, China or India in 1780.

I seem to remember learning about rampant racism in China helping to cause the Taiping rebellion? And there are enormous amounts of racism and sectarianism today outside Western countries - look at the Rohingya genocide, the Rwanda genocide, the Nigerian civil war, the current Ethiopian civil war, and the Lebanese political crisis for a few examples.

Every one of these examples should be taken with skepticism as this is far outside my area of expertise. But while I agree with the sentiment that we often conflate the history of the world with the history of white people, I'm not sure it's true in this specific case.

Yeah, you're probably right. It's just I got a strong "history=Western history" vibe from the comment I was responding to, but maybe that was unfair!


i'd be pretty surprised if almost everyone didn't have strongly racist views in 1780. Anti-black views are very prevalent in India and China today, as I understand it. eg Gandhi had pretty racist attitudes.

I think there is a "not" missing: "view if it is held by a majority of the global population."
sorry, yeah corrected

minor point but I don't think you've described citizen's assemblies in the most charitable way. Yes, it is a representative sortition of the public so they don't necessarily have expertise in any particular field but there is generally a lot of focus on experts from various fields who inform the assembly. So in reality, a citizen's assembly on climate would be a random selection of representative citizens who would be informed/educated by IPCC (or similar) scientists, who would then deliberate amongst themselves to reach their conclusions.These conclusions one would hope would be similar to what the scientists would recommend themselves as it based on information largely provided by them.

For people that might be interested, here is the report of the Climate Assembly (a citizen's assembly on climate commissioned by the UK government) that in my opinion, had some fairly reasonable policy suggestions. You can also watch a documentary about it by the BBC here.

The paper never spoke about getting rid of experts or replacing experts with citzens. So no. 

Many countries now run citizen assemblies on climate change, which I'm sure you're aware of. They do not aim to replace the role of IPCC. 

EA or the field of existential risk cannot be equated with the IPCC. 

To your second point, no this does not follow at all. Democracy as a procedure is not to be equated (and thus limited) to governments that grant you a vote every so often. You will find references to the relevant literature on democratic experimentation in the last section which focusses on democracy in the paper. 


It would help for clarity if I understood your stance on central bank independence. This seems to produce better outcomes but also seems undemocratic. Do you think this would be legitimate?

It still seems like, if I were Gates, donating my money to the US govt would be more democratic than eg spending it on climate advocacy? Is the vision for Open phil that they set up a citizen's assembly that is representative of the global population and have that decide how to spend the money, by majority vote?

As in the discussion above, I think you're being disingenuous by claiming government is "more democratic." 

And if you were Gates, I'd argue that it would be even more democratic to allow the IPCC, which is more globally representative and less dominated by special interests that the US government, to guide where you spend your money than it would to allow the US government to do so. And given how much the Gates foundation engages with international orgs and allows them to guide his giving, I think that "hand it to the US government" would plausibly be a less democratic alternative than the current approach, which seems to be to allow GAVI, the WHO, and the IPCC to suggest where the money can best be spent.

And having Open Phil convene a consensus driven international body on longtermism actually seems somewhat similar to what the CTLR futureproof report co-written by Toby Ord suggests when it says the UK should lead by, "creating and then leading a global extreme risks network," and push for "a Treaty on the Risks to the Future of Humanity." Perhaps you don't think that's a good idea, but I'm unclear why you would treat it as a reductio, except in the most straw-man form.


Hi David, I wasn't being disingenuous. Here, you say "I think you're being disingenuous by claiming government is "more democratic."  In your comment above you say "One way to make things more democratic is to have government handle it, but it's clearly not the only way." Doesn't this grant that having the government decide is more democratic? These statements seem inconsistent. 

So, to clarify before we discuss the idea, is your view that all global climate philanthropy should be donated to the IPCC?

I think there is a difference between having a citizen's assembly decide what to do with all global philanthropic money (which as I understand it, is the implication of the article), and having a citizen's assembly whose express goal is protecting the long-term (which is not the implication of the article). If all longtermist funding was allocated on the first mechanism, then I think it highly likely that funding for AI safety, engineered pandemics and nuclear war would fall dramatically. 

The treaty in the CTLR report seems like a good idea but seems quite different to the idea of democratic control proposed in the article.

Comparing how democratic government is to different things yield different results, because democratic isn't binary. Yes, unitary action by a single actor is less democratic than having government handle things, and no, having the US government handle things is not  clearly more democratic than deferring to the IPCC. But, as I'm sure you know, the IPCC isn't a funding body, nor does it itself fight climate change. So no, obviously climate philanthropy  shouldn't all go to them.   No, and clearly you need to go re-read the paper. You also might want to look into the citations Zoe suggested that you read above, about what "democratic" means, since you keep interpreting in the same simplistic and usually incorrect way, as equivalent to having everyone vote about what to do. This goes back to the weird misunderstanding that democratic is binary, and that it always refers to control. First, global engagement and the treaty are two different things they advise the UK government. Second, I'm sure that the authors can say for themselves whether they see international deliberations and treaties as a way to more democratic input, but I'd assume that they would say that it's absolutely a step in the direction they are pointing towards.

Hi David. We were initially discussing whether giving the money to govts would be more democratic. You suggested this was a patently mad idea but then seemed to agree with it. 

Here is how the authors define democracy: "We understand democracy here in accordance with Landemore as the rule of the cognitively diverse many who are entitled to equal decision-making power and partake in a democratic procedure that includes both a deliberative element and one of preference aggregation (such as majority voting)"

You say: "You also might want to look into the citations Zoe suggested that you read above, about what "democratic" means, since you keep interpreting in the same simplistic and usually incorrect way, as equivalent to having everyone vote about what to do."

equal political power and preference aggregation entails majority rule or lottery voting or sortition. Your own view that equal votes aren't a necessary condition of democracy seems to be in tension with the authors of the article. 

A lot of the results showing the wisdom of democratic procedures depend on certain assumptions especially about voters not being systematically biased. In the real world, this isn't true so so... (read more)

You're using a word differently than they explicitly say they are using the same word. I agree that it's confusing,  but will again note that consensus decision making is democratic in thes sense they use, and yet is none of the options you mention. (And again, the IPCC is a great example of a democratic deliberative body which seems to fulfill the criteria you've laid out, and it's the one they cite explicitly.)

On the validity and usefulness of democracy as a method of state governance, you've made a very reasonable case that it would be ineffective for charity, but in the more general sense that Landemore uses it, which includes how institutions other than governments can account for democratic preferences, I'm not sure that the same argument applies.

That said, I strongly disagree with Cremer and Kemp about the usefulness of this approach on very different grounds. I think that both consensus and other democratic methods, if used for funding, rather than for governance, would make hits based giving and policy entrepreneurship impossible, not to mention being fundamentally incompatible with finding neglected causes.

I really appreciate your effort defending a paper containing parts you strongly disagree with from (what you consider) bad arguments!
James Özden
I think your Open Phil example could be an interesting experiment. Do you think that if Open Phil commissions a citizen's assembly to allocate their existential risk spending  and the input  is given by their researchers / program officers, it would be wildly different to what they would do themselves? In any scenario, I think it would be quite interesting as surely if our worldviews and reasoning are strong enough to claim big unusual things (e.g. strong longtermism) we should be able to convince a random group of people that they hold? and if not, is that a problem with the people selected, our communication skills or the thinking itself? I personally don't think it would be a problem with the people (see past successes of citizen's assemblies)* so shouldn't we be testing our theories to see if they make sense under different worldviews and demographic backgrounds? and if they don't seem robust to other people, we should probably try integrate the reasons why (within reason of course). *there's probably some arguments to be made here that we don't necessarily expect the allocation from this representative group, even when informed  perfectly by experts, to be the optimal allocation of resources so we're not maximising utility / doing the most good. This is probably true but I guess the balance of this with moral uncertainty is the trade-off we have to live with? Quite unsure on this though, seems fuzzy

Hi James, I do think it would be interesting to see what a true global citizen's assembly with complete free rein would decide. I would prefer that the experiment were not done with Open Phil's money as the opportunity cost would be very high. A citizen's assembly with longtermist aims would also be interesting, but would be different to what is proposed in the article. Pre-setting the aims of such an assembly seems undemocratic.

I would be pretty pessimistic about convincing lots of people of something like longtermism in a citizen's assembly - at least I think funding for things like AI, engineered viruses and nuclear war would fall a fair amount. The median global citizen is someone who is strongly religious, probably has strong nationalist and socialist beliefs (per the literature on voter preferences in rich countries, which is probably true in poorer countries), unwilling to pay high carbon taxes, homophobic etc. 

For what it's worth, I wasn't genuinely saying we should hold a citizen's assembly to decide what we do with all of Open Phil's money, I just thought it was an interesting thought experiment. I'm not sure I agree that the pre-setting of the aims of an assembly is undemocratic, however, as surely all citizen's assemblies need an initial question to start from? That seems to have been the case for previous assemblies (climate, abortion, etc.).

To play devil's advocate, I'm not sure your points about the average global citizen being homophobic, religious, socialist, etc., actually matter that much when it comes to people deciding where they should allocate funding for existential risk. I can't see any relationship between beliefs in which existential risks are the most severe and queer people, religion or their willingness to pay carbon taxes (assuming the pot of funding they allocate is fixed and doesn't affect their taxes). 

Also, I don't think you've given much convincing evidence that a citizen's assemblies would lead to funding for key issues falling a fair amount vs decisions by OP program officers, besides your intuition. I can't say I have much evidence myself except for th... (read more)

I vaguely remember reading something about religious people worrying less about extinction, but I don't remember whether that was just intuition or an actual study. They may also be predisposed to care less about certain kinds of risk, e.g. not worrying about AI as they perceive it to be impossible. (these are pretty minor points though)

I think you're unaware of the diversity and approach of the IPCC. It is incredibly interdisciplinary, consensus driven, and represents stakeholders around the world faithfully. You should look into what they do and their process more carefully before citing them as an example.

Then, you conflated "democratically" with "via governments, through those government's processes" which is either a bizarre misunderstanding, or a strange rhetorical game you're playing with terminology.


As mentioned, the vast majority of the authors are from a similar demographic background to EAs. The IPCC also produces lots of policy-relevant material on eg the social costs of climate change and the best paths to mitigation, which are mainly determined by white males.

Here is a description of climate philanthropy as practiced today in the United States. Lots of  unelected rich people who disproportionately care about climate change spend hundreds of millions of pounds advocating for actions and policies that they prefer. It would be a democratic improvement to have that decision made by the US government, because at least politicians are subject to competitive elections. So, having the decision made by the US government would be more democratic. Which part of this do you disagree with?

It seems a bit weird to class this as a 'bizarre misunderstanding' since many of the people who make the democracy criticism of philanthropy, such as Rob Reich, do in fact argue that the money should be spent by the government.   

"the vast majority of the authors are from a similar demographic background to EAs... mainly determined by white males" A key difference is having both representation of those with other perspectives and interests, and a process which is consensus driven and inclusive. "It would be a democratic improvement to have that decision made by the US government, because at least politicians are subject to competitive elections. So, having the decision made by the US government would be more democratic. Which part of this do you disagree with?" One way to make things more democratic is to have government handle it, but it's clearly not the only way. Another way to be more democratic would, again, by being more broadly representative and consensus driven. (And the switch from "IPCC" to "climate philanthropy as practiced today in the United States" was definitely a good rhetorical trick, but it wasn't germane to either the paper's discussion of the IPCC, or your original point, so I'm not going to engage in discussing it.)
in the second bit, I wasn't talking about the IPCC, I was talking about your second point "you conflated "democratically" with "via governments, through those government's processes"". The reason I mentioned climate philanthropy was because that is what I mentioned in my original comment you responded to: if you think philanthropy is undemocratic, then that also applies to climate philanthropy, which Luke Kemp is strongly in favour of, so this is an interesting test case for their argument. 

First, are you backing away from your initial claims about the IPCC, since it in fact is consensus based with stakeholders rather than being either a direct democracy, or a unilateralist  decision.

Second, I'm not interested in debating what you say Luke Kemp thinks about climate philanthropy, nor do I know anything about his opinions, nor it is germane to this discussion.
But in your claims that you say are about his views, you keep claiming and assuming that the only democratic alternatives to whatever we're discussing are a direct democracy or control by a citizens' assembly (without expertise) or handing things to governments. Regardless of Luke's views elsewhere, that's certainly not what they meant in this paper. Perhaps this quote will be helpful;

We understand democracy here in accordance with Landemore as the rule of the cognitively diverse many who are entitled to equal decision-making power and partake in a democratic procedure that includes both a deliberative element and one of preference aggregation (such as majority voting)

As Landemore, who the paper cites several times,  explains, institutions work better when the technocratic advice is within the context of a inclusive decision procedure, rather than either having technocrats in charge, or having a direct democracy.


Hello, Yes i think it would be fair to back away a bit from the claims about the IPCC. it remains true that most climate scientists and economists are white men and they have a disproportionate influence on the content of the IPCC reports. nonetheless, the case was not as clear cut as I initially suggested. 

I find the second point a bit strange. Isn't it highly relevant to understand whether the views of the author of the piece we are discussing are consistent or not? 

It's also useful to know what the implication of the ideas are expressed actually are. They explicitly give a citizen's assembly as an example of a democratic procedure. Even if it is some other deliberative mechanism followed by a majority vote, I would still like to know what they think about stopping all climate philanthropy and handing decisions over all money over to such a body. It's pretty hard to square a central role for expertise with a normative idea of political equality. 


nor it is germane to this discussion

I do think it is germane to the discussion, because it helps to clarify what the authors are claiming and whether they are applying their claims consistently. 

I was discussing this paper, which doesn't discuss climate philanthropy, not everything they have ever stated. I don't know what else they've claimed, and I'm not interested in a discussion of it. 

You say that decisions about which risks to take should be made democratically. The implication of this seems to be that everyone, and not just EAs, who is aiming to do good with their resources should donate only to their own government. Their govt could then decide how to spend the money democratically.

I'm not fully sure that deciding which risks to take seriously in a democratic fashion logically leads to donating all of your money to the government. Some reasons I think this:

  • That implies that we all think our governments are well-functioning democracies but I (amongst many others) don't believe that to be true. I think it's fairly common sentiment and knowledge that political myopia by politicians, vested interests and other influences mean that governments don't implement policies that are best for their populations.
  • As I mentioned in another comment, I think the authors are saying that as existential risks affect the entirety of humanity in a unique way, this is one particular area where we should be deciding things more democratically. This isn't necessarily the case for spending on education, healthcare, animal welfare, etc, so there it would make sense you donate to institu
... (read more)
Not really. The IPCC   , and is not a political thinktank (even though climate risk deniers and minizers might like to claim it is), is funded at least by 65% by nation states and the UN (44% USA in 2018, 25% by  the next inheriting their democratic legitimacy  w.r.t to funding any money) and fundamentally deals with something much more narrowly defined, empirically verifiable and graspable than the TUA main causes. It suffers from a lot of the same problems w.r.t representation and democracy as all of science and society does, but it's not nearly as donor-alignment-driven as the targets of the article
The IPCC reports have hundreds of authors from all over the world:  It is misleading to say the IPCC is homogeneous and that the authors are "disproportionately white very well-educated males in the West".  Every country and a large variety of civil institutions are represented at the conference of parties, and they use a consensus process.
Guy Raveh
As always, I don't think this implication is necessarily bad. Individual philanthropy is not sustainable, undemocratic, and indeed might do more harm than good in that, in public perception, it takes the weight off the government's shoulder while in practice putting into it a fraction of the effort. I don't think this is the only side of the story, of course. I donate myself to organisations who I believe do important work but aren't in the consensus. I think governments are good for making some democratic decisions, but are very undemocratic on others, e.g. by only representing the country's population and not the entire world's, or by neglecting to represent animals or future generations. And I think organisations that operate independently of the government are good for putting checks and balances on it and preventing consolidation of power.

Regarding the risk that longtermism could lead people to violate rights, it seems to me like you could make exactly the same argument for any view that prioritises between different things. For instance, as Peter Singer has pointed out, billions of animals are tortured and killed every year. By exactly analogous reasoning, one could say that other problems 'dwindle into irrelevance' as other values are sacrificed at the altar of the astronomical expected value of preventing factory farming. So, this would justify animal rights terrorism and the like and other abhorrent actions

"Don't be fanatical about utilitarian or longtermist concerns and don't take actions that violate common sense morality"  is a message that longtermists have been emphasizing from the very beginnings of this social movement, and quite a lot.

Some examples: 

More generally, there's often at least a full paragraph devoted to this topic when someone writes a longer overview article on longtermism or writes about particularly dicey implications with outsized moral stakes. I also remember this being a presentation or conversation topic at many EA conferences.

I haven't read the corresponding section in the paper that the OP refers to, yet, but I skimmed the literature section and found none of the sources I linked to above. If the paper criticizes longtermism on grounds of this sort of implication and fails to mention that longtermists have been aware of this and are putting in a lot o... (read more)


I also agree with this. There are many reasons for consequentialists to respect common sense morality. 

I was just making the point that the rhetorical argument about rights can pretty much be made about any moral view. eg The authors seem to believe that degrowth would be a good idea, and it is a built in feature of degrowth that it would have enormous humanitarian costs


I agree that there is an analogy to animal suffering here, but there's a difference in degree I think. To longtermists, the importance of future generations is many orders of magnitude higher than the importance of animal suffering is to animal welfare advocates. Therefore, I would claim, longtermists are more likely to ignore other non-longtermist considerations than animal welfare advocates would be.

Depending on the view, legitimate self-defence and "other-defence" don't violate rights at all, and this seems close to common sense when applied to protect humans. Even deontological views could in principle endorse - but I think in practice today should condemn - coercively preventing individuals from harming nonhuman animals, including farmed animals, as argued in this paper, published in the Journal of Controversial Ideas, a journal led and edited by McMahan, Minerva and Singer. Of course, this conflicts with the views of most humans today, who don't extend similarly weighty rights/claims to nonhuman animals. EDIT: I realize now I interpreted "rights" in moral terms (e.g. deontological terms), when you may have intended it to be interpreted legally.

The longtermist could then argue that an analogous argument applies to "other-defence" of future generations. (In case there was any need to clarify: I am not making this argument, but I am also not making the argument that violence should be used to prevent nonhuman animals from being tortured.)

Separately, note that a similar objection also applies to many forms of non-totalist longtermism. On broad person-affecting views, for instance, the future likely contains an enormous number of future moral patients who will suffer greatly unless we do something about it. So these views could also be objected to on the grounds that they might lead people to cause serious harm in an attempt to prevent that suffering.

In general, I think it would be very helpful if critics of totalist longtermism made it clear what rival view in population ethics they themselves endorse (or what distribution of credences over rival views, if they are morally uncertain). The impression one gets from reading many of these critics is that they assume the problems they raise are unique to totalist longtermism, and that alternative views don't have different but comparably serious problems. But this assumption can't be taken for granted, given the known impossibility theorems and other results in population ethics. An argument is needed.

I realize now I interpreted "rights" in moral terms (e.g. deontological terms), when Halstead may have intended it to be interpreted legally. On some rights-based (or contractualist) views, some acts that violate humans' legal rights to protect nonhuman animals or future people could be morally permissible. I agree. I think rights-based (and contractualist) views are usually person-affecting, so while they could in principle endorse coercive action to prevent the violation of rights of future people, preventing someone's birth would not violate that then non-existent person's rights, and this is an important distinction to make. Involuntary extinction would plausibly violate many people's rights, but rights-based (and contractualist) views tend to be anti-aggregative (or at least limit aggregation), so while preventing extinction could be good on such views, it's not clear it would deserve the kind of priority it gets in EA. See this paper, for example, which I got from one of Torres' articles and takes a contractualist approach. I think a rights-based approach could treat it similarly. It could also be the case that procreation violates the rights of future people pretty generally in practice, and then causing involuntary extinction might not violate rights at all in principle, but I don't get the impression that this view is common among deontologists and contractualists or people who adopt some deontological or contractualist elements in their views. I don't know how they would normally respond to this. Considering "innocent threats" complicates things further, too, and it looks like there's disagreement over the permissibility of harming innocent threats to prevent harm caused by them. I agree. However, again, on some non-consequentialist views, some coercive acts could be prohibited in some contexts, and when they are not, they would not necessarily violate rights at all. The original objection raised by Halstead concerns rights violations, not merely causi
Also, I think we should be clear about what kinds of serious harms would in principle be justified on a rights-based (or contractualist) view. Harming people who are innocent or not threats seems likely to violate rights and be impermissible on rights-based (and contractualist) views. This seems likely to apply to massive global surveillance and bombing civilian-populated regions, unless you can argue on such views that each person being surveilled or bombed is sufficiently a threat and harming innocent threats is permissible, or that collateral damage to innocent non-threats is permissible. I would guess statistical arguments about the probability of a random person being a threat are based on interpretations of these views that the people holding them would reject, or that the probability for each person being a threat would be too low to justify the harm to that person. So, what kinds of objectionable harms could be justified on such views? I don't think most people would qualify as serious enough threats to justify harm to them to protect others, especially people in the far future.
This seems like a fruitful area of research—I would like to see more exploration of this topic. I don't think I have anything interesting to say off the top of my head.

@CarlaZoeC or Luke Kemp, could you create another forum post solely focused on your article? This might lead to more focused discussions, separating debate on community norms vs discussing arguments within your piece.

I also wanted to express that I'm sorry this experience has been so stressful. It's crucial to facilitate internal critique of EA, especially as the movement is becoming more powerful, and I feel pieces like yours are very useful to launch constructive discussions.

Points where I agree with the paper:

  • Utilitarianism is not any sort of objective truth, in many cases it is not even a good idea in practice (but in other cases it is).
  • The long-term future, while important, should not completely dominate decision making.
  • Slowing down progress is a valid approach to mitigating X-risk, at least in theory.

Points where I disagree with the paper:

  • The papers argues that "for others who value virtue, freedom, or equality, it is unclear why a long-term future without industrialisation is abhorrent". I think it is completely clear, given that in pre-industrial times most people lived in societies that were rather unfree and unequal (harder to say about "virtue" since different people would argue for very different conceptions of what virtue is). Moreover, although intellectuals argued for all sorts of positions (words are cheap, after all), few people are trying to return to pre-industrial life in practice. Finally, techno-utopian visions of the future are usually very pro-freedom and are entirely consistent with groups of people voluntarily choosing to live in primitivist communes or whatever.
  • If ideas are promoted by an "elitist" minority, that doesn't automa
... (read more)

Hey Zoe and Luke, thank you for posting this and for writing the paper! I just finished reading it and found it thoughtful, detailed, and it gave me a lot to think about. It is the best piece of criticism I have read, and will recommend it to others looking for that going forward. I can see the care, time, and revisions that went into the piece. I am very sorry to hear about your experience of writing it. I think you contributed something important, and wish you had been met with more support. I hope the community can read this post and learn from it so we can get a little closer to that ideal of how to handle, incorporate, and respond to criticism. 

Note: I discuss Open Phil to some degree in this comment. I also start work there on January 3rd. These are my personal views, and do not represent my employer.

Epistemic status: Written late at night, in a rush, I'll probably regret some of this in the morning but (a) if I don't publish now, it won't happen, and (b) I did promise extra spice after I retired.

I think you contributed something important, and wish you had been met with more support. 

It seems valuable to separate "support for the action of writing the paper" from "support for the arguments in the paper". My read is that the authors had a lot of the former, but less of the latter.

From the original post:

We were told by some that our critique is invalid because the community is already very cognitively diverse and in fact welcomes criticism. They also told us that there is no TUA, and if the approach does exist then it certainly isn’t dominant. 

While "invalid" seems like too strong a word for a critic to use (and I'd be disappointed in any critic who did use it), this sounds like people were asked to review/critique the paper and then offered reviews and critiques of the paper. 

Still, to the degree that ther... (read more)

This is a great comment, thank you for writing it. I agree - I too have not seen sufficient evidence that could warrant the reaction of these senior scholars. We tried to get evidence from them and tried to understand why they explicitly feared that OpenPhil would not fund them because of some critical papers. Any arguments they shared with us were unconvincing. My own experience with people at OpenPhil (sorry to focus the conversation only on OpenPhil, obviously the broader conversation about funding should not only focus on them) in fact suggests the opposite. 

I want to make sure that the discussion does not unduly focus on FHI or CSER in particular. I think this has little to do with the organisations as a whole and more to do with individuals who sit somewhere in the EA hierarchy. We made the choice to protect the privacy of the people whose comments we speak of here. This is out of respect but also because I think the more interesting area of focus (and that which can be changed) is what role EA as a community plays in something like this happening. 

I woud caution to center the discussion only around the question of whether or not OpenPhil would reduce funding in res... (read more)

Thanks for writing this reply, I think this is an important clarification.

Charles He
It's hard to see what is going on and this is producing a lot of heat and speculation. I want to present an account or framing below. I want to see if this matches with your beliefs and experiences.  I like to point out the below this isn’t favorable to you, basically, but I don't have any further deliberate goal and little knowledge in this space.   Firstly, to try to reduce the heat, I will change the situation to the cause area of global health: For background, note that Daron Acemoglu, who is really formidable, has criticized EA global health and development.     Basically, Acemoglu believes the randomista approaches used in EA could be net negative for human welfare because it supplants health institutions, reduces the functionality of the state and slows economic development. It’s also hard to measure. I don’t agree and most EA don’t agree.    The account begins: imagine with dramatically increased funding, Givewell expands and hires a bunch of researchers. GiveWell is more confident and hires less orthodox researchers that seem passionate and talented. One year later, the very first paper of one of the newly hired researchers makes a strong negative view of the randomista approach and directly criticizes GiveWell’s work.  The paper says EA global health and development is misguided and gives plausible reasons, but these closely follow Acemoglu and randomista critics. The paper makes statements that many aligned EAs find disagreeable, such as saying AMF’s work is unmeasurable. There are also direct criticisms of senior EAs that seem uncharitable.  However, there isn’t a lot of original research or claims in the paper. Also, while not stated, the paper implies the need to restructure and change the fundamental work of GiveWell, including deleting several major programs. Accompanying the paper, the new researcher also states they had very negative experiences when pushing out the paper, including getting heavily pressured to self-censor. They state p
What on Earth is this thinly-veiled attempt at character assassination? Do you actually have any substantive evidence that your “account” is accurate (your disclaimer at the start suggests not), or are you just fishing for a reaction? Honestly, what did you hope to gain from this? You think this researcher is just gonna respond and say “Yep, you’re right, I’m ill-fit for my job and incapable of good academic work!” “Not favorable to you” is the understatement of the century. Not to mention your change of the area concerned in no way lowers the temperature. It just functions as CYA to avoid making forthright accusations that this researcher’s actual boss might then be called upon to publicly refute. This is one of the slimiest posts I’ve ever seen, to be perfectly honest. Edit: I would love to see anyone who has downvoted this post explain why they think the above is defensible and how they’d react if someone did it to them.
Charles He

I enjoyed some of the discussion of emergency powers. It could be good to mention the response to covid. Leaving to one side whether such policies were justified (they do seem to have saved many lives), country-wide lockdowns were surely one of the most illiberal policies enacted in history, and explicitly motivated by trying to address a global disaster. Outside of genocide and slavery, I struggle to think of many greater restrictions on individuals freedom than confining essentially the entire population to semi house arrest. In many cases these rules were brought in under special emergency powers, and sometimes later determined to be illegal after judicial review. However, these policies were often extremely popular with the general population, so I'm not sure they fit the democracy-vs-illiberalism dichotomy the article is sort of going for.

I think it is disappointing that so many comments are focusing on arguing with the paper rather than discussing the challenges outlined in the post. From a very quick reading I don't find any of the comments here unreasonable but I do find them to be talking about a different topic. It would be better if we could separate out the discussion of "red teaming" EA from the discussion of this particular paper

The paper is very well written, crisp and communicates its points very well.

The paper includes characterizations of longtermists that seem schematic and many would find unfair. 

In the post itself, there are serious statements that add a lot of heat to the issue and are hard to approach.

I think that this is a difficult time where many people are getting/staying out away, or performing emotional labor, for what are genuinely difficult experiences of the OP. 

This isn't ideal for truthseeking. 

If I was in a different cause area with a similar issue, I wouldn't want a lot of longtermists coming in and pulling on these  threads, I don't think that is the ideal or right thing to do.

Interesting, I was thinking the opposite! I was thinking, "There's so many interesting specific suggestions in this paper and people are just caught up on whether or not they like diversity initiatives generally and what they think of the tone on this paper, how annoying."

I just mean this could have been two posts - one about the paper and one about the experience of publishing the paper. Both would be very valuable.

I agree it would have been better to have this as two posts – I'm personally finding it difficult to respond to either the paper or the post, because when I focus on one I feel like I'm ignoring important points in the other.

That said, the fact that both are being discussed in a single post is down to the authors, not the commenters. I think it's reasonable for any given commenter to focus on one without justifying why they're neglecting the other.


Yeah I agree. I disagree with most of the paper, but I find the claims about pressures not to publish criticism  troubling. 

Completely agree!

Quick thoughts from my phone:

Thanks for writing this post, Carla and Luke. I am sorry to hear about your experiences, that sounds very challenging.

I also understand why people would object to your work, as many may have had high confidence in it having negative expected value.

It was surely a very difficult situation for all parties.

I am glad you are voicing concerns, I like posts like this.

At the same time, what occurred mostly sounded reasonable to me, even if it was unpleasant. Strong opinions were expressed, concerns were made salient, people may have been defensive or acted with some self-interest, but no one was forced to do anything. Now the paper and your comments are out, and we can read and react to them. I have heard much worse in other academic and professional settings.

I think that it's unavoidable that there will be a lot of strong disagreement in the EA community. It seems unavoidable in any group of diverse individuals who are passionately working together towards important goals. Of course, we should try to handle conflict well, but we shouldn't expect that it can ever be avoided or be completely pleasant.

I also understand why people don't express criticism publicl... (read more)

At the same time, what occurred mostly sounded reasonable to me, even if it was unpleasant. Strong opinions were expressed, concerns were made salient, people may have been defensive or acted with some self-interest, but no one was forced to do anything. Now the paper and your comments are out, and we can read and react to them. I have heard much worse in other academic and professional settings.


I don't think "the work got published, so the censorship couldn't have been that bad" really makes sense as a reaction to claims of censorship. You won't see work that doesn't get published, so this is basically a catch-22 (either it gets published, in which cases there isn't censorship, or it doesn't get published, in which case no one ever hears about it).

Also, most censorship is soft rather than hard, and comes via chilling effects.

(I'm not intending this response to make any further object-level claims about the current situation, just that the quoted argument is not a good argument.)

I agree with most of what you say other than it being reasonable for some people to have acted in self-interest. 

While I do think it is unavoidable that there will be attempts to shut down certain ideas and arguments out of the self-interest of some EAs, I think it's important that we have a very low tolerance of this.

Thanks for commenting :) I intended to present the-self interest part as bad, sorry.   I agree, but I don't see this as 'shutting down' arguments. Can I just check that I am not misreading what happened? My interpretation from this was that they were strongly discouraged from publishing the paper by people who disagreed with what the paper was claiming (who may or may not have also been self-interested in maintaining their funding) and/or predicted negative outcomes from the work.  They were still able to publish the paper and participate in the community. No-one was 'shut down' in the sense that someone forced them not to publish it (though they may have strongly advised against it). Is this correct? Maybe I misunderstand what "prevent this paper from being published" actually entailed.  

Ah okay.

I think I interpreted this as ‘pressure’ to not publish, and my definition of ‘shutting down ideas’ includes pressure / strong advice against publishing them, while yours is restricted to forcing people not to publish them.