I was interested in most of the relevant cause areas in some form from childhood (the global poor, animal welfare, extinction risks), and independently formulated utilitarianism (not uncommon I’m told, both Bertrand Russell and Brian Tomasik apparently did the same), so I was a pretty easy sell.

I was assigned “Famine, Affluence, and Morality” and “All Animals are Equal” for a freshman philosophy course, and decided Peter Singer really got it and did philosophy in the way that seemed most important to me. Later I revisited Singer when working on a long final paper about animal rights and I ran into his TED talk on Effective Altruism.

At first I was sympathetic but not that involved, but gradually realized that it was much more the style of ethics/activism I was interested in promoting than the other things on the table, or at least on top of them. I founded my school’s Effective Altruism club while I still didn’t really know all that much about the movement, and started learning more, especially after a friend (Chris Webster) recommended the 80,000 Hours podcast to me.

Around this same time I read Reasons and Persons, and met my friend and long-time collaborator Nicholas Kross who introduced me to many rationalist ideas and thinkers, and by the end of undergrad, I was basically a pretty doctrinaire, knowledgeable EA. Kind of a long story, but the whole thing was pretty much in fits and starts so I don’t know a great way to compress it.

I'm really heartened by this, especially some of the names on here I independently admired who haven't been super vocal about the issue yet, like David Chalmers, Bill McKibben, and Audrey Tang. I also like certain aspects of this letter better than the FLI one. Since it focuses specifically on relevant public figures, rapid verification is easier and people are less overwhelmed by sheer numbers. Since it focuses on an extremely simple but extremely important statement it's easier to get a broad coalition on board and for discourse about it to stay on topic. I liked the FLI one overall as well, I signed it myself and think it genuinely helped the discourse, but if nothing else this seems like a valuable supplement.

For what it’s worth I haven’t gotten around to reading a ton of your posts yet, but me and pretty much everyone I showed your blog to could tell pretty quickly that it was a cut above whatever I might picture just from the title. That said, I think all the changes are good ideas on the whole. Keep up the good work!

Fully endorsed. And I would add that if you don’t mind more speculative, harder to evaluate interventions there are organizations working on risks of future astronomical suffering like the Center for Long-Term Risk, and organizations working on wild animal suffering like Wild Animal Initiative. For more measurable impacts I don’t have much to add to weeatquince’s excellent suggestions.

So this depends if you take EA to be more fundamentally interested in theories of beneficence (roughly what ought you do to positively help others) or in theories of axiology (roughly what makes a world better or worse). I’m suspicious of most theories that pull these apart, but importantly Scanlon’s work is really interested in trying to separate the two, and basically ditch the direct relevance of axiology altogether. Certainly he goes beyond telling people what they ought not to do. If EA is fundamentally about beneficence, Scanlon is very relevant, if it’s more about axiology, he’s more or less silent.

So long as we’re sharing recommendations, Parfit also has a good paper that’s relevant to this, which a good deal of the more recent partial aggregation debate is leap-frogging off of.

Does anyone know how healthy a diet mostly consisting in simple bivalves is for a cat? I’m not a bivalvegan myself out of risk aversion, but suspect that eating bivalves is probably fine, and it could be much cheaper and easier to scale at this point.

Taken as intended and not as a question for me (I am personally quite concerned abouts-risks, but think working on them involves similar insights to working on AI x-risks), I think the most common reason is people seeing these scenarios as quite unlikely. Roughly: astronomically bad outcomes are, like astronomically good outcomes, only a very tiny slice of possible outcomes of randomly selected optimization functions, but unlike astronomically good outcomes don't have existing powerful intelligences trying to aim for them. I think there are reasonable counters to this, but that's my impression of the most common thought here.

This proposal seems to have become extremely polarizing, more so and for different reasons than I would have expected after first reading this. I am more on the “this is pretty fine” side of the spectrum, and think some of the reasons it has been controversial are sort of superficial. Given this though, I want to steelman the other side (I know Yudkowsky doesn’t like steelmanning, too bad, I do), with a few things that are plausibly bad about it that I don’t think are superficial or misreadings, as well as some start of my reasons for worrying less about them:

1. “While it’s true that this isn’t ‘the same as’ calling for outright violence, if we are at least a little bit on Orwell’s side on political violence, surely the position that significantly risking nuclear war is worse than a few terrorists bombing a GPU center seems quite silly. If he supports the former but not the latter, that is quite an extreme position!”:

I’m sympathetic to this, in no small part because I lean Orwell on state violence in many cases, but I think it misunderstands Yudkowsky’s problem with the terrorists. It’s not that the fact that this is terrorism adds enough intrinsic badness to outweigh a greater chance of literal nuclear war, it’s that legitimate state authority credibly being willing to go to nuclear war is likely to actually work, while terrorism is just a naïve tactic which will likely backfire. In fact nothing even rests on the idea that nuclear war is worth preventing AI (though in a quite bad and now deleted tweet Yudkowsky does argue for this, and given that he expects survivors of nuclear war but not AI misalignment nothing about this judgement rests on his cringe “reaching the stars” aside). If a NATO country is invaded, letting it be invaded is surely not as bad as global nuclear war, but supporters of Article 5 tacitly accept the cost of risking this outcome, because non-naïve consequentialism cares about credibly backing certain important norms, even when, in isolation, the cost of going through with them doesn’t look worth it.

2. “While there are international resolutions that involve credibly risking nuclear war, like Article 5, and there are international resolutions that involve punishing rogue states, like ones governing the development of weapons of mass destruction, the combination of these two is in practice not really there, so pointing to each in isolation fails to recognize the way this proposal pushes a difference in degrees all the way to basically a difference in kind”:

I am again sympathetic to this. What Yudkowsky is proposing here is kind of a big deal, and it involves a stricter international order than we have ever seen before. This is very troubling! It isn’t clear that there is a single difference in kind (except perhaps democracy) between Stalinism and a state that I would be mostly fine with. It’s largely about pushing state powers and state flaws that are tolerable at some degree to a point where they are no longer tolerable. I think I’m just not certain if his proposal reaches this crucial level for me. One reason is that I’m just not sure what level of international control really crosses that line for me, and risking war to prevent x-risk seems like a candidate okay think for countries to apply a unique level of force to. Certainly if you believe the things Yudkowsky does. The second reason however, is that his actual proposal is ambiguous in crucial ways that I will cover in point 3, so I would probably be okay with some but not other versions of it.

3. “Yes there is a difference between state force and random acts of violence, but it isn’t clear what general heuristic we can use to distinguish the two other than ‘one is carried out by a legitimate state authority and the other isn’t’. We know what this looks like at the country level because the relevant state authority is usually pretty clear, but on the international stage this is just not something where the difference between legitimate and illegitimate governance is super obvious! How many countries have to sign up? What portion of the world population do they have to represent? How powerful on the international stage do they already have to be? What agreement mechanism/mediating body is needed to grant this authority? Surely there is a difference in the types of ways Yudkowsky cares about between violence carried out by the mafia versus the local government, but on the international stage this sort of difference is very foggy. Given the huge difference he places on illegitimate versus legitimate force, Yudkowsky should have been specific about what it would take for such a governing agreement to be legitimate. Otherwise people can fill in the details however they think one should answer this question, and Yudkowsky specifically is shielded from the most relevant sort of criticism he could face for his proposal”:

This is the objection I am most sympathetic to, and the place I wish critics would focus most of their attention. If NATO agrees to this treaty, does that give them legitimate authority to threaten China with drone strikes that isn’t just like the mafia writing threatening letters to AI developers? What if China joins in with NATO, does this grant the authority to threaten Russia? Probably not for both, but while the difference is probably at an ambiguous threshold of which countries sign up, it’s pretty clear when a country-wide law becomes legitimate, because there’s an agreed upon legitimate process for passing it. These are questions deeply tied to any proposal like this and it does bug me how little Yudkowsky has spelled this out. That said, I think this is sort of a problem for everyone? As I’ve said, and Yudkowsky has said, basically everyone distinguishes between state enforcement and random acts of civilian violence, and aside from this, basically everyone seems confused about how to apply this at the international scale on ambiguous margins. Insofar as you want to apply something like this to the international scale sometimes, you have to live with this tension, and probably just remain a bit confused.