The existential risk community’s relative level of concern about different existential risks is correlated with how hard-to-analyze these risks are. For example, here is The Precipice’s ranking of the top five most concerning existential risks: 1. Unaligned artificial intelligence[1] 2. Unforeseen anthropogenic risks (tied) 3. Engineered pandemics (tied) 4. Other anthropogenic risks 5. Nuclear war (tied) 6. Climate change (tied) This isn’t surprising. For a number of risks, when you first hear about them, it’s reasonable to have the reaction “Oh, hm, maybe that could be a huge threat to human survival” and initially assign something on the order of a 10% credence to the hypothesis that it will by default lead to existentially bad outcomes. In each case, if we can gain much greater clarity about the risk, then we should think there’s about a 90% chance we’ll become less worried about it. We’re likely to remain decently worried about hard-to-analyze risks (because we can’t get greater clarity about them) while becoming less worried about easy-to-analyze risks. In particular, our level of worry about different plausible existential risks is likely to roughly track our ability to analyze them (e.g. through empirical evidence, predictively accurate formal models, and clearcut arguments). Some plausible existential risks also are far easier to analyze than others. If you compare 80K’s articles on climate change and artificial intelligence, for example, then I think it is pretty clear that people analyzing climate risk simply have a lot more to go on. When we study climate change, we can rely on climate models that we have reason to believe have a decent amount of validity. We can also draw on empirical evidence about the historical effects of previous large changes in global temperature and about the ability of humans and other specifies to survive under different local climate conditions. And so on. We’re in a much worse epistemic position when it comes to analyzing the risk from misaligned AI: we’re reliant on fuzzy analogies, abstract arguments that use highly ambiguous concepts, observations of the behaviour of present-day AI systems (e.g. reinforcement learners that play videogames) that will probably be very different than future AI systems, a single datapoint (the evolution of human intelligence and values) that has a lot of important differences with the case we’re considering, and attempts to predict the incentives and beliefs of future actors in development scenarios that are still very opaque to us. Even if the existential risk from misaligned AI actually is reasonably small, it’s hard to see how we could become really confident of that. Some upshots: 1. The fact that the existential risk community is particularly worried about misaligned AI might mostly reflect the fact that it’s hard to analyze risks from misaligned AI. 2. Nonetheless, even if the above possibility is true, it doesn't at all follow that the community is irrational to worry far more about misaligned AI than other potential risks. It’s completely coherent to have something like this attitude: “If I could think more clearly about the risk from misaligned AI, then I would probably come to realize it’s not that big a deal. But, in practice, I can’t yet think very clearly about it. That means that, unlike in the case of climate change, I also can’t rule out the small possibility that clarity would make me much more worried about it than I currently am. So, on balance, I should feel more worried about misaligned AI than I do about other risks. I should focus my efforts on it, even if — to uncharitable observers — my efforts will probably look a bit misguided after the fact.” 3. For hard-to-analyze risks, it matters a lot what your “prior” in the risks is (since evidence, models, and arguments can only really move you so much). I sometimes get the sense that some people are starting from a prior that’s not far from 50%: For example, people who are very worried about misaligned AI sometimes use the rhetorical move “How would the world look different if AI wasn’t going to kill everyone?”, and this move seems to assume that empirical evidence is needed to shift us down from a high credence. I think that other people (including myself) are often implicitly starting from a low prior and feel the need to be argued up. Insofar as it’s very unclear how we should determine our priors, and it's even a bit unclear what exactly a "prior" means in this case, it’s also unsurprising that there’s a particularly huge range of variation in estimates of the risk from misaligned AI. (This shortform partly inspired by Greg Lewis's recent forecasting post .) ---------------------------------------- 1. Toby Ord notes, in the section of The Precipice that gives risk estimates: "The case for existential risk from AI is clearly speculative. Indeed, it is the most speculative case for a major risk in this book." ↩︎

I keep coming back to this map/cartogram. It's just so great. 

A point about hiring and grantmaking, that may already be conventional wisdom: If you're hiring for highly autonomous roles at a non-profit, or looking for non-profit founders to fund, then advice derived from the startup world is often going to overweight the importance of entrepreneurialism relative to self-skepticism and reflectiveness.[1] Non-profits, particularly non-profits with longtermist missions, are typically trying to maximize something that is way more illegible than time-discounted future profits. To give a specific example: I think it's way harder for an organization like the Centre for Effective Altruism to tell if it's on the right track than it is for a company like Zoom to tell if it's on the right track. CEA can track certain specific metrics (e.g. the number of "new connections" reported at each conference it organizes), but it will often be ambiguous how strongly these metrics reflect positive impact - and there will also always be a risk that various negative indirect effects aren't being captured by the key metrics being used. In some cases, evaluating the expected impact of work will also require making assumptions about how the world will evolve over the next couple decades (e.g. assumptions about how pressing risks from AI are). I think this means that it's especially important for these non-profits to employ and be headed by people who are self-skeptical and reflect deeply on decisions. Being entrepreneurial, having a bias toward action, and so on, don't count for much if the organisation isn't pointed in the right direction. As Ozzie Gooen has pointed out, there are many examples of massive and superficially successful initiatives (headed by very driven and entrepreneurial people) whose theories-of-impact don't stand up to scrutiny. A specific example from Ozzie's post: SpaceX is a massive and extraordinarily impressive venture that was (at least according to Elon Musk) largely started to help reduce the chance of human extinction, by helping humanity become a multi-planetary species earlier than it otherwise would. But I think it's hard to see how their work reduces extinction risk very much. If you're worried about the climate effects of nuclear war, for example, then it seems important to remember that post-nuclear-war Earth would still have a much more hospitable climate than Mars. It's pretty hard to imagine a disaster scenario where building Martian colonies would be better than (for example) building some bunkers on Earth.[2][3] So - relative to the organization's stated social mission - all the talent, money, and effort SpaceX has absorbed might not ultimately come out to even close to as much as it could have. A more concise way to put the concern here: Popular writing on talent identification is often implicitly asking the question "How can we identify future Elon Musks?" But, for the most part, longtermist non-profits shouldn't be looking to put future Elon Musks into leadership positions .[4] ---------------------------------------- 1. I have in mind, for example, advice given by Y Combinator and advice given in Talent. ↩︎ 2. Another example: It's possible that many highly successful environmentalist organizations/groups have ended up causing net harm to the environment, by being insufficiently self-skeptical and reflective when deciding how to approach nuclear energy issues. ↩︎ 3. I've encountered the argument that a Mars mission will reduce existential risk by fostering a common human identity and political unity, or hope for the future, which will in turn lead to policies that reduce other existential risks (e.g. bioterrorism or nuclear war). But I think this argument also doesn't hold to scrutiny. Focusing just at the domestic level, for example, the Apollo program had far from universal support, and the decade that followed the moon landing definitely was very from a time of optimism and unity in the US. At the international level, it was also of course largely motivated by great power competition with the Soviet Union. ↩︎ 4. A follow-up thought: Ultimately, outside of earning-to-give ventures, we probably shouldn't expect the longtermist community (or at least the best version of it) to house many extremely entrepreneurial people. There will be occasional leaders who are extremely high on both entrepreneurialism and reflectiveness (I can currently think of at least a couple); however, since these two traits don't seem to be strongly correlated, this will probably only happen pretty rarely. It's also, often, hard to keep exceptionally entrepreneurial people satisfied in non-leadership positions -- since, almost by definition, autonomy is deeply important to them -- so there may not be many opportunities, in general, to harness the talents of people who are exceptionally high on entrepreneurialism but mediocre on reflectiveness. ↩︎

Stem cell slowdown and AI timelines My knowledge of christians and stem cell research in the US is very limited, but my understanding is that they accomplished real slowdown.  Has anyone looked to that movement for lessons about AI?  Did anybody from that movement take a "change it from the inside" or "build clout by boosting stem cell capabilities so you can later spend that clout on stem cell alignment" approach? 

The goal of this short-form post: to outline what I see as the key common ground between the “big tent” versus “small and weird” discussions that have been happening recently and to outline one candidate point of disagreement.   Tl;dr: * Common ground: * Everyone really values good thinking processes/epistemics/reasoning transparency and wants to make sure we maintain that aspect of the existing effective altruism community * Impact is fat-tailed * We might be getting a lot more attention soon because of our increased spending and because of the August release of "what we owe the future" (and the marketing push that is likely to accompany it's release)[1] * A key point of disagreement: Does focusing on finding the people who produce the “tail” impact actually result in more impact? * One reason this wouldn’t be the case: “median” community building efforts and “tail” community building efforts are complements not substitutes. They are multipliers[2] of each other, rather than being additive and independent. * The additive hypothesis is simpler so I felt the multiplicative hypothesis needed some outlined mechanisms. Possible mechanisms: * Mechanism 1: the sorts of community building efforts that are more “median” friendly actually help the people who eventually create the “tail” impact become more interested in these ideas and more interested in taking bigger action with time * Mechanism 2: our biggest lever for impact in the future will not be the highly dedicated individuals but our influence on people on the periphery of the effective altruism (what I call “campground” effects) Preamble (read: pre-ramble) This is my summary of my vibe/impressions on some of the parts of the recent discussion that have stood out to me as particularly important. I am intending to finish my half a dozen drafts of a top-level post (with much more explanations for my random jargon that isn’t always even that common in effective altruism circles) at some point but I thought I’d start but sharing these rough thoughts to help get me over the “sharing things on the EA forum is scary” hump.  I might end up just sharing this post as a top-level post later once I’ve translated my random jargon a bit more and thought a bit more about the claims here I’m least sure of (possibly with a clearer outline of what cruxes make the “multiplicative effects” mechanisms more or less compelling) Some common ground These are some of my impressions of some claims that seem to be pretty common across the board (but that people sometimes talk as though they might suspect that the person they are talking to might not agree so I think it’s worth making them explicit somewhere).   1. The biggest one seems to be: We like the fact that effective altruism has good thinking processes/epistemics a lot! We don’t want to jeopardize our reasoning transparency and scout mindsets for the sake of going viral. 2. Impact is fat-tailed and this makes community-building challenging: there are a lot of uncomfortable trade-offs that might need to be made if we want to build the effective altruism community into a community that will be able to do as much good as possible. 3. We might be getting a lot more attention very soon whether we want to or not because we're spending more (and spending in places that get a lot of media attention like political races) and because there will be a big marketing push for "What We Owe the Future" to, potentially, a very big audience. [3] A point of disagreement  It seems like there are a few points of disagreement that I intended to go into, but this one got pretty long so I’ll just leave this as one point: Does focusing on “tail” people actually result in more impact? Are “tail” work and “median” work complements or substitutes? Are they additive (so specialization in the bit with all the impact makes sense) or multiplicative (so doing both well is a necessary condition to getting “tails”)?  I feel like the “additive/substitutes” hypothesis is more intuitive/a simpler assumption so I’ve outlined some explicit mechanisms for the “multiplicative/complements” hypothesis.   Mechanisms for the “multiplicative/complements” hypothesis  Mechanism 1 “Tail” people often require similar “soft” entry points to “non-tail” people and focusing on the “median” people on some dimensions actually is better at getting the “tails” people because we just model “tail” people wrong (e.g. someone could think it looks like some people were always going to be tails, but in reality, when we deep-dive into individual “tail” stories, there was, accidentally, “soft” entry points).  The dimensions where people advocate for lowering the bar are not epistemics/thinking processes, but things like  1. language barriers (e.g. reducing jargon, finding a plain English way to say something or doing your best to define the jargon when you use it if you think it’s so useful that it’s worth a definition), 2. making it easier for people to transition at their own pace from wherever they are to “extreme dedication” (and being very okay with some people stopping completely way before, and 3. reducing the social pressure to agree with the current set of conclusions by putting a lot more emphasis on a broader spectrum of plausible candidates that we might focus on if we’re trying to help others as much as possible (where “plausible candidates” are answers to the question of how we can help others the most with impartiality, considering all people alive today [4] or even larger moral circles/circle of compassion than that too, where an example of a larger group of individuals we might be wanting to help is all present and future sentient beings) Mechanism 2 As we get more exposure, our biggest lever for impact might not the people that get really enthusiastic about effective altruism who go all the way to the last stop of the crazy train (what I might call our current tent), but the cultural memes we’re spreading to friends-of-friends-of-friends of people who have interacted with people in the effective altruism community or with the ideas and have strong views about them (positive or negative), which I have been calling “our campground” in all my essays to myself on this topic 🤣.  E.g. let’s say that the only thing that matters for humanity’s survival is who ends up in a very small number of very pivotal rooms,[5] it might be much easier to influence a lot of the people who are likely to be in those rooms a little bit to be thinking about some of the key considerations we hope they’d be considering (it’d be nice if we made it more likely that a lot of people might have the thought; “a lot might be at stake here, let’s take a breather before we do X”) than to get people who have dedicated their lives to reducing X-risk because effective altruism-style thinking and caring is a core part of who they are in those rooms.  As we get more exposure, it definitely seems true that “campground” effects are going to get bigger whether we like it or not.[6]  It is an open question (in my mind at least) whether we can leverage this to have a lot more impact or whether the best we can do is sit tight and try and keep the small core community on point.   1. ^ As a little aside, I am so excited to get my hands on a copy (suddenly August doesn't seem so soon)! 2. ^ Additive and multiplicative models aren't the only two plausible "approximations" of what might be going on, but they are a nice starting point.  It doesn't seem outside the range of possibility that there are big positive feedback loops between "core camp efforts" and "campground efforts" (and all the efforts in between). If this is plausibly true, then the "tails" for the impact of the effective altruism community as a whole could be here.  3. ^ this point of common ground was edited in after first posting this comment 4. ^ This is a pretty arbitrary cutoff of what counts as a large enough moral circle to count under the broader idea behind effective altruism and trying to do the most good, but I like being explicit about what we might mean because otherwise people get confused/it’s harder to identify what is a disagreement about the facts and what is just a lack of clarity in the questions we’re trying to ask.  I like this arbitrary cutoff a lot because  1) anyone who cares about every single person alive today already has a ginormous moral circle and I think that’s incredible: this seems to be very much wide enough to get at the vibe of the widely caring about others, and  2) the crazy train goes pretty far, it is not at all obvious to me where the “right” stopping point is, I’ve got off a few stops along (my shortcut to avoid dealing with some crazier questions down the line, like infinite ethics, is “just” considering those in my light cone where it be simulated or not😅, not because I actually think this is all that reasonable, but because more thought on what “the answer” is seems to get in the way of me thinking hard about doing the things I think are pretty good which I think, on expectation, actually does more for what I’d guess I’ll care about if I had all of time to think about it.  5. ^ this example is total plagiarism, see: https://80000hours.org/podcast/episodes/sam-bankman-fried-high-risk-approach-to-crypto-and-doing-good/ (also has such a great discussion on multiplicative type effects being a big deal sometimes which I feel people in the effective altruism community think about less than we should: more specialization and more narrowing the focus isn't always the best strategy on the margin for maximizing how good things are and will be on expectation, especially as we grow and have more variation in people's comparative advantages within our community, and more specifically, within our set of community builders) 6. ^ If our brand/reputation has lock-in for decades for a really long time, this could plausibly be a hinge of history moment for the effective altruism community. If there are ways of making our branding/reputation is as high fidelity as possible within the low-fidelity channels that messages travel virally, this could be a huge deal (ideally, once we have some goodwill from the broader "campground" we will have a bit of a long reflection to work out what we want our tent to look like 🤣😝).