I recently looked through the current version of the virtual groups intro syllabus and was disappointed to see zero mention of s-risks within the sections on longtermism/existential risk. I think this is a symptom of a larger problem, where “longtermism” has come to mean a very particular set of future-oriented projects (primarily extinction risk reduction) that primarily derive from a very particular set of values (primarily classical utilitarianism). As facilitators responsible for introducing people to the ideas of EA, I think it’s important for us to diversify our readings and discussions to account for multiple reasonable starting positions. For a start, I suggest that we rework the week on existential risk to have a more general focus on cause prioritization in longtermism, including readings and discussions on the topic of s-risks.

More generally, I think that we should take the threat of groupthink very seriously. The best-funded and most influential parts of the EA community have come to prioritize a particular worldview and value system that is not necessarily definitive of EA, and one that reasonable people in the community could disagree with. Throughout my experience as a student organizer, I've seen many of my peers just defer to the views and values supported by organizations like 80,000 Hours without reflecting much on their own positions, which strikes me as quite problematic given that many want to represent EA as a question, not an ideology. Failing to include a broader range of ideas and topics in introductory fellowships only exacerbates this problem of groupthink.

I’d love to talk more about how we can diversify the range of views represented to newcomers, and in particular how we can “diversify longtermism.”

Comments10


Sorted by Click to highlight new comments since:
mic
18
0
0

Reading your suggested required readings, S-risks: An introduction and S-risk FAQ, I don't get a clear sense of why s-risks are plausible or why the suggested interventions are useful. I like S-risks: Why they are the worst existential risks, and how to prevent them (EAG Boston 2017) a bit more for illustrating why they are plausible, and I've added it as an optional reading in the uni chapter intro program I'm running. Unfortunately, it doesn't give more than a cursory overview of how s-risks could be reduced. I'd be hesitant about making an s-risk reading a required reading though since I don't know about high-quality intro-level material about s-risks for participants to learn more. I also expect that s-risks might provoke a lot of discussion, so we would want to make sure discussion facilitators have the resources to be well-informed about the issues. Right now, I wouldn't feel confident discussing s-risks with intro program participants.

By the way, if you want to make your suggestions as easy as possible to add it to the curriculum, you should also suggest discussion questions that facilitators can ask about the topic.

(Note: I'm not in charge of the EA Virtual Programs syllabus.)

Mau
14
0
0

Thanks for this! I'm not sure what I think about this--a few things might make it challenging/costly to introduce s-risks into the Introductory EA syllabi:

  1. The Introductory EA Program is partly meant to build participants' sustained interest in EA, and the very speculative / weird nature of s-risks could detract from that (by being off-puttingly out-there to people who--like most program participants--have only spent a handful of hours learning about / reflecting on relevant topics).
    1. One might wonder: if this is an issue, why introduce x-risks? I'd guess x-risks are more accessible/intuitive to people, given common discourse about climate change and our recent experiences with pandemics. And the Introductory EA Program doesn't dive into, say, the details of AI risk arguments, for a mix of this reason and the following reason.
  2. The Introductory EA Program aims to have an amount of material that's short enough (a) for many people who are busy and not totally dedicated to EA to want to apply / not drop out, and (b) for many participants to actually do the core readings (in the absence of serious external incentives). Adding core/required readings in general is costly, because it detracts from these things.
  3. Careful thinking about s-risks seems to lead people to seek cooperative ways to mitigate s-risks, which seems great. The Introductory EA Program is too rushed and shallow to promote much thorough thinking, raising risks that participants could take naive/reckless approaches to reducing s-risks (which seems terrible, including for the reputation of the s-risk community).

Still, your basic point about the importance of diversifying the ideas EAs are introduced to seems right. (This is different from the importance of diversifying the views of people in longtermism--the value of that depends on the correctness of the views that are currently most prominent.) So I'd be tentatively optimistic about some approaches like these:

  • Adding more topics to the further readings sections.
  • Adding a week where people choose readings out of a wide range of topics.
  • Having sections on s-risks in the In-Depth EA Program (if I remember correctly, this is already the case, which seems appropriate because points 1-3 above are all much less applicable to the in-depth program.)

(Also, if anyone is interested in suggesting changes to the Introductory EA Program, you might find it useful to make shovel-ready recommendations, e.g. proposing a specific reading or collection of readings for a week, because organizers are fairly time-constrained. Although broader suggestions/discussion also seems valuable.)

I agree with 2. Not sure about 3 as I haven't reviewed the Introductory fellowship in depth myself.

But on 1, I want to briefly make the case that s-risks don't have to be/seem much more weird than extinction risk work. I've sometimes framed it as: The future is vast and it could be very good or very bad. So we probably want to both try to preserve it for the good stuff and improve the quality. (Although perhaps CLR et al don't actually agree with the preserving bit, they just don't vocally object to it for coordination reasons etc)

There are also ways it can seem less weird. E.g. you don't have make complex arguments about wanting to ensure a thing that hasn't happened yet continues to happen, and missed potential, you can just say: "here's a potential bad thing. We should stop that!!" See https://forum.effectivealtruism.org/posts/seoWmmoaiXTJCiX5h/the-psychology-of-population-ethics for evidence that people, on average, weigh (future/possible) suffering more than happiness.

Also consider that one way of looking at moral circle expansion (one method of reducing s-risks) is that its basically just what many social justicey types are focusing on anyway -- increasing protection and consideration of marginalised groups. It just takes it further.

Thanks! Yeah, I think you're right; that + Sean's specific reading suggestions seem like reasonably intuitive introductions to s-risks. Do you think there are similarly approachable introductions to specific s-risks, for when people ask "OK, I'm into this broad idea--what specific things could I work on?" (Or maybe this isn't critical--maybe people are oddly receptive to weird ideas if they've had good first impressions.)

Well I think moral circle expansion is a good example. You could introduce s-risks as a general class of things, and then talk about moral circle expansion as a specific example. If you don't have much time, you can keep it general and talk about future sentient beings; if animals have already been discussed, mention that idea that if factory farming or something similar was spread to astronomical scales, that could be very bad. If you've already talked about risks from AI, I think you could reasonably discuss some content about artificial sentience without that seeming like too much of a stretch. My current guess is that focusing on detailed simulations as an example is a nice balance between (1) intuitive / easy to imagine and (2) the sorts of beings we're most concerned about. But I'm not confident in that, and Sentience Institute is planning a survey for October that will give a little insight into which sorts of future scenarios  and entities people are most concerned about. If by "introductions" you're looking for specific resource recommendations, there are short videos, podcasts, and academic articles depending on the desired length, format etc.

Some of the specifics might be technical, confusing, or esoteric, but if you've already discussed AI safety, you could quite easily discuss the concept of focusing on worst-case / “fail-safe” AI safety measures as a promising area. It's also nice because it overlaps with extinction risk reduction work more (as far as I can tell) and seems like a more tractable goal than preventing extinction via AI or achieving highly aligned transformative AI.

A second example (after MCE) that benefits from being quite close to things that many people already care about is the area of reducing risks from political polarisation. I guess that explaining the link to s-risks might not be that quick though. Here's a short writeup on this topic, and I know that Magnus Vinding of the Center for Reducing Suffering is publishing a book soon called Reasoned Politics, which I imagine includes some content on this. Its all a bit early stages though, so I probably wouldn't pick this one at the moment.

Hey Mauricio, thanks for your reply. I’ll reply later with some more remarks, but I’ll list some quick thoughts here:

  1. I agree that s-risks can seem more “out there,” but I think some of the readings I’ve listed do a good job of emphasizing the more general worry that the future involves a great deal of suffering. It seems to me that the asymmetry in content about extinction risks vs. s-risks is less about the particular examples and more about the general framework. Taking this into account, perhaps we could write up something to be a gentler introduction to s-risks. The goal is to prevent people from identifying “longtermism” as just extinction risk reduction.

  2. Yeah this is definitely true, but completely omitting such a distinctively EA concept like s-risks seems to suggest that something needs to be changed.

  3. I think the reading I listed entitled “Common Ground for Longtermists” should address this worry, but perhaps we could add more. I tend to think that the potential for antagonism is outweighed by the value of broader thinking, but your worry is worth addressing.

Mau
13
0
0

Thanks!

Ah sorry, I hadn't seen your list of proposed readings (I wrongly thought the relevant link was just a link to the old syllabus). Your points about those readings in (1) and (3) do seem to help with these concerns. A few thoughts:

  • The dichotomy between x-risk reduction and s-risk reduction seems off to me. As I understand them, prominent definitions of x-risks [1] [2] [3] (especially the more thorough/careful discussion in [3]) are all broad enough for s-risks to count as x-risks (especially if we're talking about permanent / locked-in s-risks, which I assume we are, given the context of longtermism).
  • One worry is that the proposed list might be overcorrecting--through half of its content being from CFR, it seems to suggest that about half of longtermists endorse prioritizing s-risk reduction, which is a large over-estimate.
  • As you say, we want to discourage uncritical acceptance of views presented in the syllabus, so it seems good for such a list to include criticisms of both approaches to improving the long-term future, at least in recommended readings. (Yup, the current syllabus is also light on those, although week 7 does include criticisms of longtermism.)

completely omitting such a distinctively EA concept like s-risks seems to suggest that something needs to be changed.

I'm not sure about that. The intro program omits plenty of distinctively EA concepts due to time/attention constraints--here are some other prominent ideas in EA that (if I remember correctly) are currently omitted from the core/required readings of the introductory program: consequentialism, cause X, patient longtermism, wild animal suffering, EA movement-building, improving institutional decision making, decision theory, the unilateralist's curse, moral uncertainty & cooperation, Bayesian reasoning, forecasting, history of well-being, cluelessness, mental welfare, and global priorities research.

A bunch of these (and s-risk reduction) are covered in depth in (some versions of) the in-depth EA program.

(Like I mentioned earlier, I'm pretty open to there being some discussion of s-risks in the intro syllabus. Mostly wondering about the degree to which it should be covered.)

Yeah my mistake, I should have been clearer about the link for the proposed changes. I think we’re mostly in agreement. My proposed list is probably overcorrecting, and I definitely agree that more criticisms of both approaches are needed. Perhaps a compromise would be just including the reading entitled “Common Ground for Longtermists,” or something similar.

I think you’re right that many definitions of x-risk are broad enough to include (most) s-risks, but I’m mostly concerned about the term “x-risk” losing this broader meaning and instead just referring to extinction risks. It’s probably too nuanced for an intro syllabus, but MichaelA’s post (https://forum.effectivealtruism.org/posts/AJbZ2hHR4bmeZKznG/venn-diagrams-of-existential-global-and-suffering) could help people to better understand the space of possible problems.

The EA Handbook follows a similar path to the introductory program syllabus. If you have specific suggestions for readings on S-risk (or anything else!), I'm always happy to hear them. I see the Handbook as a work in progress, and make small adjustments quite often.

Hi Aaron, thanks for your reply. I’ve listed some suggestions in one of the hyperlinks above, but I’ll put it here too: https://docs.google.com/document/d/1niRwbh3eejByFQwoiZ0NiaSZDUawn206PUmHs7aKL0A/edit?usp=sharing

I have not put much time into this, so I’d love to hear your thoughts on the proposed changes.

Curated and popular this week
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Dr Kassim
 ·  · 4m read
 · 
Hey everyone, I’ve been going through the EA Introductory Program, and I have to admit some of these ideas make sense, but others leave me with more questions than answers. I’m trying to wrap my head around certain core EA principles, and the more I think about them, the more I wonder: Am I misunderstanding, or are there blind spots in EA’s approach? I’d really love to hear what others think. Maybe you can help me clarify some of my doubts. Or maybe you share the same reservations? Let’s talk. Cause Prioritization. Does It Ignore Political and Social Reality? EA focuses on doing the most good per dollar, which makes sense in theory. But does it hold up when you apply it to real world contexts especially in countries like Uganda? Take malaria prevention. It’s a top EA cause because it’s highly cost effective $5,000 can save a life through bed nets (GiveWell, 2023). But what happens when government corruption or instability disrupts these programs? The Global Fund scandal in Uganda saw $1.6 million in malaria aid mismanaged (Global Fund Audit Report, 2016). If money isn’t reaching the people it’s meant to help, is it really the best use of resources? And what about leadership changes? Policies shift unpredictably here. A national animal welfare initiative I supported lost momentum when political priorities changed. How does EA factor in these uncertainties when prioritizing causes? It feels like EA assumes a stable world where money always achieves the intended impact. But what if that’s not the world we live in? Long termism. A Luxury When the Present Is in Crisis? I get why long termists argue that future people matter. But should we really prioritize them over people suffering today? Long termism tells us that existential risks like AI could wipe out trillions of future lives. But in Uganda, we’re losing lives now—1,500+ die from rabies annually (WHO, 2021), and 41% of children suffer from stunting due to malnutrition (UNICEF, 2022). These are preventable d
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f