Effective Altruism Forum
EA Forum

Hide table of contents
Comment Permalink
Henry Howard🔸10
15
11

I think this is a great question. The lack of clear, demonstrable progress in reducing existential risks, and the difficulty of making and demonstrating any progress, makes me very skeptical of longtermism in practice.

I think shifting focus from tractable, measurable issues like global health and development to issues that - while critical - are impossible to reliably affect, might be really bad.

Reply
Showing 3 of 5 replies (Click to show all)
8
JackM
I'm not sure how it's even theoretically possible to measure reductions in existential risk. An existential catastrophe is something that can only happen once. Without being able to observe a reduction in incidence of an event I don't think you can "measure" reduction in risk. I do on the other hand think it's fair to say that increasing awareness of existential risk reduces total existential risk, even if I'm not sure by how much exactly. I'd imagine concrete/legible actions to reduce existential risk will probably come in the form of policy change and I don't think for the most part EAs have yet entered influential policy positions. Please do say what other actions you would consider to count under concrete/legible though as that is up for interpretation. Sorry I'm not really sure what you're saying here. This is a good question. I think the best arguments against longtermism are: *  That longtermism is fanatical and that fanaticism is not warranted * On balance I am not convinced by this objection as I don't think longtermism is fanatical and am unsure if fanaticism is a problem. But further research here might sway me. * That we might simply be clueless about the impacts of our actions  * At the moment I don't think we are, and I think if cluelessness is a big issue it is very likely to be an issue for neartermist cause areas as well, and even EA altogether. I don't mind admitting that it seems unlikely that I will change my mind on longtermism. If I do, I'd imagine it will be on account of one of the two arguments above. 
3
pseudonym
I disagree - what do you think the likelihood of a civilization ending event from engineered pandemics is, and what do you base this forecast on? What % of longtermist $ and FTEs do you think are being spent on trying to influence policy versus technical or technological solutions? (I would consider many of these as concrete + legible) That was me trying to steelman your justification of lack of concrete/legible wins to "longtermism is new" by thinking of clearer ways that longtermism is different to neartermist causes, and that requires looking outside the EA space.
JackM4
1
0

I disagree - what do you think the likelihood of a civilization ending event from engineered pandemics is, and what do you base this forecast on?

As I say I don’t think one can “measure” the probability of existential risk. I think one can estimate it through considered judgment of relevant arguments but I am not inclined to do so and I don’t think anyone else should be so inclined either. Any such probability would be somewhat arbitrary and open to reasonable disagreement. What I am willing to do is say things like “existential risk is non-negligible” and ... (read more)

Reply
See in context
by Ben Snodin
1 min read4 answers 9

118

Building effective altruismExistential riskProbably GoodLongtermism
Frontpage

A motivating scenario could be: imagine you are trying to provide examples to help convince a skeptical friend that it is in fact possible to positively change the long-run future by actively seeking and pursuing opportunities to reduce existential risk.

Examples of things that are kind of close but miss the mark

  • There are probably decent historical examples where people reduced existential risk but where thoes people didn't really have longtermist-EA-type motivations (maybe more "generally wanting to do good" plus "in the right place at the right time")
  • There are probably meta-level things that longtermist EA community members can take credit for (e.g. "get lots of people to think seriously about reducing x risk"), but these aren't very object-level or concrete

118

0
0

Reactions

0
0
Mentioned in
New Answer
New Comment


4 Answers sorted by
Top

29
9
0

A lot of longtermist effort is going into AI safety at the moment. I think it's hard to make the case that something in AI safety has legibly or concretely reduced AI risk, since (a) the field is still considered quite pre-paradigmatic, (b) the risk comes from systems that are more powerful than the ones we currently have, and (c) even in less speculative fields, research often takes several years before it is shown to legibly help anyone.

But with those caveats in mind, I think:

  1. The community has made some progress in understanding possible risks and threats from advanced AI system. (See DeepMind's review of alignment threat models). 
  2. Interpretability research seems relatively legible. The basic case "we're building powerful models and it would be valuable to understand how they work" makes intuitive sense. There are also several more nuanced ways interpretability research could be helpful (see Neel's longlist of theories for impact).
  3. The fact that most of the major AGI labs have technical safety teams and governance teams seems quite concrete/legible. I'm not sure how much credit should go to the longtermist communities, but I think several of these teams have been inspired/influenced by ideas in the AI safety community. (To be fair, this might just be a case of "get lots of people to think seriously about reducing x-risk", but I think it's a bit more tangible/concrete.)
  4. In AI governance, the structured access approach seems pretty common among major AGI labs (again, a bit unclear how much credit should go to longtermists but my guess is a non-negligble amount). 
  5. In AI governance, some work on reducing misuse risks and recognizing the dual-use nature of AI technologies seems somewhat legible. A lot of people who did this research are now working at major AGI labs, and it seems plausible that they're implementing some of the best practices they suggested (which would be especially legible, though I'm not aware of any specific examples, though this might be because labs keep a lot of this stuff confidential). 
Halffull33
18
2

I think it's also easy to make a case that longtermist efforts have increased the x-risk of artificial intelligence, with the money and talent that grew some of the biggest hype machines in AI (Deepmind, OpenAI) coming from longtermist places.  

It's possible that EA has shaved  a couple  counterfactual years off of time to catastrophic AGI, compared to a world where the community wasn't working on it.

Reply
[anonymous]12
2
1

Can you say more about which longtermist efforts you're referring to?

I think a case can be made, but I don't think it's an easy (or clear) case.

My current impression is that Yudkowsky & Bostrom's writings about AGI inspired the creation of OpenAI/DeepMind. And I believe FTX invested a lot in Anthropic and OP invested a little bit (in relative terms) into OpenAI. Since then, there have been capabilities advances and safety advances made by EAs, and I don't think it's particularly clear which outweighs.

It seems unclear to me what the sign of these effects are. Like, maybe no one thinks about AGI for decades. Or maybe 3-5 years after Yudkowsky starts thinking about AGI, someone else much less safety-concerned starts thinking about AGI, and we get a world with AGI labs that are much less concerned about safety than status-quo.

I'm not advocating for this position, but I'm using it to illustrate how the case seems far-from-easy. 

Reply
MichaelStJules12
3
1

Is most of the AI capabilities work here causally downstream of Superintelligence, even if Superintelligence may have been (heavily ?) influenced by Yudkowsky? Both Musk and Altman recommended Superintelligence, altough Altman has also directly said Yudkowsky has accelerated timelines the most:

https://twitter.com/elonmusk/status/495759307346952192?lang=en

https://blog.samaltman.com/machine-intelligence-part-1

https://twitter.com/sama/status/1621621724507938816

If things stayed in the LW/Rat/EA community, that might have been best. If Yudkowsky hadn't written about AI, then there might not be much of an AI safety community at all now (it might just be MIRI quietly hacking away at it, and most of MIRI seems to have given up now), and doom would be more likely, just later. Someone had to write about AI safety publicly to build the community, but writing and promoting a popular book on the topic is much riskier, because you bring it to the attention of uncareful people, including entrepreneurial types.

I guess they might have tried to keep the public writing limited to academia, but the AI community has been pretty dismissive of AI safety, so it might have been too hard to build the community that way.

Reply
4
JakubK
Did Superintelligence have a dramatic effect on people like Elon Musk? I can imagine Elon getting involved without it. That involvement might have been even more harmful (e.g. starting an AGI lab with zero safety concerns). Here's one notable quote about Elon (source), who started college over 20 years before Superintelligence: Overall,  causality is multifactorial and tricky to analyze, so concepts like "causally downstream" can be misleading.  (Nonetheless, I do think it's plausible that publishing Superintelligence was a bad idea, at least in 2014.)
Ben Snodin3
2
0

Thanks for these!

I think my general feeling on these is that it's hard for me to tell if they actually reduced existential risk. Maybe this is just because I don't understand the mechanisms for a global catastrophe from AI well enough. (e.g. because of this, linking to Neel's longlist of theories for impact was helpful, so thank you for that!)

E.g. my impression is that some people with relevant knowledge seem to think that technical safety work currently can't achieve very much. 

(Hopefully this response isn't too annoying -- I could put in the work to understand the mechanisms for a global catastrophe from AI better, and maybe I will get round to this someday)

Reply

peterhartree

20
3
1

The CLTR Future Proof report has influenced UK government policy at the highest levels.

E.g. The UK "National AI Strategy ends with a section on AGI risk, and says that the Office for AI should pay attention to this.

Denkenberger🔸

16
4
0

I think that working out how resilient food technologies could be scaled up in a catastrophe such as nuclear winter is legible and concrete, including natural gas (methane) protein, hydrogen protein, greenhouses, seaweed, leaf protein concentrate, fat from petroleum, relocating cool tolerant crops, etc. Indeed, a survey and a poll have indicated that this work has reduced existential risk.

peterhartree

13
1
0

If you think the UN matters, then this seems good:

On September 10th 2021, the Secretary General of the United Nations released a report called “Our Common Agenda”. This report seems highly relevant for those working on longtermism and existential risk, and appears to signal unexpectedly strong interest from the UN. It explicitly uses longtermist language and concepts, and suggests concrete proposals for institutions to represent future generations and manage catastrophic and existential risks.

https://forum.effectivealtruism.org/posts/Fwu2SLKeM5h5v95ww/major-un-report-discusses-existential-risk-and-future

Comments9
Sorted by
New & upvoted
Click to highlight new comments since:
Henry Howard🔸10
15
11

I think this is a great question. The lack of clear, demonstrable progress in reducing existential risks, and the difficulty of making and demonstrating any progress, makes me very skeptical of longtermism in practice.

I think shifting focus from tractable, measurable issues like global health and development to issues that - while critical - are impossible to reliably affect, might be really bad.

Reply
JackM10
6
3

I don't think that a lack of concrete/legible examples of existential risk reduction so far should make us move to other cause areas. 

The main reason is that it might be unsurprising for a movement to take a while to properly get going. I haven't researched this, but it seems unsurprising to me that movements may typically start with a period of increasing awareness / the number of people working in the movement (a period I think we are currently still in), before achieving really concrete wins. The longtermist movement is a new one with mostly young people who have reoriented their careers but generally haven't yet reached senior enough positions to affect real change. 

If you actually buy into the longtermist argument, then why give up now? It seems unreasonable to me to think that we haven't yet achieved concrete change and  that we are very unlikely to ever do so in the future.

Reply
pseudonym22
10
2

I don't think that a lack of concrete/legible examples of existential risk reduction so far should make us move to other cause areas. 

Perhaps not, but if a movement is happy to use estimates like "our X-risk is 17% this century" to justify working on existential risks and call it the most important thing you can do with your life, but cannot measure how their work actually decreases this 17% figure, they should at the very least reconsider whether their approach is achieving their stated goals.

The longtermist movement is a new one with mostly young people who have reoriented their careers but generally haven't yet reached senior enough positions to affect real change. 

I think this is misleading, because:

Longtermism has been part of EA since close to its very beginning, and many senior leaders in EA are longtermists.

It's true that global health as an area is newer than AI safety, but given EA GHD isn't taking credit for things that happened before EA existed, like eradicating smallpox, I don't know if this is actually the "main reason".

If you actually buy into the longtermist argument, then why give up now? It seems unreasonable to me to think that we haven't yet achieved concrete change and  that we are very unlikely to ever do so in the future.

You might buy into the longtermism argument at a general level ("Future lives matter", "the future is large", we can affect the future"), but update about some of the details, such that you think planning for and affecting the far future is much more intractable or premature than you previously thought. Otherwise, are you saying there's nothing that could happen that would change your mind on whether longtermism was a good use of EA resources?

Reply
JackM8
2
1

but cannot measure how their work actually decreases this 17% figure, they should at the very least reconsider whether their approach is achieving their stated goals.

I'm not sure how it's even theoretically possible to measure reductions in existential risk. An existential catastrophe is something that can only happen once. Without being able to observe a reduction in incidence of an event I don't think you can "measure" reduction in risk. I do on the other hand think it's fair to say that increasing awareness of existential risk reduces total existential risk, even if I'm not sure by how much exactly.

Longtermism has been part of EA since close to its very beginning, and many senior leaders in EA are longtermists.

I'd imagine concrete/legible actions to reduce existential risk will probably come in the form of policy change and I don't think for the most part EAs have yet entered influential policy positions. Please do say what other actions you would consider to count under concrete/legible though as that is up for interpretation.

It's true that global health as an area is newer than AI safety, but given EA GHD isn't taking credit for things that happened before EA existed, like eradicating smallpox, I don't know if this is actually the "main reason".

Sorry I'm not really sure what you're saying here.

Otherwise, are you saying there's nothing that could happen that would change your mind on whether longtermism was a good use of EA resources?

This is a good question. I think the best arguments against longtermism are:

  •  That longtermism is fanatical and that fanaticism is not warranted
    • On balance I am not convinced by this objection as I don't think longtermism is fanatical and am unsure if fanaticism is a problem. But further research here might sway me.
  • That we might simply be clueless about the impacts of our actions 
    • At the moment I don't think we are, and I think if cluelessness is a big issue it is very likely to be an issue for neartermist cause areas as well, and even EA altogether.

I don't mind admitting that it seems unlikely that I will change my mind on longtermism. If I do, I'd imagine it will be on account of one of the two arguments above. 

Reply
pseudonym3
2
1

I'm not sure how it's even theoretically possible to measure reductions in existential risk. Without being able to observe a reduction in incidence of an event I don't think you can "measure" reduction in risk. 

I disagree - what do you think the likelihood of a civilization ending event from engineered pandemics is, and what do you base this forecast on?

I'd imagine concrete/legible actions to reduce existential risk will probably come in the form of policy change 

What % of longtermist $ and FTEs do you think are being spent on trying to influence policy versus technical or technological solutions? (I would consider many of these as concrete + legible)

Sorry I'm not really sure what you're saying here.

That was me trying to steelman your justification of lack of concrete/legible wins to "longtermism is new" by thinking of clearer ways that longtermism is different to neartermist causes, and that requires looking outside the EA space.

Reply
JackM4
1
0

I disagree - what do you think the likelihood of a civilization ending event from engineered pandemics is, and what do you base this forecast on?

As I say I don’t think one can “measure” the probability of existential risk. I think one can estimate it through considered judgment of relevant arguments but I am not inclined to do so and I don’t think anyone else should be so inclined either. Any such probability would be somewhat arbitrary and open to reasonable disagreement. What I am willing to do is say things like “existential risk is non-negligible” and "we can meaningfully reduce it”. These claims are easier to defend and are all we really need to justify working on reducing existential risk.

What % of longtermist $ and FTEs do you think are being spent on trying to influence policy versus technical or technological solutions? (I would consider many of these as concrete + legible)

No idea. Even if the answer is a lot and we haven’t made much progress, this doesn’t lead me away from longtermism. Mainly because the stakes are so high and I think we’re still relatively new to all this so I expect us to get more effective over time, especially as we actually get people into influential policy roles.

That was me trying to steelman your justification of lack of concrete/legible wins to "longtermism is new" by thinking of clearer ways that longtermism is different to neartermist causes, and that requires looking outside the EA space.

This may be because I’m slightly hungover but you’re going to have to ELI5 your point here!

Reply
Ben Snodin8
4
2

I imagine a lot of relevant stuff could be infohazardous (although that stuff might not do very well on the "legible" criterion) -- if so and if you happen to feel comfortable sharing it with me privately, feel free to DM me about it.

Reply
EdoArad4
0
0

Just out of curiosity, and maybe it'd help readers with answers, could you share why you are interested in this question? 

Reply
Ben Snodin4
1
0

I think my motivation comes from things to do with: helping with my personal motivation for work on existential risk, helping me form accurate beliefs on the general tractability of work on existential risk, and helping me advocate to other people about the importance of work on existential risk.

Thinking about it maybe it would be pretty great to have someone assemble and maintain a good public list of answers to this question! (or maybe someone did already and I don't know about it)

Reply
More from Ben Snodin
196
Mid-career people: strongly consider switching to EA work
Ben Snodin
· · 11m read
34
34
84
Incubating AI x-risk projects: some personal reflections
Ben Snodin
· · 11m read
10
10
137
My thoughts on nanotechnology strategy research as an EA cause area
Ben Snodin
· · 39m read
17
17
View more
Curated and popular this week
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Dr Kassim
 ·  · 4m read
 · 
Hey everyone, I’ve been going through the EA Introductory Program, and I have to admit some of these ideas make sense, but others leave me with more questions than answers. I’m trying to wrap my head around certain core EA principles, and the more I think about them, the more I wonder: Am I misunderstanding, or are there blind spots in EA’s approach? I’d really love to hear what others think. Maybe you can help me clarify some of my doubts. Or maybe you share the same reservations? Let’s talk. Cause Prioritization. Does It Ignore Political and Social Reality? EA focuses on doing the most good per dollar, which makes sense in theory. But does it hold up when you apply it to real world contexts especially in countries like Uganda? Take malaria prevention. It’s a top EA cause because it’s highly cost effective $5,000 can save a life through bed nets (GiveWell, 2023). But what happens when government corruption or instability disrupts these programs? The Global Fund scandal in Uganda saw $1.6 million in malaria aid mismanaged (Global Fund Audit Report, 2016). If money isn’t reaching the people it’s meant to help, is it really the best use of resources? And what about leadership changes? Policies shift unpredictably here. A national animal welfare initiative I supported lost momentum when political priorities changed. How does EA factor in these uncertainties when prioritizing causes? It feels like EA assumes a stable world where money always achieves the intended impact. But what if that’s not the world we live in? Long termism. A Luxury When the Present Is in Crisis? I get why long termists argue that future people matter. But should we really prioritize them over people suffering today? Long termism tells us that existential risks like AI could wipe out trillions of future lives. But in Uganda, we’re losing lives now—1,500+ die from rabies annually (WHO, 2021), and 41% of children suffer from stunting due to malnutrition (UNICEF, 2022). These are preventable d
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
Recent opportunities in Building effective altruism
37
AIS Netherlands is looking for a Founding Executive Director (EOI form)
7 authors
· · 5m read
4
4
12
AI Safety Collab 2025 - Local Organizer Sign-ups Open
3 authors
· · 2m read
0
0
17
There are a lot of upcoming retreats/conferences between March and July (2025)
2 authors
· · 1m read
2
2
View more