An existential risk is a risk that threatens the destruction of the long-term potential of life.[1] An existential risk could threaten the extinction of humans (and other sentient beings), or it could threaten some other unrecoverable collapse or permanent failure to achieve a potential good state. Natural risks such as those posed by asteroids or supervolcanoes could be existential risks, as could anthropogenic (human-caused) risks like accidents from synthetic biology or unaligned artificial intelligence.
Estimating the probability of existential risk from different factors is difficult, but there are some estimates.[1]
Some view reducing existential risks as a key moral priority, for a variety of reasons.[2] Some people simply view the current estimates of existential risk as unacceptably high. Other authors argue that existential risks are especially important because the long-run future of humanity matters a great deal.[3] Many believe that there is no intrinsic moral difference between the importance of a life today and one in a hundred years. However, there may be many more people in the future than there are now. Given these assumptions, existential risks threaten not only the beings alive right now, but also the enormous number of lives yet to be lived. One objection to this argument is that people have a special responsibility to other people currently alive that they do not have to people who have not yet been born.[4] Another objection is that, although it would in principle be important to manage, the risks are currently so unlikely and poorly understood that existential risk reduction is less cost-effective than work on other promising areas.
In The Precipice: Existential Risk and the Future of Humanity, Toby Ord offers several policy and research recommendations for handling existential risks:[5]
Very low confidence in this take, but I suspect that in university groups, Better Futures fieldbuilding looks much better under short timelines than technical safety research fieldbuilding. As far as I can tell, the barriers to doing important stuff in that field are much lower.
I've been thinking about how to stockpile masks in India. I plan to take this on full-time in a couple of months, but I thought I'd share a working list of questions relevant to the project, if anyone else is able to make progress on them before I can. You're also welcome to suggest questions, and please write to me at samarthsrinivas1@gmail.com if you're interested in this project! I've spoken to people at Blueprint Biosecurity and BlueDot Impact, and (the lack of) people working on this seems to be the primary bottleneck
I'm still looking for someone who's willing to bet me (for a nominal amount of money for charity) on whether the AI industry is currently in a bubble that will pop within the next ~7 years: https://forum.effectivealtruism.org/posts/eCWDZb4rcCjiCtrt7/if-you-re-confident-there-s-no-ai-bubble-consider-betting-me
I think AI is in a bubble that will pop within ~7 years, so I'm looking for someone who will take the other side, who strongly believes AI is not in a bubble.
If multiple people end up accepting, that's good too, since it's a small amount of money for charity.
This is one of the more objective, more easily resolvable bets that can be made about near-term general AI capabilities progress. Moreover, unlike, say, bets on many LLM benchmarks, the outcome can't easily be gamed by AI companies in a way that's antithetical to the spirit of what the bet was trying to capture.
Check out the bet! If you don't like any of the details, propose a change and I'll seriously consider it.
This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista.
When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.
Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it[3]. I figured that all these super smart, status-y types had it under control already.
The EMH is trivially false across all domains
In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.
I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:
Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.
And then of course something deeply upsetting happened. I realised that I was not following my own advice.
So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.
Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.
I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:
experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius
I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.
Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.
I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.
Despite being motivated, I do think there’s some legitimacy to the claim regardless.
It was my sixth EAG, and the other five had all been dominated by the topic.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.
I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.
I also have an open offer for a more general AI bubble bet.
Update (Friday, October 30th at 5:45pm Eastern):
Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
Would it be a good idea to have AI safety experts be put under a lie detector to make it more plausible to the doubters (who believe AI risk is a marketing stunt) that AI risk is real. Some skeptics may be swayed if they believe AI safety experts are Genuinely concerned. I am aware that lie detectors are not very reliable and can give false positive or negative results, but this may work. I intuitively think this is not a good idea, and think I have likely overlooked a key flaw.
Help The Unjournal prioritize AI economics and governance research
(https://uj-prioritization-dashboard.netlify.app/ai-governance/)
The Unjournal (https://info.unjournal.org/) is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We're not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work.
Much of this research seems 1) important/impactful/influential, (2) involves considerable 'firepower' from prominent researchers and practitioners, (3) not 'obviously true', and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the "six months to six years" required for the traditional (economics) journal process
Our preliminary shortlist page (https://uj-prioritization-dashboard.netlify.app/ai-governance/) includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication).
Especially looking for your input on:
Timing and next steps (our plan)
If it this works well, we hope to scale up and continue in the following months.
Alongside the evaluations, we want to help people find and use this research, and to help shape its direction. I hope the prioritization itself will be useful to students, researchers, practitioners, and other stakeholders. (NB: the "prioritization" here is mainly about "potential for impact and value for evaluation"; assessing the research credibility is for the evaluation part).
I wanted to share this quickly; planning a fuller post soon. (Feedback on this post itself is helpful too.)
Related: We’re also exploring sponsorship of evaluations/themes/questions (https://uj-prioritization-dashboard.netlify.app/sponsor/). Looking for feedback: Would this be useful, would you sponsor or recommend others to do so? Would sponsorship affect your trust in the evaluations (given the rules and safeguards mentioned in the link)?
Governor Newsom’s Nov. 18 legislative recommendations are likely our best chance to stop RSI, and California residents can have a huge impact by lobbying Newsom and their state representatives to act on them. I’m coordinating a grassroots campaign urging Newsom to call a special session of the legislature starting December 7. We’re asking the legislature to pass the report’s recommendations into law, and go further than the report, which only recommends capping automated R&D at some future point, by enacting a cap that takes effect immediately.
December 7th is obviously not that far away, so we need to act quickly. Find details and call scripts at aispecialsession.org and please spread the word!