Quick takes

This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista.  When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway. Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it[3]. I figured that all these super smart, status-y types had it under control already.                                           The EMH is trivially false across all domains In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well. I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved g

Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was: * $41 million in 2023 * $386 million in 2024 * $4.59 billion in 2025 A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply. Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027. I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice. I also have an open offer for a more general AI bubble bet. ---------------------------------------- Update (Friday, October 30th at 5:45pm Eastern): Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion

I'm still looking for someone who's willing to bet me (for a nominal amount of money for charity) on whether the AI industry is currently in a bubble that will pop within the next ~7 years: https://forum.effectivealtruism.org/posts/eCWDZb4rcCjiCtrt7/if-you-re-confident-there-s-no-ai-bubble-consider-betting-me I think AI is in a bubble that will pop within ~7 years, so I'm looking for someone who will take the other side, who strongly believes AI is not in a bubble. If multiple people end up accepting, that's good too, since it's a small amount of money for charity. This is one of the more objective, more easily resolvable bets that can be made about near-term general AI capabilities progress. Moreover, unlike, say, bets on many LLM benchmarks, the outcome can't easily be gamed by AI companies in a way that's antithetical to the spirit of what the bet was trying to capture. Check out the bet! If you don't like any of the details, propose a change and I'll seriously consider it.

Tentative thesis: China is unlikely to accept a subordinate position in AI capabilities to the US, just as the US is unlikely to accept a subordinate position in AI capabilities to China or anyone else.  This thesis suggests that there are two likely paths for the future international AI (non)regulation: 1) a continuing race between the US and China, perhaps joined by some late entrants, possibly with some rules established (e.g. mutual ban on autonomous weapons or something like that), or 2) a “pause or stop deal” that will establish some sort of ceiling on capabilities, which will be equal for China and the US. What I think is far less likely is a deal in which China would accept lower AI capabilities than the US. I simply don’t see a good reason why Chinese leaders would accept subordinate position. They know that China has the ability to catch-up to the US in various technological domains, as evidenced by the fact that in, like, 1990, China was behind in more or less everything, and now they are pushing the technological frontier in many areas.  Some estimates floating around the web suggest that, if the US would stop AI development now, China could reach current US capabilities within months. That is maybe overly optimistic/pessimistic depending on where you stand, but I very much doubt that catch-up would take more than a decade. And Chinese government, being patriotic about the abilities of the Chinese nation, probably will not have an absurdly pessimistic estimate. Moreover, the Chinese regime is in many ways oriented around this idea of catching up (this is a big difference between China and EU). It is a central plank of the official historical narrative of the People’s Republic of China (see for example preamble to their constitution: https://english.www.gov.cn/archive/lawsregulations/201911/20/content_WS5ed8856ec6d0b3f0e9499913.html) that the century between First Opium War and the Communist victory in the Chinese civil war in 1949 was the century

AI safety needs people everywhere but quickly stated, current talent bottlenecks to me look like: -- Founders -- Grantmakers -- (technical) Research leads  -- Policy entrepreneurs and implementors (which includes a lot of technical work) -- bets in international coordination and/or cooperation -- All manner of supporting talent -- program leads, ops proper, public outreach, content creators, comms   Most sought-after qualities for talent are: -- context, mission alignment, domain understanding, sophisticated views on AI strategy and threat modelling etc. -- "good judgement", "sound epistemics", "reasoning transparency" and other similar ideas/meta-skills from the EA/rationalist cannon -- a willingness to get shit done/bias for action (rather than be in learning mode, or people who need a lot of management and oversight, or folks with too many preferences/constraints) -- low ego, similar to above -- ambitious folks, since they would be really trying to be their own managers, take on bigger projects, grow themselves and their teams etc.    Finally, even having these, it's not enough to just claim to have these; job-seekers mainly trip up in being able to demonstrate and be legible about having them.

I thought Ezra's feature on recursive self-improvement was really valuable in a number of ways: 1. Generally well-done pedagogically, explaining things in a way that non AI-pilled people can understand and doing a lot of preemptive objection-handling. 2. Making the case that many of the beliefs about x-risks were held for a decade plus, before people involved had the material interests they have now. 3. Explaining well why the perceptions of AI risks diverge between people at the frontier and everyday people using ChatGPT to improve emails. 4. Explaining why recursive self-improvement is such a crucial benchmark and why going marginally slower is not a satisfactory response. 5. Defining a policy ask (control) that is broadly legible and easier to grasp than alignment. 6. Striking a balance of calling different groups out on their respective BS (the AI companies on the impossibility of defining policy responses to RSI, critics who say this is all PR, people who are too certain). Obviously a carefully crafted piece of messaging and positioning, but a pretty useful one.

Mildly against the Longtermism --> GCR shift Epistemic status: Pretty uncertain, somewhat rambly TL;DR replacing longtermism with GCRs might get more resources to longtermist causes, but at the expense of non-GCR longtermist interventions and broader community epistemics Over the last ~6 months I've noticed a general shift amongst EA orgs to focus less on reducing risks from AI, Bio, nukes, etc based on the logic of longtermism, and more based on Global Catastrophic Risks (GCRs) directly. Some data points on this: * Open Phil renaming it's EA Community Growth (Longtermism) Team to GCR Capacity Building * This post from Claire Zabel (OP) * Giving What We Can's new Cause Area Fund being named "Risk and Resilience," with the goal of "Reducing Global Catastrophic Risks" * Longview-GWWC's Longtermism Fund being renamed the "Emerging Challenges Fund" * Anecdotal data from conversations with people working on GCRs / X-risk / Longtermist causes My guess is these changes are (almost entirely) driven by PR concerns about longtermism. I would also guess these changes increase the number of people donation / working on GCRs, which is (by longtermist lights) a positive thing. After all, no-one wants a GCR, even if only thinking about people alive today. Yet, I can't help but feel something is off about this framing. Some concerns (no particular ordering): 1. From a longtermist (~totalist classical utilitarian) perspective, there's a huge difference between ~99% and 100% of the population dying, if humanity recovers in the former case, but not the latter. Just looking at GCRs on their own mostly misses this nuance. * (see Parfit Reasons and Persons for the full thought experiment) 2. From a longtermist (~totalist classical utilitarian) perspective, preventing a GCR doesn't differentiate between "humanity prevents GCRs and realises 1% of it's potential" and "humanity prevents GCRs realises 99% of its potential" * Preventing an extinction-level GCR might move u

In two days (March 21st, 12-4pm), about 140 of us (event link) will be marching on Anthropic, OpenAI and xAI in SF asking the CEOs to make statements on whether they would stop developing new frontier models if every other major lab in the world credibly does the same. This comes after Anthropic removed its commitment to pause development from their RSP. We'll be starting at 500 Howard St, San Francisco (Anthropic's Office, full schedule and more info here). This is shaping to be the biggest US AI Safety protest to date, with a coalition including Nate Soares (MIRI), David Krueger (Evitable), Will Fithian (Berkeley Professor) and folks representing PauseAI, QuitGPT, Humans First.