Featured
New & upvoted
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
PauseAI UK is holding a March Against AI Extinction on 21st November.
It will be the largest demonstration for AI safety in the world, by a substantial margin. I really hope that almost everyone in the London AI safety community will be there. If you know people in the UK, please ask them to go.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.
I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.
I also have an open offer for a more general AI bubble bet.
Update (Friday, October 30th at 5:45pm Eastern):
Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
Alignment premium and misalignment tax
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
I read this as a call to action, and I appreciate that motivation is probably needed. But I think I need more convincing that this nut can really be cracked.
Renewables is a fair comparison to aspire to, but there are also many technologies that show the opposite trend; they appear promising at first, there's some hype, and then they fail to scale. For example, cellulosic ethanol appeared super promising at first, was successfully produced, received significant investment, yet still failed to scale. GM crops are an example of a tech that's actually scalable, yet has just been banned in Europe - for reasons that are pretty likely to transfer over to cultivated meat.
I haven't read into cultivated meat for a while, but my understanding was that there are several genuinely difficult barriers to scale - contamination risk, environmental impacts, getting the structure of the meat right, and getting the price down. I'd love a post that addressed these and explained why we should still be optimistic, rather than why we need to be.
If you have engaged with Anthony's sequence, and you hadn't thought seriously about cluelessness before, I'd be interested in knowing if it changed your mind on your cause prioritisation. You can answer as a reply, or just vote:
Note: I couldn't post this at the time because there was a bug, but I'd be keen to hear responses now as well.
(We've talked about most of this outside of the public view, but I can't restrain myself from commenting none-the-less)
I agree with you strategically that people who care about mental health need to make their case in a way that's legible to more traditional metrics like income, DALYs, etc. But the general thing this points to is greater legibility --> greater uptake --> impact blah blah.
On this general point, one thing we've been thinking about and piloting is just trying to make WELLBYs more legible, especially for MH interventions by converting them to years of depression averted (YODAs).
So I could say that a fictional charity Gaya Kuides generates 1,000 WELLBYs per $1k donated. Or given that the typical case of depression (moderate depression) is associated with a -1.3. lower wellbeing compared to low depression symptoms -- this implies that actually GK averts the equivalent of 1000 / 1.3 =769 years of depression per $1k donated. FWIW we explain the YODAs more here and we're looking for feedback on whether this seems like a useful communication device.
It just seems way easier to imagine the value of a year of depression averted compared to "WELLBY".
2. Long-run effects of psychotherapy
This is very important and understudied. The effects of psychotherapy could last much longer, and we could maybe make these interventions benefit people for much longer than we. But we don't care much about LR effects in general in the research community and we do not have much understanding of the mechanisms that support long-term benefits. I obviously wish that we'd change on both accounts.
[Edit: Giving Season plans are live, but I'm still taking suggestions, here or in the comments on the post]
Aiming to finalise plans for Giving Season this week. I'm not running a donation election (for reasons I'll explain in the announcement), which gives us some more space to focus directly on having important conversations to do with funding and resource allocation in the movement.
I'll run a marginal funding week (week starting 16 Nov - message me if you'd like to take part), and a debate week (planning it for week starting Nov 30). I reckon I have capacity for two more themed weeks, as well as AMAs etc...
What would you like to see us run over Giving Season?