Featured
New & upvoted
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
PauseAI UK is holding a March Against AI Extinction on 21st November.
It will be the largest demonstration for AI safety in the world, by a substantial margin. I really hope that almost everyone in the London AI safety community will be there. If you know people in the UK, please ask them to go.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.
I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.
I also have an open offer for a more general AI bubble bet.
Update (Friday, October 30th at 5:45pm Eastern):
Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
As a welcome back gift, @Ollie Etherington reviewed my first claude-coded feature from months ago: multiple-choice polling. It only works in comments so far (though you can still put slider polls in posts.
To add a multiple choice poll to a comment or quick take, select some text, click the three dot menu in the pop up -> poll dropdown -> select multiple choice.

They look like this:
If you have engaged with Anthony's sequence, and you hadn't thought seriously about cluelessness before, I'd be interested in knowing if it changed your mind on your cause prioritisation. You can answer as a reply, or just vote:
Note: I couldn't post this at the time because there was a bug, but I'd be keen to hear responses now as well.
(We've talked about most of this outside of the public view, but I can't restrain myself from commenting none-the-less)
I agree with you strategically that people who care about mental health need to make their case in a way that's legible to more traditional metrics like income, DALYs, etc. But the general thing this points to is greater legibility --> greater uptake --> impact blah blah.
On this general point, one thing we've been thinking about and piloting is just trying to make WELLBYs more legible, especially for MH interventions by converting them to years of depression averted (YODAs).
So I could say that a fictional charity Gaya Kuides generates 1,000 WELLBYs per $1k donated. Or given that the typical case of depression (moderate depression) is associated with a -1.3. lower wellbeing compared to low depression symptoms -- this implies that actually GK averts the equivalent of 1000 / 1.3 =769 years of depression per $1k donated. FWIW we explain the YODAs more here and we're looking for feedback on whether this seems like a useful communication device.
It just seems way easier to imagine the value of a year of depression averted compared to "WELLBY".
2. Long-run effects of psychotherapy
This is very important and understudied. The effects of psychotherapy could last much longer, and we could maybe make these interventions benefit people for much longer than we. But we don't care much about LR effects in general in the research community and we do not have much understanding of the mechanisms that support long-term benefits. I obviously wish that we'd change on both accounts.
I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.
I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:
Brian subsequently wrote the essay My Confusions about the Hard Problem of Consciousness, which I think (though Vasco disagrees) basically contradicts this one.
Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to
(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)
And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.
So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.
On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.
In the latter essay, Brian writes:
Reductionism: The world ultimately reduces to physics—quarks, leptons, etc., or probably something else more fundamental. "Consciousness" is not one of those physical primitives. Thus, consciousness must be an emergent phenomenon—a process that involves many parts working together in various ways. But if consciousness is emergent, then it must have fuzzy boundaries just like any concept does. There's no "true essence" underlying an emergent phenomenon. It just is what its components are, and we decide the boundaries of labeling which emergent phenomena belong to which categories that we make up. Consciousness must be like this.
Which Vasco approves of. I think these claims are far too confident:
Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action.
I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.
I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.
Aiming to finalise plans for Giving Season this week. I'm not running a donation election (for reasons I'll explain in the announcement), which gives us some more space to focus directly on having important conversations to do with funding and resource allocation in the movement.
I'll run a marginal funding week (week starting 16 Nov - message me if you'd like to take part), and a debate week (planning it for week starting Nov 30). I reckon I have capacity for two more themed weeks, as well as AMAs etc...
What would you like to see us run over Giving Season?