Featured
New & upvoted
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
I put a little poll here to follow up on whether minds were changed by the cluelessness critiques competition. If you read the posts during that week, check it out.
Alignment premium and misalignment tax
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
FWIW, I think there are some comms risks here — I've noticed that Claude sites in particular start looking really same-y without a good design eye editing them. The new User-Friendly site I would have immediately clocked as Claude-made (and applied some automatic discount to it based on this had you not said it, based on things like):
Little glyphs and words above headings

Random italicized words for emphasis

Using 01, 02, 03 in things that don't need numbering, with the leading 0.

Along with the neutral color palette.
Here are some examples of sites that are clearly made by Claude, and have a lot of these features:
Claude Academy's example websites also look this way: https://academy.claude.com/use-cases/create-a-custom-webpage
I used Claude design with just this prompt: "Can you make me some slides? Don't use my design system - just use whatever you feel like. Fill them in with filler content."
Here are some screenshots of that:


I think that working with a good designer (like you!) seems great for folks doing this, but also, if we're already concerned about EA websites being really same-y, this poses the same kind of risk, or at least for me, as someone who has starting feeling negative about seeing endless amounts of Claude-designed things.
There's a quote in the post:
Some effective altruists expect artificial general intelligence (AGI) or transformative AI (TAI) to solve the taste and price challenges for alternative meats, and so they wonder: What's the point of work on alternative meats now?
I'd be interested to learn more about the reasoning behind this view. So far I'm only familiar with [1](https://www.lesswrong.com/posts/DuCWLYx4jAFPMXKjZ/paper-artificial-intelligence-in-life-extension-from-deep) and [2](https://forum.effectivealtruism.org/posts/TYhs8zehyybvMt5E4/).
And I have an intuition (possibly because I'm unintentionally making a strawman out of the actual position simply because I don't understand it) that the "we need AGI to solve this problem" relies on the assumption that "AGI is a thing that will solve basically all existing problems simply because it's generally much better at solving problems than we are". And the implicit conclusion is "so we don't need to think about how exactly AGI will solve this particular problem".
It seemed to me that the post was arguing against this approach.
And I actually want to argue even further:
I think we somewhat understand the problem at the gears-level where we can build models and test them against reality, given that we have narrow AI with today's SOTA capabilities at our disposal. Current models have already shown themselves to be effective in some kinds of scientific research (mostly when the task is formulated in a specific, AI-tractable way).
And overall I assign a higher probability to A than B.
And I assign a substantially higher probability to A than B. (And I'm not even talking about existential risks here, but never mind.)
Is it really true that my view is far from consensus? I thought pretty much everyone thought roughly along these lines, but after reading this post I'm somehow not so sure anymore.
If you have engaged with Anthony's sequence, and you hadn't thought seriously about cluelessness before, I'd be interested in knowing if it changed your mind on your cause prioritisation. You can answer as a reply, or just vote:
Note: I couldn't post this at the time because there was a bug, but I'd be keen to hear responses now as well.
It looks to me like OpenAI's latest model spec is the first version which explicitly asks its models to weigh animal welfare in their responses.
It’s encouraging to see OpenAI’s model spec joining Claude’s constitution in explicitly weighing animal welfare.