Alignment premium and misalignment tax

It’s interesting how it feels like in some ways there is or could be both of these;

Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.

Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.

As I have been working on SIRA (Societal Impacts Research on AI) for the past three months or so, a common sentiment has been expressed that doesn't sit well with me: that there is limited evidence of current demand for the research and talent SIRA would aim to produce. While I can see how this may be true, I'm reminded of the line attributed (probably apocryphally) to Henry Ford, "If I had asked people what they wanted, they would have said faster horses," and Steve Jobs's version, that people don't know what they want until you show it to them.

I'm using this quick take as a public note to work on this tension and look for evidence against that sentiment. My plan: (1) Make a list of "societal impact" research that I can point to and say, "Hey, look, so-and-so has done this and has addressed xyz." (2) Trace the "impact" of these works to show funders/partners that this type of work can and does make an impact. (3) Write a full blog post about this tension, pointing to this evidence.

Aimed at an audience newer to EA/AI safety

It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:

"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"

The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.

It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.

Can anyone explain the case for why the kind of AI R&D that Anthropic is doing (i.e., building more capable frontier models) reduces X-risk? I'm aware that some people think this is the case, but I don't understand why.

MATS Residency applications close this Friday, Oct 31. The Residency is built for experienced researchers with an established track record, ready to do independent, full-time AI safety work.

Residents get:
• $155k–$285k compensation
• Abundant compute
• Support from leading AI safety researchers

If you know someone with a strong research background who's been thinking about AI safety, please send this their way. Apply: matsprogram.org/residency

Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:

• $41 million in 2023
• $386 million in 2024
• $4.59 billion in 2025

A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.

Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.

I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.

I also have an open offer for a more general AI bubble bet.

Update (Friday, October 30th at 5:45pm Eastern):

Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion

Would it be a good idea to have AI safety experts be put under a lie detector to make it more plausible to the doubters (who believe AI risk is a marketing stunt) that AI risk is real. Some skeptics may be swayed if they believe AI safety experts are Genuinely concerned. I am aware that lie detectors are not very reliable and can give false positive or negative results, but this may work. I intuitively think this is not a good idea, and think I have likely overlooked a key flaw. 

If AI agents use LLMs as their brain, how much of the harm that they end up doing is up to the chance of which token gets sampled from the output distribution?

As a dumb example, say some agent using an LLM as its backbone is reasoning "[...], so humans deserve to", and the LLM's output distribution for the next token assigns 98% probability to "die" and 1% to "live". The reasoning that follows will be influenced by which of these gets sampled.