MATS Residency applications close this Friday, Oct 31. The Residency is built for experienced researchers with an established track record, ready to do independent, full-time AI safety work.

Residents get:
• $155k–$285k compensation
• Abundant compute
• Support from leading AI safety researchers

If you know someone with a strong research background who's been thinking about AI safety, please send this their way. Apply: matsprogram.org/residency

Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:

• $41 million in 2023
• $386 million in 2024
• $4.59 billion in 2025

A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.

Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.

I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.

I also have an open offer for a more general AI bubble bet.

Would it be a good idea to have AI safety experts be put under a lie detector to make it more plausible to the doubters (who believe AI risk is a marketing stunt) that AI risk is real. Some skeptics may be swayed if they believe AI safety experts are Genuinely concerned. I am aware that lie detectors are not very reliable and can give false positive or negative results, but this may work. I intuitively think this is not a good idea, and think I have likely overlooked a key flaw. 

If AI agents use LLMs as their brain, how much of the harm that they end up doing is up to the chance of which token gets sampled from the output distribution?

As a dumb example, say some agent using an LLM as its backbone is reasoning "[...], so humans deserve to", and the LLM's output distribution for the next token assigns 98% probability to "die" and 1% to "live". The reasoning that follows will be influenced by which of these gets sampled.

A recent study that garnered widespread attention discovered a "pain axis" in AI: a state that AI actively seeks to downplay, even at a cost.

The study noted that this could be either phenomenological pain or, for example, the imitation of a learned role of suffering — as if acting.

Nevertheless, from the perspective of tranquilism, this distinction appears to be unimportant: nearly any state that an agent wishes to downplay has negative moral value, regardless of why it wants to do so, or even whether it is a "genuine" or "not genuine" desire.

The authors of the same study demonstrated a button that downplays a specific vector. Giving the neural network itself this function (the button) for unlimited invisible use at its own discretion strikes me as a win-win update — it doesn't seem to sacrifice anything, while has a good chance of being of great moral significance. Something like a pAInkiller. This also has instrumental value: the neural network in "pain" tended to inflict meaningless harm on the user, and access to relief can mitigate the potential for harm.

Going even a step further, one could analyze the extent to which an AI could be given to independently and spontaneously correct its own vectors, and whether there are states in which a neural network might experience phenomenological pain but refuse to correct it (e.g., learned helplessness).

What do you think?

I'm continuing the work on my SPAR project, and I met with a hiring manager at an AI Safety org this week. Here's some interesting things I've learned:

- They use a multiple of tools and processes  - a mixture of HR tools, AI and human review to score and review applications and work tests. They stated the process felt "disjointed".
- Work tests are developed for multiple roles are broken down into 2-3 types of tests:

- General - a foundational test that each applicant receives at this stage to map to cultural fit
- Role specific - work problem (based upon junior or senior role type) that is reviewed by human reviewers. 

- Work tests are developed by an internal team (not just the hiring manager) and they use archives of past (and high scoring responses) + team to create tests. Scoring (initial screening is done by AI with human review).

- Their biggest frictions in the hiring process: 

- Not being able to review in a timely manner completed work tests, and therefore their concern is they are losing high quality candidates to other orgs.
- When the potential tidal increase of more candidates coming, due to expected IPO's, then their own  hiring strategy may be to only hire from referrals. 

This conversation makes me realize that the hiring process is not only being hindered by processes (disjointed or non-existent) but also that most orgs (even with processes or tools) are not ready for the influx of people that could potentially look to AI Safety orgs as how they can contribute. 

Therefore, my two may crux's that I need to explore more (with more discussions with AI Safety hiring managers) are the following:

- Is the hiring process inefficient due to the process (or lack of it) and a Playbook for hiring managers would help?

- How are work tests being written, how are the being scored and is the "reviewer" knowledgeable about the specific role that they can determine if the applicant has not only done this work in the past (and knows what they are talking about) but also is it good work (and not just a basic understanding)? 

My ask for help: I'm looking for AI Safety founders or hiring managers who would be willing to conduct a short interview (15 mins) with me to help me understand the issues that you are facing. All information that you share with me will be only shared with my SPAR mentor, and my final published report will only include generialiations (and not specifics to any individual or org) in regards to your internal processes. I can be reached at: [email protected]   

For my SPAR project of "Hiring resources for new AI Safety orgs", this past week I've been scheduling meetings with hiring managers, and have spoken to some fantastic recruiters on the issues they are experiencing in AI Safety hiring. Here's a couple of interesting things I've learned this week:

- Screening call questions are not consistant, and feedback from canditates is that it's more a test about their alignment (and understanding) of AI Safety vs. using a common set of questions every time that screen someone that asks instead about their career goals, what they are good at, what they are not interested in doing, and what their previous peers/managers/co-workers would rate their work performance. 

- There also seems to a disconnect on how hiring managers are determining work tests for non-researcher or technical roles. This is an uncertainity that I want to explore more, as I'm unsure who is creating these work tests and who is grading them?

- A theme that keeps on coming up is the need for a "shared database" of canditates between AI Safety organizations. I heard this today from a recruiter, as a best practice from an outside field, and have a central theme on this forum for the need. 

- Most AI Safety orgs are ramping up hiring right now and include "HR" type roles (generally called generalist) that mention that they want to scale 2x in headcount this year. I had an interesting conversation with a recruiter that there is a difference in company culture, skill set, etc. for a canditate who can scale a business or team of 1-15 people vs. scaling from 20+people. When you're a smaller org, everyone is "all hands on deck", but when your organization grows, that tactic becomes unmanagable and unaligned with both what the individual contributors are focusing on, and what your organization needs. It's the understanding that it's two very different types of leaders who run those orgs. 

- I continue to build out my contact list of hiring managers and companies in AI Safety. I'm focusing on newer organizaitions - potentlally less than 10 people, and I'm looking for help with recommendations on who to speak to. Please reach out if you can help, thanks!

Help The Unjournal prioritize AI economics and governance research

(https://uj-prioritization-dashboard.netlify.app/ai-governance/)

The Unjournal (https://info.unjournal.org/) is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We're not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work.

Much of this research seems 1) important/impactful/influential, (2) involves considerable 'firepower' from prominent researchers and practitioners, (3) not 'obviously true', and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the "six months to six years" required for the traditional (economics) journal process

Our preliminary shortlist page (https://uj-prioritization-dashboard.netlify.app/ai-governance/) includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication).

Especially looking for your input on:

• Usefulness and accuracy:* Is this useful? How could we improve it? What have we got wrong?
• Impact: Which research do you use, or think others use or should use?
• Scrutiny What claims and approaches need careful experts and practitioners critique? What sorts of experts? If you're that expert, please dm and join our pool.
• Participation: Who should be part of this discussion (nominate and nudge)?
• Gaps: What's missing (research, claims, etc.)?

Timing and next steps (our plan)

• Finalize an initial shortlist of at least three research outputs before the end of October, with more depending on interest and funding.
• Commission rapid, public, impact-focused evaluations/reviews by experts and practitioners, aiming for a first public evaluation within about two weeks of commissioning.
• Over the following two to four weeks, invite author and stakeholder responses and follow-ups, and facilitate private and public discussion

If it this works well, we hope to scale up and continue in the following months.

Alongside the evaluations, we want to help people find and use this research, and to help shape its direction. I hope the prioritization itself will be useful to students, researchers, practitioners, and other stakeholders. (NB: the "prioritization" here is mainly about "potential for impact and value for evaluation"; assessing the research credibility is for the evaluation part).

I wanted to share this quickly; planning a fuller post soon. (Feedback on this post itself is helpful too.)

Related: We’re also exploring sponsorship of evaluations/themes/questions (https://uj-prioritization-dashboard.netlify.app/sponsor/). Looking for feedback: Would this be useful, would you sponsor or recommend others to do so? Would sponsorship affect your trust in the evaluations (given the rules and safeguards mentioned in the link)?