People in AI safety/EA spheres should reorient now towards stopping continued AI capabilities escalation. 

Historically, people have been unwilling to straightforwardly say “This AI situation is disgustingly dangerous, and we need to stop” and then actually work towards making this happen. I think a lot of this is downstream of deference to high status people who weren’t willing to take positions that seemed extreme. 

The current situation is very, very bad, and there is no plan to make it better if we continue increasing AI capabilities at the current rate. 

We need to stop things as soon as possible, so the community and the world can orient and work out what to do. We don't need to have the full plan yet, we just need to get to a state where we aren't in imminent danger. 

I predict that this position will become increasingly obvious, and people will wish they reoriented earlier. I think this will be clear ex ante; when people look back they will think that they should have reoriented earlier, given the information they had at the time. 

Regarding "the organizations with the most money share the fewest details":

• I believe that AI Safety Camp commissioned that independent analysis because their fundraising was largely unsuccessful. The better-funded organizations didn't have a strong incentive to commission an independent analysis! This is a clear selection effect.
• I don't think that an independent analysis is necessarily better than the default. Coefficient Giving has conducted several highly detailed internal investigations into MATS' impact, for example, though these were never shared publicly. I expect that cG did the same for their other large grantees, which contributed to their confidence.
• Note that MATS has the most money of these programs and we try to share a lot of data. We could definitely improve, though! Let me know what data would be interesting/useful.

I've had a vague sense that the EA Forum is declining in quality posts/conversation despite having lots of posts, so had an AI tool (Astra) do a bunch of random analyses to look at various measures of activity level, and I think these vaguely confirm my sense. In particular, there has been a decline in commenting from higher karma users, and fewer posts are getting lots of upvotes, despite lots of people still posting. As a lover of the Forum, this seems bad and sad.

Quick rambly comment from my side (because I didn’t expect this to be spread outside of the publication and wasn’t aware of timelines :))

Most of the writing is done by Dirk-Jan. I gave input and part of this followed from a conversation we had on stage during EAGxAmsterdam.

I think Dirk-Jan is a rare example of an experienced professional from the ODE field who is willing to engage critically with EA ideas in good faith, and I am very happy we started this discussion :) As a community, I think we should cherish encounters with constructive outside experts

My quick take if you need to take one thing away from this thought-piece: we need to make sure that, as a community, we incentivize careful analysis of unintended consequences, and then update accordingly! The incentive is to write away unintended effects qualitatively, or handwave a modest discount (or bonus, for positive effects)

When you start a project/organisation, it makes sense to put most effort into making your theory of change very rigorous and de-risk it as much as possible. Once you scale that project/organisation, you should probably put the same effort on analyzing whether the causal links in your ToC could break-down at scale, and pay attention to the causal effects that fall outside your ToC all together! 

The best version of this, as always, employs a good mix of qualitative and quantitative thinking. If anything, I personally think discussions on unintended effects and “backfire risks” are often too qualitative, and miss a serious attempt at sizing.

Lastly, a prompt everyone can use:

• What incentives are you creating with this project, especially at scale/maturity?
• How will insiders and outsiders perceive your work when you scale it?
• And the memeable: “is there anyone you forgot to ask?!” or “who’s left out of this project, but will still experience consequences”
• Are you crowding anyone out?
• Are there (indirect) costs you are incurring on people?
  
   

A lot of AI safety people I know are excited about Anthropic and I don't fully understand why. My instinct is to distrust company because they're the one pushing the AI arms race, autonomously hacking three companies, and IPO'ing, but am likely missing something because of how respected they are within this space. Are there examples where they have counterfactually produced some result, policy, or finding that has slowed down capabilities progress more than they have themselves pushed capabilities? 

AI safety needs people everywhere but quickly stated, current talent bottlenecks to me look like:

-- Founders

-- Grantmakers

-- (technical) Research leads 

-- Policy entrepreneurs and implementors (which includes a lot of technical work)

-- bets in international coordination and/or cooperation

-- All manner of supporting talent -- program leads, ops proper, public outreach, content creators, comms

Most sought-after qualities for talent are:

-- context, mission alignment, domain understanding, sophisticated views on AI strategy and threat modelling etc.

-- "good judgement", "sound epistemics", "reasoning transparency" and other similar ideas/meta-skills from the EA/rationalist cannon

-- a willingness to get shit done/bias for action (rather than be in learning mode, or people who need a lot of management and oversight, or folks with too many preferences/constraints)

-- low ego, similar to above

-- ambitious folks, since they would be really trying to be their own managers, take on bigger projects, grow themselves and their teams etc. 

Finally, even having these, it's not enough to just claim to have these; job-seekers mainly trip up in being able to demonstrate and be legible about having them.