Quick takes

With all the recent media attention on EA, I think what's really unclear to outsiders is the relationship between EA and Anthropic (rightfully so!). To most people it doesn't make sense that a community so focused on AI safety can also be linked to one of the leading companies. I'd personally be very interested to see a survey of what EAs think of Anthropic, but maybe that's a minefield.. Even a good explainer meant for the public could go a long way.

Some quick observations on the attacks on METR, EA, and other AI safety orgs (from my perspective as a political campaigner and former Communications Director for a union and mayor during COVID): 1. If you're explaining, you're losing: Every rebuttal repeats the accusation to people who haven't heard it yet. Answer the accusation once, link to it, and stop explaining. This is the "illusory truth" effect (information is true when it's repeated). 2. Silence doesn't work either: Not responding on the record suggests something is being hidden. Make one clear statement rather than amplifying in multiple twitter threads. 3. Concede what you can: A concept I know from political comms is the "admission against interest". This buys trust, disarms cynical voters, and establishes credibility. Saying "yes, AND here's what we're doing about it" will take some air out of the critique. 4. Messenger > message: Ideally people who know the organization being attacked, are from outside the AI safety world, have independent standing, or better yet, have disagreed with the organization publicly will land the defense better. 5. Going after people's personal lives or spouses is overreach: Ordinary people will obviously find this unfair and it's where their attacks will backfire. 6. Don't quote tweet and rebut: You're just distributing their message for them. I want to write something longer on this but don't have time right now. Lots of opportunity for good crisis comms here. Let me know what's wrong about this take and if you'd like to hear more!

The "Desperate" AI Safety Talent Bottleneck is deeply misleading. Please.stop. I worked at Google. Google does not tell applicants that there is a talent bottleneck and they are desperately hiring. They say "we're cool, join us!"  I didn't get into Harvard.  Harvard does not tell applicants that they are desperately seeking students. I am not misled.  I've gotten rejected from countless AI Safety organizations. AI Safety tells applicants - we are in desperate need of talent (operators/generalists)! Please join us!  I've coached 70+ aspiring career pivoters on navigating the ecosystem. A common theme is the dejection & disappointment from all this rejection. Because of this false marketing, I am the one picking up the pieces to calibrate professionals on how difficult, competitive, and picky organizations are, how to build context, and how to endure the marathon, which is like any other job hunt.  Please.stop. Some alternatives: * We are an exciting, growing organization and you should apply for our roles! * Many people are excited to work for us - here are qualities of candidates we're especially excited about. * EA roles are competitive, and we would still love to see your application. Relevant posts: * https://forum.effectivealtruism.org/posts/B6d8Wzk4gNzHsXvdi/ai-safety-is-extremely-bottlenecked-on-grantmakers?commentId=n2Rd7RR4y9EnKP42F * https://forum.effectivealtruism.org/posts/b82SLXwEHRCs3TFJA/why-experienced-professionals-fail-to-land-high-impact-roles * https://forum.effectivealtruism.org/posts/jmbP9rwXncfa32seH/after-one-year-of-applying-for-ea-jobs-it-is-really-really 

Tentative thesis: China is unlikely to accept a subordinate position in AI capabilities to the US, just as the US is unlikely to accept a subordinate position in AI capabilities to China or anyone else.  This thesis suggests that there are two likely paths for the future international AI (non)regulation: 1) a continuing race between the US and China, perhaps joined by some late entrants, possibly with some rules established (e.g. mutual ban on autonomous weapons or something like that), or 2) a “pause or stop deal” that will establish some sort of ceiling on capabilities, which will be equal for China and the US. What I think is far less likely is a deal in which China would accept lower AI capabilities than the US. I simply don’t see a good reason why Chinese leaders would accept subordinate position. They know that China has the ability to catch-up to the US in various technological domains, as evidenced by the fact that in, like, 1990, China was behind in more or less everything, and now they are pushing the technological frontier in many areas.  Some estimates floating around the web suggest that, if the US would stop AI development now, China could reach current US capabilities within months. That is maybe overly optimistic/pessimistic depending on where you stand, but I very much doubt that catch-up would take more than a decade. And Chinese government, being patriotic about the abilities of the Chinese nation, probably will not have an absurdly pessimistic estimate. Moreover, the Chinese regime is in many ways oriented around this idea of catching up (this is a big difference between China and EU). It is a central plank of the official historical narrative of the People’s Republic of China (see for example preamble to their constitution: https://english.www.gov.cn/archive/lawsregulations/201911/20/content_WS5ed8856ec6d0b3f0e9499913.html) that the century between First Opium War and the Communist victory in the Chinese civil war in 1949 was the century

AI safety needs people everywhere but quickly stated, current talent bottlenecks to me look like: -- Founders -- Grantmakers -- (technical) Research leads  -- Policy entrepreneurs and implementors (which includes a lot of technical work) -- bets in international coordination and/or cooperation -- All manner of supporting talent -- program leads, ops proper, public outreach, content creators, comms   Most sought-after qualities for talent are: -- context, mission alignment, domain understanding, sophisticated views on AI strategy and threat modelling etc. -- "good judgement", "sound epistemics", "reasoning transparency" and other similar ideas/meta-skills from the EA/rationalist cannon -- a willingness to get shit done/bias for action (rather than be in learning mode, or people who need a lot of management and oversight, or folks with too many preferences/constraints) -- low ego, similar to above -- ambitious folks, since they would be really trying to be their own managers, take on bigger projects, grow themselves and their teams etc.    Finally, even having these, it's not enough to just claim to have these; job-seekers mainly trip up in being able to demonstrate and be legible about having them.

I think most AI safety bootcamps could be improved significantly by shifting focus off from coding. None of the researchers I know code, and I think the counterfactual activity of reading / thinking high-level about concepts is better for building context and conceptual thinking. As well as this, the technical AI safety pipeline serves more than just technical research roles (people transition into startups, grantmaking, other non-technical roles), and I think context building serves all of these roles better than coding. I think a default for a lot of bootcamps is to follow the ARENA curriculum, which is in parts outdated or marginally bad (eg I think learning about the specifics of how SAEs work, instead of the huggingface incident is clearly not optimal).

tl;dr: Stop The AI Race is organizing some rapid-response protest march this Thursday 9am-12pm in San Francisco, calling the Mayor of San Francisco and the board of supervisors to declare an AI Emergency and enforce a pause of frontier AI model development in San Francisco Over the past week, we have seen Dario Amodei and Sam Altman advocate for government intervention in pacing the frontier. However, the US Government has yet to answer their calls. Which is why on Thursday at 9am Stop The AI Race will be organizing another protest march, asking local authorities to step in. We will be marching from OpenAI to Anthropic to City Hall, asking AI company employees to join us in calling on the Mayor of San Francisco and the Board of Supervisors to declare an AI Emergency and enforce a full pacing of frontier AI model development in San Francisco. Schedule: - 9am: Rally at OpenAI - 10-10:45am: March To Anthropic - 10:45am: Rally at Anthropic - 11am-11:30am: March to City Hall - 11:30am: Rally at City Hall Sign up here.

A lot of AI safety people I know are excited about Anthropic and I don't fully understand why. My instinct is to distrust company because they're the one pushing the AI arms race, autonomously hacking three companies, and IPO'ing, but am likely missing something because of how respected they are within this space. Are there examples where they have counterfactually produced some result, policy, or finding that has slowed down capabilities progress more than they have themselves pushed capabilities?