>I made a 13x13" hole in my wall
I was away for the day and when I got home, he said "I made a hole in our wall but it can be refilled."

With all the recent media attention on EA, I think what's really unclear to outsiders is the relationship between EA and Anthropic (rightfully so!). To most people it doesn't make sense that a community so focused on AI safety can also be linked to one of the leading companies. I'd personally be very interested to see a survey of what EAs think of Anthropic, but maybe that's a minefield.. Even a good explainer meant for the public could go a long way.

Why am I not bullish on civilizational refuges for reducing bio xrisk? I believe bio xrisk is concentrated in E2H scenarios or AI takeover, and neither of these things benefit much from refuges. For both of these threat models, there is value in refuges but only if they are a safe staging area to take back control of the environment or against an AI, which means you probably need a full industrial base worth of people/infrastructure to be protected. Society-wide biohardening and PPE is therefore the better option even on purely longtermist grounds (and has the additional property of also being much more valuable on non-longtermist grounds as well!)

Wanted to share a quick update on my comments on AI safety comms from a couple weeks ago. The NY Post is now reporting that the FTC is investigating OpenAI, Anthropic and METR, with formal demands (similar to subpoenas) reportedly coming in the next few weeks.

I think Brian Martin's Backfire framework could be useful in responding here. Martin found that power holders use five methods to suppress outrage, and there's something we can do in response to make it backfire:

• They cover it up → We reveal: Most of what they do will happen out of the public view (though they seem to be more emboldened and this is less true). We can document their actions starting with the narrative they've been advancing on X to the Post stories to the eventual probe so the general public can see a pattern.
• Devalue → Redeem: They've called METR a "handpicked watchdog," they've branded EA a cult, they've attacked Lighthaven etc. We can counter their attacks using people outside of the community, who the public respects, vouching for METR's work.
• Reinterpret → Reframe: Their official line is that this is neutral fact-finding. We can name what's actually happening: they're investigating the group that published an independent report on AI agents getting out of a test environment because they want to continue accelerating A(S)I development.
• Send it to official channels → Redirect: They'll say "it's under investigation", which sounds neutral and ends the conversation. METR should not go quiet while it waits for the official action. Keep making the case in public that this is important work and explain why (without responding to the attack).
• Intimidate → Resist: This may have a chilling effect on every other evaluator, whether anyone intends it or not. Other evaluators and researchers can say publicly that they'll keep publishing their work, speaking out, and will stand with METR.

I've worked on a few similar cases in the past two years, mostly in the pro-democracy space, including the FBI's arrest of Milwaukee County Judge Hannah Dugan. We built a coalition beyond the usual suspects and brought her case to the public with polling that showed a clear message to the general public. We hosted a vigil before her trial and a "Court of Public Opinion" outside the courthouse each time she appeared before a judge.

Making this backfire will be easier to do now, before the attack lands. Crisis comms is 95% preparation and relationships, 5% the crisis itself.

Would love to talk with more people about this!

*AI use: I used AI to tighten up the backfire bullets and intend to lengthen this piece into a longer one.

Tentative thesis: China is unlikely to accept a subordinate position in AI capabilities to the US, just as the US is unlikely to accept a subordinate position in AI capabilities to China or anyone else. 

This thesis suggests that there are two likely paths for the future international AI (non)regulation:

1) a continuing race between the US and China, perhaps joined by some late entrants, possibly with some rules established (e.g. mutual ban on autonomous weapons or something like that), or

2) a “pause or stop deal” that will establish some sort of ceiling on capabilities, which will be equal for China and the US.

What I think is far less likely is a deal in which China would accept lower AI capabilities than the US.

I simply don’t see a good reason why Chinese leaders would accept subordinate position. They know that China has the ability to catch-up to the US in various technological domains, as evidenced by the fact that in, like, 1990, China was behind in more or less everything, and now they are pushing the technological frontier in many areas. 

Some estimates floating around the web suggest that, if the US would stop AI development now, China could reach current US capabilities within months. That is maybe overly optimistic/pessimistic depending on where you stand, but I very much doubt that catch-up would take more than a decade. And Chinese government, being patriotic about the abilities of the Chinese nation, probably will not have an absurdly pessimistic estimate.

Moreover, the Chinese regime is in many ways oriented around this idea of catching up (this is a big difference between China and EU). It is a central plank of the official historical narrative of the People’s Republic of China (see for example preamble to their constitution: https://english.www.gov.cn/archive/lawsregulations/201911/20/content_WS5ed8856ec6d0b3f0e9499913.html) that the century between First Opium War and the Communist victory in the Chinese civil war in 1949 was the century of humiliation, during which various imperialist powers exploited China due to the fact that it was technologically backward country unable to stand up to them. I happen to think this narrative is largely correct, but even if it isn’t, what matters is that Chinese government believes it and is determined to never again allow China to be put in the position where a technologically superior power could walk over it.

This has some implications:

A) Since China is currently behind, any sort of “pause or stop deal” between the US and China would likely have to involve either the US stopping further development and allowing China to catch up, or the US somehow crippling its own capabilities to a current Chinese level. Of course, for either option, the US would have to place a lot of trust in the Chinese government.

 B) As long as China is behind, export controls on hardware and other attempts to limit Chinese capabilities are at best of doubtful utility as a way to get to a "pause or stop" agreement. In fact, imho there are making a deal less likely. They antagonize Chinese leaders, make them even more paranoid and less inclined to a constructive agreement. But also, to the extent they are successful in limiting Chinese capabilities, they make it less likely to that the US government would agree to an equal deal, since that would mean sacrificing larger lead, which is harder to politically accept for the US. Perhaps this is a reason why nuclear arms treaties in the Cold war were agreed only after the USSR more or less caught up to US nuclear arsenal?

Now, attempts to limit Chinese capabilities can still be seen as useful if you think it is important for the US to “win” the competition, for example because there are reasons to believe US AI will be safer. Or because you think the world would be a better place if the US “wins”, even though both countries would develop similarly dangerous systems. Or of course if you think that the US victory would be preferable even if American AI would be more dangerous than Chinese AI.

Btw. for a well written snapshot of Chinese opinion/vibes, I recommend this: https://www.chinatalk.media/p/how-anthropic-became-chinas-goliath

Start with journals: a cheap governance test for AI animal-communication tools

A recent Guardian feature on AI and animal communication brought renewed attention to the PEPP Framework (Prepare, Engage, Prevent, Protect), developed by NYU’s More Than Human Life Program for the responsible development and use of nonhuman animal communication technologies.

PEPP is quite ambitious. Its 12 principles cover risk assessment, precaution, animal autonomy and best interests, transparency, responsibility, remediation, and other issues that could become increasingly important if AI systems get better at interpreting or reproducing animal signals. The framework is currently voluntary and its authors envisage it evolving as the field develops.

My tentative view is that scientific journals may be the most promising place to start turning some of these principles into actual requirements, at least while this remains predominantly a research field.

Funders have leverage, but only over the work they fund. Research ethics bodies matter earlier in the process, although their mandates and practices differ considerably across institutions and jurisdictions. Regulation will eventually matter most for some applications, especially commercial deployment or interactions with protected wildlife, but the field may still be too immature for a detailed regulatory regime.

Journals occupy an unusual leverage. Publication is a recurring bottleneck across the research ecosystem, and journals already routinely require researchers to disclose ethical approval, methods, conflicts of interest and compliance with reporting standards. For research involving AI mediated communication, they could require authors to report things such as whether playback or generated signals altered animal behaviour, how risks to the animals were assessed, what validation methods were used, and whether adverse effects occurred.

There is also a useful precedent from animal research itself.

The ARRIVE guidelines, first published in 2010, were designed to improve reporting of animal experiments. They eventually received endorsement from more than 1,000 journals as well as funders and universities. Yet evidence of improved reporting remained limited. The lesson seems important: endorsement alone did very little. ARRIVE 2.0 explicitly identifies active involvement by journal and editorial staff as one of the key factors affecting whether reporting standards actually change practice.

A randomised trial at PLOS ONE makes the point even more clearly. Simply asking authors to submit an ARRIVE checklist, without editors checking compliance, did not improve reporting. Other interventions involving shorter requirements and greater editorial follow up performed considerably better.

That makes me think the first useful experiment for PEPP could be quite modest: take a small subset of its principles that can already be operationalised as disclosure requirements and persuade one or more journals publishing animal communication research to require them.

The effects could then actually be observed: Do researchers comply? Do the requirements reveal risks or methodological weaknesses that would otherwise remain invisible? Are some PEPP principles too vague to use this way? Which requirements create useful information, and which simply create paperwork?

Animal communication technology is probably a relatively small AI × animals issue in terms of current welfare scale. I still think it could be an unusually useful test case for a much bigger governance question: how do we move from voluntary animal inclusive principles to institutional constraints without locking in rules before we know whether they work?