Motivation: If we want to move from Plan D to Plan A or S, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do.
Abstract:
We're plausibly only a few years from a catastrophe. Fable 5 cracked open a brief window of attention, but policymakers are still worried about the wrong risks. This is our chance to wake them up.
⚠️ Epistemic status: I have skin in this game, which is either a conflict of interest or two years of data, depending on how you see things: I run a think tank that does this type of activity, so discount accordingly. I preferred to ship quickly rather than not ship at all, or ship too late. I expect some claims not to be stable under reflection, but the core argument is one I hold with reasonable conviction. See this as a bottle in the ocean. My point is not to dunk on research. I think that research is how we keep finding unknown unknowns; nothing in this post argues for stopping it. AI safety is one of the hardest fields to navigate, and I’ve often wondered if what I do is pointless. I might be wrong about the net-positiveness of some types of AI regulations, but I feel that the level of the discourse is really bad, the conversation is not happening, and I want this conversation to happen before irreversible things start happening.
Thanks to Epi Gedeon, Ryan Greenblatt, Arthur Grimonpont, Alexandre Variengien, Jack Stennett, Lovkush Agarwal, Monika Jotautaite, and Jonathan Salter for useful feedback and suggestions.
PS. I wrote a short FAQ in the comment section one week after the publication.
This section defines what I mean by political will and then argues that we are not applying basic best practices, so having more ideas is clearly not the bottleneck.
Start with a single policymaker. They have to move along a pipeline like:
"Political will" in the aggregate is just this funnel run across the people who set, enforce, and narrate policy.
Spoiler: Level 0 sounds like a low bar. It is. The median meeting I've had with a senior policymaker starts below it. In the international forums I've attended, my rough estimate is that at least a third of the policymakers I've personally met are not even at Level 0. Domestic legislatures are a bit further along: 40 current members of US Congress have now publicly discussed AGI or loss of control, up from a handful in early 2023, doubling roughly every 5.5 months[4]. Those public discussions are at Level 0 or 1. This covers 7% of Congress. At Level 3, I count roughly 3.[5]
According to SaferAI’s rating, 35% is currently the highest overall assessment score, given to Anthropic. 59% is currently the best overall assessment score if a company adopted all the industry best practices found across companies.
Where’s my DNA synthesis screening? DNA Screening is a reasonable, long-advocated measure that was even mentioned in the American AI Action Plan, but is still not mandated. It's the canonical example of how glacially even obvious, relatively low-cost regulatory measures move forward (global synthesis map).
I like this quote from Buck Shlegeris: "Five years ago I thought of misalignment risk from AIs as a really hard problem that you'd need some really galaxy-brained fundamental insights to resolve. Whereas now, to me the situation feels a lot more like we just really know a list of 40 things where, if you did them — none of which seem that hard — you'd probably be able to not have very much of your problem. But I've just also updated drastically downward on how many things AI companies have the time/appetite to do."
The priority bottleneck is not finding more best practices: I agree we don't know robustly how to align a superintelligence - but at the same time we are not even willing to implement already existing best practices. The 80/20 playbook against scheming may not be enough, but we aren't even doing the 80/20. For example: Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes.[6]
Basic measures like transparency are still not applied: companies reporting incidents are more the exception than the norm, and we don't know how AIs are aligned concretely (we have very little confirmed public information about why frontier AIs end up being apparently behaviorally aligned). Most of those best practices are basic ideas that have been on the table from the start. And since companies don’t seem to adopt them under current competitive pressure, implementation has to come from enforcement.
I could go on and on with this, but I think this makes the point.
Greenblatt has tried to put numbers on what political will buys. He sketches a spectrum of how hard the world is trying, from Plan D, roughly today's world, where maybe ten people inside each company are trying to implement safety measures, up to Plan A, a strong international agreement with real enforcement and a slowdown. His tentative estimate: conditional takeover risk falls from ~45% (Plan D) to ~7% (Plan A). An ~84% relative cut, most of it unlocked by political will, partly directly, partly by buying the time for technical safety work to actually be applied (Redwood Research).[7]
Obviously, the exact numbers are not the point here: directionally, strong political will and taking the risks seriously would tremendously reduce them. (PS: Greenblatt recently updated, and now estimates a smaller reduction than this, even if the direction holds[8])
Aligning a superintelligence may well be genuinely hard, and even plan A might be insufficient. That's an argument for more political will, not less: it's what buys the time, and above all, the seriousness that a hard problem demands.[9]
P.S. added on the 11th July: This week, the AI Futures Project, Greenblatt included, published AI 2040: Plan A, a detailed scenario of how this could go. The whole thing hinges on a US–China agreement by 2029. The best plan the research community can produce depends on a political precondition we are far from, and I think that there are substantial ways to improve upon the baseline: one of my main critiques has always been that the framework treats political will as a fixed variable rather than a strategic lever.
There are two funnels that are mostly independent:
Both funnels matter, and I won't adjudicate between them here; but the belief funnel is the one that's barely started, and it's the one the rest of this post is about.
Why not just target Trump? Why do we need conversation with the top 100 people?
So the target is not one man but a layer of ~100–1,000 people whose beliefs will decide on the level of political will.
We won't get governance without agreeing on the problem, and we won't agree without waking up.
Unfortunately, we are still sleep-walking.[13]
There is much that I’d like policymakers to know, but if I only get 5 paragraphs, I think it would be the following high-level ideas (obviously, not necessarily written like this):
Superintelligence is a very real possibility. Most decision-makers are busy regulating bias and deepfakes, even while the open secret is out: AI companies are openly racing toward systems more capable than humans across the board.
It's near, not sci-fi. The people building these systems put transformative capability within a few years. This sense of urgency is largely absent from AI policy. And that’s a problem because nobody prioritizes a problem they expect to arrive after they've left office.
Companies themselves can't robustly control it, and don't even claim to. The implicit belief in the room is "surely the people building it know how to make it safe." It's false. Companies have never claimed to have solved superalignment. They made very public announcements that it was still unsolved.
Recursive self-improvement changes the whole picture. With RSI, everything accelerates, including risks such as mirror life, interestingly lethal pandemics, and large-scale cyberattacks.[14]
No one else is handling this, and the tail is mass death. There are no hidden adults in the room, and we are completely in the fog, operating under wide uncertainty. Risk of loss-of-control or engineered pandemics, with a large fraction of people dead in expectation, is very much on the table.
Sadly, these five beliefs are almost completely absent in the rooms where AI policy is made. It will be hard to mitigate the risks of a problem that you don’t name.
We scraped the submissions of the UN Global Dialogue. Out of the 1,534 UN Global Dialogue submissions:[15]
So, yeah: almost no one talks about x-risks[16], even among the organizations that showed up to an AI-governance consultation.
(To be clear: I'm not saying cyber, deepfakes, or bias don't matter. Talking about cyber takes up a significant fraction of my time, and we are working with the EU AI Office to develop evaluations of harmful manipulation for the AI Act. My claim is that a healthy ecosystem would have a more balanced set of organizations covering x-risks).
Salience is low but rising: Prompted concern about AI risk is rising (when you ask: "what do you think about AI?"), but top-of-mind salience remains very low (when you ask: "what's on top of your mind, dear citizen?"). In most general issue-priority polls, AI barely appears as a category, and it is discussed far less than inflation, purchasing power, immigration, jobs, health care, crime, or security during elections.[17]
The good news is that this top-of-mind salience, while still low, is climbing fast.
But don't say 'alright' too fast:
Within AI risks, catastrophic risks lag behind: conditional on talking about AI, catastrophic risk sits below near-term concerns, though the gap is modest.[18] But not being in first place has a cost: the main AI risk prioritized at the last G7 was child safety, and loss of control didn’t make it into the ministerial declaration of the G7 (cyber and CBRN made it finally; that’s a positive development).
Sections 1 and 2 were about the world: the bottleneck is belief among the people who decide. This section is about us: if that's the bottleneck, why isn't our field attacking it? This is the meat of my critique.
There are a few factors we don't control:
But I think that beyond those factors, we are simply under-investing in engagement & advocacy. So here are a few elements that explain why we are not on the ball:
A large majority of the people who organize the summits, sit at the UN, work for the OECD, or staff the Commission have simply never had the conversation (to be clear, some of them had the conversation and dismissed it).
According to Corporate Europe Observatory, of 97 senior Commission meetings on AI in 2023, 84 were with industry, 12 with civil society, and 1 with academics; Google alone had 10, nearly matching the total for civil society combined.[20]
Even when we are in the room, we self-censor.
In October 2025, Yoshua Bengio posted about the first Key Update to the International AI Safety Report. Connor Leahy replied:
"While I highly respect Yoshua and the titanic effort that goes into compiling such reports, it is disheartening to see the complete absence (even downplay) of discussion of superintelligence, existential risk and loss of control."
If the people closest to the problem self-censor, the signal never reaches the deciders.
I've found empirically that almost all the think tanks whose members discuss x-risks freely with me obfuscate their messages in public.
I also did, and still do to some extent, at CeSIA. For example, we recently revamped our website, and at some point someone convinced us to remove the risk page. I now think this was an error, and that even if we look a bit more institutional without it, we're losing in the long term.
A larger example, where I was nearly complicit myself: a joint submission to an international consultation from around ten civil society organizations, most of which had signed the Global Call for AI Red Lines. The final text names no specific risk at all. Not even cyber. The reasoning was pretty sophisticated: the document was meant as a door-opener, and the worry was that naming a risk high officials might disagree with would get the whole letter dismissed before anyone read the rest. CeSIA was invited to co-sign. I went back and forth for two weeks, pushed to name the risks, and in the end we declined to co-sign and submitted our own, explicitly naming the risks. I want to be fair here: these were thoughtful people making a defensible bet about a specific audience. But there is a pattern: a coalition of organizations that privately take catastrophic risk seriously produced a public document that refers repeatedly to "shared understanding of unacceptable risks" without ever naming the one that matters most to them.
To be clear, there are multiple schools of thought on institutional engagement, and I still think it sometimes makes sense not to be maximally blunt about AI risks in a first meeting with a policymaker (say, if you can get a win with a recommendation that doesn't depend on understanding catastrophic risk). But overall, I've been surprised by the relative absence of risk explanations in major think-tank submissions.
An elegant option for organizations that believe in the risks but won't name them publicly: use your convening power to invite researchers who will:
Have you heard about AI Safety Connect? I’d bet most people on LessWrong have not - but I can tell you that the team working there is doing heroic coordination work. They organized very large side events during the series of international AI summits and created some space for the community, while inviting people outside the field who have to sit down and listen to Yampolskiy’s 99.999% Doom argument. Some policymakers were shaken after the event. Hilarious and effective.
– from the Invisible side of AI governance.
US AI governance has roughly 3.6 researchers per advocate [21].
Research is high-status; the work that moves policy is often invisible and unrewarded. And think tanks are often evaluated by nerds with research instincts whose hobby is often reading blog posts and fascinating new arguments.
There is no h-index for minds changed.
Obviously, research is safer to fund, and its downside risk might just be wasted effort, whereas the standard argument is that advocacy can backfire, for example by making AI policy partisan or locking in a flawed regulatory regime. But look: AI policy is already partisan, and we are already in a flawed regulatory regime.
Let's be clear: More research is the right call for genuinely open questions, such as digital sentience. The error is applying it to risks we already understand well enough to act on, in which further study becomes a form of avoidance.
Most likely, no one will read your 50-page paper.
I don't know why CAIP, one of the few AI policy shops in Washington, didn't get funded. Its strategy looked sound to me, the numbers impressive, and the director's LessWrong sequence was early and didactic on many of the points in this memo. [22]
Meanwhile, applications to safety programs have multiplied many times over. Still, there's no lobbying pipeline, few execution seats (per the MATS talent study), and the ecosystem might not be scaling adequately to absorb this talent.
Humans are not superintelligences; they rarely update immediately in response to evidence.
Changing opinions takes time; even the numbers from the best methodologies are sobering. Broockman and Kalla's deep-canvassing study found that ten-minute doorstep conversations produce only ~0.08 standard deviations of attitude shift, roughly nudging someone from "somewhat opposed" to "slightly less opposed," not from opposed to supportive. Deep canvassing is the gold standard. By contrast, brief campaign contact has nearly zero effect on voting choices.[23]
If extended, personal, face-to-face conversation with motivated people is the most effective persuasion format we know, and it moves the needle so little, then what should we expect from a 30-minute meeting with a minister's chief of staff who has twenty other priorities?
The answer is almost nothing.
Single conversations cannot be the plan.
Repetition is how you get things done: by slowly growing salience in the ecosystem, doing the invisible work of agenda-setting or coalition-building, to get independent voices amplifying others.
The number we currently deliver to most policymakers is zero. Agenda-setting research (McCombs and Shaw) shows issues need repeated appearances across multiple channels, from multiple actors, before they become priorities.
I often won’t listen to a problem that’s raised to me unless I get the same message from an independent source.
In research, novelty is the main value. In governance it's almost the opposite: several people pushing the same thing, independently, is what works. Authority arguments like: “This respected person also pushes for this” are how to get things done.
I'd most like funders to stop treating redundancy as a reason not to fund AI governance work.[24]
(Consider this post my own Level 0 conversation with the field. The model predicts I'll need to repeat it two to four times).
Many times, senior people told me, "that institution is already covered," and then I'd find near-virgin land. Coverage, on inspection, sometimes means one person gave one talk there in 2023. To my knowledge, CeSIA was the first org to present (privately) on loss of control in 3 major international institutions.[25] That's why I think we are dropping the ball at the ecosystem level: the work is so under-resourced that one small team keeps finding itself first.
And there are still massive low-hanging fruit everywhere.
Being outgunned by industry is only half the problem: the entire field is undersized against any comparable effort.
As of COP 30 in 2025, the UNFCCC has admitted 3,907 NGOs as observers, and Climate Action Network alone spans more than 2,500 organizations across over 150 countries. AI safety governance has roughly 45 non-technical organizations and ~500 people, most founded in the last five years (McAleese 2025).
To my knowledge, fewer than 5 core AI Safety organizations engage with the UN in some fraction of their time (note: not counting what’s happening at the UN Global Dialogue). That's two orders of magnitude smaller than the field that fought climate change[26]. (And remember that only about a fifth of those ~500 people are knocking on doors rather than researching.[27])
I'd say that in France, maybe 5 people are meeting policymakers and journalists in total (while juggling an insane amount of work). Not more. Is this sufficient to wake up a whole country? I'm not sure the number is much higher when counting people working in Brussels who are willing to talk candidly about risks.[28]
(I’m less knowledgeable about what’s happening in the US, so maybe I’m overindexing on my experience)
Maybe the strongest counterargument to all of this is that the slow advocacy work doesn't matter, because the evidence will eventually speak for itself, whether through a crisis or an event like Mythos. In most other fields, the safety regime arrived primarily after a disaster: aviation after a string of crashes, nuclear after Three Mile Island. So why not wait for AI's equivalent?
Because, first, a crisis is partly constructed. A warning shot is just an event; it becomes a regulatory moment only if the environment is ready to notice it, and channel it into concrete regulation. Holly Elmore explains why we can't just wait for the cavalry: for a warning shot to update someone, three things must hold at once: a) the event provides information they already believe would confirm AI is dangerous, b) it does so in a quickly recognizable way, and c) it points at an obvious next action. Notice the word already. People need the dominoes pre-loaded: from capability to dangerous capability to short timelines to, finally, catastrophic risk without meaningful action. Otherwise the event means nothing to them.
For example, the AI Safety community broadly agreed that observing deceptive alignment would be an "absolute shut-it-down moment." Then Anthropic published the alignment-faking paper, and within days experts were debating whether it counted as the warning they'd imagined, and the moment dissolved.
The smoking gun, if it fires, won't fire cleanly. This is also why I'm skeptical of the view that technical evidence converts more efficiently than advocacy. Even one of the best strategies for political will, catching an AI red-handed, is probably unreliable. Yes, if this happened clearly, this would move political will faster than any realistic number of ministerial meetings. But first, the catch would need to be converted in a legible way to the media, and even if it comes, it probably won't be legible enough to convert decision-makers, and it may arrive only once development is too rushed and positions too entrenched to act on (Greenblatt, "How will we update about scheming?").
Anecdotally, I've felt this in miniature. I've shown someone an impressive video (to me) of a new robot. I expected "wow, amazing" and instead got "it's so slow and clunky, what are you talking about?" People don't see how insane it is that this already exists, and forget all the magic necessary for sand to process this information in the first place; the evidence is right there, and it slides off.
Mythos confirmed this again. In my own ministerial-cabinet meetings, it helped with tangential risks like cyber, but people stopped there, and even then I was met with "isn't it just hype?" Most policymakers didn't wake up. They remained suspicious of Anthropic, and in every meeting since Mythos, I've had to be the trusted person in the room, saying, "Yes, Anthropic is very good at marketing and the raw capabilities are somewhat inflated… but the underlying trend is real."
Beyond Mythos, I claim in this post and this comment that we probably won't get convincing-enough warning shots before crossing the event horizon.
I agree that to some extent, Mythos, and the Anthropic–White House meeting that followed, did more for political awareness (on Cyber) than the work of every CSO combined. Maybe. But the missing mood is still there: the executive order almost didn't pass, the provisions remain fragile, and they still don't touch the risks arriving next: bio, loss of control. To the contrary, we got a big push for sovereignty in the EU, and safety feels even lower on the list of priorities. When people in charge of AI in a government don't know what a jailbreak is, that should be informative of where we stand, to put it mildly.
I hope it will become easier and easier to talk about risks as AI becomes more capable, but I strongly feel that we shouldn't just wait for a crisis. Mythos is already behind us. Let's not wait for something else to happen: it's now or potentially never.
Objection 1: "Policy now risks premature action that locks in the wrong frame." This is roughly Dean Ball's position. He takes superintelligence relatively seriously, but thinks a bias to action produces bad lock-in, and that the US government is incompetent and self-serving enough that light-touch regulation is the safer bet; his confidence threshold for intervention is much higher than mine. But the main crux is obviously the p(Doom): in the Tegmark–Ball debate, I lean heavily on Tegmark, and the disagreement boils down to Dean's low p(Doom). If expected takeover risk is about a coin flip, the risk of inaction dwarfs the risk of lock-in.
Objection 1b: Another backfire objection appears in On Pessimization, by Richard Ngo: awareness-raising pessimizes when advocates lack concrete proposals, and then the energy flows into negative spirals, such as racing and creating new labs like OpenAI. I agree to some extent, but the main failure mode is awareness without asks. Fortunately, we now have many clear asks ready to go today, with regulations soon to be enforced in need of support (e.g., the Code of Practice of the AI Act, and some state regulations).
Objection 2: "If we're seen as advocates, we lose our seat at the table." Ok, this one is real, and I feel it personally: part of why we get invited into technical and diplomatic rooms is that we're perceived as a serious organization™. There is a glass ceiling for purely advocacy organizations in international institutions. Two responses: First, this is an argument about who does advocacy and how, not about the aggregate allocation in the field: mature policy fields differentiate inside voices and outside voices. Second, I believe there are elegant ways to present the dire situation we are in, and the cost of talking frankly has been greatly reduced today, given all the public statements and recent developments with Mythos/Fable.
Objection 3: "Political will is low now, but it will rise on its own (as in AI-2027, where governments wake up late on their own), so the real bottleneck will be the verification mechanism when the time comes." Political will probably won't rise quickly enough on its own, and this doesn't exclude advocacy now. And I think basic verification mechanisms are already good enough to get started (see here); waiting for political will to rise on its own forfeits the preparation that determines whether a later crisis converts.
Objection 4: "SuperPACs in the US are already doing this at scale." Good! But they're aimed at the general public to shape elections, and they rarely reach the ~100–1,000 decision-makers who can make the real difference.
Objection 5: “Policymakers have heard about AGI and choose to dismiss it.” Dean Ball made this point after the Delhi Summit. He says that in global policy circles, talking about powerful AI is considered impolite, even a little discrediting, because “AGI” is heard as an American imperialist construct. But Ball himself changed some minds in Delhi with a report in hand. And if the dismissal comes from distrust of the messenger, I believe this is an argument for advocates independent of American labs, not for more papers.
Naturally, I'm more confident about the problem than on the solutions. This section is more speculative.
Knock on the doors of media, policymakers, and influential institutions, and keep knocking.
Both the level of investment in advocacy & engagement with stakeholders and the allocation away from pure research have to change.
Judge this work by the number of minds moved.
A rough back-of-envelope: US AI-governance work currently runs at something like the ~1:3.6 advocates-to-researchers ratio documented in Section 3. I think it should be closer to 1:1, maybe even 3:1.
Relatedly, Asya Bergal from Coefficient Giving just made a structurally identical argument for a different margin: many marginal hires at research organizations (Apollo, Redwood, METR, RAND, GovAI…) would have more impact founding or joining capacity-building organizations (The case for AI safety capacity-building work). Their surveys and testimonials point in the same direction as this post: the multiplier work is systematically undervalued relative to object-level research. I'd simply add advocacy and policy engagement to their list of multipliers.[29]
ControlAI did a good job creating the playbook for the outsider game (the Direct Institutional Plan). Here’s my tentative playbook for the insider game.[30]
Samuel Buteau, alone at ControlAI, followed the playbook for the Canadian Parliament, formed a cross-party group of MPs who publicly signed the statement, and triggered a series of parliamentary hearings on superintelligence risk (Canada Campaign Statement | ControlAI). It seems to work!
A cheap way to contribute is to submit to an open consultation. As we saw in Section 2, almost nobody raises the risks that matter most in these consultations, making a marginal submission unusually visible. If you want more people to talk about what matters to you, submit something in the next such consultation.
CEOs have an insane level of access to Heads of State, so getting them to speak more candidly about risk would be unusually effective. Employees are among the few people positioned to create the internal pressure that shifts what a CEO is willing to say.
They can sign internal and public statements, push for stronger commitments, dissent on the record when safety pledges are quietly weakened, or even quit with a viral tweet when it's time to speak up more loudly. I think this is probably high-leverage.
A recommendation adopted without its underlying rationale is quite brittle: the moment it's inconvenient, or the situation changes, no one downstream can defend it because no one truly understands why it's there.
Also, making good recommendations is (very) hard, takes time; if you say something dumb, you lose credibility.
There is no universal rule here, but I'd lean towards an environment where policymakers are exposed to the risks worldview rather than one where they receive only shallow recommendations that do not generalize.
As I argued above, a crisis only converts if the ground is already prepared. So, concretely: have the analysis and, most importantly, the relationships ready before the event lands.
I tried to convert as much as possible from within CeSIA in the Mythos moment, but we made mistakes and were too slow. We need to be more prepared.
For example, if at some point we get clear architecture using Neuralese in production[31], I predict it won't be clear at all to the media why this is bad - and this requires a lot of awareness-raising and explanation ahead of time that I'm not seeing much of.
I think safety people should win Bayes points and credit for being early, but we're not really getting those points. The fix is to state publicly and collectively, before the event: "this specific observation will happen." If the event lands, you point at the registration and, hopefully, you get listened to a bit more carefully.
If you work in AI governance, make your worldview public; it's an investment (on top of enabling us to discuss and engage with each other). We could have won a ton of points when AI started eating Erdős problems.
This follows the previous point. Some skeptics won't be moved, and that's fine; the goal isn't unanimity. You don't have to convert the irredeemable; you have to make their claims progressively less credible to the people watching, by being the side whose predictions are borne out over time. Luc Julia, the second most prominent skeptic in France after LeCun, never changed his mind, but Mr Phi, a prominent French YouTuber, made a very visible video showing that several of his claims were factually false, and it stuck and went viral. We need more of this type of analysis.
Obviously easier said than done, but viral comm can be hugely effective, and there are ways to manufacture it. AI-2027 and Europe-2031[32] reached some of the relevant 100–1,000 stakeholders. We should make more of these: tune each one to a different constituency (national security, EU competitiveness, labor, biosecurity) while carrying the asks on the Pareto cheapness/effectiveness frontier.
Yes, superintelligence and human extinction sound like sci-fi, but those concepts are the actual variables driving the risk. If we hide them, we prevent the Overton window from shifting at all.
So we need intentional strategies to normalize these conversations. The CAIS statement and the superintelligence statement were good first steps. I think that more is possible in this direction.[33]
The task of industry lobbyists is much easier than ours because they are aligned in what they ask for: they all want less regulation. By contrast, AI safety advocates are often far less united in their demands.
We could coordinate around a few shared demands to create a voice that's actually unavoidable at the UN Global Dialogue and other summits, rather than arriving, as we do now, as scattered voices.
This is what the International Campaign for the Abolition of Nuclear Weapons (ICAN) did: it got hundreds of organizations to say one thing (ban them) until it was on the agenda, and won a Nobel Prize for it.
Concretely, this means being willing to move toward a common ask rather than holding onto my own variant. Be it the IAEA for AI, specific red lines, or anything else. We should have this discussion publicly. The specific ask might matter less than the convergence: whatever we choose, the value is in saying it together.
Over 200 of the 1,534 submissions to the UN Global Dialogue spontaneously call for “red lines”[34]. CEOs have recently asked for international standards, and for an IAEA for AI. An IAEA for AI sounds utopian until you remember the actual IAEA was built in four years by people who had just finished bombing each other. Maybe this can be the basis for coordination.[35]
I started drafting this post under the title "Political Will, Not Research," and softened it to "the current bottleneck" for a reason: some research bears directly on the bottleneck. Here are a few directions:
Research on how to convince people of the problem. If the bottleneck is understanding, then how to build understanding that converts is itself a neglected research question, and it seems almost nobody studies it systematically. Seismic's report On the Razor's Edge: AI vs. Everything We Care About (2025) is a start, and its findings are counterintuitive. It might be the case that the vast amount of advocacy to date was for nothing, and the best strategy is "issue bundling," where people reach AI-risk concern through what they already care about (see, for example, job loss or mental health), and only then talk about catastrophic risk. But it's nearly the only systematic work I know of, and we need far more: What actually moves a cabinet advisor from "cyber" to "loss of control"? Which framings convert? CeSIA had to experiment from scratch, and that’s probably the same for many orgs.
Research that helps turn will into requirements. Prioritizing the asks the AI Office should make of companies, and the risk-modelling methodology to hold them to it (argued in A Call for Better Risk Modelling: this is urgent since CoP enforcement starts on August 2); auditing the thresholds and mitigations companies publish, continuing what AI Lab Watch has been doing (OpenAI's red line for AI self-improvement is fundamentally flawed); operationalizing and harmonizing red lines across jurisdictions (AI Red Lines: A Research Agenda), or the recent draft treaty proposal signed by a coalition of international experts.
Research that measures the progress in political will. The effect of advocacy is mostly illegible.[36] But even if the causal chain is hard to be sure of, we can still measure the aggregate effect, and potentially fund more METR-graph-for-policy, like the AIPN tracker cited in Section 1, and better understanding of what’s happening under the hood.
Research that creates demonstrations of risks. Model organisms like agentic misalignment: I use this paper in all my presentations to policymakers now - I think that it is the best paper to demonstrate that frontier models could be dangerously unaligned, which is still one of the main bottlenecks (if not the main one), for AI risk to be taken seriously and prioritized accordingly. I thought in the past that we already had enough risk demonstration (Sleeper agent, alignment faking, Mecha Hitler, ChaosGPT...) but no, this is really a substantial improvement.
Engineering that makes "yes" cheap. Factorize technical mitigations across labs: a shared, off-the-shelf library of safety techniques (constitutional-classifiers-style) that Chinese labs or Mistral could adopt seamlessly.[37]
Research that could flip the strategy's sign. Advocacy without red-teaming is how you lock in the wrong ask: Human takeover might be worse than AI takeover. I'd like to fund the research that proves this post wrong and tells me what to do instead. There is no shortage of such cruxes[38].
Two closing notes on why I expect this thesis to become more true over time.
Don’t take all of this personally; AI Governance being slow is not an AI safety-specific concern.
Ok, yes, to some extent, political will being the bottleneck doesn't apply to causes that are still genuinely researchy with no clear ask, where "keep doing the research" is the right call. But for most causes with a known ask, the constraint is coordination and political will, not more analysis.
I see some people in the ecosystem starting to take space governance seriously, and writing about this. Yet the UN has worked on it for roughly fifty years, and I don't think the community has started ngaging with the UN on this.[39]
Same for power concentration: if politicians were AGI-pilled, they would act much more rationally around this.[40]
(This probably won’t happen in the short term, and it should only be considered after we handle AGI, but getting politicians to be more rational, scale-sensitive, and epistemologically sound in the long term would, in general, be incredibly useful and beneficial for society.)
AI will clearly accelerate technical safety work, and even governance inputs. I expect AI to be good at finding ideas at some point and to become excellent at forecasting. But it seems, empirically, that people don’t care about forecasts; they don't care that top forecasters with a good track record, like the authors of AI-2027, are raising the alarm.
AI won't, by default, be able to accelerate agreement, consensus-building and human engagement. Consensus is a human process, and it scales differently than research does.
So, as automation reduces the research bottleneck, the human-coordination bottleneck becomes even more important.
...
So yep, that's bad news for us introverts; we'll have to engage with the external world at some point.
(not disagreement after consideration, but absence of the conversation itself)
Update: ~7% engage the broader concept of loss of control, though explicit existential framing stays under 1%. See the discussion in the comment.
This isn't the first time LessWrong has heard the case for reallocating toward advocacy. The post “Instead of technical research, more people should focus on buying time” made a version of it in 2022, and the community's verdict was that outreach backfires when done poorly, and that indeed it will be done poorly (Wentworth–Larsen), and this was probably reasonable on the evidence of the time; Katja Grace diagnosed the underlying aversion the same year. Then, that verdict has been reopened: MIRI pivoted its entire strategy toward communications, Ruthenis argued in 2025 that awareness is the bottleneck, and gave up on policymakers, which I'll dispute below. What the reopening has lacked is concrete evidence from inside the rooms. That's what I try to add: a model of what advocacy produces (the funnel) with 2 years of insider experience across European and multilateral institutions.
It would be nice if this law was as solid as the METR’s doubling trend of AI capabilities.
Senators Hawley and Blumenthal introduced the AI Risk Evaluation Act, a mandatory pre-deployment evaluation of frontier systems for loss-of-control and scheming behavior, with penalties, and Hawley did it while his own party's administration was pulling the other way. Bernie Sanders talks about superintelligence and introduced a recent bill. I might be forgetting other initiatives.
Chain-of-thought (CoT) is the model's step-by-step reasoning trace. One of our better safety hopes is that we can read it to catch misbehaviour, but that only works if labs don't optimise the CoT to look good, which destroys its faithfulness as a signal. Anthropic has acknowledged inadvertently training against the CoT on more than one occasion. Doing it by accident is exactly the kind of basic process failure that should be easy to avoid. See the AI safety Atlas for an explainer on this.
Note that I value Greenblatt’s opinion since he has an excellent track record of forecasting AI capabilities.
His current AI-2040 numbers put Plan A at ~19% takeover risk vs ~45% for Plan D: a ~58% relative cut rather than ~84% (that requires a near-ideal Plan A).
(Note that we don't need to skip from Plan D to A to get substantial improvement - we can also push continually along the spectrum of political will, where each increment gets a worthwhile improvement)
(and people who wield power, which includes some people in AI companies)
(at least, if you have the time to read this)
Enforcement is a crude lever compared to a request for information and can also introduce adversariality into the relationship with the provider, so it is not entirely surprising to see that the Commission is very careful when using those powers.
That’s not surprising considering that even at NeurIPS, the biggest gathering of AI researchers in the world, a third of the AI researchers don’t know what AGI stands for, but yeah, we are nowhere near.
I now think that loss of control is probably more urgent than biorisks. See this analysis from PourDemain. I think loss of control is orders of magnitude more likely than irreversible x-risks from AI-enabled pandemics, even if both are probably already at intolerable levels.
We matched on exact strings.
PS: In the comments, cloud notes that ~7% of submissions raise loss-of-control-type concerns, but loss of control and existential risk differ, and the "<1%" refers to the latter. Reading the submissions flagged under "AI pursuing its own goals," only around ten frame it in genuinely existential terms (extinction, human disempowerment, uncontrollable ASI), still under 1% of the 1,534. That low explicit signal, even among the actors most predisposed to raise it, is itself the self-censorship dynamic described in the post.
When people are asked about AI directly, they often express concern; but when they are asked what political issues matter most, AI risk is usually absent or marginal. Concretely: YouGov's "AI will negatively affect society" rose from 34% (Dec 2024) to 47% (June 2025), yet AI does not appear at all in Pew's 24-item ranking of top national problems (Feb 2025) and sits near 1% in Gallup's open-ended "most important problem."
And the comparison is cross-pollster rather than head-to-head (no single neutral poll ranks them against each other). Pew (Aug 2024) finds people "highly concerned" about misinformation (66%), loss of human connection (57%), job loss (56%) and bias (55%); a separate YouGov poll (June 2025) puts concern about AI-driven human extinction at 43%.
Did you know that the Mexican government has already been hacked by an unattributed hacker using Claude? (Bloomberg, Feb 2026) It seems that, in an authorized red-team test, a frontier model reportedly compromised most of the NSA's classified systems within hours (NYT). The fact that this is not the top story in the world is also revealing about the information ecosystem.
But again, this is not really a factor under our control, and I argue in this post that convincing warning shots are unlikely. I come back to this in the sub-section “The main objection: Let’s just wait for a warning shot? I don’t think this works.”
There are more lobbyists working on the EU's digital files (890 FTE in 2025) than there are MEPs (720). (Corporate Europe Observatory, Big Tech lobby budgets hit record levels)
202.5 vs 55.75 FTE, Green-Lowe / CAIP, An Activist View of AI Governance, 2025; author's estimate.
I don't know the grantmakers' specific reasons, and they may have been good ones. But I have the feeling that the ecosystem's revealed preference is that a new research org is easier to fund than an advocacy org.
In advertising, the exact threshold of repetition has been debated since Krugman's 'Why Three Exposures May Be Enough' (1972) — two, three, four? — but the general effect is well established.
(or at least to reduce this parameter in their weighted factor models)
I cannot name them publicly, but DM if you want private proof.
The comparison is loose; UNFCCC observers also include industry groups and all kinds of NGOs, while I'm counting only non-technical AI safety organizations (which is also generous in some respects). Regardless, I think that even with heavy discounting, the gap is enormous.
Remember also that the environmental movement is over fifty years old. Rome wasn't built in a day.
Michael Dickens's 2025 donor review also concludes advocacy is far more neglected than research and that the few advocacy orgs do not get much grantmaker support.
I'm proud to say that I've spent four years on this problem by founding ML4Good (20+ bootcamps in 10 countries). Of ~400 alumni, ~150 now work at places like the EU AI Office, MATS, and UK AISI..
The insider game has documented integrity costs, and there is also a missing mood (Integrity in AI Governance and Advocacy). But I also think that it can be done correctly. E.g., I don’t think the most effective strategy is necessarily to open with the full Doom argument, and there are many strategies that lead to effective results and indirectly lead to more political will.
Why are Neuralese bad? See this: Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
I’m not sure Europe2031 was net positive. See.
Maybe a wild proposal, but one way to bypass institutional stigma is to create safe channels for civil servants, policymakers, and lab insiders to express what they already think privately. Maybe organizing anonymous joint statements?
I must now admit that we should have pushed harder to explain the risks during the red lines campaign: removing the detailed explanation from the FAQ was a mistake.
The article "diplomacy on a deadline" explains how, historically, we were able to negotiate treaties quickly. Spoiler: feeling a sense of urgency helps tremendously, among other things.
“A lot of semi-invisible, ongoing-over-years, and hard-to-definitively-attribute work has gone into many of the policy 'successes' of recent years. And sometimes part of the difficulty in attribution is actually down to things like the social proofing of multiple groups providing independent evidence and testimony that adds up to a credible body of expert input on a topic.” – source, Seán Ó hÉigeartaigh
It's been more than a year since the constitutional classifier paper was published, and there is still no ready-to-use library. This is a coordination failure because the same work is duplicated over and over, and because this is currently one of the most effective strategies for reducing misuse. Currently, a small team in each lab might be tasked with reimplementing the best mitigation strategies (which include CC), and safety teams at smaller labs are often just a handful of people. If this library were maintained and updated as new vulnerabilities emerge, my guess is that it would free up a lot of time for safety teams across different labs. This is high-leverage because it would raise the floor for the whole ecosystem.
See Zvi's Crux List for much more than you have ever asked for.
More generally, I think that our community has a tendency to ignore the ecosystem and work in silos, disconnected from the institutions and international fora, while that’s where the governance discourse happens.
Yes, politics is the mind killer, but only because smart people disengage from it. This is a hyperstition that needs to stop.