Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: If acceleration compounds, the naive extrapolation implies superintelligence within a year
This post discusses a public METR report on AI R&D acceler...
TL;DR:
* When we started Rethink Priorities (RP) in 2018, our mission was, simply put, to close the knowledge gaps that stopped people from improving the world as much as they could. That hasn’t changed much, but the landscape we’re doing it in is shifting.
* In 2027, we plan to grow our core GHD, Animal Welfare, and Worldview Investigations teams to take on emerging priorities across causes, including those that the forthcoming transformative AI is creating.
...
This was cross-posted from the Animal Charity Evaluators blog.
We update our list of recommendations annually. This year, we announced our recommendations on November 10.
Most people who care about animals arrive at the same impas...
We now know the makeup of Congress for what may turn out to be the two most important years in AI’s development. Democrats won big and AI was maybe the fourth or fifth biggest driver of the backlash.
So is strong and smart AI legislation finally on the way? Not yet, says host Rob Wiblin. Overriding a presidential veto takes 67 votes in the Senate and 290 in the House, and support for tough rules isn’t at that level today.
But there are several ways Congress can make progress right away:
Passing measures already popular in both parties, like requiring companies to secure their model weights against theft, restoring chip export controls on China, and protecting AI whistleblowers
Using must-pass funding bills to build out CAISSI, the likely home of US AI regulation. It runs on about $15 million a year, while the UK, adjusted for the size of its economy, spends something like 50 times as much
Holding hearings where company staff answer under oath: What’s Anthropic’s safety case? How big are your biggest models now? Which safety results were too embarrassing to publish?
Still, Rob thinks none of that will be enough.
Meaningfully slowing the AI race while models are around human level will probably take another warning shot at least as shocking as the Hugging Face hacks.
When that moment comes, what gets done will depend on the ideas lying around, so the time to carefully think through proposals is now. The ideas Rob is most keen on:
Making ‘ban superintelligence’ workable by requiring companies to spend a minimum share of their compute serving customers — creating a dial a regulator could turn up or down
Throttling recursive self-improvement by only letting companies internally use models that finished training at least six months ago
Banning models that can do too much without writing anything down in their chain of thought
Placing government auditors inside the companies
Letting companies quickly overturn clearly political decisions in court
Leaving state AI laws in place rather than preempting them
Rob also explains why halting data centre construction — probably the most popular demand of all — is a useful step but not enough by itself, and why people who understand how frontier models get trained are so badly needed for this work.
This episode was recorded on November 6, 2026.
Our production team includes:
Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour
Producers: Elizabeth Cox and Nick Stockton
Coordination and support: Katy Moore and Lou Moran
Camera operator: Dominic Armstrong
Subtitle: The midterms make a lot of things possible. But it will be far from easy.
They’ll have at least 250 seats in the House of Reps, and 56 out of 100 seats in the Senate — including seats in Mississippi, South Carolina, and Kansas, a place they hadn’t won in 100 years. And they won Texas, of all places, by 11 points, sweeping every office in the state.
AI wasn’t a top-three issue in this election, but it was maybe the fourth or fifth most significant driver of the voter backlash. It’s a place where government policy and what voters say they want are just clearly, massively out of sync.
All of which makes it more likely that Congress takes major action on AI during this crucial window in its development. So what’s likely to happen, and what are the most likely ways for things to go really well in the next two years?
Legislating AI will still be difficult
Let’s start with the bad news, at least from my perspective. Despite the massive shift in Congress and the role that AI played in it, passing a strong standalone AI bill still won’t be at all simple.
The first threshold one has to cross to have a shot at passing legislation is to get 60 votes in the Senate to overcome the filibuster. That is actually probably possible for an AI bill.
Among the general public, Democrats and Republicans feel roughly the same about AI — which is to say, remarkably negative. Four in five Americans want the government to keep AI safety rules even if it slows down development, including almost 80% of Republicans.
And while 67% of Americans say they think AI is advancing too quickly, just 2% think that it’s advancing too slowly.
Plus, a number of Republican senators have been leaders in the push to act on AI, including Josh Hawley, John Kennedy, and Marsha Blackburn, and we can guess that many more are quietly sympathetic.
But the president has a veto over any legislation passed by Congress, and overcoming that would require 67 votes in the Senate and 290 in the House of Reps. With overall concern at today’s levels, that is going to be pretty hard to achieve — or it would require watering down the rules to the point that they’re not especially useful.
But that is far from the end of the road.
Four things Congress can do immediately
There are at least four ways Congress can make progress right away, even with a veto hanging over it.
The first is to pass the commonsense measures that are already popular in both parties, some of which might get 70, 80, or even 90 votes in the Senate.
For example, today it seems entirely possible that the Chinese government can and is stealing model weights from US AI companies — in a sense that actually gives them more extensive access to frontier models than the US government itself has.
Nobody in Congress likes this situation, and legislation could demand that companies fix it before they keep going training more powerful models.
As the water level rises, the range of provisions that can attract majority support in both parties is only going to keep growing.
Second, there’s must-pass legislation: the bills that keep the government funded, like the annual defence bill. These are hard for a president to veto, because it means a government shutdown or no defence budget. So this is where negotiated deals between parties get done.
The most obvious thing to try to include in such a deal is proper funding to build out CAISSI, the likely home of US AI regulation — something that I’ll come back to later.
The third pathway is the least significant, but I’ll mention it anyway. Cabinet secretaries and many other senior officials need to be confirmed by the Senate, so senators can grill nominees for AI-relevant roles, and vote down or hold up anyone whose views on AI are badly out of step with those of the Senate itself.
The fourth pathway is quite big. It’s to hold hearings and collect information. There are so many things that the Senate and the House should demand that company staff answer under oath.
OpenAI also says it has evidence that using looped transformers didn’t reduce monitorability much. What are all of the experiments that they conducted to assess that?
What models are OpenAI, Anthropic, and Google training right now, and how much compute are they using?
And have these companies conducted any safety tests with results so embarrassing that they didn’t publish them?
These are all great questions to ask, and probably quite fun for members of Congress as well. Between them, these four pathways could get a lot done over the next two years. But they’re all about working around the veto, not overcoming it.
Why workarounds won’t be enough
Unfortunately, I don’t think just working around it will be enough to make it through this. We need a much bigger shift in policy than anything I’ve been describing.
At a minimum, we need to slow down the AI race while models are around the human level, to give AI companies breathing room to test the nature and personality of their models, check over the work that they’re doing, and hone their alignment techniques.
Without that extra breathing room, it seems more likely than not that we produce superintelligent machines that are misaligned but completely capable of hiding that fact — and that this, or some other screw-up, leads to total disaster.
That is a very serious problem, because more and more evidence is coming out suggesting that it will be possible to fully automate AI R&D as soon as 2027, and that this will lead to a big speed-up in research progress — including a new report from METR that landed last week.
For that reason, we desperately need California to take big steps immediately as a stopgap measure until Congress can step in, and I hope to cover that in a separate video.
To change Trump’s perspective, or get a supermajority in Congress to overcome his veto, my guess is that it will take another demonstration of what AI can do, or is willing to do, that’s at least as shocking to people as that was.
At the low end, we could have another Hugging Face incident, but one that demonstrates long-term power seeking by the AI, or that interrupts services ordinary people use.
The days and weeks after such an event could well be the most pivotal ones in AI policy — the critical moment that sets us on a better trajectory, or saves us from a much worse disaster later.
Some people should be trying to improve policy today, both for the direct benefit and to build experience and relationships. But if willingness to act can predictably soon be 10 or 100 times greater than it is today, clearly many people should also be preparing for that moment. That means being ready to work 100-hour weeks, and above all, thinking through and drafting excellent legislation now, while there’s still time to think carefully and get it right.
Why preparation matters
Anyone preparing for that moment should expect a very different political environment from the one we’re in today. I see three main ways that things can predictably change.
Up until now, AI regulation has done everything it can to improve safety while doing as little as possible to interfere with company activities. If a warning shot is severe enough, the public mood may be so furious that that constraint could disappear, or even flip on its head.
Interfering with model training or deployment, or just generally lashing out and interfering with AI companies, may come to be viewed by the public or politicians as actively desirable, rather than something that needs to be minimised.
The second big shift is that the typical legislator in AI regulation discussions is then going to know much less about the topic, so things are going to have much less of a technocratic vibe.
And thirdly, the public is going to be a lot more involved, and politicians will have to sell the steps they’re taking to the general public. That’s going to mean boiling it down to a core simple message, and communicating why what they’re doing is actually going to deal with the problem.
The slogan will be ‘ban superintelligence’
My guess is that the simple message more informed groups are going to rally around will be ‘ban superintelligence‘, or ‘ban superhuman AI’.
It’s a simple idea that’s easy to explain. It makes intuitive sense that it would help. Unlike banning AI entirely, it’s going to seem feasible.
And polling shows the US public is already overwhelmingly against the training of superintelligent machines — ones that totally dominate human experts in all domains. It’s a sufficiently natural idea that there’s already legislation introduced in the House and Senate to accomplish that goal: ban superintelligence.
Needless to say, there are massive implementation challenges that people preparing for this moment need to figure out ahead of time. Such as: what’s the best treaty to immediately propose to China to enable both countries to pace the frontier? What specifically are we going to be limiting or banning? And how does the policy shift over time?
One promising answer is that rather than limiting training runs, we require AI companies to spend a minimum share of their compute serving inference to outside customers. If you set that minimum to 100%, that’s basically a research pause that keeps every AI product running — and it’s actually immediately good for AI users, who will be able to buy inference at much lower prices.
The exit plan would be to gradually tone this constraint down to, say, 70%. But if we get new warning signs, that’s a dial that we can turn back up again to pace the frontier more strictly. Effectively, a regulator could set this number in order to achieve whatever speed of AI progress it thinks is desirable — the same way the Federal Reserve sets interest rates in order to keep the economy not too hot and not too cold.
This stuff needs careful forethought and legislative drafting involving technical experts familiar with how the frontier companies work.
Limit recursive self-improvement
Beyond the big public slogans, there are a few ideas that are too niche to ever be rallying cries, but that I think are worth developing out now, and probably worth including in any emergency response.
The thing I’m most excited by is the idea to throttle recursive self-improvement using the specific mechanism of prohibiting AI companies from using recent models for their research or any other work.
The rule would be something like: for internal purposes, AI companies can only use models that finished training at least, say, six months ago, and which have already been released publicly.
The companies couldn’t complain that it’s ridiculous and makes their work impossible. Presumably it wasn’t impossible to do their jobs six months ago. And for that reason it wouldn’t by itself stop AI research progress entirely — but it would definitely slow it down, and it would make it much harder for it to ever become truly explosive, or for alignment to go off the rails very abruptly.
That’s because any capabilities gains couldn’t be fed back into the recursive self-improvement cycle for at least six months. We’d also have at least six months for extensive alignment tests of a model before it gets its own hands on the control wheels and is able to design its successor.
The policy also has a more significant slowing effect the stronger and faster recursive self-improvement would otherwise be going, which is exactly the effect we want. If six months turned out to be too little, we could just increase the delay. And if the figure is extended out to one year, or 18 months, or two years, you’re incrementally coming closer and closer to an outright ban on recursive self-improvement.
We ideally don’t want to interfere with research aimed at improving safety and alignment in particular, and to that end, companies could potentially apply to CAISSI for exemptions for particular projects. To give companies less reason to pass off capabilities work as alignment work, the rules could require that any work conducted with the help of more recent models has to be published in full as soon as it’s completed — which would mean that any capabilities research would effectively immediately be shared with competitors.
This approach does have some weaknesses. If a company found an older model was misaligned in some way that they hadn’t appreciated at the time, they wouldn’t just be able to replace it with a new one that they think is more aligned. We’d need to have some workaround to address that.
But overall, this really seems like an idea with legs that’s worth developing in detail. If Congress is willing to take any significant steps to limit AI company research practices in the near future, this is the most valuable rule that I’m aware of to attach to a must-pass bill.
In short, you don’t ban models that are too capable, full stop. Rather, you ban models that are able to accomplish too much without writing anything down in their chain of thought along the way.
If models can’t do very much without a chain of thought, it’ll be a lot easier for us to monitor them, and they’ll have a much harder time scheming against users. A rule like that would force companies to go back and figure out how to design AI with a fundamentally safer architecture that has much less serial depth.
Another approach people have suggested is banning excessive use of reinforcement learning to generate superpersistent agents, because those RL environments seem to play a very outsized role in causing AI misbehaviour, at least currently. Perhaps you could set a maximum amount of RLVR-generated data that can be used while training models.
Another possibility is that frontier AI training and research should have to be conducted in fully sandboxed, air-gapped facilities.
I don’t know if those ideas can be made to work well in practice. We’re going to need clever technical people to think hard and figure that out.
Is it useful to stop data centres?
Those more technical ideas are where I’d put my own energy. But alongside ‘ban superintelligence’, I expect the slogan with the broadest support to be something much more concrete: stop building data centres.
And assuming a serious warning shot, preventing more compute coming online is a step in the right direction, for sure. It’s a great place to start building a broader coalition as well.
The thing is, if we’re already at the point where loss of control is possible, simply not adding more chips doesn’t make the problem go away. It at best holds it steady.
To lower the risk, you need to restrict company research and deployment practices like I’ve been describing, or you need to start actively disconnecting chips from existing data centres. And you also can’t allow chips to be relocated to data centres outside the US — that’s not much of an improvement. So to be properly effective, a ban on data centre construction would have to be paired with stronger export controls as well.
So data centre bans by themselves aren’t top of my list, but they could be one useful part of a broader response package that avoids effectively pouring more fuel on the fire, while we use other policies to help put the fire out.
The government needs visibility
Whatever mix of bans and limits we end up with, there’s one other piece of the puzzle that would be really easy to underrate, which is making sure the government can actually accurately tell how much danger we’re still in.
If you want to pace the frontier of AI development for years rather than just months, it becomes necessary to progressively add global coordination, and proper enforcement, and compute controls — all of which is possible in principle, but raises the difficulty quite a bit.
To keep escalating like that, it will be necessary for political willingness to act to keep rising in line with the seriousness of the threat. And experience so far shows that simple facts about what AI models can do, and the misaligned things they are doing, and the sloppy company practices that make it all possible — that has done far more to change minds than anything else.
That means it’s incredibly valuable for Congress and the executive to get more transparency into AI companies.
For instance, companies should be required to immediately share any significant accidents or warning signs. CAISSI should automatically get access to any tests or research results conducted by the companies. And really, they should have their own auditors placed inside the companies, keeping tabs on what they’re doing, following the embedded bank supervisor model.
We need more AI experts in CAISSI
On top of this, the US government needs its own expert AI body that can make sense of this enormous amount of incoming information, implement regulations that are passed, and adapt them to circumstances over time.
Unfortunately, an organised government agency with top AI experts can’t be thrown together in a few weeks in a crisis, which is why it’s urgent that Congress properly fund the body best placed to play this role, which is the Center for Advancing Innovation and Standards for Super Intelligence, or CAISSI.
There are great people there, but it’s just way too small and underfunded relative to the scale of the role that we need it to play. It’s getting outclassed — mogged, really — by the equivalent agencies in the UK.
CAISSI runs on about $15 million a year with just a few dozen staff, while the UK’s AI Security Institute has over 100 technical staff alone, and about eight times the budget. Britain just committed another £300 million to it over the next four years, plus hundreds of millions more for new agencies on AI biosecurity, cyber, and rogue-AI incidents.
Adjusting for its economy, the UK is now spending something like 50 times as much building state capacity in AI. Which is a massive credit to the UK. But it’s also completely nuts.
The effect is that in a serious emergency, the US might be forced to delegate a lot of AI risk assessment to the UK government, because it hasn’t built the capacity to perform core functions for itself. Both parties in Congress and the White House should want to urgently rectify this.
Fortunately, this is exactly the kind of thing that can go into the next government funding bill. Even the White House has asked to nearly double CAISSI’s budget next year, and I think Congress should go much further.
Checks and balances
Again, there’s another high-level challenge to deal with here. Almost every step that I’ve described, including a bigger CAISSI — all of it — hands more power to the executive branch. And that power could be used to strong-arm companies into doing things that have nothing to do with safety, if that’s what the executive wants to do.
Ideally, we’d have built-in safeguards against this, such as well-thought-through options for companies to challenge decisions in the federal courts if they’re capricious, or violate the First Amendment, or just aren’t credibly justifiable on safety grounds.
And the details are really going to matter here. It’s probably too dangerous to allow companies to get a preliminary injunction to pre-emptively block rulings from, say, CAISSI. But we do want them to be able to object and overturn clearly political decisions promptly — within two weeks, if at all possible.
In the worst case, if Congress concludes that the executive simply isn’t going to enforce rules that it passes, or it’s only going to use them to pursue partisan political objectives, Congress may be forced to just play an assisting role, helping states like California and Texas do the heavy lifting, at least for some period.
The tradeoffs here are pretty difficult to deal with, and they’re another thing that has to be thought through ahead of time, not in the middle of an emergency.
Don’t stop the states
Here’s a final trap to watch out for. Until now, the expectation has been that when Congress passes a big set of AI rules, it’s going to pre-empt the states and severely limit their ability to regulate AI. By default, though, I think anything like that should be kept out of an emergency bill after a warning shot of some type.
By that point, states may well have strong rules on the books, and regulatory systems and agencies that have actually been set up and are functioning. And given the goal is to immediately strengthen safeguards, it would be totally perverse to delete those rules, which could have the net effect of at least temporarily loosening controls.
Have good ideas lying around
The midterm results substantially increase the chance that Congress will act to reduce AI risk, which is great. But it’s nowhere near guaranteed that this will happen in time, or happen in a really smart way.
If you understand how frontier models actually get trained, this work needs you really badly.
It’s a cliché that when a crisis finally comes, what gets done depends on “the ideas that are lying around.” But it’s a cliché because it’s mostly true.
Political will on this topic is only going to go in one direction. Our job is to make sure that when the moment comes, the ideas lying around are good ones — ideas that will actually fix the problem. That’s something people listening to this show are unusually well placed to help with.
If we pull it off, these might just be the two years during which an AI catastrophe is averted.
5
More posts like this
400
The Game Board has been Flipped: Now is a good time to rethink what you’re doing
