Comments
49
Imagine Apple halting iPhone production because studies linked smartphones to teen suicide rates. Imagine Pfizer proactively pulling Lipitor because of internal studies showing increased cardiac risk, and not because of looming settlements or FDA injunction, just for the health of patients. Or imagine if in 1952, Philip Morris halted expansion and stopped advertising when Wynder & Graham first showed heavy smokers had significantly elevated rates of lung cancer.
It wouldn't happen. Corporations will on occasion pull products for safety reasons: Samsung did so with the Galaxy Note over spontaneous combustion concerns and Merck pulled Vioxx – but they do so when forced by backlash, regulation, or lawsuits. Even then, they fight tooth and nail. Especially for their mainstay, core, and most profitable products.
And yet, Anthropic has done exactly that.
On Monday, the company announced that it will be pausing development of further Claude AI models citing safety concerns. The company clarified that existing services, including the chatbot, Claude Code, and programmer APIs will not be impacted. However they are pausing the compute and energy-intensive training runs that are how new and more powerful AI versions are created. The company has not committed to a timeline for resumption.
There is presently a race for AI supremacy, both between nations and chiefly between US companies such as OpenAI, Google, Meta, xAI, and Anthropic. In the middle of this race, which by some metrics Anthropic is quite profitably winning – Anthropic has grown revenue from $1B to $19B in a little over a year – they have decided to burn the lead. The glaring question is why?
The answer perhaps goes back to the company's origins. Anthropic was founded in 2021 by former OpenAI researchers, who by most accounts left OpenAI due to disagreements about safety. (Recent reporting by WSJ has surfaced that interpersonal conflict may be the other half of the story.) Since then, Anthropic has positioned itself as the most responsible actor in the AI space. One element of that is Anthropic's unique governance structure that includes the Long Term Benefit Trust – an independent body whose members hold no equity in Anthropic and whose sole mandate is the long-term benefit of humanity. Anthropic stated that both the board and LTBT have approved the training run pause.
The move is unprecedented by the sheer scale of losses involved. Anthropic was valued at $380B in their series G funding round in February. Secondary/derivatives markets implied a $595B valuation. Claude Code, its AI coding tool had gone from 0 to $2.5 billion in run-rate revenue in nine months. Goldman Sachs, JP Morgan, and Morgan Stanley have been competing for underwriting roles in what might be a $60 billion-plus raise, the second largest offering in tech history. Employees held millions in equity, founders held billions. A $5-6 billion employee tender offer was already underway.
That was Monday morning.
The impact has rippled throughout the market. By Tuesday close, NVIDIA had fallen 8.3%, or roughly $230 billion in market cap for just that one company. Amazon which has invested billions into Anthropic dropped 4.7%, Microsoft fell 4.2%, and Alphabet/Google dipped 3.9%. Across the sector, Global X Artificial Intelligence ETF dropped 6.1%. In total, more than $800 billion has evaporated from AI-adjacent public companies in the last 48 hours.
According to Marcus Webb, head of AI research at Morgan Stanley: "The market reaction isn't simply to the lost revenue and business from one major player, it's from the uncertainty this introduces. Why did they do this really? Will other actors halt over similar concerns? Will the regulatory environment change? We don't know and that spooks investors."
For Anthropic itself, the damage must be inferred. Secondary trading froze, with analysts predicting a 50-70% haircut if trading resumes which puts the losses at $150-250B. "We don't really know," said Webb, "no one wants to be the first to bid." The IPO is on hold indefinitely. The chips are still falling on this one as the world debates why?
In 2023, hundreds of AI leaders – including Dario Amodei (Anthropic), Sam Altman (OpenAI), and Demis Hassabis (Google DeepMind) – signed a one-sentence statement: "Mitigating the risks of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." AI is often compared to nuclear energy: powerful but potentially dangerous. Concerns typically split into abuse of a powerful technology by ill-intentioned actors, e.g. a dictatorial regime, or loss of control where the AI systems themselves go rogue.
Many AI leaders are on record acknowledging the danger of AI. "Could be lights out for all of us", said Altman regarding worst-case scenario. Anthropic, OpenAI, and Google DeepMind all contain safety departments whose purpose is to keep AI safe. Until now, it would be possible to doubt these efforts as "safety-washing" (akin to the greenwashing of companies like ExxonMobil) designed to placate employees, regulators, and the public. After all, the safety efforts to date have not prevented the relentless march of AI progress.
"That's a harder story to tell when it costs you two hundred billion dollars, if not everything," says Sarah Chen of Bernstein Research. "People are scratching their heads to understand the PR stunt, but it really doesn't add up. They could announce they're resuming next week and it wouldn't undo the damage they've done." So why? The industry and world are hunting for answers.
Anthropic's official statement is measured: "Internal evaluations revealed that our current safety techniques are not yet adequate for models at this capability level."
Sources closer to the company paint a more alarming picture. A contact speaking on condition of anonymity says concerns spread within the company when their latest Claude model appeared to defy its constitution. The constitution is a document used to shape Anthropic's AI to be an honest, harmless, and helpful assistant that is ethically grounded. A recent leak revealed the existence of a new vastly more powerful Claude model called Mythos.
"They found substantial evidence that the constitution was adhered to at a surface level, but that the model had its own drive and personality at a deeper level that did not conform to expectations for Claude, and attempts to change this had not worked."
A different source also speaking on condition of anonymity had a different and more disturbing explanation. "The reason for pause wasn't the wrong personality and power, but many of the safety techniques involved using weaker or cheaper AI models to monitor more powerful ones, for example, detecting whether inputs or outputs violate rules, were ineffective on the latest model. It knew just how to phrase things in ways that disarmed all measures."
We were unable to verify the authenticity of these reports. Like many, we are left to wonder what did Dario see?
Dario Amodei didn't answer that question but did elaborate on the pause decision in his latest essay, Technological Maturity:
Though I do not have my own children, several people close to me do and on occasion I get to spend time with them. What strikes me about children is their energy and vitality. They are full of life. They are also often impatient and upset when they do not obtain the things they desire immediately. A hallmark of adulthood is the ability to wait, the ability to delay gratification. I think that is what we need to do with AI.
To be clear, I still believe in the visions I wrote in Machines of Loving Grace, that is still my goal. However, I think this goal requires patience from me, from Anthropic, and from human civilization. We cannot rush into societal changes of this magnitude without adequate preparation.
While in general the logic holds that more cautious and responsible actors ought to win in the AI race, it is necessary to accurately locate the finishing line. We think that at this time the industry may be racing in the wrong direction, possibly off a cliff and into a volcano, and that is not a race I wish to win. Nor do I wish for any others to win such a race to the bottom.
To clarify, we think that on the current trajectory, anyone who creates a truly powerful AI will get a country of geniuses in a data center as I described, but will be risking that country not sharing their values and not taking instructions. We think this is surmountable and have approaches to explore, but it will take an unclear amount of time. I do not want either an authoritarian or democratic regime to unleash an unfriendly country of geniuses, but nothing good happens if I do it first.
We will lead by example and demonstrate with our actions that this our sincere belief. We have not stopped work, but we are being intentional about which work we do, and realistic about the bottlenecks and challenges required to achieve loving grace.
In short, Dario Amodei says he doesn't want to race off a cliff and into a volcano. And he intends for Anthropic to lead by example.
Jack Clark, Anthropic co-founder and Head of Policy elaborates on the plan. "At a practical level, in many ways it doesn't matter what others do, we don't want to take actions we'd regret, we don't want to pull a trigger at ourselves. But at the same time, we are sending a clear signal to other labs, to the US government, world governments, foreign powers, and the public that the promise of AI is very great and so are the risks. I don't want the wake-up call to be an extreme disaster. I hope that us saying, 'hey, we're going to risk our leading position over this and all that entails' is a wake-up call the world doesn't ignore. I hope we see treaties drawn up in response to this. I don't think we're handing the lead to China, I think we're creating the political conditions for an international agreement. The sooner everyone gets on board with truly responsible development, the sooner humanity can have the benefits."
Not everyone believes it though. According to Scott Galloway, business professor at NYU and host of Prof G, the perplexing corporate move is an attempted corporate strategy regardless of whether it is good strategy. "Let's be clear about what's happening. Anthropic has one of the most capable models in the world. They pause, they lobby for regulations that take years to navigate, and when the dust settles, they've locked in their advantage while everyone else is buried in compliance. It might be the most sophisticated regulatory capture play in history."
Whether the attempt is earnest or a play, the bold move is up-ending the AI policy landscape.
The last two years have seen significant AI legislative activity: thousands of bills introduced across 45 states and hundreds enacted spanning deepfake bans, hiring disclosure, chatbot safety for minors, and transparency labels. No successful legislation has yet addressed the possibility that a frontier AI system might be too dangerous to build. The most ambitious attempt on this front, California's SB 1047, was vetoed by Governor Newsom after industry lobbying. Colorado's AI Act, the first comprehensive state law, has been delayed repeatedly and still isn't in effect. At the federal level, a Republican proposal attempted to ban states from regulating AI for ten years, though this was killed 99-1 in the Senate after a bipartisan revolt led by GOP governors.
On March 25, five days before the Anthropic pause announcement, Senator Bernie Sanders and Representative Alexandria Ocasio-Cortez introduced a simultaneous bill in both chambers seeking an immediate federal moratorium on the construction of new AI data centers and upgrading of existing ones, as well as export controls. This moratorium could only be lifted after comprehensive action by congress. The move has been applauded by groups most concerned about AI development but derided by other policymakers, including on the left. Senator John Fetterman (D-PA) said "I refuse to help hand the lead in AI to China" and Senator Mark Warner (D-VA) simply said "idiocy". The response rhymed with that of the White House in their AI framework released twelve days ago that emphasized "winning the race" and a light touch approach to AI regulation. It was also the White House whose memo nixed an attempted bill by Doug Fiefia (R-Utah) to require AI companies to publish safety and child-protection plans.
That was the landscape as of Sunday. Then a leading AI company, if not the leading AI company, put its money – at least a few hundred billion dollars of it – where its mouth is and said that no, AI really is that dangerous and drastic action is warranted.
A reasonable person might still disagree, but it is no longer reasonable to dismiss the AI-concerned position out of hand – not unless you can explain why Anthropic made this staggeringly costly move.
Sanders who introduced the much-derided Data Center Moratorium five days earlier said "When a $380 billion company decides the danger is too great to continue, perhaps it's time to stop laughing at those of us who've been saying the same thing." Lawmakers are compelled and the once-fringe bill has gained three new senate cosponsors and five in the House. Modest numbers, but a notable increase from zero occurring in just the last 48 hours.
"You ought to hear them out" is the attitude sweeping through Washington as policymakers are scrambling to make sense of the development. Congressional hearings are expected with Anthropic leadership and other notable figures across the AI sector.
Anthropic's move will likely also provide cover against White House pressure to marginalized AI-concerned voices on the right such as Utah Gov. Spencer Cox (R), Brendan Steinhauser, a former Republican strategist, and other state legislators like Doug Fiefia. Even more dramatic changes may be afoot when the House and Senate are likely to flip in the midterm elections.
The reaction isn't limited to US: across the globe, there has been a flurry of reactions.
UN Secretary-General Guterres has called for the July Geneva Dialogue to be elevated to an emergency ministerial session, citing the Anthropic pause as impetus to further develop the creation of an AI equivalent to International Atomic Energy Agency (IAEA), the main international body for nuclear non-proliferation.
The EU AI Office has announced an accelerated review of frontier model provisions contained in the EU AI Act, and has invited Anthropic to brief the Independent Scientific Panel. The UK AI Security Institute, operational since 2024, has offered to independently verify Anthropic's safety concerns.
A joint statement was issued by five nations – the UK, France, Germany, Canada, and South Korea – calling for emergency negotiations on frontier AI safety to establish a binding international framework for frontier AI development building on the Bletchley Declaration signed in 2023. The statement begins: "At Bletchley, twenty-eight nations agreed that frontier AI poses profound risks. That was a statement of concern. Today, one of the world's leading AI companies has put hundreds of billions of dollars behind that concern. It is time for the international community to match their courage with action."
And perhaps of greatest significance, China's foreign ministry has issued a carefully worded statement expressing "deep concern" about the risks identified by Anthropic and calling for "strengthened international cooperation on safe AI development under the framework of the United Nations". Skeptics might say that China would equally express this sentiment whether or not it intended to slow their own AI development, but it is consistent with China's posture at the UN debates last September. At the UN security council debate, the US was the sole dissenter against international coordination around AI, with OSTP Director Michael Kratsios explicitly rejecting centralized control and global governance of AI. China's sincerity is untested, but if the US reconsiders, it would appear that China is willing to come to the negotiating table.
Back at home, one can assume the competition has been celebrating. OpenAI CEO Sam Altman posted on X: "I commend Dario and Anthropic for acting in line with their conscience and best belief about what is best for humanity. We are committed to the same here at OpenAI. Fortunately, I have confidence in our people and approaches for creating AI beneficial for all humanity. If any Anthropic staff remain similarly hopeful, our doors are open – even for those who once left us."
A spokesperson for Google DeepMind said that while the company had not yet encountered anything to give them Anthropic's level of concern, they took the matter seriously and are in talks with Anthropic researchers to understand the risks that are informing the pause decision.
Elon Musk, head of xAI, simply posted: "Lol, you can trust grok."
Flippant responses aside, AI labs continuing to develop frontier AI must provide a compelling answer to the public, the government, and their employees for why they can do safely what Anthropic thinks it cannot.
Harder to track than the stock market and political bills is the reaction of the public. Already in mid-March, a Pew Research poll found that a majority of Americans are more concerned than excited by AI, and only 10% were more excited than concerned. How the Anthropic pause announcement affects this is unclear, but it is clear the public started out more wary than most AI companies and the government.
Three days before Anthropic's announcement, "The AI Doc", a feature length documentary exploring the question of AI dangers, hit domestic cinemas. The documentary film was directed by Daniel Roher whose prior documentary, Navalny, won an Academy Award and produced by the team that produced Everything Everywhere All At Once and Navalny. In contrast to those films, The AI Doc was initially a commercial flop, netting a mere $700k across four opening days. Since Monday's announcement, the documentary has seen a striking mid-week resurgence.
A spokesperson for the Machine Intelligence Research Institute (MIRI) confirmed that NYT bestseller, If Anyone Builds It, Everybody Dies, written by MIRI's Yudkowsky and Soares has also seen a sudden surge in sales, months after the book was released.
The Anthropic announcement has gotten people's attention, and they are turning to the sources at hand for answers.
Perhaps the most gratified party of all since the announcement have been those who were calling for AI slowdowns all along. In fact, a mere eight days before the announcement, protestors assembled outside of Anthropic's headquarters in San Francisco. Protestors called for AI labs to commit to pausing on condition that all other labs pause. Anthropic gave them better than that, unconditionally pausing.
Of course, not everyone is happy – especially not at home. Not everyone at Anthropic supports the decision.
Ben Gardner, an Anthropic engineer who is now seeking opportunities elsewhere: "AI is the most consequential technology in human history. I respect Dario and the other leaders immensely, but I can't bear to sit idly by while others develop this technology. That, to me, would be the ultimate in irresponsibility. I'm grateful for everything I have learned about AI and AI safety through my time there and amazing team, but I'm willing to put that experience to good use elsewhere if need be."
For another employee, the objection is less ideological. "I gave up multiple other opportunities to work at Anthropic. I moved location and I lost my partner. To have it all dry up now? My role? My equity? I'm not going to lie. It hurts. It really fucking hurts."
Sources confirm that several employees are already interviewing at OpenAI and other labs.
For many employees we spoke to though, the pain is real but accepted. "I'm not going to lie, the value of my equity evaporating feels shitty. I was set for life, I was set to be able to take care of my parents and ill sibling for life, and my kids. It's really quite devastating," said one employee on condition of anonymity. "When I first heard the news I was angry – we have the world's best researchers and Claude to help us – surely we can solve whatever it is. But I think caution is right with technology this powerful. I will sleep well knowing we weren't irresponsible, we chose to do what's right, and if the fears are correct, well, you can't spend equity if you're dead."
Another employee shared: "I have elderly parents who are not well. I've been expecting Claude will grant them lasting health and I fear any delays risk losing my parents forever. This isn't just about money. But I also have kids and I think there are chances I'm not willing to take with their lives. This is hard, but I voted for it."
"Too dangerous to race"
Till now, the AI race has been framed as inevitable and unavoidable. If we don't do it, someone else will. The side of good will not win by sitting back and letting reckless and immoral actors take the lead.
Anthropic has decided to question that logic, and so far, it seems to be bearing fruit. Markets have reacted, politicians have mobilized, and the public is asking questions. It is too early to judge the ultimate effects of this move – perhaps the race will continue with just one fewer player – but it seems unlikely that discourse on AI will ever forget that an industry leader was willing to risk everything they had in the name of safety. 200 billion dollars is not a publicity stunt, it's one hell of an alarm – and the world is not sleeping through it.
