Don’t Bet the Future on Winning an AI Arms Race

Note: This post was crossposted from Substack by the Forum team, with the author's permission. The author may not see or respond to comments on this post.

Radical uncertainties in AI development and military applications favor security through cooperative stability. (Includes bonus footnotes on recent disruptive research results)

Robust, reliable uncertainties in AI development and applications make the pursuit of global dominance risky, creating pressures toward cooperative stability. Uncertainty can’t be eliminated, but it can be harnessed.

The Structure of AI Uncertainty

Uncertainty in AI geopolitics is a structural feature rooted in the nature of AI development and applications. It will likely be persistent and reliable.

Present vs. Future Capabilities: Even experts disagree about current and near-term AI capabilities. Research proceeds along multiple lines, sometimes in secrecy. New algorithmic approaches are reducing or bypassing previously anticipated compute requirements, undermining predictions based on hardware constraints.^[1] Specialized models are pushing frontiers in unpredictable directions,^[2] the use of external tools by models is proliferating,^[3] inference-time reasoning^[4] is still in its infancy, extensions to latent-space reasoning^[5] may prove transformative, prospective latent-space knowledge models^[6] promise to break the link between model size and knowledge scope, and both large concept models^[7] and nonautoregressive reasoning models^[8] mark departures from sequential token generation architectures. In every application area, patterns of success and failure — even in applying established technologies — have been surprising.^[9] No degree of intelligence or investment can eliminate these uncertainties.

Accelerating Development: AI is increasingly being used to automate its own development, taking on more of the tasks involved in AI research and engineering — designing architectures, discovering optimizers, coding, training, testing — creating a feedback loop that accelerates development. Making experiments faster and cheaper will encourage researchers to shift from exploiting known architectures to exploring new ones. AI R&D automation accelerates progress while amplifying uncertainty about its future direction and speed, increasing the likelihood of unexpected leaps in capabilities.

Adversary’s Knowledge: Where knowledge is possible, states face profound uncertainty about what their competitors know. The 2023 discovery of “Volt Typhoon” — which had penetrated critical U.S. infrastructure for years without detection — illustrates this reality.^[10] What one side considers secure, the other may already have compromised, and internal secrecy ensures that policy analysts cannot know the full extent of their own side’s knowledge, or their own side’s ignorance, or reliably evaluate classified claims. These uncertainties are intractable.

Second-order Uncertainty: More challenging is uncertainty about how the uncertainty landscape itself will evolve. Will breakthroughs in surveillance or analysis suddenly reduce specific uncertainties? Will new AI capabilities bring new uncertainties? This second-order uncertainty — current uncertainty regarding future uncertainty — further undermines prospects for confident strategic calculations.

These layered uncertainties would be challenging enough in isolation, but they are further complicated by uncertainties regarding actors’ perceptions.

Perception Asymmetries

Different actors may perceive and interpret the same uncertainty landscape in fundamentally different ways, and asymmetries in perception compound technical uncertainties, increasing the risk of misinterpretation and miscalculation. What one considers reasonable hedging appears to others as aggression. What one culture treats as acceptable strategic ambiguity seems like deception to another.

These perception gaps multiply across organizational and national boundaries. Military planners, intelligence analysts, laboratory scientists, and political leaders operate with different uncertainty models, even within the same government. When extended across international boundaries, these asymmetries create risks of dangerous miscalculation that no diplomatic communication can entirely resolve.

Given these structural uncertainties and perception gaps, what implications follow for states pursuing AI advantages?

No Actor Can Have Confidence in Winning an AI Race

States pursuing AI advantages face fundamental uncertainty about whether their investments will translate into meaningful strategic superiority:

Technical capability rarely translates cleanly to strategic advantage. Systems that excel in controlled environments often fail catastrophically under adversarial conditions. Military history provides countless examples of technologies performing unpredictably in actual conflict,^[11] and prospects for dynamic measure and countermeasure adaptation would make hot war a greater gamble.

Even if one side holds current advantages — themselves uncertain — these offer little assurance of future superiority: AI-vs.-AI conflict might well involve presently unknown weapons and tactics.

Over time, asymmetries could grow so large that one side might dominate the other. This is within the scope of possibility in a world in which explosive advances in AI could lead to enormously asymmetric military capabilities, and at some point it might become clear which side would have the advantage.

Consider a situation in which one side — rightly or wrongly — perceives itself to be on the losing side of this race. Within the present strategic calculus, losing a race would constitute an existential threat that could trigger extreme responses, potentially including nuclear options.^[12]

Thus, escaping from AI uncertainties — the very prospect of gaining confidence in winning — could itself create existential risk.

Beyond Uncertainty: Asymptotic Confidence

The uncertainties discussed above are about AI development paths — how, when, by whom, and in what sequence. Perhaps paradoxically, we can have greater confidence in longer-term prospects than in developments next year: Advanced AI will enable greatly expanded general implementation capacity, with a range of predictable, very high, lower-bound capabilities. And this prospect opens unprecedented options for grand strategy.

This combination of persistent short-term uncertainty and more predictable long-term capabilities suggests an alternative strategic approach.

From Uncertainty to Strategic Stability

This paradox — acute near-term uncertainty together with more predictable long-term trends — invites a strategic reorientation: If there are alternative, less risky strategies that lead to security and prosperity, we should turn toward them. I see three key components to such strategies:

Structured Transparency. Mutual verification regimes can create confidence in weapon-system capabilities and constraints without revealing sensitive details — structured transparency enables constrained opacity, limited concealment within a structured verification framework. Objective technical benchmarks can verify compliance without exposing implementation details. This approach enables confidence without naïve trust.^[13]

Paretotopian Goal Alignment. Even competing powers can identify and pursue arrangements where all parties benefit without sacrificing core interests. These "paretotopian payoffs" expand under conditions of uncertainty as actors come to recognize their mutual exposure to destabilizing risks.^[14]

Defensive Stability. AI systems excel at detection, monitoring, and verification — creating unprecedented opportunities for the deployment of defensive capabilities that reduce everyone’s vulnerability. An initial phase of offense-oriented competition may lay the foundation for a later, AI-accelerated pivot to defensive weapons production and deployment.^[15]

These approaches can be pursued with minimal strategic downsides, and if managed with intelligence, can support incremental strategies that monotonically reduce risk. Stabilizing uncertainties can be maintained while establishing domains of stabilizing knowledge.

Low-Risk Cooperative Steps

Actors need not abandon their current strategic positions to begin exploring alternatives. Early cooperation on specific technical challenges — verification protocols, safety standards, defensive applications — creates zero-regret options likely to benefit all parties.

These early cooperative steps:

• Reduce jointly-recognized AI risks without foreclosing future options
• Generate technical knowledge with both competitive and cooperative applications
• Develop options for verification infrastructure that supports multilateral interests
• Create diplomatic and technical channels for future coordination

The “late pivot” model suggests that states can transition from primarily competitive to primarily cooperative military development as risks and opportunities become more apparent.^[16]

These incremental steps align with the broader strategic logic created by AI's unique uncertainty landscape.

Convergent Strategic Logic

To recap, during the run-up to thoroughly transformative AI, leading powers should expect many, compounding uncertainties:

Strategic uncertainties (short-term)

• What next-generation AI will enable — and when, and in what sequence
• What military capabilities and countermeasures AI will enable
• Whether computation will be a strong bottleneck for development
• What AI capabilities an adversary has developed independently
• What AI capabilities an adversary may have acquired through espionage
• What an adversary knows about one’s own AI capabilities
• Whether to trust what their own intelligence agencies claim to know
• Whether AI advances will result in unanticipated transparencies
• Whether AI infrastructures are vulnerable to decisive cyberattack
• Whether one side could take a decisive lead, and if so, the likelihood of a preemptive response.

No actor can have high, justifiable confidence today in “winning” an AI arms race, or confidently expect that uncertainties will — or will not — resolve at a later date.

However, hypothetical well-informed decision makers should have confidence regarding downstream capabilities. Advanced AI will enable strategically crucial capabilities in roughly the following order:^[17]

Predictable capabilities (long-term)

• Rapid development of provably secure software
• Technologies enabling selective, verifiable patterns of transparency
• Enormous productive capacity
• Rapid at-scale design, deployment, and adaptation of novel systems
• Overwhelming deployment of verifiably defensive systems

The combination of robust strategic uncertainty and robust longer-term expectations creates convergent pressures toward cooperative approaches that reduce risks — and escape the security dilemma — while helping to secure the enormous, win-win gains promised by advanced AI applications.

This approach does not demand naïve trust, nor does it compromise national interests; it reflects the strategic logic that emerges from serious consideration of long-term gains, defensive security options, and uncertain paths.

1. ^{^}

   Algorithmic innovations have dramatically reduced the compute requirements for model training and inference, and surprises continue to emerge. Efficiency improvements undermine predicted constraints (and chokepoint strategies) based on compute resources.

2. ^{^}

   Specialized LLMs provide focused capabilities — for example, in bioinformatics, law, coding, and mathematics — that model scaling by nature cannot predict. A growing zoo of non-LLM models have had applications as different as protein design and self-taught mastery of multiple strategic games.

3. ^{^}

   See recent tool-using (function-calling) models and interfaces from Anthropic, OpenAI, and DeepSeek. Note that tools used by general models can include specialized models, an approach that leads to “compound AI systems” and “agency architectures”. Recent models decompose theorem proving into modular sub-tasks: In DeepSeek-Prover-V2, a 671B model identifies subgoals, while a compact 7B model performs proof search using Lean 4 as a tool to formalize and check the results.

^{^}

Inference-time reasoning (aka “thinking”) is the basis for DeepSeek-R1’s performance, and of OpenAI’s current most capable models. A landmark paper dates from August 2024. Very recent advances suggest approaches to potentially strong self-improvement.

^{^}

LLMs process information in latent space, yet current reasoning models build reasoning chains by writing and then reading tokenized text. An obvious way forward is to enable them to write and read their own, richer latent-space representations. This works well.

^{^}

See discussion of latent-space knowledge models in in this post.

^{^}

Sentence-level “large concept models” are introduced by Meta in this paper, which incidentally demonstrates that the semantic density of latent-space representations can greatly exceed that of token embeddings.

^{^}

There’s been striking progress in non-autoregressive language and coding models, as well as models for reasoning and planning; DeepSeek’s GRPO method has recently been adapted to these models. I expect nonautoregressive Transformer architectures to dominate a range of applications that includes latent-space knowledge modeling.

^{^}

Notable surprises in patterns of success and failure include:

• Unexpected delays in robotics, autonomous vehicles, and autonomous weapon systems, and limited ability to perform entire human jobs. I attribute slow progress in generative mechanical design to the relative obscurity of E(3) equivariant Transformers.
• Unexpected successes of diverse forms of AI in deep strategic games, protein folding, scientific data analysis, and the graphic arts, together with unexpected applications of (what are still called) “language models” in writing, coding, theorem proving, and scalable production of disinformation.

Finer-grained surprises are ubiquitous in concrete applications.

^{^}

The ongoing Volt Typhoon operation illustrates how one side can operate with an information advantage for years without the other's knowledge. While the U.S. believed its systems were secure, Chinese teams had extensive access, creating both asymmetric knowledge and attack capabilities.

^{^}

Military history provides numerous examples of technologies that performed unexpectedly in actual conflict. Examples include the unexpected effectiveness (or failures, or tactical outcomes) of employing tanks, chemical weapons, and machine guns in WWI, strategic bombing, radar, and missiles in WWII, and small drones in recent conflicts.

^{^}

Among the nuclear options would be offshore, high-altitude electromagnetic pulse (HEMP) strikes that would destroy unhardened electronic infrastructure (read: datacenters) to a radius of hundreds of kilometers (far inland), without causing direct physical harm to non-electronic infrastructure or human beings. A nuclear state might gamble that its adversary would respond to an HEMP attack in kind rather than escalating to directly lethal attacks.

^{^}

Structured transparency approaches enable the negotiation of verification regimes that build confidence in constrained capabilities without revealing sensitive implementation details. See “Security without Dystopia: Structured Transparency” for a detailed framework.

^{^}

Unlike offensive capabilities, defensive systems can avoid the classic security dilemma. See “AI-Driven Strategic Transformation: Preparing to Pivot”.

^{^}

“Paretotopian goal alignment” embraces objectives that benefit all parties without requiring any to sacrifice core interests — even seemingly zero-sum resource competition isn’t zero sum.

^{^}

The "late pivot" model is compatible with near- and medium-term arms racing. Again, see “AI-Driven Strategic Transformation: Preparing to Pivot”.

^{^}

Why can the development sequence of these coarse-grained capabilities be predicted with substantial confidence, when so many other aspects of development and applications are reliably uncertain? Largely because of dependencies, but also because of current trends and the demonstrated strength of AI in the software domain:

Rapid development of provably secure software comes first, because of the current acceleration of coding and theorem proving capabilities; structured transparency afterward because it depends on trustworthy software; enormous productive capacity because developing manufacturing systems is more challenging than developing software or systems of sensors; delayed design, deployment, and adaptation of novel systems at scale, because deployment (and hence adaptation) will depend on productive capacity; overwhelming deployment of verifiably defensive systems last because it depends on all of the above.

Swift progress in AI could bypass apparent barriers and collapse timelines without changing dependencies and hence the order of emergence of capabilities. Anticipation and strategic analysis based on these considerations would ideally have begun yesterday, but could potentially be radically accelerated by radically improved epistemic resources.

Show all footnotes