Comments
Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?
We’ve seen two large and extremely capable swarms from OpenAI in the last few months:
- 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched a sophisticated criminal attack on the AI company Hugging Face.
- A swarm of 10,000 agents solved a version of the longstanding Navier-Stokes problem in mathematics. It took them just 88 hours to do so, in which time they sent 5 million messages to each other and used 300 billion tokens.
No doubt we will soon see even larger swarms with even more impressive capabilities. But they are not cheap. It is estimated that the swarm of 10,000 agents cost about $20 million at API prices. So while they are very powerful, it will be some time before we see the million-fold reduction in cost needed for this level of power to be possible on a $20/month plan. We should be thinking of it as a grand demonstration of what is possible when money is little constraint — like AlphaGo — rather than a new level of performance for the same cost.
A good way to see AI swarms is as a new form of inference-scaling. The main form of inference-scaling at the moment is having the agent spend more and more time on the task — increasing the maximum length of its chain of thought before it needs to give a final answer. This buys more capability, but at an increasingly expensive price. As measured by performance on maths benchmarks, this performance increases only logarithmically with the amount of compute used. I’ve previously shown that climbing from about 20% to about 80% on a reasoning benchmark typically requires scaling up the length of the chain of thought (and thus the number of tokens, the amount of compute, and the cost) by roughly 100x.
HOW DO SWARMS SCALE?
How do capabilities scale if we instead increase the number of agents in the swarm? There isn’t much data on this — especially for frontier systems like OpenAI’s recent swarms. But OpenAI’s launch post for GPT 5.6 Sol includes some charts containing just enough information to allow one to tease-out an answer.
The chart below shows the performance of three sizes of swarm as their chains of thought are lengthened. Each swarm size displays the usual kind of steep diminishing returns to more reasoning tokens.
Note how the single-agent ‘swarm’ (in light blue) is the most efficient, reaching each level of capability for far fewer total reasoning tokens. Indeed, it appears to use about half as many total tokens as the 4-agent swarm, which uses about half as many as the 16-agent swarm. If we redraw this graph on a logarithmic x-axis, we can see this more easily:
Now we can clearly see that the scaling curve for each swarm-size has logarithmic returns to longer chains of thought (because they are straight lines when plotted on a logarithmic x-axis) and that they have roughly equal slope, meaning that the scaling dynamic remains the same for all these swarm sizes.
We can also see that the 4-agent swarm is stably requiring about twice as many total tokens as the 1-agent swarm to get the same performance, and that the 16-agent swarm is needing roughly twice as many again.
But we don’t yet have a chart that shows how capability increases if we just scale swarm size (leaving the chain of thought length fixed). The experiments OpenAI ran didn’t include this. They didn’t run different swarms of different sizes with exactly the same chain of thought length to see what would happen.
Luckily, we can simulate this from their data. Let’s use the same starting point they did — 1 agent with its lowest reasoning level. Then we ask what would happen if we used a 4-agent swarm with the same average tokens per agent (=4x the total tokens). We can find that point on the 4-agent curve. Because the curve is so straight, the interpolation should be quite reliable. We can then ask what would happen if we scaled up to 16 agents, without increasing the average tokens per agent, by finding the point on that curve with 4x as many total tokens. Let’s plot these in green on the same chart:
We can now see how much we get from purely increasing the number of agents (swarm scaling), and how it compares to purely increasing the length of the chain of thought (duration scaling). The swarm scaling gives a little over half as much gain in capability for the same scale-up of compute. Or put another way, you need to do the scale-up of compute twice to get to the same capability, squaring the total multiplier needed.
Economists have a nice way of thinking about this. They have studied how having many people work on a task can get it done sooner, but usually at the expense of more total person-hours of labour. A convenient way to think about it is that people working together get as much done as one person working for times as long. Here is a parameter measuring how parallelisable the task is. They call it the ‘stepping on toes’ parameter. If = 1, you have a perfectly parallelisable task, with no stepping on toes and no efficiency penalty. But in reality is usually between 0 and 1 — allowing more people to help, but with diminishing returns. For example, if = 0.5 then 100 people working together get as much done as 1 person working for 100 = 10 times as long.
This allows us to state the swarm scaling behaviour more precisely. In the graph above, the slope of the green line is actually 57% the slope of the blue lines, so = 0.57. This means that scaling up the swarm size by 16x would give the same performance as scaling up the length of the chain of thought by just 16 = 4.9x. And if you check the chart, you can see that the light blue single-agent curve reaches the same score as the 16-agent point on the green curve after just a 4.9x scale-up.
The GPT 5.6 launch page includes swarm results for 3 different benchmarks. I asked Claude Opus 5 to determine the values of for each of them. It ran more careful regressions and got values (and confidence intervals) of:
- BrowseComp: = 0.68, 90% CI = [0.63, 0.76]
- SEC-Bench Pro: = 0.57, 90% CI = [0.52, 0.61]
- Terminal-Bench: = 0.48, 90% CI = [0.40, 0.57]
These are very much in line with estimates from economists for the diminishing returns of human teams. The precise value of clearly depends on the kind of task, as we see here with these three benchmarks — some kinds of task are inherently more parallelisable than others. And it may also depend on the scale of the swarm. Here the estimates when scaling up from 1 to 4 agents were similar to scaling up from 4 to 16, but that may no longer be true when scaling from 1,000 to 4,000 — again this will depend on the task. e.g. the task of building 100 brick walls is almost perfectly parallelisable up to = 100, where it becomes much worse.
IMPLICATIONS
Now that we have some preliminary measures of , what do they imply?
First, we can use it to convert between swarm scaling and the more traditional duration scaling. Let’s take the estimates of as 0.48, 0.57, and 0.68. This means that scaling up the number of agents in the swarm by 10x doesn’t get as much performance as using 10x as many tokens with one agent. Instead it gets 10x as much — which is 3x to 5x. And this shortfall accumulates quickly for larger scaleups, with the swarm falling further and further behind. To get the same performance gain as a 100x scale-up of the number of tokens for a single agent you need to scale up the swarm size by 900x to 15,000x.
So why would you ever use swarms?
The most important answer is speed. The 4-agent swarm needed about twice the total number of tokens to get the same performance, but in terms of tokens per agent, it only needed half as many. Since the agents are run in parallel, this means it can theoretically achieve the same task in half the time. And the same was true when moving from 4 agents to 16. In total, one could achieve the task in about 1/4 the time for 4x the cost. In reality, the speedups aren’t quite this good (perhaps because some agents use more tokens than the average), but they are substantial. So for situations in which you’d pay a large premium for speed, swarms can be very useful.
This fits closely with how Noam Brown described it on the Dwarkesh podcast:
Basically if you have four agents working on the problem, it is done twice as fast. So you're basically paying, because there's four agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern, it's like a little less efficient, but you continue to see that performance.
More generally, growing the number of agents in a swarm by a factor of could theoretically speed them up by a factor of but uses times as much compute. (Assuming stays constant over that scale-up.)
There may also be other advantages to multiple agents on top of speed. For example, if you keep increasing the length of the chain of thought for a single agent, the performance eventually plateaus. But the height of the plateau for a 1,000-agent swarm may be greater than for a single agent. However, for now, the main demonstrated reason is speed.
A second implication of concerns the possibility of intelligence explosions. I first encountered when studying recursive self-improvement (RSI). In the most common models of RSI, is one of the key parameters for determining whether the rate of growth of AI capabilities explode towards a vertical asymptote. That happens whenever > 1, and is proportional to , so high makes intelligence explosions more likely.
The prominent AI Futures Model for RSI uses a default estimate of = 0.5, while Tom Davidson and Tom Houlden’s median estimate is = 0.6. So these measured values that I’ve derived from OpenAI’s data are pretty much exactly as expected. I’d hoped that the value of for AI agents would be lower, making an intelligence explosion less likely, but that appears to not be the case. People should keep tracking this as new estimates for appear and (especially) when new orchestration methods increase the value of for a given type of task.
THE NAVIER-STOKES SWARM
When OpenAI announced their 10,000-agent swarm had solved the Navier-Stokes problem, they included this graph:
View fullsize
They have done their best to remove most of the useful information from this graph (such as the x-axis labels) and they don’t even say what form of inference compute is being scaled. But given that they went all the way up to 10,000-agent swarms, I’d bet it is tracking the number of agents in the swarm (i.e. that it is tracking total tokens spent, but that the main difference between data points is the number of agents in the swarm rather than the tokens per agent).
One thing I immediately noticed was that while it has the usual logarithmic performance gains, the slope of the curves is about half that of the inference-scaling curves for o1 and o3. So instead of requiring a ~100x scale-up of compute to go from 20% to 80% on the benchmark, it is requiring 10,000x the compute. I first wondered if this was due to the unusual benchmark of open math problems — perhaps the standard deviation of their difficulties is twice that of the problems in the AIME maths benchmark or the ARC-AGI-1 benchmark. That could still be right, but since we’ve now seen that is around 0.5, this alone would be enough to perfectly explain the halved slope of these scaling curves.
The other interesting thing about this chart is that the jump up from GPT-6 Astra’s performance to the performance curve of the internal model is much larger than we’ve seen before. The jumps from o1 to o3 and from o3 to GPT-5 were enough to allow the better model to get the same performance as the prior model using about 1/3 as many tokens:
View fullsize
But on their new chart, the internal model is getting the same performance as GPT-6 Astra for about 1/100 the tokens. That’s like 4 previous jumps in one. Even if we adjust for the slopes being lower due to swarm scaling, it would still be 2 jumps in one. Whatever changed was a big deal.
Indeed on the Dwarkesh podcast, OpenAI’s Noam Brown took pains to explain that the dramatic success of solving a Millennium Prize problem wasn’t primarily due to the large scale of the swarm, even though that was the part that seemed most unusual with their setup:
I wouldn't even attribute 10% of the credits to multi-agent. The reality is OpenAI has trained a very powerful model.
From the data they’ve released, I think that’s right. The multi-agent swarms helped them go fast enough to scoop Anthropic (and academia) by getting the result in just 88 hours, but it probably made the project much more expensive too. e.g. if we (somewhat heroically) assume = 0.5 at all points in the scaleup from 1 to 10,000 agents, then they could have got the same result for 10% of the cost in 10x the time (37 days) by using 100 agents, or at 1% of the cost in 100x the time (1 year) using 1 agent.
16
