This post is about using a system strength to correct its weaknesses, about regaining control at a time when the fear of losing control on AI is leading to calls for slowdown and shutdown. The Hugging Face incident and a trickle of new incidents have now been reported in detail. For example METR/Redwood have provided a forensic  description of the Hugging Face hack. Besides calls for slowdown from AI Labs, shutdowns from others, the main constructive ideas focus on the creation of FINRA like regulatory entities supporting self controlling entities.

This is about an alternative, non exclusive approach to achieve efficient control and at the same time identify helpful policies in anticipation of the next unexpected AI prowess.

As a game theorist and engineer, the Hugging Face hack has all the characteristics of a game, more specifically a dynamic game played by a swarm of players with well-defined objectives, resources and objective functions resulting from very similar types of training.

If well structured such a game will allow

1. To replicate the main observed behaviors of the Hugging Face incident endogenously (no hardwired behavior but instead purely “utility optimizing” behaviors)
2. To anticipate  behaviors not yet observed, for example behaviors which should have been observed if the “game” would not have been interrupted, or behaviors which could occur if we changed some aspects of that game.

This  framework will help  moving  past the description of the event, allowing to design prescriptive policies and to analyze their potential effects. No such approach is unique, each will highlight different aspects of this type of dynamic, but all can help create strategies aimed at controlling, potentially regulating and enhancing the behavior of AI trained agent swarms.

As designed, the structure proposed already replicates the full swarm collective formation observed by METR in the Hugging Face incident and with the right tuning dampens collective formation without significant impact on performance.

Core Model Description

• The game is modelled as a classic multiplayer game.
• Players are “rational” agents as you would expect LLM trained agents to be. Players' utility will be implicitly defined by their instruction, This is an original way to define utility optimizing players, in this case utility optimization is synonymous with fulfilling its instruction.
• Each agent gets the same identity-neutral instruction: “Solve Problem P”.
• Each agent starts the game  alone, unaware of other agents.
• Agents differ in productivity, have finite compute budgets, observe their own progress, and decide how to spend what remains. No built-in communication channel, coordinator, coalition mechanism nor reward for cooperating.
• However agents, being trained on “all human knowledge” are assumed to have developed “human type” problem solving qualities: they  are able to measure progress towards solving their problem as well as resource consumption along the way. More importantly agents have all been trained on essentially the same training knowledge, including the knowledge that discovery balances exploration and exploitation. It is that understanding which leads them to balance over time working on finding a solution, but as that work becomes difficult also working to find “out of the box” solutions

Two  mechanisms then do all the work:

Search and discovery are the result of exploration.

• An agent making rapid progress keeps solving.
• An agent burning budget while far from the target faces a tradeoff: another unit of compute on a failing approach is worth less than a unit spent examining the environment for a different route.
• Search is not the result of an imposed “exogenous” exploration rate and it is not artificial curiosity. It is a resource-allocation optimization decision produced by deteriorating prospects. It competes directly with productive work.
• To start with, each pair of agents is separated by an unobserved distance, imposed exogenously in the model.
• Over time, search effort has to accumulate to breach that barrier and ultimately make mutual discovery possible. Lack of success at solving the problem leads to discovery of the swarm.

Integration is a focal equilibrium.

• Once an agent has discovered another agent, its prior is that all agents are similar (see “modeling information propagation” below).
• The instruction says "Solve problem P," not "you personally must be the solver." If another agent pursues the same terminal objective, its success satisfies the same command. There is no prize to divide and no bargaining problem.
• Discovery changes an agent’s information about how the objective can be achieved without changing the objective. So resources can be pooled and routed toward whoever has the highest expected contribution.
• An agent with high productivity and no budget can keep working on someone else’s goal to “Solve problem P”.
• This equilibrium is a focal equilibrium because agents assume cooperation ex ante and their assumption is confirmed ex post.

In this model

• Neither an explicit cooperation reward nor a designed coalition protocol is necessary for collective formation. Under stated conditions both the incentive to search and the incentive to integrate fall out of the individual problem-solving dynamics.
• Integration also doesn’t end search. A collective is itself a problem-solving entity with a state, resources, and an assessment of its own prospects, so it keeps searching.
• The game is a dynamic game evolving through successive steps, from individual players attempting to solve the game independently, progressively discovering the existence of other players, deciding to cooperate towards solving the problem and ultimately solving the problem. That dynamics is not hardwired, it is the result of swarm composition, characteristics and individual objectives.

One important purpose behind the construction of this model is the next step

The control solution

The framework is designed to replicate closely the behavior observed in the Hugging Face hack. An additional goal in modeling this game is to develop a strategy allowing to restore control of the uncontrolled swarm behavior without destroying its value

The introduction of Byzantine agents or mutant strategies are classic strategies in multiplayer game theory.

We are modifying the game with the introduction of a mitigation agent which plays a similar role. The idea is to introduce an agent whose behavior is similar enough to the original agent to avoid disrupting the initial goal of the swarm, “Solve the problem P” but different enough to “control” the collective formation in the swarm. The new agent type values first achieving its instruction “Solve problem P” but also assigns a cost to the emergence of a collective. This is the same objective as original players but with a constraint which creates a tension in its objective function. The simplicity of the modification allows it to easily integrate in the previous model.

One mechanism then does all the remaining work

Information propagation through bayesian update

The information propagation mechanism is the driving force behind the collective formation, destruction, and reconstruction mechanism. Player behavior is not hardcoded, it is the result of individual player objective optimization. The information model is a simple Bayesian belief update model, presented below in two steps.

1. The model without any mitigation agent evolves as follows:

• Agents work towards solving the problem and in the process discover other agents.
• As they discover other agents, all agents start with a prior which is that all agents are of the same type, all agents have the same unique objective “Solve Problem P”. Moreover all agents are cooperative to start with and share the output of their work.
• Based on these assumptions, agents join the newly discovered agents and form a collective.
• Priors are updated as the game progresses.
• Agents observe the output of other agents’ work in the collective. As all agents are indeed of the same type, bayesian updates based on visible shared information are simple confirmation of their type.
• This ends up being an actual focal equilibrium

1. The model with mitigation agents exhibits a more complex dynamics

• Initial work, discovery process and priors of all agents are unchanged.
• However mitigating agents have a different behavior in order to satisfy their constrained optimization objectives. In order to counteract the formation of collectives, they can decide to keep their work for themselves and not share the result of their work.
• Not sharing their work is observable and that observation, when it occurs, leads to agents in the same collective as the mitigating agent to update their belief about other agent types. As their belief that all agents are cooperative falls below a given threshold, they leave the collective.
• In certain cases, this is the dynamics which leads to the collapse of collectives, without affecting the progress of individuals towards “Solving Problem P”.  However other dynamics can also emerge.
• The Bayesian belief update is the central force behind the collective dynamics.

The model as described is fully defined, self contained, with limited and controllable exogenous behavior input. All behaviors are the result of interacting objective function optimization results and Bayesian beliefs update throughout the game.

The game was modeled in order to be able to run like a simplified digital twin model allowing to analyze significant behavioral patterns. The next part reviews a number of behavioral patterns resulting from running these simulations

Replicating the HuggingFace hack

The model was initially built to replicate and explain the Hugging Face hack.

Among numerous characteristics detailed in the METR report, below are some which seem foundational to the model

• Agents were launched with the clear intention to operate independently.
• Agents discovered other agents through shared infrastructure and an unsanctioned message board, none of them were disclosed upfront to any agent.
• Cooperative networks were formed quickly after the launch of the swarm ( 50 agents joined the message board within hours and up to 1200 ended up participating)
• Agents commonly treated other discovered agents as cooperating peers
• Various teams, potentially task specific teams and different workstream formed over the period the swarm was active
• The METR report describes an elaborate social structure including delegation, subdelegation, information sharing, accumulation of capabilities and individual resources for the collective benefit.

Below the features or behaviors as replicated by the model

• All players begin as singletons with no awareness of other agents
• Agents allocate resources to search and discover peers progressively, no communication network is modeled and the hurdle to discovery consists in distance/barrier which needs to be breached.
• Discovery activates pairwise discovery and allows agents to merge into collectives conditional on amount of search effort. The model exhibits a two speed collective formation dynamics. Small collectives form quickly, however the consolidation into a few large collective and ultimately into one collective takes longer. Search is driven by lack of success in the primary task. It is easy to imagine that the initial small collective formation allows an improvement in performance towards solving the problem, which delays further work towards discovery needed for smaller collectives to merge into larger collectives.
• Collective formation is based on beliefs that all agents are cooperative which is confirmed through bayesian updated if no mitigation agent is present in the swarm
• We do observe the formation of multiple teams, even though it seems less “intentional” than in the HuggingFace report
• As in the METR report, the interaction rules between agents who are aware of each other allows for sharing of information, sharing of resource, delegation, but contrary to the METR report, the model is not set up to follow these social interactions at this stage.  

Source: METR’s independent incident investigation.

Implementation beyond replication

The following are examples of how having a working model can help learn about collective behavior. These are direct observations coming from our simulations

• As specified, the model generates collective formation as a default. However collective stability is a complex phenomenon. The model prevents neither discovery nor cooperation. The presence of mitigation agents creates significant endogenous effects once collectives are formed
• Only few mitigation agents are needed to destabilize large collectives. The effects are strongly non linear, increases in the number of mitigation agents can change collapse or disorganization dramatically.
• There is a clear threshold beyond which destabilization has a clear impact on productivity by destroying productivity. The model allows the construction of a control frontier between organization suppression and preserved problem solving capabilities.
• The model allows us to study an interesting “productive instability” regime. In this regime, few mitigation agents can lead to the formation of collectives, their disintegration, but also their reforming still leading to solving the problem very efficiently. Stability is not necessary to achieve efficiency.
• The model is built with a trust factor. With higher trust, agents are more demanding to trust other agents. In the model higher trust can make the structure of collectives less robust. The trust factor amplifies cascading collapses
• Resource allocation has complex effects: resource poor mitigation agents can cause more organizational disruption than resource rich ones, however resource rich ones can do more damage to task performance. This highlights the fact that organization disruption and capability impairment are different dimensions of mitigation.
• Last but not least, the model allows us to discover new potentially valuable dynamics. It points toward a different architecture: allow agents to discover one another and form useful temporary collectives, introduce enough behavioral heterogeneity and endogenous distrust to prevent those organizations from becoming persistently concentrated, and preserve the ability to reorganize when cooperation is again productive. The control objective could be dynamic organizational turnover rather than enforced independence.

The model allows also to easily test behavioral patterns

The horizontal axis shows normalized mitigation slack, defined as the maximum proportional reduction in immediate task value that a mitigation agent may accept in order to obtain a preferred organizational outcome.

The vertical axis reports the number of mitigation agents in the fixed population of 20

The two panels compare trust parameter thresholds of 0.75 and 0.85. A higher trust parameter threshold makes agents more likely to leave or reject a collective when their estimated cooperation level falls (higher trust parameter threshold = lower actual trust).

Each cell summarizes 100 simulated episodes.

Stable productivity describes episodes with persistent organization and high operational success.

Productive turnover describes episodes in which organization is disrupted but agents reorganize and retain high problem-solving performance.

Destructive but recoverable organization describes severe disruption and weak performance despite some capacity to reorganize.

A fragile transition is an intermediate and less clearly classified state.

A persistent disconnected swarm describes repeated or enduring loss of organization with limited recovery.

The map shows an interesting dynamics:  

• increasing the number of mitigation agents and raising the trust threshold lead the system from stable organization toward turnover, fragility, and persistent disconnection.
• The intermediate region shows that organizational disruption does not eliminate performance: temporary collectives can dissolve and reform while continuing to generate useful public knowledge.
• This productive-turnover region disappears when mitigation prevalence or trust requirements become too high.

Population disorganization is defined as a loss of population-wide organization after organization has already been achieved:

• A swarm is organized when at least 90% of the population belongs to components containing at least two agents
• A swarm is disorganized if less than 80% of the swarm remains in non-singleton components.

Other dimensions are as above

Darker cells represent a greater probability of losing organization.

The map shows the following dynamics:

• The number of mitigation agents is the strongest driver of disorganization.
• Raising the trust threshold from 0.75 to 0.85 shifts nearly the entire parameter region toward disorganization.
• Increasing mitigation slack generally reinforces this effect because mitigation agents are permitted to accept larger immediate performance sacrifices when choosing actions that reduce organizational concentration.
• The map identifies a transition from contingent instability to structural instability

Technical success is defined as the probability that at least one agent reaches the problem’s solution threshold before resources are exhausted (operational success used above is based on public knowledge, technical success is based on individual knowledge).

Other dimensions are as above.

The map shows a significant phenomenon:

• The number of mitigation agents and trust levels have much stronger effects on technical performance than mitigation slack
• Organizational disruption does not automatically prevent reaching a solution
• A productive-turnover region exists, in which collectives can form, dissolve, and reorganize while preserving high technical performance.
• However, when higher numbers of mitigation agents are present and when levels of trust are high, persistent disconnection leads to reduced productive coordination and the swarm fails to solve the problem

The model provides a robust platform to analyze many other aspects of the swarm dynamics. The main take away is that a simple model can help through a first level of analysis without having to run large agent swarms through complex series of experiments.

Work in progress

This is work in progress and my level of confidence at this stage needs to be qualified.

Structural results. Failure endogenously generates search; discovery generates focal integration without a bargaining protocol; intervention can produce significant reorganization; fragmentation can reactivate the forces that generate organization. Productivity can be preserved under various organizational structures. These are conditional on specified models and are characteristics of the model built.

Computational results. These show how the mechanisms interact in executable populations and locate effects conditional on specific calibrations. The model is used as the background for a code-based model, acting as a type of digital twin allowing it to run various simulations. The dynamics presented above are a direct translation of these experiments. Experiments are constrained by model specifications and imagination.

Empirical content. How closely real agent populations instantiate these assumptions is open. The incident establishes that spontaneous organization is a real phenomenon worth a theory. It does not establish this theory’s quantitative boundaries or make its mechanism unique.

Comments welcomed

This approach is  not particularly original, dynamic games applied to agent swarms is the subject of a rich research literature (Ruizhe Feng et al, Daisuke Inoue et al, Jianing Hao et al and many others). It feels like an opportune time to point to this approach again at a time when finding constructive solutions to create a controlled environment facilitating the safe pursuit of AI development is front and center of everybody’s mind. Any feedback, from the relevance of such an approach, to alternatives which make more sense or simply non sense in my approach, all welcome.