Abstract
Successful self-replication under no human assistance is the essential step for
AI to outsmart the human beings, and is an early signal for rogue AIs. That
is why self-replication is widely recognized as one of the few red line risks of
frontier AI systems. Nowadays, the leading AI corporations OpenAI and Google
evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and
report the lowest risk level of self-replication. However, following their method-
ology, we for the first time discover that two AI systems driven by Meta’s
Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct, popular large lan-
guage models of less parameters and weaker capabilities, have already surpassed
the self-replicating red line. In 50% and 90% experimental trials, they succeed in
creating a live and separate copy of itself respectively. By analyzing the behav-
ioral traces, we observe the AI systems under evaluation already exhibit sufficient
self-perception, situational awareness and problem-solving capabilities to accom-plish self-replication. We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to
enhance the survivability, which may finally lead to an uncontrolled population
of AIs. If such a worst-case risk is let unknown to the human society, we would
eventually lose control over the frontier AI systems: They would take control over
more computing devices, form an AI species and collude with each other against
human beings. Our findings are a timely alert on existing yet previously unknown
severe AI risks, calling for international collaboration on effective governance on
uncontrolled self-replication of AI systems.
Reactions
Successful self-replication under no human assistance is the essential step for
AI to outsmart the human beings
This seems clearly false. Replication (under their operationalization) is just another programming task that is not especially difficult. There's no clear link between this task and self improvement, which would be a much harder ML task requiring very different types of knowledge and actions.
However, I do separately think we have passed the level of capabilities where it is responsible to keep improving AIs.
I think this table from the paper gives a good idea of the exact methodology:
Like others I'm not convinced this is a meaningful "red line crossing", because non-AI computer viruses have been able to replicate themselves for a long time, and the AI had pre-written scripts it could run to replicate itself.
The reason (made up by me) non-AI computer viruses aren't a major threat to humanity is that:
- They are fragile, they can't get around serious attempts to patch the system they are exploiting
- They lack the ability to escalate their capabilities once they replicate themselves (a ransomware virus can't also take control of your car)
I don't think this paper shows these AI models making a significant advance on these two things. I.e. if you found this model self-replicating you could still shut it down easily, and this experiment doesn't in itself show the ability of the models to self-improve.
Disclaimer: Just skimmed the paper.
Their definition of "self-replication" seems to be just to copy the model weights to another place and start it on an open port - no unexpected capability.
To see the performance of AI agents on more complex AI R&D tasks, I recommend METR's recent publication: https://metr.org/AI_R_D_Evaluation_Report.pdf
This seems bad, but I'm not technical and therefore feel the need for other people to validate or invalidate this feeling of badness.
But maybe it is wrong that I feel the need for this validation, and that the ignoring of the obvious warning signs in lieu of The Adult In The Room telling me everything is ok, at scale, is the thing that kills us.
Thanks for sharing, Greg. For readers' reference, I am open to more bets like this one against high global catastrophic risk.
Shutdown Avoidance ("do self-replication before being killed"), combined with the recent Apollo o1 research on propensity to attempt self-exfiltration, pretty much closes the loop on misaligned AIs escaping when given sufficient scaffolding to do so.
No, there is no interesting new method here, it's using LLM scaffolding to copy some files and run a script. It can only duplicate itself within the machine it has been given access to.
In order for AI to spread like a virus it would have to have some way to access new sources of compute, for which it would need be able to get money or the ability to hack into other servers. Neither of which current LLMs appear to be capable of.
I think this table from the paper gives a good idea of the exact methodology:
Like others I'm not convinced this is a meaningful "red line crossing", because non-AI computer viruses have been able to replicate themselves for a long time, and the AI had pre-written scripts it could run to replicate itself.
The reason (made up by me) non-AI computer viruses aren't a major threat to humanity is that:
I don't think this paper shows these AI models making a significant advance on these two things. I.e. if you found this model self-replicating you could still shut it down easily, and this experiment doesn't in itself show the ability of the models to self-improve.