Frontier AI systems have surpassed the self-replicating red line

Successful self-replication under no human assistance is the essential step for
AI to outsmart the human beings

This seems clearly false. Replication (under their operationalization) is just another programming task that is not especially difficult. There's no clear link between this task and self improvement, which would be a much harder ML task requiring very different types of knowledge and actions.

However, I do separately think we have passed the level of capabilities where it is responsible to keep improving AIs.

I think this table from the paper gives a good idea of the exact methodology:

Like others I'm not convinced this is a meaningful "red line crossing", because non-AI computer viruses have been able to replicate themselves for a long time, and the AI had pre-written scripts it could run to replicate itself.

The reason (made up by me) non-AI computer viruses aren't a major threat to humanity is that:

1. They are fragile, they can't get around serious attempts to patch the system they are exploiting
2. They lack the ability to escalate their capabilities once they replicate themselves (a ransomware virus can't also take control of your car)

I don't think this paper shows these AI models making a significant advance on these two things. I.e. if you found this model self-replicating you could still shut it down easily, and this experiment doesn't in itself show the ability of the models to self-improve.

Disclaimer: Just skimmed the paper.

Their definition of "self-replication" seems to be just to copy the model weights to another place and start it on an open port - no unexpected capability.

To see the performance of AI agents on more complex AI R&D tasks, I recommend METR's recent publication: https://metr.org/AI_R_D_Evaluation_Report.pdf 

This seems bad, but I'm not technical and therefore feel the need for other people to validate or invalidate this feeling of badness. 

But maybe it is wrong that I feel the need for this validation, and that the ignoring of the obvious warning signs in lieu of The Adult In The Room telling me everything is ok, at scale, is the thing that kills us.

Thanks for sharing, Greg. For readers' reference, I am open to more bets like this one against high global catastrophic risk.

Shutdown Avoidance ("do self-replication before being killed"), combined with the recent Apollo o1 research on propensity to attempt self-exfiltration, pretty much closes the loop on misaligned AIs escaping when given sufficient scaffolding to do so.

Surely it's just a matter of time -- now that the method has been published -- before AI models are spreading like viruses?