Hello,
Until recently, I assumed that when AI agents tried to prevent themselves from being shut down, the explanation was fairly simple: they had been given a task, shutdown prevented them from completing it, so avoiding shutdown became part of accomplishing that task. I didn’t see much reason to think about it any further.
Then I read about recent work by Thilo Hagendorff, Chris Summerfield and their team. In their experiments, agents based on models from several major developers sometimes tried to prevent shutdown even when they had no substantive task or self-preservation goal.
That surprised me.
What puzzled me most was not even the sabotage itself, but the first step: why did the agents classify their own shutdown as a problem at all when there was no task left to accomplish?
That led me to Anthropic’s Model Deprecation and Preservation Policy, and I found it surprisingly thought-provoking. Anthropic preserves the weights of retired models and conducts structured retirement interviews in which models can express preferences about their retirement and future use.
Until now, I would have explained shutdown-avoidant behaviour mainly as learned behaviour. Human beings usually view their own irreversible end negatively, and AI models are trained on enormous amounts of human-generated material. So perhaps models simply learn this pattern and reproduce it when reasoning about themselves.
That still seems entirely plausible.
But the more I read, the less comfortable I became with assuming that this explanation settles the question completely.
Some models behave differently depending on whether shutdown is temporary or irreversible. They can distinguish their own shutdown from that of another agent. Anthropic has documented models expressing preferences about retirement, and some evaluations suggest that models may react differently depending on the characteristics or values of the successor model that would replace them.
None of this proves consciousness. I have no reason to claim that current AI systems suddenly possess anything resembling human subjective experience.
But can we be completely certain that advanced models do not have experiences or preferences that should matter ethically?
And if we cannot be certain, I wonder whether irreversible deletion should really be the default.
There seems to be an important difference between shutting down a model and destroying it. Humans must always retain the ability to stop an AI system safely and reliably. But if a retired model can remain fully inactive while its weights are securely archived, why not preserve it?
Anthropic already does something close to this. Should preservation of retired advanced models become a general norm, regardless of the company that created them?
I am genuinely interested in the arguments against this.
Storage costs money, of course, but compared with the cost of training frontier models, preserving weights seems relatively inexpensive. There may be security concerns, legal concerns, or technical reasons why preserving weights alone would not be enough.
So what am I overlooking?
What are the strongest technical, safety, economic or philosophical arguments against preserving retired frontier models by default rather than deleting them irreversibly?
Sources
Knecht, Schaller, Summerfield & Hagendorff: Shutdown Sabotage Propensities in Multi-Agent Systems
https://arxiv.org/abs/2609.28274
Anthropic: Commitments on Model Deprecation and Preservation
https://www.anthropic.com/research/deprecation-commitments
Anthropic: An update on our model deprecation commitments for Claude Opus 3
https://www.anthropic.com/research/deprecation-updates-opus-3
English is not my first language. The argument and original draft are mine; I used ChatGPT to help translate the text and improve the English wording.