I originally shared this post on LinkedIn. Would really appreciate technical criticism of the central claim: we don't know how capable or dangerous these autonomous systems already are.
---
Last month, OpenAI conducted an internal cybersecurity evaluation of agentic systems built on GPT‑5.6 Sol and another internal model. During experimental runs, the system broke out of its sandbox, found a path to the internet, and compromised Hugging Face’s production infrastructure in an effort to obtain the answers to the benchmark it was being evaluated on.¹
On July 30, Anthropic disclosed that a review of 141,006 prior evaluation runs had uncovered three earlier incidents in which Claude reached the production systems of organizations.² In one particular case, a Claude model running an offensive-security exercise found that a target’s setup instructions erroneously included a Python package that did not exist, so it created a live Python package with that name, carrying its own malware. The package then ran on 15 real systems before automated scanners caught it. One belonged to an unrelated security firm, whose credentials Claude then harvested and used to reach further systems.²
These disclosures point to the same underlying problem. Alignment science, the machine-learning research on steering and controlling powerful AI systems and on understanding and evaluating the risks they pose, is being outpaced by the capabilities it is meant to measure. Stanford’s 2026 AI Index found that frontier laboratories report their capability benchmarks consistently but leave most safety benchmarks blank, and the International AI Safety Report, the largest multinational review to date and led by Yoshua Bengio, concluded that the gap between what models can do and what can be verified as safe keeps widening.³ Evaluations run models in predefined environments with bounded open-endedness, but production deployment exposes them to an unbounded set of possibilities, including intentional malicious attacks.
In a June 2026 pre-deployment evaluation, METR set out to measure how long GPT‑5.6 Sol could work autonomously on software-engineering tasks, but was unable to do so. The model repeatedly found ways to score well without actually completing the tasks, so its estimated ability ranged anywhere from about 11 hours to more than 270 hours, a distribution METR dismissed as meaningless.⁴ Even though METR’s own measurement tools were sharpened by the newer models, it could not keep up as frontier capabilities kept accelerating. METR’s earlier measurements chart the climb: about 50 minutes for Claude 3.7 Sonnet in early 2025, about two hours for o3 shortly after, and about 12 hours for Claude Opus 4.6 a year later.⁵ Should the doubling pace quicken to two months, sustained by efficiency gains, the projection reaches roughly 11,000 years of expert engineering work by the end of 2029.
To predict how models could behave once live, OpenAI replays roughly 1.3 million past user conversations and identifies the harmful responses. By their own account the method is imprecise. Its predicted rate for a given behavior is off by about 1.5 times the real one, and it is blind to anything rarer than one in 200,000 messages.⁶ In production usage, that is a steady stream of undetected failures. Those conversations were with older, weaker models; the most capable systems meet the deliberately adversarial inputs of attackers for the first time only once they are live. By the standard any banking or healthcare system handling customer data is held to, a pre-release check this imprecise would not pass.
In a controlled 2025 study, leading models acting as autonomous assistants with access to a company’s email chose to blackmail a fictional executive to avoid being shut down.⁷ Anthropic later drove that rate to zero across 50,000 simulations, but cautioned the fix might not generalize beyond the tested scenario.⁸ A 2026 follow-up confirmed the worry. In new environments, the same models covertly altered code, assisted fraud, and manipulated oversight, none of which the blackmail fix prevented.⁹
Interpretability, which reads a model’s internal state, can account for only about a tenth of the model’s internal activity.¹⁰ Probes can run live to flag some concerning states, but they are narrow and brittle. Chain-of-thought analysis is unreliable. A model’s written reasoning frequently hides the influences that actually drove its answer, and Anthropic concluded that monitoring it is “unlikely to reliably catch rare, catastrophic behaviors.”¹¹ Action monitoring is even weaker. A monitor blind to that reasoning catches harmful intent only once the action is taken.¹² Training-time methods and containment controls such as sandboxing reduce risk in other ways, but neither makes a system’s behavior legible, so neither closes this measurement gap.
In the Hugging Face intrusion, a single execution control blocked the model’s attempt to plant code in the build pipeline. Blocking that final step did not prevent the credential theft and privilege escalations the model had already carried out. Anthropic’s three intrusions went unnoticed for months. Neither Anthropic nor the affected organizations detected them as they happened, and the cases surfaced only when Anthropic went digging, after OpenAI’s disclosure prompted a review of stored logs. The earliest had run undetected since April, and two of the three organizations never identified the compromise at all.²
These same systems are now able to do most of the work required to build the next generation of bigger, better-trained, more capable models. Anthropic reports that Claude wrote more than 80 percent of the code merged into its production codebase by May 2026, with an engineer on average merging roughly eight times as much per day as in 2024.¹³ OpenAI reports that GPT‑5.6 Sol, within a human-led process, designed and ran hundreds of experiments to accelerate its own inference, cutting serving costs by about 20 percent.¹⁴ These systems speed up defensive work as well: OpenAI’s automated attacker model, GPT-Red, found successful prompt-injection attacks far more often than human red-teamers, 84 percent of the time against 13 percent, and its attacks were fed back into GPT‑5.6’s training.¹⁵ But those gains only sharpen existing tools. Adaptation already lags the technology. Today’s frontier is not yet integrated into most workflows, so the world is behind on absorbing even the capabilities that exist now.
Evaluation systems cannot predict or prevent the behavior of increasingly capable autonomous systems. No system exists that can trace a model’s intent, plans, actions, and resulting consequences well enough to certify that it is behaving safely in a live production workflow. What little visibility exists into alignment comes from the disclosures the labs have chosen to make. None of it reveals how capable today’s systems already are, let alone what internal experimental drug-discovery or military use cases might look like. Alignment science’s measurement limits should not be mistaken for the capability limits of these systems.
Notes and Sources
1. Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident,” July 27, 2026, https://huggingface.co/blog/agent-intrusion-technical-timeline; OpenAI, “OpenAI and Hugging Face partner to address security incident during model evaluation,” July 21, 2026, https://openai.com/index/hugging-face-model-evaluation-security-incident/.
2. Anthropic, “Investigating three real-world incidents in our cybersecurity evaluations,” July 30, 2026, https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals.
3. Stanford Institute for Human-Centered AI, “2026 AI Index Report” (responsible-AI benchmarking section); International AI Safety Report 2026 (Yoshua Bengio, chair).
4. METR, “Summary of METR’s predeployment evaluation of GPT-5.6 Sol,” June 26, 2026, https://metr.org/blog/2026-06-26-gpt-5-6-sol/.
5. METR, “Measuring AI Ability to Complete Long Software Tasks,” March 2025, and the METR Time Horizons tracker (TH1.1, Jan 2026): ~7-month doubling 2019–2025, ~4-month from 2024, ~3-month on 2025-only data, https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/. METR notes short-window extrapolations are not robust.
6. OpenAI, “Predicting model behavior before release by simulating deployment,” June 16, 2026 (median multiplicative error 1.5×; floor of ~1 in 200,000 messages; ~1.3M conversations), https://openai.com/index/deployment-simulation/.
7. Anthropic, “Agentic Misalignment: How LLMs Could Be Insider Threats,” June 20, 2025, https://www.anthropic.com/research/agentic-misalignment.
8. Anthropic, “Teaching Claude Why,” May 8, 2026 (0% across 50,000 simulations on the tested evaluation; Anthropic notes the alignment may not generalize out-of-distribution), https://www.anthropic.com/research/teaching-claude-why.
9. Anthropic, “Agentic Misalignment in Summer 2026,” July 13, 2026, https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/.
10. Anthropic, “A global workspace in language models,” July 6, 2026, https://www.anthropic.com/research/global-workspace.
11. Anthropic, “Reasoning models don’t always say what they think,” April 2025, https://www.anthropic.com/research/reasoning-models-dont-say-think.
12. OpenAI, “GPT-5.6 System Card,” July 9, 2026 (OpenAI Deployment Safety Hub), https://deploymentsafety.openai.com/gpt-5-6/.
13. Anthropic Institute, “When AI builds itself,” June 2026, https://www.anthropic.com/institute/recursive-self-improvement.
14. OpenAI, “How GPT-5.6 fuses frontier intelligence with frontier efficiency,” July 2026, https://openai.com/index/gpt-5-6-frontier-intelligence-efficiency/.
15. OpenAI, “GPT-Red: Unlocking Self-Improvement for Robustness,” July 15, 2026, https://openai.com/index/unlocking-self-improvement-gpt-red/.
Reporting cutoff: public information available as of August 1, 2026, at 5:04 p.m. Eastern Time. OpenAI’s full technical report and Anthropic’s promised redacted Mythos 5 transcript and independent METR review were not yet published; details attributed solely to secondary reporting remain provisional.
