Do you guys remember when for a brief moment of time in late 2025/early 2026, people on X would not stop asking Grok to generate undressed photos of real people? It was truly a frenzy, there was no way to avoid the constant stream of adulterated images of people, and it got to a point where it extended to children as well. X responded by limiting image generation to paid users eventually and X said some safeguards had been added, and the trend died down but users still reported finding workarounds with Grok afterwards.
Deepfakes are used to make inappropriate edits of women. This sort of vice is something women have always faced and not new due to AI. But the consequences are not the same everywhere, because in some countries this can lead to physical assault, acid attacks and honor killings.
Similarly political opponents can use deepfakes to make someone a target of terrorism, political abductions, false accusations or generally cause a lot of problems for them in the area.
Now the question is:
Didn't Photoshop capabilities already exist?
Non-consensual intimate imagery is something women and children have always had to deal with. But you needed someone with expertise, decent computer-use knowledge, and the resources to afford expensive editing software. Even using illegal software required prior knowledge of piracy, which implied decent access to computing and internet resources.
This was a bottleneck for proliferation. The results were also not always perfect: people could identify edited images if they were not made by a very skilled person. Video and audio editing were even more resource-heavy.
Generative models, even very cheap versions, removed the dependency on seeking out an expert to carry out this task. A lot of people can now make a lot of fakes at a very low price compared with a high-end photo-editing PC setup.
Then there are bot farms and bot families that proliferate this content at a higher rate. After that, social media algorithms give more visibility to such content as it gains engagement.
TikTok access means someone near you is likely to get exposed to this video: immediate or distant family, someone in your neighbourhood, a local politician with a lot of influence, or local blackmailers. They can decide to respond in whatever way seems appropriate to them.
So the whole system amplifies every part of this pipeline:
generation → bot amplification → algorithmic amplification via ranking algorithms → distribution to your real social graph where it has a local echo that no one controls.
We do not need superintelligence for any of this. We can describe the broad mechanisms at each step, and we know some of the ways in which they can cause harm.
And yet, once this chain has been set off, can you intervene? If you are a victim, can you completely stop the content from spreading or reaching someone you don't want it to reach?
Now imagine a superintelligent system, with components we won't understand, or mechanisms too internal for us to observe. A system where we don't know which action will set off which chain of risks.
Do you think we are prepared to intervene against risks arising in such systems?
When I first heard about AI x-risk in the news, I was like, "there is no way this could happen." It reminded me of the Age of Ultron movie, and I was sure we weren't living in an Avengers world. I thought surely, if something like that kicks off, people will realise it and stop it. Or, you know, we can always just unplug it.
I also thought that some countries would be less connected to these systems, and not everywhere in the world would be affected in the same way. I also assumed that critical systems, such as nuclear launch systems, will still be off-grid and airtight from the internet, so some factions would survive.
In the worst case scenario, maybe I just needed to move to a remote forest and wait it out while eating bananas with monkeys or something.
At the same time, being a Machine Learning Engineer, I was closely following research and researchers in the field and looked up to a lot of them, admired their mathematical sophistication and philosophical approaches. When even some of those people, who also had no industrial gain from contributing to hype, started talking about loss of control and existential risk, I was nudged into starting my own investigation.
Even then, my idea of the failure mode still had a human at the centre of it all: a power-seeking, mischievous overlord, willing to sacrifice millions of people and plunder the earth for some obscure power trip. We would all become slaves in his techno-feudal dystopia. I am not saying that concern has worn off or is any less serious, but it was Stuart Russell's Human Compatible that knocked my train of thought off its tracks.
In the beginning of Human Compatible, Stuart Russell uses social-media algorithms as an example. He describes how systems optimising for click-through can influence users' preferences, making their responses more predictable. These systems do not have to be particularly intelligent to affect the world and yet they already reach billions of people.
This caused me to think deeply about the evidence we had of how much at risk we are from existing systems, and how much our ability to stay safe is degraded, and how in that context, a superintelligent system might not even have to work super hard or have a long multi-turn run on an optimisation problem to make us all disappear into thin air.
I explored this idea in the beginning of this article with the example of Grok's incident, but a similar chain of events could set off any kind of catastrophic consequences, such as a large scale war.
If a military catastrophe happens, other parts of the system also get involved. Targets will now be identified with AI. AI can choose weapons, define strategy and give false positives.
In spring 2026, according to CNN, an analyst at a US command used a chatbot to examine the manifest of a Chinese vessel. The chatbot falsely claimed that the vessel contained components for a nuclear weapons program, and the analyst used AI to format this into a standard intelligence report. With armed personnel preparing to board the ship and planes already in the air, the mistake was spotted by a more experienced analyst. One source said the incident "almost started a war".
During the India-Pakistan conflict in May 2025 between two nuclear-armed states, both sides were treated to a barrage of AI-generated videos and cloned voices of political and military leaders "admitting defeat" or "apologising", purportedly burnt-out fighter jets, generals' purported statements, etc. Fact-checkers on both sides were still debunking them weeks after the four-day war had ended.
What would happen if a fabricated video reached someone in a military or political chain of command at a particularly tense moment?
We would need to check whether decision-makers in military and political chains of command see this content, how they fact-check it and how it impacts their decisions.
The risk assessment cannot stop at whether the model can generate a convincing enough video.
More generally, many critical infrastructure, financial, and internet systems already have failure points and dependencies. Increasingly, many institutions may rely on a small number of AI providers, so a change or failure in one place could affect multiple institutions. As new capabilities get integrated into these systems, it's important to understand what that enables across the whole stack.
I found Atoosa Kasirzadeh's Two Types of AI Existential Risk: Decisive and Accumulative (Philosophical Studies, 2025) to be a useful way to think about existential risk from AI. In it, Kasirzadeh contrasts a) risks from a powerful AI leading to a decisive catastrophe and b) accumulative risks. In the accumulative case, AI leads to a series of disruptions that cumulatively weaken societal resilience; some shock then leads to irreversible collapse. She uses a systems analysis and a boiling-frog analogy for gradual deterioration. Disruptions are modelled as they propagate and interact across political, economic, and military subsystems.
I think this is the right way to think about existential risk from AI, though I have four thoughts/disagreements:
1. "Smaller disruptions":
What counts as a "smaller disruption" is relative. From a global, macro-level perspective, the kinds of disruptions Kasirzadeh describes as lower-severity (e.g. manipulation, erosion of trust) are "smaller disruptions" that only lead to existential risk in aggregate. But for certain actors in certain contexts, capabilities like deepfakes can lead to people being killed by their families; this would hardly count as a "smaller disruption" for those people/families/villages. The same capability leads to different outcomes depending on the social and legal environment it's applied in; the multiplier isn't in the model, but in institutions downstream. Kasirzadeh's paper mentions inequality, but it doesn't analyse how vulnerability is distributed across regions, and I think that is where a lot of the action is. The frog is not boiling evenly; some areas of the pot are boiling.
2. Accumulative vs decisive risk:
Kasirzadeh's illustrative trigger is an AI-enabled cyberattack on power grids across three continents. I think accumulative and decisive risk are not alternatives but coupled; accumulative weakening of societal resilience increases the chance of a decisive catastrophe. I think the most likely triggers are local: at a flashpoint between nuclear neighbours, where all of the accumulated damage (eroded trust, saturated infosphere, AI in the intelligence and targeting loop) has lowered the threshold. In this scenario, the decisive event is not separate from the accumulative trajectory, but is the end of it. This is why I think we should be looking not just at frontier labs, but also at the weakest links in the system.
3. Single big events don't need superintelligence.
In the paper, the decisive pathway is linked to AGI or ASI: a single, very capable intelligence that does something catastrophic and fast. The accumulative pathway takes years of erosion and many individual disruptions. Between these two extremes, I think we should be worried about events that are fast, but don't require extraordinary intelligence. Ordinary (i.e. not very intelligent) AIs acting at critical chokepoints could cause decisive events; or multiple AIs could interact at speeds faster than humans can follow. For example, in the 2010 Flash Crash, automated trading systems interacting with each other caused close to a trillion dollars to temporarily evaporate from US stock markets within minutes. This was not an intended consequence. In 1983 a Soviet early-warning system indicated that American missiles were en route; Stanislav Petrov judged that it was a false alarm and the world was spared. This was a Petrov moment: the world was saved by the judgement of a single human at a critical chokepoint. In the incident mentioned above, it was a Petrov moment with a chatbot: an AI system made up a threat and presented it as a standard intelligence report. Happily, an experienced analyst spotted what was going on. But we are increasingly using AIs at critical chokepoints, are making these systems ever faster, and giving the humans less time and fewer reasons to question what the AIs are telling them. So I would suggest a middle category: the risk from integration, rather than from intelligence per se.
4. The proposed governance solution to accumulative risk in the paper is distributed monitoring, with sector-specific bodies keeping an eye on when disruptions are approaching critical thresholds, e.g. financial regulators monitoring systemic risk. I agree that we need monitoring, but I think we need to develop more break points in the chain as well, i.e. some point at which intervention reliably stops propagation, and someone with the incentive and authority to do so.
1. By zooming out, do you mean mechanistic looks inside the model are pointless?
Not at all, I used this term to describe the perspective of risk analysis and threat modelling that accounts for AI systems' interactions with our present world. AI models are a part of the system and thus mechanistic interpretability is a very useful tool because it contributes to the study of the intrinsic properties of a major part of the threat system. It can reveal its weaknesses too.
2. The harms that you discussed today cannot turn into existential risk
The structure of harms we see today is the legacy that more capable systems will be born into. They can also erode the self-correction we would need when something catastrophic DOES occur.
3. Are you mixing up correlation with causation?
I agree that causation is hard to prove.
I think in the case of x-risk, however, the proof that AI ‘caused’ it matters less than the existentially threatening event happening.
4. Can't we just fix the platforms?
We probably can for today's problems, but how do we do so for increasing capabilities of AI systems in the future?
5. ASI might never be achieved
I don't think the argument relies on it that much. Today's shocks (wars, crime waves, pandemics) already find the breaking points in our economies, societies and infrastructure, so more capable systems will only stretch them thinner.
6. "Systems Thinking" is very vague
It is indeed not valuable if you use it to just say "everything is connected" and leave it there. But if you take it link by link and ask who's responsible for this step, what would stop something from propagating here, and how do we know that stopped working, you might end up with concrete steps to prevent harm.
7. What would change your mind?
If accumulative damage turns out not to lower the threshold for decisive events, then my second disagreement with Kasirzadeh is wrong. And if humans at chokepoints keep catching AI errors as reliably as Petrov did, even as AI outputs get faster and more official-looking, my third point matters less than I think.
One of the many useful things I learnt from BlueDot's Technical AI Safety cohort is that if you have delved into a problem, and truly understood it, you should be able to at least give a high level idea of where you can be useful. The threat chains I discussed above belong to platforms, governments and communities.
However, AI developers also have a significant amount of control over evaluations, monitors and AI judges. They decide if a model gets deployed, if an agent's action is flagged, or if a training signal rewards the right behaviour. As models get more capable (and less observable) these types of controls become a bigger and bigger slice of what stands between a model and the world.
Zooming out to focus on AI risk means that the harms of today act systemically, that how severe a harm is also depends on where it lands, and that our bottleneck is breaking the chain, not tracing it. That's why I believe we'll need far stronger safeguards than we have now to actually catch issues (evaluations, monitors, AI judges).
So these are the questions I have started my investigations with:
Do our tests measure the model, or the wording?
Can the thing being judged talk the judge out of it?
What actually drives agents' behaviour towards each other and the tasks they perform?
I think behavioural and mechanistic investigations are both fruitful and needed, as complements, to answer these questions. In the end usefulness is the best guide to where effort should go. Right now, that means making our evaluations robust: not only for today's LLMs, but for different kinds of AI and for the methods coming next (continual learning, labs working towards recursive self-improvement, and so on).
In the end, along with the idea of gradual collapse, I maintain my belief in the probability of a single powerful catastrophic event, but if you don't, what is your opinion on the present realities of our world and how they interact with AI? What is your thinking about how incapable we are at handling current complexities and possible domino effects that can lead to both x-risks and s-risks today?
"Once he went into the mountains on a clear, sunny day, and wandered about for a long time with a tormenting thought that refused to take shape.
Before him was the shining sky, below him the lake, around him the horizon, bright and infinite, as if it went on forever.
For a long time he looked and suffered.
He remembered now how he had stretched out his arms to that bright, infinite blue and wept.
What had tormented him was that he was a total stranger to it all...
…And everything has its path, and everything knows its path, goes with a song and comes back with a song;
only he knows nothing, understands nothing, neither people nor sounds, a stranger to everything and a castaway."
- The Idiot, Fyodor Dostoevsky.
- Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. Philosophical Studies 182(7). arXiv:2401.07836.
- Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control.
- Grok image-editing incident, January 2026. CNN: https://edition.cnn.com/2026/01/14/tech/grok-elon-musk-image-generation · OECD AI incident: https://oecd.ai/en/incidents/2026-01-03-5c4c
- CNN (18 Sep 2026), reporting on the AI-assisted intelligence error involving a Chinese vessel. See also Gizmodo: https://gizmodo.com/almost-started-a-war-us-military-nearly-boarded-a-chinese-ship-based-on-bad-intel-from-ai-2000814290
- BOOM, misinformation report, May 2025 (India–Pakistan conflict): https://www.boomlive.in/boom-research/boom-misinformation-report-may-2025-operation-sindoor-india-pakistan-armed-conflict-28733
