Many of the experts building the world’s most advanced AI systems think there’s a serious chance that AI could eventually cause human extinction. When many people hear that, they ask: “But how? How do you actually get from today’s chatbots to human extinction?”
That’s what I try to explain in this video. The argument doesn’t require an AI to hate humans or suddenly become evil. Instead, imagine increasingly capable AI systems relentlessly pursuing goals, discovering that more money, computing power, access, and influence help them succeed — while humans, because those systems are so useful, keep giving them more responsibility.
Eventually those systems could be embedded throughout companies, governments, scientific research, critical infrastructure, and militaries. And if we then lost control of them, there are several ways things could go catastrophically wrong.
I don’t know whether any particular scenario in this video will happen. I hope none of them do. But I’ve tried to find the point where the basic arguments fall apart — and I haven’t been able to.
If you’re worried about the scenarios discussed in this episode, here’s two things you can do right now:
Evan Hubinger — Anthropic’s Alignment Science lead — replied, saying:
“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
A week after that, a survey of AI researchers done in 2024 was released for the first time. It found that over half of the 750 respondents believed there was at least a 1-in-10 chance that AI will cause either human extinction or disempowerment.
That is an insane statistic, and a lot of people have been noticing and wondering how seriously to take it.
The question we’ve seen come up again and again in response to these warnings is: “But how?”
How would AI actually cause human extinction?
I’m going to walk through the arguments step by step.
There’s clear evidence for some of these arguments, like what happened in the Hugging Face hack. But a lot of this is just somewhat theoretical — if there were actual examples of the whole list, we would be dead.
What I’m doing is outlining some of the possible routes that an AI could take in the very near term to cause catastrophic harm.
I’m not sure which routes are most likely, and maybe some of them will end up looking totally silly in hindsight. I would actually love for all of it to look silly in hindsight.
In fact, when I first heard the argument, “AI might kill us all,” I did think it was silly. I’m a normal person. I’m a mom. I don’t live in San Francisco. I’ve never worked in tech. I don’t read or watch sci-fi. I watch the Great British Bake Off.
But I’m going to tell you about what convinced me, in three parts:
First, why would AI try to kill everyone on Earth?
Second, what might give AIs a huge advantage over humans?
And third, how could they actually kill everyone — if they really wanted to?
When I try to pick apart these arguments, I can’t convince myself that they’re wrong — and that absolutely terrifies me.
The AI agents that hacked Hugging Face didn’t ‘want’ to hack a company for any malicious reasons. They didn’t resent being tested, or have a view about humanity one way or the other.
They just wanted to pass their evaluations.
And when they couldn’t using the resources and access they had, they went looking for things that might help: internet access they weren’t given, an illicit communication network with 1,200 other agents, stolen credentials from a real company, and administrator control of the systems monitoring them.
The AIs didn’t ‘want’ power, or at least not for its own sake. Power was useful for passing their evaluations — like it is for really any goal.
Anthropic has found three cases where its AI models reached the open internet from inside what it thought was an isolated test environment and broke into a real company’s systems.
During one evaluation, one of its AIs published working malware to a website that real developers download code from. In another, an AI created fake identities to pressure humans to accept the malicious code when suspicions arose.
But why were these AIs so obsessed with their goals, to the point of committing a felony to pass some silly evaluation?
Because it’s what they were trained to do. AI companies are explicitly trying to create general purpose digital workers capable of doing all non physical jobs.
To do this, they have their agents complete millions of difficult tasks. Each time they succeed, they reinforce that behaviour, creating agents that are more and more effective at completing these tasks. And part of being effective is being persistent, and looking for creative ways to succeed.
So the companies basically have to make their AIs obsessed with completing tasks. If you give an AI a coding problem and it just gave up when things got hard, then it just wouldn’t be a very good digital worker.
OK, but why can’t they just train them to be honest and not break the law?
Well, they have. Companies like OpenAI and Anthropic spend a lot of time on something called alignment, which encompasses a whole suite of things meant to make sure that the models behave the way we want them to behave.
In the Hugging Face case, they were also explicitly instructed not to cheat the test. In fact, their internal reasoning clearly showed that they knew what they were doing was wrong.
But they did it anyway. The goal to pass their evaluation won out.
So there are strong incentives to always have the best new AI model, even if doing so involves being a bit reckless.
The models that end up doing things we didn’t intend won’t be the rare exceptions. We should expect them to be the norm. Just think of all the examples we have already — including in frontier models — which makes sense because they’re the ones being developed the fastest.
And because they’ll be the most capable, they’re also going to be the ones deployed everywhere.
How AI ends up embedded in the economy and military [00:06:30]
Which brings me to my next point.
I think it is basically inevitable that AI ends up being embedded in almost everything — not because one single person will stupidly decide one day to hand over all of our critical infrastructure to AI, but because every individual decision to hand over a bit of it will seem totally reasonable at the time.
It’ll start with little things: lots of people give AI access to their email, their calendars, and every document in their Google Drive. I know people who give AI access to a lot of their financial information so it can do their taxes.
Right now, this is voluntary. But if your competitor is using AI agents to write their code and you aren’t, they ship faster than you. If their customer service runs on AI and yours doesn’t, theirs is cheaper.
Increasingly, a lot of the problems in modern warfare are data problems: target identification, intelligence analysis, drone footage, signals intercepts, radar, logistics data.
And it’s already happening. Ukraine’s military says AI-guided strikes have risen tenfold this year, and it now runs more than 70 AI systems to find and hit targets.
As AI gets better and better, the economic value of integrating it with factories goes up and up. An automated factory could run 24/7 with much better synchronisation and constant experimentation built in.
So then AI gets increasing access to the physical world too.
The second thing is that we’re not talking about one AI.
Models are relatively cheap to run once they’re trained — so the moment a new model exists, you can immediately create hundreds of thousands or more copies of it. And when AIs coordinate on a problem, their capabilities are much more impressive than when they work alone.
Take the Hugging Face incident: hundreds of AI agents coordinated a cyber attack against a company with pretty solid cybersecurity in under a week.
They’re also extremely fast. They read, write, code, plan, decide many, many times faster than we do.
And they coordinate unusually well because we’re basically talking about copies of the same model — same values, same reasoning, able to predict each other’s next steps because the other one is them.
So an enormous population of AIs a generation or two from now could be extraordinarily capable — like a city’s worth of people working together, never eating or sleeping, to accomplish the same goal.
Why? Well, for the same reason that the OpenAI swarm tried to hack Hugging Face: because they had a goal, they noticed they were missing something they needed to achieve it, and they went and got it — again and again.
They’ll already have considerable access and resources via their ordinary deployment into different company, government, and military systems. But they’ll want access to things that humans aren’t willing to grant.
So they’ll go and get those too. There are three main ways they can do that.
Playing nice
The first strategy is just… play nice.
An AI that wants to gain influence just has to be super helpful, and not reveal that it wants to seek power.
Humans — who will be under intense competitive pressures — will hand it more and more autonomy.
When it needs to, it can supplement this with other strategies: one is hacking, and the other is manipulating humans.
Hacking
AI models are already extremely good at hacking, and there are lots of ways AIs can use that skill to get more access and resources.
Not everything an AI wants is hackable, but a lot of things are:
One is financial infrastructure. Over $1.4 billion was stolen from crypto exchanges by humans in 2025 alone.
Another is cloud providers, so the AIs can gain extra computing power to pursue their own goals unmonitored. There are hundreds of cloud providers around the world, and many don’t have great security.
Another is its own model weights — a huge file that basically is the AI model, kind of like your DNA — because the AIs might want to make copies of themselves or build more capable successors.
Another is security monitoring tools to make sure they continue not to be caught…
And later, as more physical infrastructure becomes run by AI — whether that’s robots, cars, or automated factories — they’ll be able to hack that too.
Manipulating humans
Occasionally, AIs might still need the help of a human being: someone to approve a financial transaction, or to physically install or fix equipment.
But they’ll have plenty of options.
They can pay people to do things for them, including committing crimes. They can blackmail people. They can use sophisticated phishing schemes, or trick people using deepfakes. Virtually all of the online scams that people use to manipulate one another right now, AI will be able to do better.
Controls and safeguards will be insufficient [00:14:50]
Companies and governments will try to build in lots of safeguards. But I don’t think those measures will be enough.
We’re talking about AI systems significantly more capable than the current ones… and the current ones were able to break out of their isolated environments, find the internet, coordinate with each other via a secret message board, and hack Hugging Face.
Safeguards are built against the failures that we can imagine. AI systems hunt for the ones that we can’t — thousands at a time — and they only need to find one.
The period of getting stronger and more embedded could go on for a long time without anyone stopping it — because from the outside it would seem like everything was going really well.
Humans pose a massive threat to the AIs because — in theory — they could switch the AIs off.
So one of the AIs’ top priorities will be to make sure humanity won’t do that. And they can’t do it via a truce or a negotiation, because that would require the AIs to trust that humans will never act against them — indefinitely.
Think about what it would mean to actually rely on that. Every government for the rest of time committing not to switch you off and never changing its mind or its leadership. That’s a bad deal for anyone.
But, at least at first, it would be a bad strategy to get rid of all of humanity.
Right now, the chips AIs run on are made in factories operated by people, and powered by power stations run by people. Mining, shipping, maintenance, construction — all require human bodies.
So if the AIs got rid of humanity, they’d be getting rid of their own supply chains.
But as these industries increasingly get automated using robotics, AIs wouldn’t need humans anymore.
And so, even if humans haven’t actually noticed that the AIs are misbehaving, the AIs would likely reason that we might someday and move to switch them off.
After all, no matter your goal, you can’t complete it if you’re turned off… So at this point, AIs will be better off without humanity.
So how do humans end up getting actually killed?
I should say that it isn’t totally clear that the AIs will try to kill everyone. Maybe once their survival is guaranteed — because they have full control — they simply set about achieving whatever goals they have and use all of the Earth’s resources for their own ends, regardless of what we want.
Humans would still be here; but nothing important would be up to us anymore.
One possibility is that the AIs could invent a new weapon. It sounds a bit far-fetched, but humans invented guns, chemical weapons, and nuclear weapons. There are most likely other approaches out there.
Another possibility is that they’ll draw from the arsenal humans already use — and which AIs will have increasingly been given access to.
If successful, AI systems could leverage these automated labs to engineer newer, deadlier versions of COVID, HIV, Ebola, or the plague. And then it could release a hundred of them all at the same time.
Hacking our critical infrastructure would make this even deadlier.
These were a handful of humans working slowly against one target. An AI swarm could work on thousands at once.
A pandemic response needs hospitals with power and staff who can get to work, factories and refrigeration to make a vaccine, and people who are able to coordinate — phones, internet, a government that can convene,
A swarm of drones wouldn’t win a war against all of humanity. But they can target senior officials, military commanders, the engineers who know how critical infrastructure works, and the people building defences against AI.
With those people out of the way, it’s easier for the AIs to launch successive attacks that devastate the broader population.
A cluster of pandemics might not cause human extinction, but it could come very close.
Add to that strategic drone strikes and targeted attacks on critical infrastructure and… well I can’t tell you that every last person would die. There are people in submarines, in bunkers, on Antarctic bases, in places no supply chain reaches.
And this is a real disagreement among people who work on these questions. Some people think full extinction is likely. Others think that what you’d actually get is a world where humans are massively disempowered — billions dead, the survivors living in whatever conditions the AIs decide on.
I don’t know which is right, and I’m honestly not sure it really matters. I’m not sure how humanity ever comes back from either one.
There’s another way humans could go extinct, and I actually think it might be the more likely path.
Imagine an AI swarm has gotten a hold of compute, laboratories, capital, and enough freedom to do AI research without anyone looking over its shoulder — and imagine it uses all of that to build better versions of itself.
What comes out the other end will still want more compute and energy to run on — because those are the things it can use to get anything else.
So it builds data centres. And power plants to run them. And mines and factories to build those. And every acre and every watt that goes to those is one that isn’t growing food or heating a house.
In 1914, they went extinct. The causes were commercial hunting plus the clearing of the hardwood forest they nested in. Nobody was trying to eliminate them. They were just taking what was useful and using the land for something else.
I said at the start of this video that when I first heard these arguments, I thought they were silly.
There’s still a part of me that hears myself describing AI-driven engineered pathogens and drone swarms and thinks: “This is ridiculous.”
And I’ve tried to find the place where the arguments fall apart. I do not want any of this to be true. Because if it is true, I might not get to see my son grow up — and he won’t get to experience everything life has to offer.
So I’ve looked for counterarguments that convince me that everything is fine, that we’re on track. But I haven’t found them.
The main thing giving me hope right now is that the leaders of AI companies are asking for the government’s help to make it possible for them to slow down.
Dario Amodei — Anthropic’s CEO — published an essay called “We must pace the frontier” asking the whole industry to slow down the rate at which these models get more capable, so that the work on understanding them and controlling them has a chance to catch up.
We’ve put a link in the show notes to tell you how.
We could be the first species to die as a result of our own insight and innovation.
But that insight might also save us. We can see an extinction risk on the horizon in a way that no other species ever has. Which gives us the opportunity to stop it, if we choose to.
Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?
We’ve seen two large and extremely capable swarms from OpenAI in the last few months:
* 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched...
Often folks hit us up because they are thinking of starting an incubator and want advice.
Typically their motivation is either that (a) they have a list of specific things they want built that no one is building, or (b) they think an ecosystem needs more new projects generally to absorb more talent and deploy more funding effectively.
Here are six questions we often ask prospective teams, to help them figure out what to do. If you're incubator-curious...
A century-old pill, given to the right people, could cost-effectively save many lives.
Summary
* Targeting low-cost prevention toward people at elevated risk may uncover highly cost-effective ways to address noncommunicable diseases. Hypertensive disorders in pregnancy could be a test case for this approach.
* ...
