Comments
The video is about extrapolating the future of AI progress, following a timeline that starts from today’s chatbots to future AI that’s vastly smarter than all of humanity combined–with God-like capabilities. We argue that such AIs will pose a significant extinction risk to humanity.
This video came out of a partnership between Rational Animations and ControlAI. The script was written by Arthur Frost (one of Rational Animations’ writers) with Andrea Miotti as an adaptation of key points from The Compendium (thecompendium.ai), with extensive feedback and rounds of iteration from ControlAI. ControlAI is working to raise public awareness of AI extinction risk—moving the conversation forward to encourage governments to take action.
You can find the script of the video below.
In 2023, Nobel Prize winners, top AI scientists, and even the CEOs of leading AI companies signed a statement which said “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
But how do we go from ChatGPT to AIs that could kill everyone on earth? Why do so many scientists, CEOs, world leaders expect this?
Let’s draw a line of AI capabilities over time. Back here in 2019 we have GPT2, which could answer short factual questions, translate simple phrases, and do small calculations. Then in 2022 we get models like GPT3.5, which can answer complex questions, tell stories, and write simple software. By 2025 we have models that can pass PhD-level exams, write entire applications independently, and perfectly emulate human voices. They’re beginning to substantially outperform average humans, and even experts. They still have weaknesses, of course, but the list of things AI can’t do keeps getting shorter.
What happens if we extend this line? Well, we’d see AIs become more and more capable until this crucial point here, where AIs can design and build new AI systems without human help. Then instead of progress coming from human researchers, we’d have AIs making better AIs, and the line would get a lot steeper.
If we keep going from there, we hit this point, where AIs are superintelligent, better than humans at every intellectual task—better than all of humanity put together—spontaneously producing breakthroughs in research and manufacturing, responsible for most economic growth, completely transforming the world.
And if we keep extending the line, eventually we could get to this point here: incomprehensible machine gods which tower above us the way we tower above ants, seemingly able to reshape the world however they want, with or without our involvement or permission. If we do reach this point, probably humanity would go extinct for the same reason we drove so many other species extinct: not because we wanted to wipe them out, but because we were busy reshaping the world and it wasn’t worth our effort to keep them around.
When would this happen? We’re not sure. Are there other ways the future could go? Certainly. But if we keep building more powerful AIs the way we’re doing it right now, then this looks like the default path. And this default path is terrible for humans.
Why? Because we’re great at making more powerful AIs, but we haven’t yet figured out how to make them do what we want. We’ve made a bit of progress: AI assistants are usually resistant to things like explaining how to make bombs. But they’re not that resistant.
And so far we’ve just been trying to control things weaker than us, where we can catch when they try to lie to us. But once we hit this point, that stops being true, and the problem becomes much harder. In fact, nobody knows how to do it. That’s why experts including Nobel prize winners and the founders of every top AI company have spoken out about the risk that AI might lead to human extinction.
The point of this video is not to say that this outcome is unavoidable, but instead to say that we do have to actually avoid it. So let’s look at this line in a bit more detail: what happens if AIs do just keep getting smarter?
Let’s think about this bit of the line first: from AIs right now to AGIs - Artificial General Intelligences. What does ‘general intelligence’ mean? For the sake of this video, let’s say it means whatever a human could do on a computer, an AGI could also do.
AIs can already do the basics: we have systems you can connect to a regular computer and then they can look at the screen and figure out what’s going on, send messages, search the internet, and so on. They’re also getting pretty good at using external tools. Obviously they can’t video call people, but they… well no, actually they can totally do that.
So they’ve already got the building blocks: the question is how well can they combine them? They can send slack messages, but can they run a company? They can write code, but can they write a research paper?
And the answer is ‘not yet, but eventually, and maybe quite soon’. After all, they just keep getting better, and if we just extrapolate forwards, it sure looks like they’ll reach this point sooner or later. The jump from ‘write a short story’ to ‘write an excellent novel’ is pretty big, but remember that five years ago they could barely write coherent sentences.
Now, many people doubt that this can happen, usually for one of two reasons.
One is that AIs aren’t ‘really thinking’ deep down. It’s interesting to think about, but whether AIs have genuine understanding isn’t really the relevant question here: what matters is what kinds of tasks they can do and how well they perform at them, and the fact is they just keep getting better. It doesn’t matter whether AIs really understand chess in the way humans do, what matters is that they can easily crush human world champions.
The other main reason for doubt is the idea that AI is going to hit some kind of wall, and some tasks will just be too hard. But even if this does happen, it’s very hard to say when, and people have been predicting it incorrectly for years. From chess to open-ended conversation to complex images, people keep expecting AI progress is about to reach a limit, and they keep being wrong.
The way we train AIs draws on fundamental principles of computation that suggest any intellectual task humans can do, a sufficiently large AI model should also be able to do. And right now, several of the biggest and most valuable companies in the world are throwing tens of billions at making that happen.
At a certain point, this line is probably going to start curving upward. Why? Because AI will become capable enough to help us make better AI. This is Recursive Self-Improvement: every generation of AIs making the next generation more powerful.
This will happen very easily once we have AGI: if an artificial general intelligence can do any computer-based task that a human can do, then it can work on making better AIs. But in fact, AI is already starting to contribute to its own development, for example by
and
Over time we’ll only see more of this: AIs doing better and more independently, and even doing things like developing new algorithms, and discovering novel ways to interact with the world.
As far as we can tell, it’s not like there’s going to be one specific day where AIs tell researchers to step away from the computer. If anything, it will probably look more like what we’re currently seeing: humans deliberately putting AI systems in the driver’s seat more and more because they keep getting faster, cheaper, and smarter.
Beyond a certain point they’ll be doing things humans can’t do—they’ll be far more capable, coming up with innovations we wouldn’t be able to think of and perhaps wouldn’t even be able to understand. And every innovation will make them better at discovering the next one.
So what comes next? What happens if AIs start making better AIs?
Well, the next step after artificial general intelligence, is artificial superintelligence - ASI. An ASI isn’t just more capable than any human, it’s more capable than every single human put together. And after AGI this might not take long.
After all, once you have one AGI, it’s very easy to make more: you can just copy the code and run it on a different server. This is already what happens now: once OpenAI was finished training ChatGPT, it was able to put out millions of copies of it running in parallel shortly after.
And there are other advantages AIs have. For one, they’re much faster: an AI like ChatGPT can easily produce a page of dense technical writing in under a minute and read thousands of words in seconds. The best chess AIs can beat the best human players even if the AI is only given a second to pick each move.
On top of that, it’s much easier for AIs to share improvements. If one of these AIs can figure out a way to make itself smarter, it can copy that across to the other million AIs that are busy doing something else.
What’s more, there’s no reason to think individual AIs will stall at whatever level humans can reach: Historically, that’s just not how this works. If we look at the tasks where AI reached human level more than five years ago—board games, facial recognition, and certain types of medical diagnostics—they are still continuing to get better.
If you just imagine an average human who miraculously had the ability to memorise the entire internet, read and write ten times faster, and also clone themselves at will, it’s pretty obvious how that could get out of hand. Now imagine that the clones could figure out ways to edit their own brains to make every single one of them more capable. It’s easy to see how they might quickly become more powerful than all of humanity combined.
So where does it end? Let’s zoom out the chart a little bit and see what’s going on up here at the top, at this point we’ve somewhat fancifully labelled incomprehensible machine gods.
Let’s be clear: AI isn’t going to start doing magic. We can be confident they aren’t going to invent perpetual motion machines or faster-than-light travel, because there are physical limits to what’s possible. But we are absolutely nowhere near those limits.
Modern smartphones have a million times as much memory as the computer used on the Apollo moon mission, but they’re still another million times less dense than DNA, which still isn’t all the way to the physical limit.
Let’s think about how our current technology would look to even the best scientists from a hundred years ago. They would understand that it’s hypothetically possible to create billions of devices that near-instantly transmit gigabytes of information across the world using powerful radios in space, but they’d still be pretty surprised to hear that everyone has a phone that they can use to watch videos of cartoon dogs.
We don’t know what artificial superintelligence would be able to do, but it would probably sound even more crazy to us than the modern world would sound to someone from 1920.
Think about it this way. So, progress happens faster and faster as AI improves itself. Where does this end? The self improvement process wouldn’t stop until there were no more improvements that could be made. So, try to imagine an AI system so powerful that there’s no way to make it any smarter even by using the unimaginably advanced engineering skills of the most powerful possible AI. Such a system would be bumping up against the only limits left: the laws of physics themselves. At that point, AI abilities still wouldn’t be literally magic, but they might seem like it to us. If that happens, we will be completely powerless compared to them.
For this to go well, we need to figure out how to make AIs that want to do what we want. And there are lots of ideas for how we could do that. But they are nowhere near close to done.
Our best attempts to make AIs not do things that we don’t want them to are still pretty unreliable and easy to circumvent. Our best methods for understanding how AIs make choices only allow us to glimpse parts of their reasoning. We don't even have reliable ways to figure out what AIs systems are capable of. We already struggle to understand and control current AIs: as we get closer to AGI and ASI this will get a lot harder.
Remember: AIs don’t need to hate humanity to wipe it out… The problem is that if we actually build the AIs that build the AGIs that build the ASIs, then to those towering machine gods we would look like a rounding error. To the AIs, we might just look like a bunch of primitive beings standing in the way of a lot of really useful resources. After all, when we want to build a skyscraper and there’s an ant hill in the way, well, too bad for the ants. It’s not that we hate them—we just have more important things to consider.
We cannot simply ignore the problem – just hoping for powerful AIs to not hurt us literally means gambling the fate of our entire species.
Another option is to stop, or at least slow things down, until we’ve figured out what’s going on. We’re a long way away from being able to control millions of AIs that are as smart as us, let alone AIs that are to us as we are to ants. We need time to build institutions to handle this extinction-risk level technology. And then we need time to figure out the science and engineering to build AIs we can keep control of.
Unfortunately, the race is on, and there are many different companies and countries all throwing around tens of billions of dollars to build the most powerful AI they can. Right now we don’t even have brakes to step on if we need to, and we’re approaching the cliff fast.
We're at a critical time when our decisions about AI development will echo through history. Getting this right isn't just an option—it's a necessity. The tools to build superintelligent AI are coming. The institutions to control it must come first.
We’d like to thank our friends at ControlAI for making this explainer possible. It's because of their support that you can watch it now! But, as you can tell from our other videos, this subject has always been important to us.
ControlAI is mobilising experts, politicians, and concerned citizens like you to keep humanity in control. We need you: every voice matters, every action counts, and we’re running out of time.
Visit controlai.com now and join the movement to protect humanity’s future. Check the description down below for links to join now.
Help us ensure humanity's most powerful invention doesn't become our last.
Executive summary: This explainer video by Rational Animations and ControlAI argues that, unless AI development is slowed or better controlled, current trends suggest we are on a path toward superintelligent, recursively self-improving systems that could lead to human extinction—not out of malice, but through indifference—making urgent action by governments and the public essential to avert catastrophic outcomes.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.