Hide table of contents

As part of a presentation I'll be giving soon, I'll be spending a bit of time explaining why AI might be an x-risk.

Can anyone point me to existing succinct explanations for this, which are as convincing as possible, brief (I won't be spending long on this), and (of course) demonstrating good epistemics.

The audience will be actuaries interested in ESG investing.

If someone fancies entering some brief explanations as an answer, feel free, but I was expecting to see links to content which already exists, since I'm sure there's loads of it.




New Answer
New Comment

7 Answers sorted by

Here is an intuitive, brief answer that should provide evidence that there is risk:

In the history of life before humans, there have been 5 documented mass extinctions. Humans--the first generally intelligent agent to evolve on our planet--are not causing the 6th mass extinction.

An intelligent agent that is superior to humans, clearly has potential to be another mass extinction agent--and if it turns out humans are in conflict with that agent, the risks are real. 

So it makes sense to understand that risk--and, today, we don't, even though development of these agents is barrowing forward at an incredible pace. 



Hi Sanjay, There is this post.

I think my explainer on the topic does a good job:


Due to the hierarchical manner in which I wrote the piece, it's brief as long as you don't go down too deep following too many of the claims.

This is a presentation with slides/graphics you might find helpful (but it is not succinct in its explanation) AI safety intro talk 

How about something like:

  • AI systems are rapidly becoming more capable.
  • They could become extremely powerful in the next 10-50 years.
  • We basically don't understand how they work (except at high levels of abstraction) or what's happening inside them. This gets even harder as they get bigger and/or more general.
  • We don't know how to reliably get these systems to do what we want them to do. One, it's really hard to specify what exactly we want. Two, even if we could, their goals/drives may not generalize to new environments.
  • But it does seem like, whatever objectives they do aim for, they'll face incentives that conflict with our interests. For example, accruing more power, preserving option value, avoiding being shut off and so on is generally useful, whatever goal you pursue.
  • It's really hard to rigorously test AIs because they (1) are the result of a "blind" optimization process (not a deliberate design), (2) are monolithic (i.e. don't consist of individual testable components), and (3) may at some point be smarter than us.
  • There are strong incentives to develop and deploy AI systems. This means powerful AI systems may be deployed even if they aren't adequately safe/tested.

Of course this is a rough argument, and necessarily leaves out a bunch of detail and nuance.

As far as I'm aware, the best introduction to AI safety is the AI safety chapter in The Precipice. I've tested it on two 55-year-olds and it worked. 

It's a bit long (20 minute read according to lesswrong), but it's filled to the brim with winning strategies for giving people a fair chance to understand AI as an X-risk. A list of names of reputable people who wholeheartedly endorse AI safety, for example.

Sorted by Click to highlight new comments since: Today at 5:18 AM

Tyler John asked the same question on Twitter and got good responses:


I think it's important to give the audience some sort of analogy that they're already familiar with, such as evolution producing humans, humans introducing invasive species in new environments, and viruses. These are all examples of "agents in complex environments which aren't malicious or Machiavellian, but disrupt the original group of agents anyway".

I believe these analogies are not object-level enough to be arguments for AI X-risk in themselves, but I think they're a good way to help people quickly understand the danger of a superintelligent, goal-directed agent.

More from Sanjay
Curated and popular this week
Relevant opportunities