Tyler John asked the same question on Twitter and got good responses:

https://twitter.com/tyler_m_john/status/1641061269116538881

Here is an intuitive, brief answer that should provide evidence that there is risk:

In the history of life before humans, there have been 5 documented mass extinctions. Humans--the first generally intelligent agent to evolve on our planet--are not causing the 6th mass extinction.

An intelligent agent that is superior to humans, clearly has potential to be another mass extinction agent--and if it turns out humans are in conflict with that agent, the risks are real. 

So it makes sense to understand that risk--and, today, we don't, even though development of these agents is barrowing forward at an incredible pace. 

https://en.wikipedia.org/wiki/Holocene_extinction

https://www.cambridge.org/core/journals/oryx/article/briefly/03807C841A690A77457EECA4028A0FF9
 

Hi Sanjay, There is this post.

I think my explainer on the topic does a good job:

https://forum.effectivealtruism.org/posts/CghaRkCDKYTbMhorc/the-importance-of-ai-alignment-explained-in-5-points

Due to the hierarchical manner in which I wrote the piece, it's brief as long as you don't go down too deep following too many of the claims.

How about something like:

• AI systems are rapidly becoming more capable.
• They could become extremely powerful in the next 10-50 years.
• We basically don't understand how they work (except at high levels of abstraction) or what's happening inside them. This gets even harder as they get bigger and/or more general.
• We don't know how to reliably get these systems to do what we want them to do. One, it's really hard to specify what exactly we want. Two, even if we could, their goals/drives may not generalize to new environments.
• But it does seem like, whatever objectives they do aim for, they'll face incentives that conflict with our interests. For example, accruing more power, preserving option value, avoiding being shut off and so on is generally useful, whatever goal you pursue.
• It's really hard to rigorously test AIs because they (1) are the result of a "blind" optimization process (not a deliberate design), (2) are monolithic (i.e. don't consist of individual testable components), and (3) may at some point be smarter than us.
• There are strong incentives to develop and deploy AI systems. This means powerful AI systems may be deployed even if they aren't adequately safe/tested.

Of course this is a rough argument, and necessarily leaves out a bunch of detail and nuance.

Some answers here: https://forum.effectivealtruism.org/posts/p3eiBqnijXPv5pCMA/usd20k-in-prizes-ai-safety-arguments-competition#comments

AI Risk for Epistemic Minimalists (Alex Flint, 2021).

I think it's important to give the audience some sort of analogy that they're already familiar with, such as evolution producing humans, humans introducing invasive species in new environments, and viruses. These are all examples of "agents in complex environments which aren't malicious or Machiavellian, but disrupt the original group of agents anyway".

I believe these analogies are not object-level enough to be arguments for AI X-risk in themselves, but I think they're a good way to help people quickly understand the danger of a superintelligent, goal-directed agent.