In late 2021, MIRI hosted a series of conversations about AI risk with a number of other EAs working in this problem area. As of today, we've finished posting the (almost entirely raw and unedited) results of that discussion.
To help with digesting the sequence now that it's out, and to follow up on threads of interest, we're hosting an AMA this Wednesday (March 2) featuring researchers from various organizations (all speaking in their personal capacity):
- Paul Christiano (ARC)
- Richard Ngo (OpenAI)
- Rohin Shah (DeepMind)
- Nate Soares (MIRI)
- Eliezer Yudkowsky (MIRI)
You're welcome to post questions, objections, etc. on any vaguely relevant topic, whether or not you've read the whole sequence.
The AMA is taking place on LessWrong, and is open to comments now: https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/34Gkqus9vusXRevR8. If you don't have a LessWrong account, feel free to post questions below and I'll cross-post them.
Do you believe that AGI poses a greater existential risk than other proposed x-risk hazards, such as engineered pandemics? Why or why not?
Thanks for the question! I cross-posted it here; Nate Soares replies:
I responded here - cross-posting here for convenience:
Toby Ord's definition of an existential catastrophe is "anything that destroys humanity's longterm potential." The worry is that misaligned AGI which vastly exceeds humanity's power would be basically in control of what happens with humans, just as humans are, currently, basically in control of what happens with chimpanzees. It doesn't need to kill all of us in order for this to be a very, very bad outcome.
E.g. the enslavement by the steel-loving AGI you describe sounds like an existential catastrophe, if that AGI is sufficiently superhuman. You describe a "large portion of humanity" enslaved in this scenario, implying a small portion remain free — but I don't think this would happen. Humans with meaningful freedom are a threat to the steel-lover's goals (e.g. they could build a rival AGI) so it would be instrumentally important to remove that freedom.
The AGI would rather write programs to do the grunt work, than employ humans, as they can be more reliable, controllable, etc. It could create such agents by looking into its own source code and copying / modifying it. If it doesn't have this capability it will spend time researching (could be years) until it does. On a thousand-year timescale it isn't clear why an AGI would need us for anything besides say, specimens for experiments.
Also as reallyeli says, having a single misaligned agent with absolute control of our future seems terrible no matter what the agent does.