Participants

Rohin Shah — PhD student at the Center for Human-Compatible AI, UC Berkeley
Asya Bergal – AI Impacts
Robert Long – AI Impacts
Sara Haxhia — Independent researcher

Summary

We spoke with Rohin Shah on August 6, 2019. Here is a brief summary of that conversation:

Before taking into account other researchers’ opinions, Shah guesses an extremely rough~90% chance that even without any additional intervention from current longtermists, advanced AI systems will not cause human extinction by adversarially optimizing against humans. He gives the following reasons, ordered by how heavily they weigh in his consideration:
- Gradual development and take-off of AI systems is likely to allow for correcting the AI system online, and AI researchers will in fact correct safety issues rather than hacking around them and redeploying.
  - Shah thinks that institutions developing AI are likely to be careful because human extinction would be just as bad for them as for everyone else.
- As AI systems get more powerful, they will likely become more interpretable and easier to understand because they will use features that humans also tend to use.
- Many arguments for AI risk go through an intuition that AI systems can be decomposed into an objective function and a world model, and Shah thinks this isn’t likely to be a good way to model future AI systems.
Shah believes that conditional on misaligned AI leading to extinction, it almost certainly goes through deception.
Shah very uncertainly guesses that there’s a ~50% that we will get AGI within two decades:
- He gives a ~30% – 40% chance that it will be via essentially current techniques.
- He gives a ~70% that conditional on the two previous claims, it will be a mesa optimizer.
- Shah’s model for how we get to AGI soon has the following features:
Shah is skeptical of more ‘nativist’ theories where human babies are born with a lot of inductive biases, rather than learning almost everything from their

...

AI Impacts

AI Impacts's Posts

AI Impacts's Comments