I listened to "If Anyone Builds It, Everyone Dies" today.
I think the first two parts of the book are the best available explanation of the basic case for AI misalignment risk for a general audience. I thought the last part was pretty bad, and probably recommend skipping it. Even though the authors fail to address counterarguments that I think are crucial, and as a result I am not persuaded of the book’s thesis and think the book neglects to discuss crucial aspects of the situation and makes poor recommendations, I would happily recommend the book to a lay audience and I hope that more people read it.
I can't give an overall assessment of how well this book will achieve its goals. The point of the book is to be well-received by people who don't know much about AI, and I’m not very good at predicting how laypeople will respond to it; seems like results so far are mixed, and reviews from people who are familiar with AI risk are fairly negative. So I’ll just talk about whether I think the arguments in the book are reasonable enough that I want them to be persuasive to the target audience, rather than whether I think they’ll actually succeed.
Thanks to several people for helpful and quick comments and discussion, especially Oli Habryka and Malo Bourgon!
Synopsis
Here's a synopsis and some brief thoughts, part-by-part:
I personally really liked the writing throughout (unlike e.g. Shakeel I didn't find the sentences torturous at all). I'm a huge fan of Eliezer's fiction and most of his non-fiction that doesn't talk about AI, so maybe this is unsurprising. I often find it annoying to read things Eliezer and Nate write about AI, but I genuinely enjoyed the experience of listening to the book. (Also, the narrator for the audiobook does a hilarious job of rendering the dialogues and parables.)
My big disagreement
In the text, the authors often state a caveated version of the title, something like "If anyone builds it (with techniques like those available today), everyone dies". But they also frequently state or imply the uncaveated title. I'm quite sympathetic to something like the caveated version of the title. But I have a huge problem with equivocating between the caveated and uncaveated versions.
There are two possible argument structures that I think you can use to go from the caveated thesis to the uncaveated one, and both rely on steps that are IMO dubious:
Argument structure one:
- If anyone built ASI with current techniques in a world that looked like today's, everyone would die.
- Tricky hypothesis 1: ASI will in fact be developed in a world that looks very similar to today's (e.g. because sub-ASI AIs will have negligible effect on the world; this could also be because ASI will be developed very soon).
- Therefore, everyone will die.
This is the argument that I (perhaps foolishly and incorrectly) understood Eliezer and Nate to be making when I worked with them, and the argument I made when I discussed AI x-risk five years ago, right before I started changing my mind on takeoff speeds.
I think Eliezer and Nate aren’t trying to make this argument—they are agnostic on timelines and they don’t want to argue that sub-ASI AI will be very unimportant for the world. I think they are using what I’ll call “argument structure two”:
- If anyone built ASI with current techniques in a world that looked like today's, everyone would die.
- The big complication: However, ASI might be built in a world that looks very different from today's: it might be several decades in the future, pretty powerful AI might be available for a while before ASI is developed, researchers might be way more experienced getting AIs to do stuff than they currently are.
- Tricky hypothesis 2: But the differences between the world of today and the world where ASI will be developed don't matter for the prognosis.
- Therefore, everyone will die.
The authors are (unlike me) confident in tricky hypothesis 2. The book says almost nothing about either the big complication or tricky hypothesis 2, and I think that’s a big hole in their argument that a better book would have addressed.
I think that explicitly mentioning the big complication is pretty important for giving your audience an accurate picture of what you're expecting. Whenever I try to picture the development of ASI, it's really salient in my picture that that world already has much more powerful AI than today’s, and the AI researchers will be much more used to seeing their AIs take unintended actions that have noticeably bad consequences. Even aside from the question of whether it changes the bottom line, it’s a salient-enough part of the picture that it feels weird to neglect discussing it. (See also Lukas's articulation of this complaint.)
And of course, the core disagreement that leads me to disagree so much with Eliezer and Nate on both P(AI takeover) and on what we should do to reduce it: I don't agree with tricky hypothesis 2. I think that the trajectory between here and ASI gives a bunch of opportunities for mitigating risk, and most of our effort should be focused on exploiting those opportunities. If you want to read about this, you could check out the back-and-forth me and my coworkers had with some MIRI people here, or the back-and-forth Scott Alexander and Eliezer had here.
(This is less relevant given the authors’ goal for this book, but from my perspective, another downside of not discussing tricky hypothesis 2 is that, aside from being relevant to estimating P(AI takeover), understanding the details of these arguments is crucial if you want to make progress on mitigating these risks.)
If they wanted to argue a weaker claim, I'd be entirely on board. For example, I’d totally get behind:
- It is pretty unclear whether we will survive or not. There are various reasons to think we might be able to prevent AI takeover. But none of those reasons are airtight, and many of them require that all AI companies with dangerous models implement safety measures competently, and it's very unclear that that will happen.
- We should demand an extremely low probability of extinction from AI developers, because extinction would be really bad. And we are not on track to getting to justified confidence in the safety of powerful AI.
But instead, they propose a much stronger thesis that they IMO fail to justify.
This disagreement leads to my disagreement with their recommendations—relatively incremental interventions seem much more promising to me.
(There’s supplementary content online. I only read some of this content, but it seemed somewhat lower quality than the book itself. I'm not sure how much of that is because the supplementary content is actually worse, and how much of it is because the supplementary content gets more into the details of things—I think that the authors and MIRI staff are very good at making simple conceptual arguments clearly, and are weaker when arguments require attention to detail.)
(I will also parenthetically remark that superintelligence is less central in my picture than theirs. I think that there is substantial risk posed by AIs that are not wildly superintelligent, and it's plausible that humans purposefully or involuntarily cede control to AIs that are less powerful than the wildly superintelligent ones the authors describe in this book. This causes me to disagree in a bunch of places.)
I tentatively support this book
I would like it if more people read this book, I think. The main downsides are:
- Some people will be persuaded by the parts of the book that I think are wrong, which will have slightly bad consequences but is not a huge problem, and seems overall better than them never engaging with a serious argument about AI x-risk.
- Some people will be turned off by the book, especially the most unreasonable parts of it, and we will have missed the opportunity to have someone more reasonable (according to me) than Eliezer and Nate write a similar book and then do a tour etc. I'm less worried about this after reading the book, because the book was good enough that it's hard for me to imagine someone else writing a much better one.
- Relatedly, success of this book will lead Eliezer and Nate to be more prominent public intellectuals on the topic of AI. I don't know whether this is good or bad. It really depends on who they're displacing.
- I don't love them as representatives of AI safety to the public for a few reasons. Despite the book being impressively cleaned up compared to Eliezer’s usual writing style, I expect them to be somewhat worse at being likable and persuasive to mass audiences in unscripted settings. I think their arguments are often unpersuasive to informed audiences (partially because of the flaws in the arguments that I complained about above, and partially because they don’t know much about empirical ML or empirical evidence about alignment and sometimes come across as blowhards to ML researchers). And I disagree with their recommended actions.
- I think it would be suboptimal if important stakeholders tried to get advice from them (though again, it depends who they’re displacing), because I don't think that they have good recommendations for what people should do.
Despite my complaints, I’m happy to recommend the book, especially with the caveat that I think it's wrong about a bunch of stuff including the thesis. Even given all the flaws, I don't know of a resource for laypeople that’s half as good at explaining what AI is, describing superintelligence, and making the basic case for misalignment risk. After reading the book, it feels like a shocking oversight that no one wrote it earlier.
"This disagreement leads to my disagreement with their recommendations—relatively incremental interventions seem much more promising to me."
What's the reasoning here?