I tend to disagree with most EAs about existential risk from AI. Unfortunately, my disagreements are all over the place. It's not that I disagree with one or two key points: there are many elements of the standard argument that I diverge from, and depending on the audience, I don't know which points of disagreement people think are most important.
I want to write a post highlighting all the important areas where I disagree, and offering my own counterarguments as an alternative. This post would benefit from responding to an existing piece, along the same lines as Quintin Pope's article "My Objections to "We’re All Gonna Die with Eliezer Yudkowsky"". By contrast, it would be intended to address the EA community as a whole, since I'm aware many EAs already disagree with Yudkowsky even if they buy the basic arguments for AI x-risks.
My question is: what is the current best single article (or set of articles) that provide a well-reasoned and comprehensive case for believing that there is a substantial (>10%) probability of an AI catastrophe this century?
I was considering replying to Joseph Carlsmith's article, "Is Power-Seeking AI an Existential Risk?", since it seemed reasonably comprehensive and representative of the concerns EAs have about AI x-risk. However, I'm a bit worried that the article is not very representative of EAs who have substantial probabilities of doom, since he originally estimated a total risk of catastrophe at only 5% before 2070. In May 2022, Carlsmith changed his mind and reported a higher probability, but I am not sure whether this is because he has been exposed to new arguments, or because he simply thinks the stated arguments are stronger than he originally thought.
I suspect I have both significant moral disagreements and significant empirical disagreements with EAs, and I want to include both in such an article, while mainly focusing on the empirical points. For example, I have the feeling that I disagree with most EAs about:
- How bad human disempowerment would likely be from a utilitarian perspective, and what "human disempowerment" even means in the first place
- Whether there will be a treacherous turn event, during which AIs violently take over the world after previously having been behaviorally aligned with humans
- How likely AIs are to coordinate near-perfectly with each other as a unified front, leaving humans out of their coalition
- Whether we should expect AI values to be "alien" (like paperclip maximizers) in the absence of extraordinary efforts to align them with humans
- Whether the AIs themselves will be significant moral patients, on par with humans
- Whether there will be a qualitative moment when "the AGI" is created, rather than systems incrementally getting more advanced, with no clear finish line
- Whether we get only "one critical try" to align AGI
- Whether "AI lab leaks" are an important source of AI risk
- How likely AIs are to kill every single human if they are unaligned with humans
- Whether there will be a "value lock-in" event soon after we create powerful AI that causes values to cease their evolution over the coming billions of years
- How bad problems related to "specification gaming" will be in the future
- How society is likely to respond to AI risks, and whether they'll sleepwalk into a catastrophe
However, I also disagree with points made by many other EAs who have argued against the standard AI risk case. For example, I think that,
- AIs will eventually become vastly more powerful and smarter than humans. So, I think AIs will eventually be able to "defeat all of us combined"
- I think a benign "AI takeover" event is very likely even if we align AIs successfully
- AIs will likely be goal-directed in the future. I don't think, for instance, that we can just "not give the AIs goals" and then everything will be OK.
- I think it's highly plausible that AIs will end up with substantially different values from humans (although I don't think this will necessarily cause a catastrophe).
- I don't think we have strong evidence that deceptive alignment is an easy problem to solve at the moment
- I think it's plausible that AI takeoff will be relatively fast, and the world will be dramatically transformed over a period of several months or a few years
- I think short timelines, meaning a dramatic transformation of the world within 10 years from now, is pretty plausible
I'd like to elaborate on as many of these points as possible, preferably by responding to direct quotes from the representative article arguing for the alternative, more standard EA perspective.
I currently don't think these normative arguments make much of a difference in prioritization or decision making in practice. So, I think this probably isn't that important to argue about.
Perhaps the most important case in which they would lead to very different decision making is the case of pausing AI (or trying to speed it up). Strong longtermists likely want to pause AI (at the optimal time) until the reduction in p(doom) per year is around the same as the exogenous doom. (This includes the chance of societal disruption which makes the situation worse and then results in doom. For instance, nuclear war induced societal collapse which results in building AI far less safely. Gradual changes in power over time also seem relevant, e.g. china.) I think the go-ahead-point for longtermists probably looks like 0.1% to 0.01% reduction in p(doom) per year of delay for longtermists, but this might depend on how optimistic you are about other aspects of society. Of course, if we could coordinate sufficiently to also eliminate other sources of risk, the go-ahead-point might lower considerably.
ETA: note that waiting until the reduction in p(doom) per year of delay is 0.1% does not imply that the final p(doom) is 0.1%. It's probably notably higher, maybe over an order of magnitude higher.
[Low confidence] If we apply the preferences of typical people (but gloomier empirical views about AI), then it seems very relevant that people broadly don't seem care that much about saving the elderly, life extension, or getting strong versions of utopia for themselves before they die. But, they do care a lot about avoiding societal collapse and ruin. And they care some about the continuity of human civilization. So, the go-ahead-point in reduction in doom per year if we use the preferences of normal people might look pretty similar to longtermists (though it's a bit confusing to apply somewhat incoherant preferences). I think it's probably less than a factor of 10 higher, maybe a factor of 3 higher. Also, normal people care about the absolute level of risk: if we couldn't reduce risk below 20%, then it's plausible that the typical preferences of normal people never want to build AI because they care more about not dying in a catastrophe than not dying of old age etc.
If we instead assume something like utilitarian person-affecting views (let's say only caring about humans for simplicity), but with strongly diminishing returns (e.g. logarithmic) above the quality of live of current americans and with similarly diminishing returns after 500 years of life, then I think you end up do 1% reduction in P(doom) per year of delay as the go-ahead point. This probably leads to pretty similar decisions in most cases.
(Separately, pure person-affecting views seem super implausible to me. Indifference to the torture of arbitrary number of new future people seems strong from my perspective. If you have asymmetric person-affecting views, then you plausible get dominated by the potential for reducing suffering in the long run.)
The only views which seem to lead to a pretty different conclusions are views with radically higher discount rates, e.g. pure person-affecting views where you care mostly about the lives of short lived animals or perhaps some views where you care about fulfilling the preferences of current humans (who might have high discount rates on their preferences?). But it's worth noting that these views seem indifferent to the torture of an arbitrary number of future people in a way that feels pretty implausible to me.
I don't think this depends on the concept of "the human species". Personally, I care about the overall utilization of resources in the far future (and I imagine many people with a similar perspective agree with me here). For instance, I think literal extinction in the event of AI takeover is unlikely and also not very importantly worse relative to full misaligned AI takeover without extinction. Similarly, I would potentially be happier to turn over the universe to aliens instead of AIs.
Separately, I think scope-sensitive/linear-returns person-affecting views are likely dominated by the potential for using a high fraction of future resources to simulate huge number of copies of existing people living happy lives. In practice, no one goes here because the actual thing people mean when they say "person-affecting views" is more like caring about the preferences of currently existing humans in a diminishing returns way. I think the underlying crux isn't well described as person-affecting vs non-person-affecting and is better described as diminishing returns.