From my post,

Eliezer Yudkowsky, perhaps the most influential person in the AI risk community, has already demanded an “indefinite and worldwide” moratorium on large training runs. This sentiment isn’t exactly new. Some effective altruists, such as Toby Ord, have argued that humanity should engage in a “long reflection” before embarking on ambitious and irreversible technological projects, including AGI. William MacAskill suggested that this pause should perhaps last “a million years”. Two decades ago, Nick Bostrom considered the ethics of delaying new technologies in a utilitarian framework and concluded a delay of "over 10 million years" may be justified if it reduces existential risk by a single percentage point.

You are assuming that AI could be massively economically beneficial (significantly) before it causes our extinction (or at the least, a global catastrophe). I don’t think this is likely, and this defeats a lot of your opposition to an indefinite pause.

If you don't think AI will be economically significant before extinction, I'm curious whether you'd say that your view has been falsified if AI raises economic growth rates in the US to 5, 8, or 10% without us all dying. At what what point would you say that your model here was wrong?

(This isn't a complete reply to your comment. I appreciate your good-faith engagement with my thesis.)

Assuming the world accepts the reason for the pause being that the default outcome of AGI is extinction, then this wouldn’t be necessary. A strong enough taboo would emerge around AGI development. How many human clones have ever been born in our current (non-police-state) world?

This won't address all the arguments in your comment but I have a few things to say in response to this point.

I agree it's possible that we could just get a very long taboo on AI and halt its development for many decades without a world government to enforce the ban. That doesn't seem out of the question.

However, it also doesn't seem probable to me. Here are my reasons:

• AGI is something that several well-funded companies are already trying hard to do. I don't think that was ever true of human cloning (though I could be wrong).

• I looked it up and my impression is that it might cost tens of millions of dollars to clone a single human, whereas in the post I argued that AGI will eventually be possible to train with only about 1 million dollars. More importantly, after that, you don't need to train the AI again. You can just copy the AGI to other hardware. Therefore, it seems that you might really only need one rich person to do it once to get the benefits. That seems like a much lower threshold than human cloning, although I don't know all the details.

• The payoff for building (aligned) AGI is probably much greater than human cloning, and it also comes much sooner.

• The underlying tech that allows you to build AGI is shared by other things that don't seem to have any taboos at all. For example, GPUs are needed for video games. The taboo would need to be strong enough that we'd need to also ban a ton of other things that people currently think are fine.

• AGI is just software, and seems harder to build a taboo around compared to human cloning. I don't think many people have a disgust reaction to GPT-4, for example.

Finally, I doubt there will ever be a complete global consensus that AI is existentially unsafe, since the arguments are speculative, and even unaligned AI will appear "aligned" in the short term if only to trick us. The idea that unaligned AIs might fool us is widely conceded among AI safety researchers, and so I suspect you agree too.

contra Matthew Barnett, treaties aren’t permanent

I don't really know what you mean by this, but I never said treaties are permanent. Can you please not strawman me?

In my opinion, "X is dubious" lines up pretty well with "X is 75% likely to be false". That said, enough people have objected to this that I think I'll change the wording.

We can also proactively create AI regulation aimed specifically at promoting individual autonomy and freedom. Some general objectives for such policies could include:

• Establish a “right to refuse simulation”, as a way of preempting the most extreme forms of targeted manipulation.

Can you elaborate on how you think such a regulation could be implemented? Currently the trend seems to be that AI will be able to emulate anything that it's trained on. In essence, your proposal might look like ensuring that AIs are not trained on human data without permission. In practice, this might take the form of a very strict copyright regime. Is that what you suggest?

One alternative is that AI should be allowed to be trained on other people's data without restriction, but AIs should refuse any request to emulate specific individuals during inference. That sounds more sensible to me, and is in line with what OpenAI seems to be doing with DallE-3.

(Clarification about my views in the context of the AI pause debate)

I'm finding it hard to communicate my views on AI risk. I feel like some people are responding to the general vibe they think I'm giving off rather than the actual content. Other times, it seems like people will focus on a narrow snippet of my comments/post and respond to it without recognizing the context. For example, one person interpreted me as saying that I'm against literally any AI safety regulation. I'm not.

For a full disclosure, my views on AI risk can be loosely summarized as follows:

• I think AI will probably be very beneficial for humanity.
• Nonetheless, I think that there are credible, foreseeable risks from AI that could do vast harm, and we should invest heavily to ensure these outcomes don't happen.
• I also don't think technology is uniformly harmless. Plenty of technologies have caused net harm. Factory farming is a giant net harm that might have even made our entire industrial civilization a mistake!
• I'm not blindly against regulation. I think all laws can and should be viewed as forms of regulations, and I don't think it's feasible for society to exist without laws.
• That said, I'm also not blindly in favor of regulation, even for AI risk. You have to show me that the benefits outweigh the harm
• I am generally in favor of thoughtful, targeted AI regulations that align incentives well, and reduce downside risks without completely stifling innovation.
• I'm open to extreme regulations and policies if or when an AI catastrophe seems imminent, but I don't think we're in such a world right now. I'm not persuaded by the arguments that people have given for this thesis, such as Eliezer Yudkowsky's AGI ruin post.

What is your credence in doom conditional on AIs not caring for humans?

How much do they care about humans, and what counts as doom? I think these things matter.

If we're assuming all AIs don't care at all about humans and doom = human extinction, then I think the probability is pretty high, like 65%.

If we're allowed to assume that some small minority of AIs cares about humans, or AIs care about humans to some degree, perhaps in the way humans care about wildlife species preservation, then I think the probability is quite a lot lower, at maybe 25%.

For precision, both of these estimates are over the next 100 years, since I have almost no idea what will happen in the very long run.

What is your response to the standard arguments that 'just train them hard to be ethical' won't work? E.g. Ajeya Cotra's writings on the training game.

In most of these stories, including in Ajeya's story IIRC, humanity just doesn't seem to try very hard to reduce misalignment? I don't think that's a very reasonable assumption. (Charitably, it could be interpreted as a warning rather than a prediction.) I think that as systems get more capable, we will see a large increase in our alignment efforts and monitoring of AI systems, even without any further intervention from longtermists.

Can you say more about what model you have in mind? Do you have a model?

I'm happy to meet up some time and explain in person. I'll try to remember to DM you later about that, but if I forget, then feel free to remind me.

If this debate were about whether we should do anything to reduce AI risk, then I would strongly be on the side of doing something. I'm not an effective accelerationist. I think AI will probably be good, but that doesn't mean I think we should simply wait around until it happens. I'm objecting to a narrower point about whether we should view AI as an exception to the general rule that technology is good.

I also think existential risk from AI is way too high. That's why I strongly support AI safety research, careful regulation and AI governance. I'm objecting to the point about whether AI should be seen as an exception to the rule that technology is good. In the most probable scenario, it may well be the best technology ever!