Due to current outsourcing being of data labeling, I think one of the issues you express in the post is very unlikely:

My general worry is that in future, the global south shall become the training ground for more harmful AI projects that would be prohibited within the Global North. Is this something that I and other people should be concerned about?

Maybe there's an argument about how: 

• current practices are evidence that AI companies are trying to avoid following the laws (note I mostly don't believe this), 
• and this is why they're outsourcing parts of development, 
• so then we should be worried they'll do the same to get around other (safety-oriented) laws. 

This is possible, but my best guess is that low wages are the primary reason for current outsourcing. 

Additionally, as noted by Larks, outsourcing data-centers is going to be much more difficult, or at least take a long time, compared to outsourcing data-labeling, so we should be less worried that companies could effectively get around laws by doing this. 

This line of argument suggests that slow takeoff is inherently harder to steer. Because pretty much any version of slow takeoff means that the world will change a ton before we get strongly superhuman AI.

I'm not sure I agree that the argument suggests that. I'm also not sure slow takeoff is harder to steer than other forms of takeoff — they all seem hard to steer. I think I messed up the phrasing because I wasn't thinking about it the right way. Here's another shot:

Widespread AI deployment is pretty wild. If timelines are short, we might get attempts at AI takeover before we have widespread AI deployment. I think attempts like this are less likely to work than attempts in a world with widespread AI deployment. This is thinking about takeoff in the sense of deployment impact on the world (e.g., economic growth), rather than in terms of cognitive abilities. 

On a related note, slow takeoff worlds are harder to steer in the sense that the proportion of influence on AI from x-risk oriented people probably goes down because the rest of the world gets involved, also the neglectedness of AI safety research probably drops; this is why some folks have considered conditioning their work on e.g., high p(doom).

Thanks for your comments! I probably won't reply to the others as I don't think I have much to add, they seem reasonable, though I don't fully agree. 

I think these don’t bite nearly as hard for conditional pauses, since they occur in the future when progress will be slower

Your footnote is about compute scaling, so presumably you think that's a major factor for AI progress, and why future progress will be slower. The main consideration pointing the other direction (imo) is automated researchers speeding things up a lot. I guess you think we don't get huge speedups here until after the conditional pause triggers are hit (in terms of when various capabilities emerge)? If we do have the capabilities for automated researchers, and a pause locks these up, that's still pretty massive (capability) overhang territory. 

While I’m very uncertain, on balance I think it provides more serial time to do alignment research. As model capabilities improve and we get more legible evidence of AI risk, the will to pause should increase, and so the expected length of a pause should also increase [footnote explaining that the mechanism here is that the dangers of GPT-5 galvanize more support than GPT-4]

I appreciate flagging the uncertainty; this argument doesn't seem right to me. 

One factor affecting the length of a pause would be the (opportunity cost from pause) / (risk of catastrophe from unpause) ratio of marginal pause days, or what is the ratio of the costs to the benefits. I expect both the costs and the benefits of AI pause days to go up in the future — because risks of misalignment/misuse are greater, and because AIs will be deployed in a way that adds a bunch of value to society (whether the marginal improvements are huge remains unclear, e.g., GPT-6 might add tons of value, but it's unclear how much more GPT-6.5 adds on top of that, seems hard to tell). I don't know how the ratio will change, which is probably what actually matters. But I wouldn't be surprised if that numerator (opportunity cost) shot up a ton. 

I think it's reasonable to expect that marginal improvements to AI systems in the future (e.g., scaling up 5x) could map on to automating an additional 1-7% of a nation's economy. Delaying this by a month would be a huge loss (or a benefit, depending on how the transition is going). 

What relevant decision makers think the costs and benefits are is what actually matters, not the true values. So even if right now I can look ahead and see that an immediate pause pushes back future tremendous economic growth, this feature may not become apparent to others until later. 

To try and say what I'm getting at a different way: you're suggesting that we get a longer pause if we pause later than if we pause now. I think that "races" around AI are going to ~monotonically get worse and that the perceived cost of pausing will shoot up a bunch. If we're early on an exponential of AI creating value in the world, it just seems way easier to pause for longer than it will be later on. If this doesn't make sense I can try to explain more. 

Sorry, I agree my previous comment was a bit intense. I think I wouldn't get triggered if you instead asked "I wonder if a crux is that we disagree on the likelihood of existential catastrophe from AGI. I think it's very likely (>50%), what do you think?" 

P(doom) is not why I disagree with you. It feels a little like if I'm arguing with an environmentalist about recycling and they go "wow do you even care about the environment?" Sure, that could be a crux, but in this case it isn't and the question is asked in a way that is trying to force me to agree with them. I think asking about AGI beliefs is much less bad, but it feels similar. 

I think it's pretty unclear if extra time now positively impacts existential risk. I wrote about a little bit of this here, and many others have discussed similar things. I expect this is the source of our disagreement, but I'm not sure. 

I don't think you read my comment:

I don't think extra time pre-transformative-AI is particularly valuable except its impact on existential risk

I also think it's bad how you (and a bunch of other people on the internet) ask this p(doom) question in a way that (in my read of things) is trying to force somebody into a corner of agreeing with you. It doesn't feel like good faith so much as bullying people into agreeing with you. But that's just my read of things without much thought. At a gut level I expect we die, my from-the-arguments / inside view is something like 60%, and my "all things considered" view is more like 40% doom. 

Yep, seems reasonable, I don't really have any clue here. One consideration is that this AI is probably way better than all the human scientists and can design particularly high-value experiments, also biological simulations will likely be much better in the future. Maybe the bio-security community gets a bunch of useful stuff done by then which makes the AI's job even harder. 

there will be governance mechanisms put in place after a failure

Yep, seems reasonably likely, and we sure don't know how to do this now. 

I'm not sure where I'm assuming we can't pause dangerous AI "development long enough to build aligned AI that would be more capable of ensuring safety"? This is a large part of what I mean with the underlying end-game plan in this post (which I didn't state super explicitly, sorry), e.g. the centralization point

centralization is good because it gives this project more time for safety work and securing the world

I'm curious why you don't include intellectually aggressive culture in the summary? It seems like this was a notable part of a few of the case studies. Did the others just not mention this, or is there information indicating they didn't have this culture? I'm curious how widespread this feature is. e.g., 

The intellectual atmosphere seems to have been fairly aggressive. For instance, it was common (and accepted) that some researchers would shout “bullshit” and lecture the speaker on why they were wrong.

we need capabilities to increase so that we can stay up to date with alignment research

I think one of the better write-ups about this perspective is Anthropic's Core Views on AI Safety. 

From its main text, under the heading The Role of Frontier Models in Empirical Safety, a couple relevant arguments are: 

• Many safety concerns arise with powerful systems, so we need to have powerful systems to experiment with
• Many safety methods require large/powerful models
• Need to understand how both problems and our fixes change with model scale (if model gets bigger, does it look like safety technique is still working)
• To get evidence of powerful models being dangerous (which is important for many reasons), you need the powerful models.