Yes, I'm aware of more formal models with estimates based on expert surveys. Sadly, this work isn't public yet I think.

This argument neglects improvements in speed and capability right? Even if parallel labor and compute are complements, shouldn't we expect it is possible for increased speed or capabilities to substitute for compute? (It just isn't possible for AI companies to buy much of this.)

(I'm not claiming this is the biggest problem with this analysis, just noting that it is a problem.)

Might be true, doesn't make that not a strawman. I'm sympathetic to thinking it's implausible that mechanize would be the best thing to do on altruistic grounds even if you share views like those of the founders. (Because there is probably something more leveraged to do and some weight on cooperativeness considerations.)

The main reason not to wait is... missing the opportunity to cash in on the current AI boom.

This is a clear strawman. Matthew has given reasons why he thinks acceleration is good which aren't this.

From my perspective, a large part of the point of safety policies is that people can comment on the policies in advance and provide some pressure toward better policies. If policies are changed at the last minute, then the world may not have time to understand the change and respond before it is too late.

So, I think it's good to create an expectation/norm that you shouldn't substantially weaken a policy right as it is being applied. That's not to say that a reasonable company shouldn't do this some of the time, just that I think it should by default be considered somewhat bad, particularly if there isn't a satisfactory explanation given. In this case, I find the object level justification for the change somewhat dubious (at least for the AI R&D trigger) and there is also no explanation of why this change was made at the last minute.

Conditional on no intentional slow down, maybe median 2035 or something? I don't have a cached 25th percentile for this, but maybe more like 2031.

Hmm, 10k$ is maybe too small size to be worth it, but I might be down to do:

• You transfer $50k to me now.
• If AIs aren't able to automate a typical senior individual contributor software engineer / research engineer by 2031 (based on either credible reports about what's happening inside AI companies or testing of externally deployed systems), I send $75k to you. ($75k = $50k * (1 + 1/5) * (1.045^5.5)  The 1.045 comes from interest rates.)
  • More precise operationalization: typical in AI development or some other moderate importance sector where the software engineering doesn't require vision. Also, the AI needs to be able to automate what this job looked like in 2025 (as this job might evolve over time with AI capabilities to be what AIs can't do). 

I'd like to bet on a milestone that triggers before it's too late for human intervention if possible, so I've picked this research engineer milestone. We'd presuambly have to operationalize further. I'm not sure if I think it's worth the time to try to operationalize enough that we could do a bet.

Somewhat relatedly, Anthropic quietly weakened its security requirements about a week ago as I discuss here.