I think I disagree with a bunch of what you said there: I don't think good AIS involves being on good side of AI labs necessarily (tho I think there are good arguments for this), I think large movement building without getting a full pause would be a big win for PauseAI and that many EAs would agree with this (despite the fact that I and maybe them know little about social movements), and I do think making simple statements reflects negatively from a whole host of perspectives and should be taken pretty seriously. 

I'd be interested in hearing how you/others at pause AI have tracked how much of this marginal improvements to advocacy you have been doing so far is. What are the wins? What were the costs? Happy to also get on a call with someone about this (tho, tbh, the stunt at EAG just makes me pretty skeptical of Pause AI's efficacy in general). 

If there are real numbers on this (I know it's hard in spaces like yours), I'd be curious about hearing why they aren't often posted on the forum/LW, as that, I think, would be more helpful to people (and tracking the cost-effectiveness of AIS in general is super underrated, so this would be good). 

If you don't have the numbers, I would ask why you are so confident that this is actually the right approach. 

Can you say more about what these EA assumptions are? 

Disagreement is cool and awesome. Even intense disagreement (“I think this view is deeply misguided”). I really see no room for antagonism among two people that could be having an epistemically healthy conversation. 

Sorry about your father. 

I think there’s a much more mundane and much more epistemically healthy way to understand this disagreement.

Perhaps this is naive but my view currently just is: most people are disagreeing about a few concrete empirical and strategic parameters: the likely effectiveness and public reception of a Pause movement, how much meaningful safety work can be done from inside labs, and (probably) estimates of p(doom). Given how uncertain and high-stakes these questions are, it seems completely unsurprising that reasonable people would land in very different places.

It’s fine to worry about incentives and institutional bias — that could matter — but treating this as if the disagreement is obviously resolved, or as if it cleanly divides the world into “accelerationists” and “pause-ers,” strikes me as bad epistemics.

Interesting. A few thoughts:

Beyond strengthening the case for non-existential risks, if Sisyphus risk is substantial it also weakens arguments that place extreme weight on reducing existential risk at a specific time. Some of the importance of the Time of Perils comes from comparative advantage, which is diluted if civilization plausibly gets multiple runs.

One additional Sisyphean mechanism worth flagging is resource exhaustion: collapsing before reaching renewable resource self-sufficiency could permanently worsen later runs. This probably relies on a setback happening much later or a large amount of resources being used before, but it’s worth flagging. 

A caveat on donation timing: even if post-AGI x-risk declines slowly, aligned AGI plausibly generates enormous resources, so standard impatient-philanthropy arguments may still apply if we have setbacks. And if we assume those resources are lost in a collapse, the same would likely apply to resources saved in advance.

Finally, the plausible setbacks all seem to hinge on something like the loss of knowledge. Other worries (e.g. Butlerian backlash) tend to rely on path-dependent successes—historically contingent timing, unusually alignable models, or specific public perceptions that don’t automatically replicate—seem hard to change conditional on setbacks. If those aren’t mostly luck-based and the relevant knowledge survives, a post-setback society could plausibly re-instantiate the same mechanisms, making Sisyphus risk primarily an epistemic rather than, say, a governance problem.

I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.

That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.

This matters to me for two reasons.

First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.

Second, these track-record arguments are sometimes used—explicitly or implicitly—to bolster the case for concern about AI risks. If the evidential support here rests on past forecasting success, then the strength of that support depends on how well those earlier cases actually hold up under scrutiny. If the success was mostly at the level of identifying broad structural risks (e.g. incentives, tail risks, coordination failures), that’s a very different kind of evidence than being right about timelines, concrete outcomes, or specific mechanisms.

I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.

That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.

This matters to me for two reasons.

First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.

Second, these track-record arguments are sometimes used—explicitly or implicitly—to bolster the case for concern about AI risks. If the evidential support here rests on past forecasting success, then the strength of that support depends on how well those earlier cases actually hold up under scrutiny. If the success was mostly at the level of identifying broad structural risks (e.g. incentives, tail risks, coordination failures), that’s a very different kind of evidence than being right about timelines, concrete outcomes, or specific mechanisms.

I can’t join this Sunday (finals season whoo!), but this is a really good idea. I’d love to see more initiatives like this to encourage writing on the Forum—especially during themed weeks.

Also, I’m always down to do (probably remote) co-working sessions with people who want to write Forum posts.

Strongly agreed. Organizing a group is probably one of the best things one could do for both their present and future impact. 

I (and many others) would be happy to get on a call/help anyone willing to take over (I have a bunch of experience from organizing the UChicago group)! Dm me here to take me up on that.  

Many questions in this space rely on assumptions about whether insect lives are positive or negative, though I haven't seen much discussion of this explicitly (mostly just heard them in conversations). Is there not much that can be done to learn more other than what has already been done? 

If so, the insect welfare initiatives that are interested in creating more or less are going to need to be dropped by 1-2 OOMs to account for massive uncertainty (say, 5% higher that they are positive), which is weird. It also wouldn't be very robust in that the probability we do something good vs bad is quite fragile (especially a problem for many non hedonistic utilitarian views). 

Do you have further takes here, Vasco?