Am I taking crazy pills? Why aren't EAs advocating for a pause on AI capabilities?

[Question]

Am I taking crazy pills? Why aren't EAs advocating for a pause on AI capabilities?

[anonymous]

1 min readAug 15, 2023

Comments 21

Sorted by

New & upvoted

Larks

There's a big gap between "believe we should press it if presented with a magic global pause button" and "think pro-pause advocacy is the most efficient use of their time on the current margin". I suspect the majority of Xrisk-concerned EAs would press such a button if given the chance.

[anonymous]

-3

i think you'd be surprised how many wouldn't press such a button
what publicly available thinking have you seen on the potential impact of advocacy in this context (within EA)? i'm interested how you've formed this opinion

Larks

You can run a poll to try to find out the answer to 1) if you want!

Zach Stein-Perlman

Messy practical reasons.

I agree with Larks that most of us would press a magic button to slow down AI progress on dangerous paths.

But we can't, which raises two problems:

Tractability.
Effectiveness of moderate success. If you get a non-global slowdown, or a slowdown that ends too early, or a slowdown regime that's evadable, or if you differentially slow cautious labs, or even if you just differentially slow leading labs, the effect is likely net-negative. (Increasing multipolarity among labs + differentially boosting less-cautious actors + ~~compute overhang~~ enabling rapidly scaling up training compute. See Slowing AI: Foundations.)

(I'd be excited to talk about proposals more specific than 'push for a pause,' or outcomes more specific than 'pause until proven <0.1% doom.' Who is doing the pausing; what are the rules? Or maybe you don't have specific proposals/outcomes in mind, in which case I support you searching for new great ideas, but it's not like others haven't tried and failed already.)

[anonymous]

Thanks for the comment Zach.

1. Can you elaborate on your comment "Tractability"?

2. I'm less worried about multipolarity because the leading labs are so far ahead AND I have short timelines (~ 10 years). My guess is if you had short timelines, you might agree?

3. If we had moderate short term success, my intuition is that we've actually found an effective strategy that could then be scaled. I worry that your thinking is basically pointing to 'it needs to be an immediately perfect strategy or don't bother!'

Zach Stein-Perlman

Pushing a magic button would be easy; affecting the real world is hard. Even if slowing is good, we should notice whether there exist tractable interventions (or: notice interventions' opportunity cost).
Nope, my sense is that DeepMind, OpenAI, and Anthropic do and will have a small lead over Meta, Inflection, and others, such that I would be concerned (re increasing multipolarity among labs) about slowing DeepMind, OpenAI, and Anthropic now. (And I have 50% credence on human-level AI [noting this is underspecified] within 9 years.)
Yeah, maybe, depending. I'm relatively excited about "short term success" that seems likely to support the long-term policy regimes I'm excited about, like global monitoring of compute and oversight of training runs with model evals for dangerous capabilities and misalignment, maybe plus a compute cap. I fear that most pause-flavored examples of "short term success" won't really support great long-term plans. (Again, I'd be excited to talk about specific proposals/outcomes/interventions.)

Pato

it appears many EAs believe we should allow capabilities development to continue despite the current X-risks.

Where do you get this from?

Also, this:

have < ~ 0.1% chance of X-risk.

means p(doom) <~ 0.001

[anonymous]

"Where do you get this from?"

the lack of content and discussion regarding a pause on the forum / podcasts / twitter
the dismal success rate of grant requests to drive such a strategy
vibe

Ben_West🔸

This sequence is still in progress but is the best collection of resources that I know of regarding slowing AI (including an indefinite pause).

Greg_Colbourn ⏸️

Worth cross-posting to the EA Forum. @Zach Stein-Perlman

[anonymous]

Thank you Ben I will take a look :)

Ariel Simnegar 🔸

I think the best reason is that it's not within the Overton window :)

Peter Berggren🔸

Last I checked, the whole point of the Overton window is that you can only shift it by advocating for ideas outside of it.

[anonymous]

I'm confused - where is the evidence it is outside the overton window?

Peter Berggren🔸

Don't really think there is any; in fact, there's plenty of evidence to the contrary, from the polls I've seen.

David_Moss

We found 51% support, 25% opposition in our polling here.

[anonymous]

-9

the overton window is a crutch for EAs who don't believe in the power of advocacy

Tyler Johnston

I've worked in advocacy for EA causes for a bit, so I definitely believe in the power of it, but I also think the Overton window is a pretty crucial consideration for folks who are trying to mobilize the public. I'm guessing this is a popular view among people who work in advocacy for EA causes, but I might be wrong.

To be fair, I do think there could be value in making bold asks outside the Overton window. James Ozden has a really good piece about this. I think groups like DxE and PETA have done this for the animal movement, and it seems totally plausible to me that this has had a net positive effect.

But on the other hand, I think lots of the tangible changes we've seen for farmed animals have come from the incremental welfare asks that groups like Mercy For Animals and The Humane League focus on (disclaimer: I worked at the latter). The fact that these groups have been very careful to keep their asks within the Overton window has had the benefit of (1) helping advocates gain broad-based public support; and (2) getting corporations and policymakers on board and willing to actually adopt the changes they are asking for.

It seems likely to me that the second point applies for AI safety, but I'm not sure about the first and would probably need to see more polling or message testing to know. Nonetheless I suspect these concerns might be part of why the AI pause ask hasn't been as widely adopted among EAs (although a number of them did sign the FLI letter).

[anonymous]

The public is very concerned about powerful AI and want something done about it.

If anyone is outside the overton window its EAs.

Tyler Johnston

I agree that the public has been pretty receptive to AI safety messaging. Much more than I would have expected a few years ago.

It sounds like you already have some takes on this question — in that case, it could be worth writing something up to make the case for why EAs should be advocating for a pause. I’d be happy to offer feedback if you do.

[anonymous]

Thats very generous of you, thanks Tyler!

Comments

Curated and popular this week

Cultivating hope: calibrating the expectations for cultivated meat to end factory farming

PabloAMC 🔸·1w ago·Curated 1d ago·22m read

GWWC's 2025 impact evaluation (executive summary)

Aidan Whitfield🔸, Giving What We Can🔸·3d ago·2m read

This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...

Maybe do the thing you wish CEA would do

alejoacelas 🔸·15h ago·2m read

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts: * Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do * NEST is an opinionated ideas-fi...