I think that's strongly contra Eliezer's model, which is shaped something like "succeeding at solving the alignment problem eliminates most sources of existential risk, because aligned AGI will in fact be competent to solve for them in a robust way".  This does obviously imply something about the ability of random humans to spin up unmonitored nanofactories push a bad yaml file.  Maybe there'll be some much more clever solution(s) for various possible problems?  /shrug

I do agree with this, in principle:

A system being ‘cognitively efficient wrt humanity’ doesn’t automatically entail ‘whatever goals the system has – and whatever constraints the system might otherwise face – the cognitively efficient system gets what it wants’.

...though I don't think it buys us more than a couple points; I think people dramatically underestimate how high the ceiling is for humans and think that a reasonably smart human familiar with the right ideas would stand a decent chance at executing a takeover if placed into the position of an AI (assuming speedup of cognition, + whatever actuators current systems typically possess).

However, I think this is wrong:

LLMs distill human cognition

LLMs have whatever capabilities they have because those are the capabilities discovered by gradient descent which, given their architecture, improved their performance on the test task (next token prediction).  This task is extremely unlike the tasks represented in the environment where human evolution occurred, and the kind of cognitive machinery which would make a system effective at next token prediction seems very different from whatever it is that humans do.  (Humans are capable of next token prediction, but notably we are much worse at it than even GPT-3.)

Separately, the cognitive machinery that represents human intelligence seems to be substantially decoupled from the cognitive machinery that represents human values (and/or the cognitive machinery that causes humans to develop values after birth), so if it turned out that LLMs did, somehow, share the bulk of their cognitive algorithms with humans, that would be a slight positive update for me, but not an overwhelming one, since I wouldn't expect an LLM to want anything remotely relevant to humans.  (Most of the things that humans want are lossy proxies for things that improved IGF in the ancestral environment, many of which generalized extremely poorly out of distribution.  What are the lossy proxies for minimizing prediction loss that a sufficiently-intelligent LLM would end up with?  I don't know, but I don't see why they'd have anything to do with the very specific things that humans value.)

None of those obviously mean the same thing ("runaway AI" might sort of gesture at it, but it's still pretty ambiguous).  Intelligence explosion is the thing it's pointing at, though I think there are still a bunch of conflated connotations that don't necessarily make sense as a single package.

I think "hard takeoff" is better if you're talking about the high-level "thing that might happen", and "recursive self improvement" is much clearer if you're talking about the usually-implied mechanism by which you expect hard takeoff.

I think people should take a step back and take a bird's-eye view of the situation:

• The author persistently conflates multiple communities: "tech, EA (Effective Altruists), rationalists, cybersecurity/hackers, crypto/blockchain, Burning Man camps, secret parties, and coliving houses".  In the Bay Area, "tech" is literally a double-digit percentage of the population.
• The first archived snapshot of the website of the author's consultancy ("working with survivors, communities, institutions, and workplaces to prevent and heal from sexual harassment and sexual assault") was recorded in August 2022.
• According to the CEA Community Health team: "The author emailed the Community Health team about 7 months ago, when she shared some information about interpersonal harm; someone else previously forwarded us some anonymous information that she may have compiled. Before about 7 months ago, we hadn’t been in contact with her."
  • This would have been late July 2022.
• From the same comment by the CEA Community Health team: "We have emailed the author to tell her we will not be contracting her services."
  • Implied: the author attempted to sell her professional services to CEA.
• The author, in the linked piece: "To be clear, I’m not advocating bans of the accused or accusers - I am advocating for communities to do more, for thorough investigations by trained/experienced professionals, and for accountability if an accusation is found credible. Untrained mediators and community representatives/liaisons who are only brought on for their popularity and/or nepotistic ties to the community, without thought to expertise, experience, or qualifications, such as the one in the story linked above (though there are others), often end up causing the survivors greater trauma." (Emphasis mine.)
• The author: "In February 2023, I calculated that I personally knew of/dealt with thirty different incidents in which there was a non-trivial chance the Centre for Effective Altruism or another organization(s) within the EA ecosystem could potentially be legally liable in a civil suit for sexual assault, or defamation/libel/slander for their role/action (note: I haven’t added the stories I’ve received post-February to this tally, nor do I know if counting incidents an accurate measure (eg, accused versus accusers) also I’ve gotten several stories since that time; nor is this legal advice and to get a more accurate assessment, I’d want to present the info to a legal team specializing in these matters). Each could cost hundreds of thousands and years to defend, even if they aren’t found liable. Of course, without discovery, investigation, and without consulting legal counsel, this is a guess/speculative, and I can’t say whether they’d be liable or rise to the level of a civil suit - not with certainty without formal legal advice and full investigations."  (Emphasis in original.)
• The author: "In response to my speculation, the community health team denied they knew of my work prior to August 2022, and that it was not connected to EA. Three white community health team members have strongly insinuated that I’ve lied and treated me – an Asian-American – in much the gaslighting, silencing way that survivors reporting rape fear being treated. Many of the women who have publicly spoken up about sexual misconduct in EA are of Asian descent. As I stated in the previous paragraph, I haven’t yet consulted with lawyers, but I personally believe this is defamatory. Additionally, the Centre and Effective Ventures Foundation are in headquartered in a jurisdiction that is much more harsh on defamation than the one I’m in."  (Emphasis in original.)
• The author: "Unlike most of these mediators and liaisons, I have training/formal education, mentorship, and years of specific experience. If/When I choose to consult with lawyers about the events described in the paragraph above, there might be a settlement if my speculations of liability are correct (or just to silence me on the sexual misconduct and rapes I do know of). If (again, speculative) that doesn’t happen and we continue into a discovery process, I’m curious as to what could be uncovered." (Emphasis in original.)

I don't doubt that the author cares about preventing sexual assault, and mitigating the harms that come from it.  They do also seem to care about something that requires dropping dark hints of potential legal remedies they might pursue, with scary-sounding numbers and mentions of venue-shopping attached to them.

I think you are somewhat missing the point.  The point of a treaty with an enforcement mechanism which includes bombing data centers is not to engage in implicit nuclear blackmail, which would indeed be dumb (from a game theory perspective).  It is to actually stop AI training runs.  You are not issuing a "threat" which you will escalate into greater and greater forms of blackmail if the first one is acceded to; the point is not to extract resources in non-cooperative ways.  It is to ensure that the state of the world is one where there is no data center capable of performing AI training runs of a certain size.

The question of whether this would be correctly understood by the relevant actors is important but separate.  I agree that in the world we currently live in, it doesn't seem likely.  But if you in fact lived in a world which had successfully passed a multilateral treaty like this, it seems much more possible that people in the relevant positions had updated far enough to understand that whatever was happening was at least not the typical realpolitik.

2. If the world takes AI risk seriously, do we need  threats? 

Obviously if you live in a world where you've passed such a treaty, the first step in response to a potential violation is not going to be "bombs away!", and nothing Eliezer wrote suggests otherwise.  But the fact that you have these options available ultimately bottoms out in the fact that your BATNA is still to bomb the data center.

3. Don't do morally wrong things 

I think conducting cutting edge AI capabilities research is pretty immoral, and in this counterfactual world that is a much more normalized position, even if consensus is that chances of x-risk absent a very strong plan for alignment is something like 10%.  You can construct the least convenient possible world such that some poor country has decided, for perfectly innocent reasons, to build data centers that will predictably get bombed, but unless you think the probability mass on something like that happening is noticeable, I don't think it should be a meaningful factor in your reasoning.  Like, we do not let people involuntarily subject others to russian roulette, which is similar to the epistemic state of the world where 10% x-risk is a consensus position, and our response to someone actively preparing to go play roulette while declaring their intentions to do so in order to get some unrelated real benefit out of it would be to stop them.

4. Nuclear exchanges could be part of a rogue AI plan

I mean, no, in this world you're already dead, and also nuclear exchange would in fact cost AI quite a lot so I expect many fewer nuclear wars in worlds where we've accidentally created an unaligned ASI.

He proposes instituting an international treaty, which seems to be aiming for the reference class of existing treaties around the proliferation of nuclear and biological weapons.  He is not proposing that the United States issue unilateral threats of nuclear first strikes.

This advocates for risking nuclear war for the sake of preventing mere "AI training runs".  I find it highly unlikely that this risk-reward payoff is logical at a 10% x-risk estimate. 

All else equal, this depends on what increase in risk of nuclear war you're trading off against what decrease in x-risk from AI.  We may have "increased" risk of nuclear war by providing aid to Ukraine in its war against Russia, but if it was indeed an increase it was probably small and worth the trade-off^[1] against our other goals (such as disincentivizing the beginning of wars which might lead to nuclear escalation in the first place).  I think approximately the only unusual part of Eliezer's argument is the fact that he doesn't beat around the bush in spelling out the implications.

1. ^{^}

   Asserted for the sake of argument; I haven't actually demonstrated that this is true but my point is more that there are many situations where we behave as if it is obviously a worthwhile trade-off to marginally increase the risk of nuclear war.

Many things about this comment seem wrong to me.

Yudkowsky's suggestions seem entirely appropriate if you truly believe, like him, that AI x-risk is probability ~100%. 

These proposals would plausibly be correct (to within an order of magnitude) in terms of the appropriate degree of response with much lower probabilities of doom (i.e. 10-20%).  I think you need to actually run the math to say that this doesn't make sense.

unproven and unlikely assumptions, like that an AI could build nanofactories by ordering proteins to be mixed over email

This is a deeply distorted understanding of Eliezer's threat model, which is not any specific story that he can tell, but the brute fact that something smarter than you (and him, and everyone else) will come up with something better than that.

In the actual world, where the probability of extinction is signficantly less than 100%, are these proposals valuable?

I do not think it is ever particularly useful to ask "is someone else's conclusion valid given my premises, which are importantly different from theirs", if you are attempting to argue against someone's premises.  Obviously "A => B" & "C" does not imply "B", and it especially does not imply "~A".

It seems like they will just get everyone else labelled luddites and fearmongerers, especially if years and decades go by with no apocalypse in sight. 

This is an empirical claim about PR, which:

1. does not seem obviously correct to me
2. has little to say about the object-level arguments
3. falls into pattern of suggesting that people should optimize for how others perceive us, rather than optimizing for communicating our true beliefs about the world.

Separate from object-level disagreements, my crux that people can have inside-view models which "rule out" other people's models (as well as outside-view considerations) in a way that leads to assigning very high likelihoods (i.e. 99%+) to certain outcomes.

The fact that they haven't successfully communicated their models to you is certainly a reason for you to not update strongly in their direction, but it doesn't mean much for their internal epistemic stance.