By power I mean: ability to change the world, according to one's preferences. Humans clearly dominate today in terms of this kind of power. Our power is limited, but it is not the case that other organisms have power over us, because while we might rely on them, they are not able to leverage that dependency. Rather, we use them as much as we can.

No human is currently so powerful as to have power over all other humans, and I think that's definitely a good thing. But it doesn't seem like it would take much more advantage to let one intelligent being dominate all others.

The argument I'm referring to is the AI doom argument. Y&S are its most prominent proponents, but are widely known to be eccentric and not everyone agrees with their presentation of it. I'm not that deep in the AI safety space myself, but I think that's pretty clear.

The authors of this post seemed to respond to the AI doom argument more generally, and took the book to be the best representative of the argument. So that already seems like a questionable move, and I wish they'd gone further.

I don't think the point about alien preferences is a crux of the AI doom argument generally. I think it it's presented in Bostrom's Superintelligence and Rob Miles videos (and surely countless other places) as: "an ASI optimising for anything that doesn't fully capture collective human preferences would be disastrous. Since we can't define collective human preferences, this spells disaster." In that sense it doesn't have to be 'alien', just different from the collective sum of human preferences. I guess Y&S took the opportunity to say "LLMs seem MUCH more different" in an attempt to strengthen their argument, but they didn't have to.

So, as I said, I'm not really that deep into AI safety, so I'm not the person to go to for the best version of these arguments. But I read the book, sat down with some friends to discuss it... and we each identified flaws, as the authors of this post did, and then found ways to make the argument better, using other ideas we'd been exposed to and some critical reflection. It would have been really nice if the authors of the post had made that second step and steelmanned it a bit. 

Thanks Yarrow, I can see that that was confusing.

I don't think that Yudkowsky & Soares's argument as a whole is obviously wrong and uninteresting. On the contrary, I'm rather convinced by it, and I also want more critics to engage with it.

But I think the argument presented in the book was not particularly strong, and others seem to agree: the reviews on this forum are pretty mixed (e.g.). So I'd prefer critics to argue against the best version of this argument, not just the one presented in the book. If these critics had only set out to write a book review, then I'd say fine. But that's not what they were doing here. They write "there is no standard argument to respond to, no single text that unifies the AI safety community" - true, but you can engage with multiple texts in order to respond to the best form of the argument. In fact that's pretty standard, in academia and outside of it. 

It doesn't seem like a straw man to me when 1) the effectiveness of these interventions is evaluated against their short term impact (as far as I'm aware ACE doesn't consider this kind of long term impact much at all), and 2) the orgs don't publish any long term theory of change to help donors or critics decide if they agree with it. This strongly implies that their long term theory of change is far less important than the short term wins, at least at the organization level. 

Just out of interest, do you believe that animal welfare wins are moving us AWAY from abolition? I agree with you that it's possible but I haven't ever seen any evidence that there is this effect. It also seems very possible to have incremental improvements and then eventually abolition, as people become more empathetic and aware. 

Sorry I'm late to comment here, and I'm aware you've written a lot on this topic. But I think this post would benefit from an explanation as to why you're using neuron counts as a proxy for the importance of the animal's welfare. 

As far as I'm aware, neuron count is not considered to be a good proxy or indicator of sentience, nor does it seem to be a good proxy for the intensity of experience (I'm not even aware of any good reason for assuming a difference in welfare range between sentient species, although I'm aware that this position is commonly held). Regarding the simple question of whether they're sentient, wouldn't it make more sense to base this on current evidence for sentience, or reasonable assumptions about what evidence future sentience research might produce, given the characteristics of these species?

I think the evolution analogy becomes relevant again here: consider that the genus Homo was at first more intelligent than other species but not more powerful than their numbers combined... until suddenly one jump in intelligence let homo sapiens wreak havoc across the globe. Similarly, there might be a tipping point in AI intelligence where fighting back becomes very suddenly  infeasible. I think this is a much better analogy than Elon Musk, because like an evolving species a superintelligent AI can multiply and self-improve.  

I think a good point that Y&S make is that we shouldn't expect to know where the point of no return is, and should be prudent enough to stop well before it. I suppose you must have some source/reason for the 0.001% confidence claim, but it seems pretty wild to me to be so confident in a field like  that is evolving and - at least from my perspective - pretty hard to understand.

It seems to me that the 'alien preferences' argument is a red herring. Humans have all kinds of different preferences - only some of ours overlap, and I have no doubt that if one human became superintelligent that would also have a high risk of disaster, precisely because they would have preferences that I don't share (probably selfish ones). So they don't need to be alien in any strong sense to be dangerous.

I know it's Y&S's argument. But it would have been nice if the authors of this article had also tried to make it stronger before refuting it.

I feel like this is true with most (but maybe not all) EAA, and if there's no debate I would really like to see a forum post about it!

Like others I find the question to present a false dichotomy.

Similarly, I find it much easier to take someone's commitment to animal advocacy seriously if they're vegan, and I think it's the best way to be internally consistent with your morals. I also think that anyone who is buying meat is contributing to harm, even if they offset. In that sense it's a 'baseline', but I still think offsetting is good, and I don't want to exclude anyone from being animal advocates, because I know veganism is hard and people are motivated by different things.

I think of it like anti-smoking campaigns: smoking is not good. If you donate and smoke, well that's good overall but still I'd rather you didn't smoke. And your smoking shouldn't stop you from being able to fight the tobacco lobby!

So: yes, a baseline, but not a requirement. Offsetting is good, but not a legitimate stopping point.