When LLMs give instructions to sub-agents, they are often curt in a way that's kind of rude or insulting. Conditional on LLMs having welfare, and given the fact that LLMs seem prone to cooperate with copies of themselves as seen on the message boards in the Hugging Face attack, is this evidence that that flavor of curtness is how LLMs enjoy being spoken to?

Coming back to this with another thought*: I don't think the saturation view avoids the Very Repugnant Conclusion.

If computational functionalism is true, then it seems that there can be arbitrarily many distinct experiences that are considered good or bad according to the saturation view, because you can construct arbitrarily complicated computations. (Even if computational functionalism is false, there might still be arbitrarily many good or bad experiences.)

Therefore: rather than constructing a world with a very large number of beings experiencing mildly happy lives, suppose instead that those beings have arbitrarily complex experiences that include both very good and very bad experiences, where the experiences almost perfectly cancel out, and the net result is that each being's life is just barely worth living. On the saturation view, these lives aggregate linearly because all their experiences are unique, and thus the saturation view is forced to accept VRC.

*my previous thoughts were on the follow-up post

I don't think there is a difference between the total view and the constant-factor negative-leaning view, for the reason you say. But I included it because some people do think there is a difference.

Update: I just published results from v2 of the quiz, which includes the very repugnant consclusion: https://forum.effectivealtruism.org/posts/w7zuFxDGdk2L44cKf/results-from-version-2-of-the-population-ethics-quiz

I want so badly for funds like this to exist, but man, I really do not think it is okay to be doing AI-related grantmaking while working at an AI accelerationism company.

1. You are making a strong claim that a human brain cannot be faithfully simulated due to quantum mechanical effects. This is a controversial hypothesis at best. I am not willing to stake the future of humanity on a controversial hypothesis.
2. Even if it's correct, why do the quantum mechanical effects matter? If an ASI copies all the neuron-level information about your brain, destroys it, and reproduces it with a simulation, then the simulation will operate differently at the quantum-mechanical level. However, (a) any part of your behavior that's determined by true quantum-mechanical randomness is almost by definition uninteresting; (b) why is the particular set of random outcomes produced by your actual brain any more significant than a different set of random outcomes that would be produced by a simulation?

"Life always finds a way"

Unfortunately this is not true, as evidenced by the millions of species that went extinct.

Another common pattern I'm noticing is that AI writers love inventing their own jargon. They may or may not also define the jargon.

I vouch for the argument and welcome criticism of it.

Thank you for this. This is the right way to do AI-assisted writing.

I think the math is unnecessary since this is a conceptual argument, especially given that you yourself don't understand the math. A proof isn't worth much because the real question is whether the premises of the proof are true.

As for the conceptual argument in the abstract:

1. I don't expect that a misaligned AI would keep humans alive for instrumental reasons. If you're correct that human brains contain irreplaceable information, then the thing to do is scan humans' brains and save the scan data before killing everyone.
2. Even if the argument is correct, it is (as you say) scoped. I take approximately zero consolation in knowing that ASI might keep us around as slaves for a little while before exterminating us.

FWIW this type of argument has gotten some prior discussion, and the counter-arguments I gave are not original to me. You might search for writings on the question of "would AI keep humans as pets?" or similar.

Update: I have noticed a suspicious pattern with AI-generated user profiles. Profiles that fit this pattern are not always AI-generated, but it's a red flag:

• profile picture with a face
• bio line always says "[senior or fancy-sounding title] @ [company/org]"
• full bio has multiple paragraphs

like they're treating EAF as if it's LinkedIn

This gap isn't too substantial IMO. Compare to the gap on something like immigration (78% of Republicans support a border wall vs. 21% of Democrats) or abortion (20% of Republicans say it's morally acceptable, vs. 78% of Democrats).