Animal welfare concerns are dominated by post-ASI futures

I've claimed before that animal welfare concerns should probably be dominated by post-ASI futures where humans survive.  Let me outline a hypothetical scenario to illustrate the shape of my concern.

Ten years from now, some lab builds an ASI with the assistance of AIs that are superhuman at AI R&D.  The ASI is not corrigible; that turned out to either be an incoherent target for a general system like that, or an unsafe target (because making it corrigible necessarily traded off against making it act in our interests in situations where we were wrong about either the long-run consequences of what we were asking for, or because we were wrong about our own values).

The lab founders and employees are smart, well-meaning, and have heard of CEV.  The alignment work goes implausibly well and we live in an implausibly friendly universe where CEV as an alignment target has a pretty wide basin of attraction.  We end up with a friendly singleton aligned to humanity's CEV.

CEV, as an alignment target, is about implementing the overlapping (extrapolated) preferences between humans.  As Eliezer wrote:

If your CEV algorithm finds that "People coherently want to not be eaten by paperclip maximizers, but end up with a broad spectrum of individual and collective possibilities for which pizza toppings they prefer", we would normatively want a Friendly AI to prevent people from being eaten by paperclip maximizers but not mess around with which pizza toppings people end up eating in the Future.

Not mess around with which pizza toppings people end up eating in the Future, you say?

In the section titled Role of 'coherence' in reducing expected unresolvable disagreements, Eliezer tries to address the question of animal suffering directly:

A CEV is not necessarily a majority vote. A lot of people with an extrapolated weak preference* might be counterbalanced by a few people with a strong extrapolated preference* in the opposite direction. Nick Bostrom's "parliamentary model" for resolving uncertainty between incommensurable ethical theories, permits a subtheory very concerned about a decision to spend a large amount of its limited influence on influencing that particular decision.

This means that, e.g., a vegan or animal-rights activist should not need to expect that they must seize control of a CEV algorithm in order for the result of CEV to protect animals. It doesn't seem like most of humanity would be deriving huge amounts of utility from hurting animals in a post-superintelligence scenario, so even a small part of the population that strongly opposes* this scenario should be decisive in preventing it.

(ADDED 2023: Thomas Cederborg correctly observes that Nick Bostrom's original parliamentary proposal involves a negotiation baseline where each agent has a random chance of becoming dictator, and that this random-dictator baseline gives an outsized and potentially fatal amount of power to spoilants - agents that genuinely and not as a negotiating tactic prefer to invert other agents' utility functions, or prefer to do things that otherwise happen to minimize those utility functions - if most participants have utility functions with what I've termed "negative skew"; i.e. an opposed agent can use the same amount of resource to generate -100 utilons as an aligned agent can use to generate at most +1 utilon. If trolls are 1% of the population, they can demand all resources be used their way as a concession in exchange for not doing harm, relative to the negotiating baseline in which there's a 1% chance of a troll being randomly appointed dictator. Or to put it more simply, if 1% of the population would prefer to create Hell for all but themselves (as a genuine preference rather than a strategic negotiating baseline) and Hell is 100 times as bad as Heaven is good, compared to nothing, they can steal the entire future if you run a parliamentary procedure running from a random-dictator baseline. I agree with Cederborg that this constitutes more-than-sufficient reason not to start from random-dictator as a negotiation baseline; anyone not smart enough to reliably see this point and scream about it, potentially including Eliezer, is not reliably smart enough to implement CEV; but CEV probably wasn't a good move for baseline humans to try implementing anyways, even given long times to deliberate at baseline human intelligence. --EY)

Let's suppose that the lab employees responsible for implementing the CEV target had even read this 2023 addendum.  Maybe they decide that the appropriate solution is to simply forgo any structure that allows vetoes, and to simply tank the fact that some small percentage of the population will have unpalatable-to-them extrapolated preferences which they will exercise with their shared allocation of natural resources.

Current human preferences are confusing to current humans.  They seem quite sensitive to contingent environmental circumstances.  I would be very surprised if extrapolated human preferences were not also quite sensitive to contingent details of the extapolation procedure.  This suggests that there might not be an "obvious" or "privileged" endpoint of such extrapolations, unless it turns out that there's a principled meta-procedure which enables trade/averaging across them, or at least finding the maximum overlapping subset.  (This sounds like a familiar problem.)

Let's put that problem aside, for now, and assume there is such a principled meta-procedure (or assume that actually current human preferences cohere extremely reliably in ways that aren't sensitive to the extrapolation procedure).

How confident are you that no current humans will have an extrapolated volition that non-strategically cares about preserving parts of the Earth's environment in their "natural" state?  What about an extrapolated volition that just wants to live out the rest of the universe's lifespan in a non-simulated pastoral lifestyle, with live, unmodified animals and everything?

Remember, we're definitely pressing the button, the other labs aren't too far behind and it's a miracle that we've hit the target at all.  "Failing" closed at the end of the three-stage procedure described in the "Selfish bastards" problem section isn't an option. Unfortunate that we couldn't find and verify an extrapolation procedure that doesn't have much worse downsides, while still remaining principled enough that we don't feel like we're unilaterally seizing the reigns of power to impose our preferences on everyone else.

Alas, naive solutions to the problem I've described above seem pretty unprincipled, since they approximately round off to "try to optimize over the environment to shape how other people's preferences change".  Also, that kind of thing seems like something that CEV would explicitly want to avoid being influenced by, so it's not even obvious that it'd help at all, as opposed to wasting a bunch of resources playing games that smarter agents with better decision theory would avoid playing in the first place.

Having said that, what things might improve the state of animal welfare in the post-ASI future?

(If you're a negative utilitarian, well, the situation seems kinda bleak for you if we do get an aligned singleton, since it doesn't seem like there'll be literally zero suffering in the future.  It might be even worse if get an unaligned singleton, since it's not clear that there won't be suffering subprocesses all over the place - very unclear.)

The obvious next answers are "making sure we don't die to an unaligned superintelligence", and "figuring out philosophy and meta-philosophy, so that we don't end up implementing a dumb extrapolation procedure which leaves a bunch of value on the table", since at least that way the number of animals living happy, fulfilled lives is likely to vastly dominate animals suffering for contingent reasons.  (Also, it seems pretty likely that most people's extrapolated preferences will be willing to engage in trades where they trade off the "animals suffering" bits of the things that they value for other things that that they value more, assuming that those don't get treated as utility inverters in the first place.)

Not very satisfying, and I'm naturally kind of suspicious of that being the bottom line, given my beliefs, but such is life.

Thanks to Habryka for discussion on how CEV would handle historical attempts to strategically influence CEV outcomes.