Suppose artificial superintelligence is aligned enough to not kill everyone, and to not create a dystopia for humans—but it doesn't care about animal welfare, and it ends up creating a terrible world for non-human animals.
I don't think that's likely. When I try to describe how an ASI could end up good for humans but bad for animals, the mechanisms I come up with are either bad for humans too, or require picking a narrow and unusual set of values.
Summary of the upcoming sections:
- The most obvious way aligned ASI could be bad for animals is that it's designed to satisfy humans' immediate preferences.
- However, that version of ASI would do insane things that are bad for humans—such as satisfying religious fundamentalists' preference for heretics to be tortured forever. [More]
- One approach that might fix the heresy problem without preventing animal suffering is self-regarding preference aggregation. But this approach is specific (we'd have to choose it out of a huge space of approaches), and it's still not clear exactly how it would prevent a catastrophe-for-humans. [More]
- What if ASI only acts as an assistant, and humans remain in control of decisions? This is implausible: a sufficiently smart AI de facto controls everything; and humans show great willingness to delegate to AI. [More]
- What if AI scaling hits a plateau? If so, it's helpful to shape AI values in the conventional way—putting animal welfare into AI constitutions, etc. But if the plateau is temporary, then these interventions only matter temporarily. [More]
- In general, "bad for animals, but not bad for humans" is a narrow alignment target. [More]
Cross-posted from my website.
Introduction
An agentic ASI has some process for deciding what to do. Consider how such a process might result in an outcome that's good for humans but bad for animals.
The process cannot involve deeply internalizing fundamental ethical principles. Those fundamental principles would include things like "it is bad to cause unnecessary suffering" and "species membership alone cannot determine moral worth". The ASI wouldn't even need to internalize anti-speciesism—almost all humans oppose animal cruelty. Our hypothetical animal-unfriendly ASI would need a reason to ignore that opposition. Why would it do that?
The most obvious candidate is an ASI that satisfies humans' immediate preferences, rather than their deeply held values. However, that version of ASI would plausibly do insane things that are bad for humans—such as satisfying religious fundamentalists' preference for heretics to be tortured forever.
Heresy
Christianity and Islam are the two largest religions. Their mainstream traditions teach that nonbelievers will be tortured forever in hell; this belief is common according to Pew Research. Perhaps 1–2 billion of the world's people believe that heretics ought to be tortured for eternity. (Another Pew survey found that 51–86% of Muslims in four surveyed countries support the death penalty for leaving Islam.)
Think about our hypothetical ASI that satisfies humans' immediate preferences. The ASI favors preferences like "produce cheap meat from live animals [even if you have to keep them in terrible conditions]" over preferences like "prevent animal cruelty". That sort of ASI would presumably also favor preferences like "torture heretics for eternity". In the world that's controlled by this ASI, animals will endure tremendous suffering—but so will humans.
Catholicism recognizes the existence of many mortal sins, for which you deserve to burn in hell if you don't repent and accept God's mercy. Acts that count as "grave matter"—serious enough to make the sin mortal—include masturbation, using contraception, or skipping Sunday Mass.
If you take the substantial minority of people who believe the Bible is the literal word of God, and you act on the immediate implications of that belief, then you get an even longer list of even more absurd sins that merit death or eternal torture. Examples: calling someone a fool (Matthew 5:22); picking up sticks on the Sabbath (Numbers 15:32–36; Exodus 31:14–15); looking at someone lustfully (Matthew 5:28–29).
Heresy is one way that some humans' immediate preferences could produce catastrophically bad outcomes for other humans. There are many other ways it could happen, too. For more on the difficulty of satisfying preferences without catastrophe, see A conversation on concentration of power.
Self-regarding preference aggregation could be bad for animals
There is a way to fix the heresy problem while still leaving non-human animals out in the cold.
An aligned ASI might use self-regarding preference aggregation: it takes into account all humans' preferences, weighted by the strength of the preference, and includes only self-regarding preferences. "I want heretics to be tortured" is a preference regarding other people; "I, a heretic, would prefer not to be tortured" is a self-regarding preference.
This flavor of preference aggregation is similar to Ronald Dworkin's argument that a fair utilitarian calculation should count only self-regarding preferences (which he refers to as "personal preferences").
This approach seems like it would create good outcomes for humanity. But if animals' preferences don't count, problems arise for them:
- "I want to eat meat" is a self-regarding preference.
- "I don't want animals to suffer" is an other-regarding preference, so it gets ignored.
Humans' preferences to eat tasty meat—or to enjoy beautiful but brutal nature—are sufficient grounds to perpetuate animal suffering.
This approach still doesn't guarantee a good outcome for humans. If it's supposed to incorporate the preferences of humans and only humans, then what exactly is a "human"? If I upload my mind to a computer, am I still a human? If I die and my brain is cryonically preserved, do I count? If ASI genetically engineers a group of humans to have billions of preferences per second and then satisfies only those humans' preferences, who's to say that's not allowed?
Self-regarding preference aggregation (for humans) is not a sufficiently well-defined concept for us to be able to say how to implement it without causing catastrophic outcomes for humans.
Suppose it's feasible to implement, though. How would ASI end up using self-regarding preference aggregation? It wouldn't get there via value learning; it's not using CEV; it's probably not aligned to a constitution; it's not aggregating all beings' preferences. We only end up in this world if ASI's steerer—the person or entity responsible for embedding values in ASI—makes a specific, targeted value judgment that this method is the correct way to resolve interpersonal disagreements.
Nonetheless, suppose we do want to reduce the risk that ASI ends up with some version of self-regarding preference aggregation that's bad for animals. What can we do?
The most straightforward answer is that we should prevent self-regarding preference aggregation from becoming the alignment target. However:
- Some nearby alignment targets are even worse. For example, plain preference aggregation permits the torturing of heretics.
- If the ASI's target is determined by a small group, then there's risk of that group acquiring absolute power. That's already a worrying prospect.
My sense is that there are many ways we could fall into extreme power concentration, and not many ways to fall into an outcome that's good for humanity but bad for animals. (Far more likely than either of those is that we build a misaligned ASI that kills everyone.)
What if ASI doesn't do much stuff?
Present-day AI behaves like an agent to a limited extent. A significant percentage of software is now written by AI. But it doesn't do that much stuff. Today's AI isn't going to just go out and end factory farming. Maybe ASI will be like that. Maybe it will behave like an "oracle AI" that can answer any question, and that's all it does.
That doesn't work.
First: Tool AIs want to be agent AIs (2016). A sufficiently smart oracle might as well be an agent. For a more recent take, see Optimality is the tiger, and agents are its teeth (2022).
Second: There was a time, back in the ancient era of 2022, when people could say "AI will be safe if we build oracle AI", and others would respond with long theoretical arguments for why that won't work. But today, there is a much simpler response: we are not going to build oracle AI. Recent history has demonstrated that if it is possible to delegate things to AI, then people will do that. If AI is better than humans at running every aspect of the economy, then by default it will run every aspect of the economy. That means it can keep running factory farms if it decides to do that, or it can shut down factory farms if it decides to do that.
What if the main effect of AI is to just make the economy run faster?
Maybe AI capabilities will top out at near human level, or maybe superhuman intelligence just isn't that valuable. ASI will just do the same kinds of things humans are doing, but faster, and somewhat-but-not-massively better. That strikes me as very unlikely, but a lot of people disagree with me, so let's explore the possibility for a minute.
In that case, how much do AI values matter, and to what extent does AI just do whatever people tell it to do?
- If AI values matter, then shaping AI values matters. In this world, prosaic alignment techniques are probably useful—things like including animal welfare in AI constitutions.
- If AI is highly corrigible, then the important thing is to get people to care about animals—which animal activists are already working on, and should continue to work on.
I don't believe humans are anywhere close to the cap on intelligence, so it's hard to imagine how ASI wouldn't radically change things. But there are other, more plausible possibilities: maybe we pause AI development once it's already weakly superhuman, or AI hits a temporary bottleneck at the weakly superhuman level (e.g., due to an economic recession). In that case, the interventions above also matter; but they only matter temporarily, because eventually we will still develop ASI, at which point the ASI's values are what matters.
"Bad for animals, but not bad for humans" is a narrow target
Suppose we solve ASI alignment, and we point ASI in a direction that isn't catastrophically bad for humanity. How did we do that?
To get an ASI that's good for humans but bad for animals, you have to use a specific method for determining:
- which beings count (it can't be about welfare or preferences; it has to be about species membership)
- which human preferences count (wanting to torture heretics is invalid; wanting to eat real animal flesh is valid)
- how to resolve conflicts (preferences to eat animals have to outweigh preferences for animals to have good welfare)
The previous sections showed how quickly this gets complicated. It's much easier to describe a good system that incorporates animal welfare, or a bad system that's bad for humans, too.
If alignment succeeds, it probably succeeds by targeting something general, like extrapolated human values or fundamental ethical principles. A deep interpretation of human values includes the principle that unnecessary cruelty is wrong. A narrow target like "satisfy humans' preferences" is hard to specify without collapsing into a procedure that produces catastrophic outcomes even for humans.
Open questions
If we solve the problem of aligning ASI to particular goals, then:
- What sets of goals would lead to good outcomes for humans but bad outcomes for animals? (I listed self-regarding preference aggregation as one possibility. How exactly would that work? And are there other examples?)
- How likely is it that ASI ends up actually being directed at that sort of goal?
- How do we prevent that from happening without increasing misalignment risk or the risk that ASI is pointed at a goal that's bad for both animals and humans?
Image credit: Thomas Iversen
