By Zershaaneh Qureshi & Robert Wiblin   |    Watch on Youtube   |   Listen on Spotify   |    Read transcript

Episode summary

Most debates about the moral status of AI systems circle the same question: is there something that it feels like to be them? But what if that’s the wrong question to ask?

Andreas Mogensen — a senior researcher in moral philosophy at the University of Oxford — argues that so-called ‘phenomenal consciousness’ might be neither necessary nor sufficient for a being to deserve moral consideration.

For instance, a creature on the sea floor that experiences nothing but faint brightness from the sun might have no moral claim on us, despite being conscious.

Meanwhile, any being with real desires that can be fulfilled or not fulfilled can arguably be benefited or harmed. Such beings arguably have a capacity for welfare, which means they might matter morally.

And, Andreas argues, desire may not require subjective experience. Desire may need to be backed by positive or negative emotions — but as Andreas explains, there are some reasons to think a being could also have emotions without being conscious.

There’s another underexplored route to moral patienthood: autonomy. If a being can rationally reflect on its goals and direct its own existence, we might have a moral duty to avoid interfering with its choices — even if it has no capacity for welfare.

However, Andreas suspects genuine autonomy might require consciousness after all. To be a rational agent, your beliefs probably need to be justified by something, and conscious experience might be what does the justifying. But even this isn’t clear.

The upshot? There’s a chance we could just be really mistaken about what it would take for an AI to matter morally. And with AI systems potentially proliferating at massive scale, getting this wrong could be among the largest moral errors in history.

In today’s interview, Andreas and host Zershaaneh Qureshi confront all these confusing ideas, challenging their intuitions about consciousness, welfare, and morality along the way. They also grapple with a few seemingly attractive arguments which share a very unsettling conclusion: that human extinction (or even the extinction of all sentient life) could actually be a morally desirable thing.

This episode was recorded on December 3, 2025.

Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Coordination, transcripts, and web: Katy Moore

Highlights

Does consciousness on its own merit moral consideration?

Zershaaneh Qureshi: If we were able to determine that an AI was conscious, would that on its own be enough to oblige us to take its moral interests into consideration?

Andreas Mogensen: I think there might be two slightly separate questions here. One is whether phenomenal consciousness on its own is enough for having some kind of moral standing — because you might think if there were a current AI system that was phenomenally conscious, probably it would not only be phenomenally conscious; it would be phenomenally conscious and exhibiting plausibly quite sophisticated cognitive capacities of a kind that maybe, amongst animals, only human beings have, and that most other animals lack at least, or something like that.

And I think those are potentially two quite different questions. So considering the question of whether phenomenal consciousness on its own suffices to make you the sort of being that we ought to be morally concerned for, I think it’s somewhat difficult to say, but my instinct is to say no.

It seems like we can imagine these very simple creatures, so there might be some kind of organism that is bolted to the sea floor and it has some degree of conscious experience, but the only conscious experience it has is an impression of a slight brightness every now and then. Something like that: there are no thoughts, there are no emotions. There’s just this faint feeling of brightness that flips through its experiential field every now and then. [Note: this example is borrowed from a paper by Andrew Lee. —AM]

So there’s this question: does it matter morally what happens to this creature, more so than an otherwise exactly similar organism sitting next to it where the lights are totally off? It’s very difficult to say exactly, but my intuition is that it certainly seems fine to say no: the fact that it’s phenomenally conscious, that just doesn’t actually suffice to give it any kind of claim on us.

And I think there’s also a question of what kind of claim would it even have on us? It doesn’t feel pleasure or pain. It has no desires. It just has this experience of a faint brightness every now and then. So I’m inclined to think no, in that sort of case. So in that sense, phenomenal consciousness does not suffice for moral standing or moral patienthood.

So then there’s a separate question of: might phenomenal consciousness suffice for moral standing in conjunction with other mental states? And certainly if you put it like that, then the answer is definitely yes, because those other mental states could be emotions and feelings of pain.

One question that’s maybe more interesting is something like, if you had a creature that was phenomenally conscious and it did have the kind of sophisticated cognitive abilities that you and I have, and that sort of characterise human beings as animals, and it was phenomenally conscious, but say it felt no emotions, never felt pleasure or pain, would it have moral standing?

This is more or less identical to this philosophical Vulcan thought experiment that David Chalmers has come up with, which I think may have been debuted on the 80K podcast many years ago.

Zershaaneh Qureshi: Possibly. What a claim to fame.

Andreas Mogensen: Yeah. He asked us to imagine what he calls “philosophical Vulcans.” So these are supposed to be sort of extreme versions of the Vulcans from Star Trek. I think the Vulcans from Star Trek do feel emotions and they suppress them. But these are beings that never feel any emotion; they never feel pleasure nor pain nor happiness nor sorrow. Nonetheless, they’re conscious beings, very much like you or me. They have sophisticated intellectual and moral desires.

And there’s a question of whether they have moral standing. Chalmers has the intuition that they do. It would be wrong to mow down hundreds or thousands of these Vulcans for the sake of providing some trivial benefit to some being that’s sentient — in the sense of not only having phenomenal consciousness, but experiencing pleasure, pain, emotional feelings, and that sort of thing. So like surely it would be wrong to kill millions of these Vulcans in order to give a hamster a juicy treat or something like that.

I definitely share that intuition, so I’m definitely inclined to think that beings of this kind, Vulcan-like beings, would have some kind of moral standing. But it’s not just because they’re conscious.

Another route to moral patienthood: deserving autonomy

Zershaaneh Qureshi: When people ask, “Should AIs be given moral consideration?” the conversation often ends up being about this question of: do they have some capacity for welfare? Or in other words, are there ways that this AI could be benefited or harmed by my actions?

But I want to understand whether an AI having this capacity for welfare is even really necessary for it being worthy of our moral concern. And you think that it’s not: you think there’s maybe another route to moral patienthood that doesn’t even involve being the kind of thing whose life can go better or worse. Can you explain what that route is?

Andreas Mogensen: Yeah. Very roughly, the thought is that something like autonomy or capacity for autonomy might suffice to make you a moral patient, in the absence of any capacity to be benefited or harmed.

Zershaaneh Qureshi: And when you talk about autonomy, what do you mean here?

Andreas Mogensen: Good. It’s a good question. I guess one thing I should say is — well, maybe it doesn’t need saying, because it’s true of all things in philosophy — but it’s very controversial exactly what autonomy is.

Zershaaneh Qureshi: Of course it is.

Andreas Mogensen: I might think, for example, that autonomy is a kind of psychological capacity for self-government, for the ability to sort of direct your life based on rational reflection — of a kind that human adults, say, possess to a much greater degree than human children and nonhuman animals or something like that. So that’s roughly what it is that we’re sort of pointing to.

Then there’s questions about exactly what it is we’re pointing to when we try to point to that thing. And this is where it gets really controversial amongst philosophers. So there’s this thought that, generally speaking, in order to be autonomous, you need to have a capacity for rational reflection. You aren’t simply led about blindly by impulses. I think very often this is conceived as involving a capacity for second-order desires.

What this means is there are first-order desires, which are basically just desires for things out there in the world: desires for chocolate cake or desires to win a football match or something like that. And then there are desires about desires: these are second-order desires. So I might desire that I don’t want to eat the chocolate cake, because maybe I’m on a diet or something like that.

And some philosophers, especially inspired by the work of Harry Frankfurt, think that what’s really essential to autonomy is that you’re able to formulate these second-order desires, or even higher-order desires — through which you identify yourself with some of these first-order desires you might have and you distance yourself from others.

Zershaaneh Qureshi: Yeah. So that means you choose to pursue or not pursue some of these first-order desires in accordance with what your second-order desire is, right?

Andreas Mogensen: Yeah, exactly. Absolutely. So a lot of people think that’s really essential for autonomy, or something that looks a little bit like that capacity.

Other things that people often associate with autonomy is things like being minimally rational in some sense, so satisfying certain minimal criteria for having consistent beliefs and desires, for not just having a totally jumbled mind or something like that.

Over and above just satisfying these formal coherence criteria, some philosophers think to really count as autonomous, you need to be rational in a more substantive sense. So they think you need to genuinely value the things that genuinely are valuable or something like that. That’s slightly more controversial, but I think it’s a view that has quite a bit of purchase amongst philosophers who think about autonomy.

Another thing that’s often associated with autonomy is some idea that to count as properly autonomous, you sort of need to have the right kind of history — so you can’t tell whether someone’s autonomous just by looking at them as they are right now.

This often goes off these thought experiments where you can imagine that someone’s been going through their life and then someone spreads some nanobots into their brain and rewires their neural circuitry such that they become a completely different person. Maybe they become a duplicate of somebody else or something like that, but they have all these alien desires and preferences and values implanted within them which they might then act in a rational and reflective way in order to realise. But there’s a sense that this individual is not autonomous because of the way that these goals and values originated in them.

So there’s this thought that there needs to be the absence of a certain kinds of manipulation in our past in order for us to count as genuinely autonomous. Exactly what needs to be absent is a matter of significant controversy. But a lot of people also buy this sort of view.

So these are just some sketches as to what it would take to count as autonomous. But as I said, everything is very controversial, especially so when it comes to what it means to be autonomous.

…

Zershaaneh Qureshi: One of the things you mentioned is this manipulation point: you talked about the example of someone doing the sort of complicated neural fiddling, and that the person who’s had their brain fiddled with doesn’t count as autonomous. Does that kind of rule out AIs from meeting the criteria? You could think that we’ve manipulated AIs in basically that way.

Andreas Mogensen: It’s a good question. I think the honest answer is I don’t really know, in large part because I think we as a community of philosophers or something like that don’t quite know what it means to be manipulated in the relevant sense.

You might think that in some sense, certainly if the universe is genuinely deterministic, then our psychological state will always have been determined by factors beyond that occurred prior to our birth, that were outside of our control. So there definitely is a worry that many philosophers have articulated that these kind of intuitions about manipulation cases, if you take them to their logical conclusion, mean that there couldn’t be genuine autonomy in a deterministic universe.

And similarly you might think, assuming that LLMs count as having preferences and values, those preferences and values are sculpted through things like reinforcement learning from human feedback and so forth. You might wonder, that might look a lot like manipulation, but how different is it from the ways in which we instil values in human children through ordinary parenting practices?

I feel like I probably don’t understand enough about how reinforcement learning from human feedback works to be able to say with much confidence exactly how different these processes are. But I think it’s certainly a worry that the way in which the values and preferences of LLMs are sort of installed in them by us might very much raise this worry that they fall foul of this anti-manipulation clause. But there is also this worry that we, if we really thought about it, would fall foul of this anti-manipulation clause, because our preferences and values are also sculpted by factors beyond our control.

Zershaaneh Qureshi: Yeah. So you might think there’s some sort of line somewhere, where if you fall on one side of the line you do count as autonomous, and if you fall on the other side of the line you don’t count as autonomous. And maybe the thing at issue here is like the extent to which you’ve been manipulated or something like that.

Maybe there's no objective truth about any of this

Andreas Mogensen: So imagine you live in a cabin somewhere. There’s a mountain some distance away. Each morning you go out, you look on the horizon, and you see, moving along the mountainside, some kind of shape. It’s all really fuzzy at this distance, so you can’t really work out what it is that you’re looking at. And you don’t really give it much thought. You don’t spend lots of time thinking about, what is that? You just sort of notice that there’s this shape moving along. You don’t give any more thought. You go back inside to have your breakfast or whatever.

Suppose this keeps happening day after day. You go out in the morning, you look onto the mountainside, and there’s this kind of fuzzy shape. You don’t give it much thought, but you do think to yourself something like, “It’s one of those again.” And you do this day after day. You think, “It’s one of those again. There it is again.”

Suppose what you’ve been seeing are deer, like real living deer. Real living deer are the things which give this kind of appearance from that distance. Suppose nonetheless that one day some scientists have built a robotic deer. This deer looks quite a bit like the deer that you’ve been seeing. And indeed, it gives the same appearance, let’s say, from this distance or something like that. So this robotic deer has wandered out onto the mountainside. You’re looking out and you think, “It’s one of those again.”

So this is a question: Is that true? Is it one of those again? I think a kind of natural reaction is, well, I don’t know. There isn’t really a fact of the matter here. Because your usage of this term, this concept of “one of those,” you haven’t really put enough thought into your usage of that concept to determine whether it refers specifically to deer, a living thing, or whether it refers to anything that has the appearance of a deer, anything that has the outward form of a deer, or maybe anything that gives off the kind of visual appearance at this distance that these deer give off, or something like that — which might in principle be something that a totally different kind of thing could give off, or something like that.

You’ve just been kind of gesturing somewhat blindly at something you know not what, and you’ve been kind of pointing to it, and it’s not really clear that the general kind you’ve been pointing to, whether that’s deer, things that look like deer but could be robots, or something else.

And the thought’s supposed to be that something similar is going on with our concepts of our conscious states. So when I think there’s that horrible feeling, pain, because it’s totally opaque to me what’s really going on inside my mind, my usage of this concept “that horrible feeling” is just not specific enough to determine whether I’m pointing to a neurophysiological state in particular or an abstract computational state in particular.

And for this reason, there might very well be no fact of the matter about whether something that’s shared in the abstract computational state but lacked the underlying neurophysiology, whether that would count as having “that” — the feeling that’s horrible — just as there might be no fact of the matter about whether this robot deer is “one of those.”

Zershaaneh Qureshi: Yeah, yeah, this makes sense to me. I think what strikes me here is just how much this is coming down to how our concepts work and how we’re using terms, rather than what’s actually out there in the world — like whether there is a deer or a robot deer out there in the world or something.

Why not just let superintelligence figure this out for us?

Zershaaneh Qureshi: I really want to get a sense of just how important these topics feel right now. There is some reason to believe that at some point we’re going to get really advanced AI systems that can help us reason through really tricky philosophical problems like these ones about moral patienthood. And maybe there’s some reason to think that in the future we’ll just be way better equipped to answer all of this messy stuff than we are right now. Should we just wait for that, or should we be working on this right now?

Andreas Mogensen: It’s a good question. If you were sort of certain that you would have a superintelligent AI system that radically surpassed human capacity for moral reasoning or moral philosophy, and that such a system would arrive before there were any AI systems that were moral patients, then yeah, obviously it would make sense to just wait until we have such a system and allow it to guide us or something like that.

It nonetheless seems to be a very real possibility that things won’t appear in that order, that you might have a world in which there are potentially a large number of AI systems of a kind that exhibit morally significant mental properties like emotion, phenomenal consciousness, whatever you think is the really important stuff — and that predates a world in which superintelligent AI systems of a kind that might far surpass our capacity for moral reasoning and moral philosophy appear.

So plausibly it hinges on these timelines-y questions. But it does seem to me that, on the one hand, the possibility that we’ll get the arrival order such that in some sense we have to face up to these questions before we can punt these questions to a more intelligent being, if such a being comes to exist, we’re going to sort of face that crunch, that point in time, that seems relatively likely. And I think it especially seems working on these issues because they seem very neglected by comparison with other kinds of issues that you might associate with risks and opportunities thrown up by the possibility of transformative artificial intelligence or whatever you want to call it.

Zershaaneh Qureshi: Yeah. And I guess it’s not just a case of like, if we get things wrong and realise later that we were wrong, no harm done. It’s like, if we get things wrong, it means we are enslaving species of morally important beings or something like that.

Andreas Mogensen: Yeah, something like that.

Zershaaneh Qureshi: Hopefully that’s not the direction things go. But there are some stakes here which suggest that working on things sooner, when there’s uncertainty about what order things are going to happen in, seems wise.

Andreas Mogensen: Yeah. You might also think that there’s the potential for something like a kind of weak lock-in effect or something like that. It’s somewhat implausible that you would lock in the kind of way of treating digital minds as strongly as, say, some kind of extinction event that killed all human beings would lock in a world without humans. Extinction is very final.

But nonetheless, if you think about, say, our treatment of intensively farmed or factory farmed animals, in the early 20th century when factory farming was first coming online, when this kind of industrialised agriculture was still being put in place, if we’d anticipated the future we would be bringing about, and we thought, “Is this a good idea? Is this morally OK?” we would probably have thought, no, we shouldn’t be doing this.

But now we’re in this world where industrialised agriculture and the intensive farming of nonhuman animals is an entrenched practice. And once a practice becomes entrenched like this, it becomes much harder to give it up, because people have become accustomed to being able to buy meat at very low prices and things of that nature. So it becomes much harder to quit a practice once you’ve begun it, once it’s become institutionalised and entrenched and once people are used to it, rather than to sort of preempt a given practice before it really properly gets going, and people come to depend on it and expect that things will work a certain way.

Zershaaneh Qureshi: Yeah. You can imagine a situation with AIs where we get some regulation in place, and even on the basis of new evidence that comes to light further down the line, where we’re like, “Maybe we should be kinder to our AIs,” there will be a lot of resistance because it’s like, “Hey, we already came up with the regulations. Why are you changing it?”

Or you reach a point where lots of people have lots of interests embedded in the way things have been set up. They’re making profits out of the status quo, and we’ve come to expect that we can run these things very cheaply and this incurs costs. So I can totally imagine how it pans out that it becomes especially difficult to reverse things once they’ve been set in motion. Though definitely not impossible. Not as possible as if we had literally gone extinct at this point. That seems much harder to get back from.

Andreas Mogensen: Yeah. And in some sense you might think… I mean, maybe there wasn’t a moral superintelligence around at the time, but certainly once factory farming had got going and had become entrenched, there were moral philosophers like Peter Singer and others, who you might think in some sense have a greater-than-ordinary capacity for moral reasoning or for sort of ferreting out what’s actually morally important, what’s actually right and what’s wrong. And those people did tell society at large that what they were doing was morally unacceptable or something like that.

And certainly there were many people who were receptive to that message, but it’s certainly not put a stop to factory farming and associated practices. So even if there might be some future moral sage that could awaken in a data centre somewhere, I’m not sure that one should expect that its advice would be able to sway us from abandoning a practice that had become entrenched in something like the way factory farming has become entrenched.

Zershaaneh Qureshi: Yeah. Especially if we’re like, “Well, obviously you, the AI, are going to say that you deserve moral rights. Why should we listen to you?”

Andreas Mogensen: Exactly, yeah.

How could human extinction be a good thing?

Zershaaneh Qureshi: You’ve done a lot of research into some quite tricky moral questions, and some of the stuff that really stands out to me is about how we should weight suffering and whether that means human extinction could actually be a morally desirable thing.

At 80,000 Hours, as listeners probably know, we’re pretty keen to have lots of people working on preventing human extinction, on the grounds that we think the future of our species could be incredibly valuable and worth preserving. So I really want the answer to the question “Is human extinction morally desirable?” to be no. But can you maybe tell me in brief terms what assumptions or facts about the world might lead you to believing it?

Andreas Mogensen: Yeah. So the view that human extinction would be a very bad outcome and perhaps one of the worst tragedies imaginable is definitely a very common view.

Zershaaneh Qureshi: Yep!

Andreas Mogensen: I think amongst people who are inclined to doubt this, generally speaking, those people are most concerned or most driven by concerns about the impact of human activities on nonhuman animals: on the one hand, the many nonhuman animals that we farm for meat and so forth, and then also wild animals whose habitats and whose populations we decimate or something like that.

…

Zershaaneh Qureshi: It seems like it makes sense that if we’re causing a lot of harm to these animals, maybe it would be better if we didn’t exist. But my understanding is that the argument is not quite that simple, and we have to make some further assumptions before we conclude “…and therefore humans should go extinct.” Could you walk us through a little bit about what else is going on here?

Andreas Mogensen: Yeah. So on the one hand, you have to make an assumption that these trends will continue or worsen or something like that. And that might not be at all obvious. I don’t think it is at all obvious. Human beings have previously succeeded in, to a large extent, enforcing a kind of global elimination of certain practices that were once prevalent and morally catastrophic. For example, the near-universal acceptance of a prohibition on enslaving other human beings is something that in some sense happens relatively suddenly and relatively recently in the history of our species. And a similar kind of moral revolution could take hold when it comes to the interests of nonhuman animals. I think this is quite unclear.

But there are also potentially certain philosophical considerations that might be brought to bear on this question, which are about how you sort of add up and weigh harms and benefits to different kinds of individuals. So let’s start by just focusing on wild animals.

…

Viewed certainly through a kind of utilitarian lens, in order for it to be a bad thing that human activities reduce the size of wild animal population, we’d certainly have to assume that, generally speaking, wild animals have good lives — they have lives that are worth living, as we might put it. And I think certainly that is the kind of intuitive view. Many people I think very naturally assume that animals that live in the wild, that live in their natural habitats, that they have a kind of positive existence — that they’re flourishing, that they’re happy, they’re fulfilled, things of that kind.

I think there’s good reason to think that this idyllic conception of what life in the wild is like is a mistake, and that in fact, the lives of many wild animals are plagued to a large extent by starvation, suffering, disease, fleeing from predators in fear, or suffering within the jaws of predators or things of that kind. … And in some sense, if you add up all the suffering and all the happiness that these animals experience, you find that there’s more suffering in total.

Zershaaneh Qureshi: Right. But I’m guessing that the important question here, if you have utilitarian assumptions about maximising welfare, is not just about the question of, “Do these wild animals have lives worth living?” but, “How good are they? How above that threshold are they?” Because it needs to be enough for them to then outweigh, in aggregate, the combined wellbeing of continued human lives. So I guess it has to be above some kind of bar as well, right?

…

Andreas Mogensen: Even if it is positive, as you said, it matters how positive. But it might also matter in somewhat surprising ways. So if you have a straightforward, total utilitarian view where we assign numbers to how well people are doing — they’re positive if they’re doing well, and they’re negative if they’re doing badly — and then we just add up all these individual numbers, and whatever gives the biggest number, that’s the best outcome.

If you have a view like that, then the following is possible: maybe in general, wild animals have lives that are only just above neutral or something like that, or maybe this is true of very many of them. Sufficiently many of these lives that are only just above the neutral point, in total, it could be better to have that many lives than to have a much smaller number of lives that are really good in terms of everything that can make a life go well or something like that.

This is a version of the famous or infamous repugnant conclusion. The repugnant conclusion says: for any population, no matter how big and no matter how well off the people in it are doing, there is a better population which would be bigger, much bigger, even though the people in it have lives that are only barely worth living or something like that.

So you could imagine some utopian civilisation where everybody has these wonderful, flourishing lives: you know, they’re successful as artists and inventors and scientists, and they have these deep, meaningful relationships, et cetera. Compare that with a much bigger population of people who basically go through life more or less feeling nothing, like there’s nothing good or bad really in their lives. They exist in a kind of stupor. Every now and then they hear a slightly pleasant jingle or something like that. A kind of standard total utilitarianism says that that second outcome would be better if you just have enough of these people who live these stupor lives or something like that.

One thing you might try to say in response to this concern that it would be better for human beings if there were no more human beings because of the extent to which human beings decimate wild animal populations: if you think that in fact very many of these wild animals have lives that if above neutral, are only slightly above neutral, and you reject the repugnant conclusion — and you think maybe that there are certain kinds of goods like autonomy or meaningfulness or knowledge of deep truths about the universe that really ennoble human lives and make them really good or at least the very best possible human lives — then if you take this position where you reject the repugnant conclusion, you might think that it would be better to have some number of human beings who achieve these pinnacles. Even if we could instead have a much larger number of wild animals, if those wild animals, if they have lives that are worth living, that are above neutral, might nonetheless be only just above neutral.

So there seems to be some case for thinking that this issue that arises in thinking about whether you should accept or reject the repugnant conclusion might be relevant to thinking about how we should trade off the interests of human beings, or think about the human beings and nonhuman animals, wild animals, and how we should think about the desirability of the continued survival of our species.

So... should we still try to prevent extinction?

Zershaaneh Qureshi: Many of our listeners are longtermists or compelled by some of the ideas of longtermism. We want to influence the long-term future as a moral priority. And if you’re a longtermist, the obvious thing to do is work to try to prevent the extinction of future generations of humans. But some of these arguments might make that seem less obviously a good thing.

What do you make of this? Should longtermists be less sure they want to prevent extinction, or should we embrace the intuitive view that extinction is a really bad thing?

Andreas Mogensen: Ultimately I suspect we should still embrace the view that extinction would be a very bad thing. Certainly I think these considerations should make us maybe less certain of this than we were previously — but there’s a question about how much.

And with respect to something like this question of whether the harm that we do to nonhuman animals outweigh whatever good we might experience within human populations, it’s very difficult to know exactly what the future holds. I mean, it might be the case that we get our act together and we succeed in ending or suitably limiting the worst of these practices.

Zershaaneh Qureshi: Right.

Andreas Mogensen: I think it’s also the case… We’ve been speaking about the possibility of human extinction or something like that, but we haven’t really said very much about how humans might go extinct. I guess maybe the sort of extinction programme that might be justified by this, if any, would be a voluntary human extinction or something like that. If you’re instead worried about human beings going extinct from some kind of disastrous natural catastrophe, like an asteroid strike or something like that, that’s not only going to take out humans, but it’s also going to take out nonhuman animals. It’s going to take out a large chunk of the population of wild animals or something like that.

And similarly, say what you’re worried about is existential risks associated with advanced AI systems: you might think quite reasonably that if such advanced AI systems come to disempower humanity, or perhaps even to bring about human extinction, it’s probably not going to be good news for the biosphere as a whole.

So if you imagine the classic paperclip maximiser story: the paperclip maximiser is given some goal to maximise the output of paperclips. It’s a superintelligent system, it’s been given a very bad goal, and it converts the known universe into paperclip factories. That brings about human extinction, but it also completely destroys all life on Earth or anywhere else in the immediate vicinity or something like that.

So if you’re worried about the impact of human civilisation on wild animals, there are many extinction scenarios that longtermists and others might worry about that would be also pretty terrible for these wild animals.

Zershaaneh Qureshi: Yeah, makes sense. It’s interesting to think about AI here, because I think we’ve been having most of our conversation imagining life gets wiped out or not wiped out or something like that. But in fact what we might be dealing with here is the prospect of humans getting replaced by some new dominant species.

And I sort of see the argument that in the animal case, if AIs wipe out humans, there will be some kind of disruption to nature that comes with that that’s going to be bad for the animals too. But is there any reason to think that longer term, life for animals on Earth might be better if AIs are the dominant species? This is totally speculative.

Andreas Mogensen: Yeah, it’s obviously very hard to know. Certainly you might think an AI system has no need to eat, and probably no desire for animal flesh or something like that. So the intensive farming of animals would disappear.

The question of whether, if human beings were replaced by a civilisation of AI systems, would that be worse for wild nonhuman animals? I’m not totally sure about this. If we’re talking about radically misaligned AI systems that have strange values that are entirely orthogonal to human values, you might very well suspect that they would be also indifferent to a large extent to how well or badly things go with these wild animals or something like that. But it’s a strange and very difficult question, and I guess I’m not totally sure.

Zershaaneh Qureshi: Yeah, that’s completely fair. It was a wild card.