I believe that most people in that situation would feel compelled to grant the robots basic rights.
As noted in some of the other comments, I think this is quite debatable. I personally would lean strongly towards dissecting the robot, conditional on the fact that I have the assurance that there is no phenomenal experience going on inside the mother robot. After all, whatever I do to the baby robot, the mental state of the mother robot will not change - because it doesn't exist!
Of course, the question of how I would gain assurance that the mother robot or child robot has no subjective experience is a different issue. If the mother robot pleaded like a human mother would, I would need an extremely strong assurance that these robots have no subjective experience and would probably lean strongly against dissecting, just in case. But (correct me if I'm wrong), this problem is assumed away in the thought experiment.
I stick by my intuition, but it is really just an intuition about how human behavior. Perhaps some people would be completely unbothered in that situation. Perhaps most would. (I actually find that itself worrisome in a different way, because it suggests that people may easily overlook AI wellbeing. Perhaps you have the right reasons for happily ignoring their anguished cries, but not everyone will.) This is an empirical question, really, and I don’t think we’ll know how people will react until it happens.
Here's a thought experiment similar to Kagan's that I expect could get different responses from people:
There are humans for which pain is not unpleasant (pain asymbolia). However, suppose one such child has been taught to act as if their pain is unpleasant. They will cry out, plead to make it stop, do things to avoid pain, talk about pain as if it's unpleasant, and so on, with consistently appropriate responses. To anyone who doesn't know they have pain asymbolia, the child seems to experience pain like any other.
That child and a typical (non-asymbolic) child are injured and both need surgery. The surgery for the typical child would be very painful to them. The surgery the asymbolic child needs would be even more painful than the surgery the non-asymbolic child needs, if performed on a typical non-asymbolic child. You only have enough painkillers to totally eliminate the pain in one of the surgeries (if performed on a non-asymbolic child, who finds pain unpleasant in the typical ways), or even less than that. The painkillers also have diminishing marginal returns to greater dosages.
How do you split the painkillers between the two children?
The right answer here seems to be to give all of the painkillers to the typical/non-asymbolic child, and none to the child with pain asymbolia, no matter how much more painful the surgery the asymbolic child needs would typically be for non-asymbolic people, if we're ~100% sure the asymbolic child is fully (or almost fully) asymbolic. We know from their history that they didn't respond to pain in the right way, and only came to do so due to being taught to act like it. Maybe they're also missing some physiological responses to unpleasant pain, e.g. heart rate, tears, stress hormones, neural responses, but maybe they've found a way to fake some of those.[1]
I think this would get different responses from people partly because I've made it much more concrete why the child does not find pain unpleasant in a way that's easy to entertain and believe. "our best science reveals to us—correctly—that they are not sentient" is pretty vague. Why should I believe that our best science is correct in general or about this case in particular?
This case is interesting, but I think it touches on a slightly different issue. The symbolic presumably doesn’t care about their pretend pain. There is a more complicated story about their actions that involves their commitment to the ruse. In the robot case, I assume we’re supposed to imagine that the robots care about each other to whatever extent that unconscious things can. Their motivational structure is close to ours.
I think the case is less clear if we build up the extent to which the asymbolic child really wants the painkillers. If they constantly worry about not getting them, if they are willing to sacrifice lots of other things they care about to secure them (even though they know that it won’t help them avoid pain), etc. I’m less inclined to think the case is clear cut.
In the robot case, I assume we’re supposed to imagine that the robots care about each other to whatever extent that unconscious things can.
I think so, but without more detail about what exactly they're missing, my intuitive reaction is that they are conscious or reasonably likely to be conscious. It's hard to trust or entertain the hypothetical. How could they not be conscious?
If you fill in the details in specific ways, then you might get different responses. If the robots are like today's LLMs or a giant lookup table, then I'm inclined to say they aren't really conscious to any significant degree: they've been designed (or assumed into existence) by Goodharting the behavioural outputs of conscious beings.
There's another question about whether I'd actually dissect one, and maybe I still wouldn't, but this could be for indirect or emotional reasons. It could still be very unpleasant or even traumatic for me to dissect something that cries out and against the desperate pleas of its mother. Or, it could be bad to become less sensitive to such responses, when such responses often are good indicators of risk of morally significant harm. People who were confident nonhuman animals don't matter in themselves sometimes condemned animal cruelty for similar reasons.
Or, maybe the robots' consciousness is very probably minimal, but still enough to warrant some care. This could be in line with how many people treat insects or spiders: they wouldn't give up much to help them, but they might still take them outside when found inside or otherwise avoid killing them when the costs are very low.
If they constantly worry about not getting them, if they are willing to sacrifice lots of other things they care about to secure them (even though they know that it won’t help them avoid pain), etc. I’m less inclined to think the case is clear cut.
This could all follow from a great commitment to pretending to be capable of unpleasant pain like a typical person.
I guess if they're subjectively worse off the less convincing they think they are to others, they could be worse off finding out they won't get painkillers, if and because they find out they failed to convince you.
You could just lie and say there aren't any painkillers available, but then this gets into the issue of whether they care about actually being convincing, or just believing they're convincing (contact with reality, experience machine, etc.), and which of the two you care about on their behalf.
It is rare for theories of consciousness to make any demands on motivational structure.
Global workspace theory, for instance, says that consciousness depends on having a central repository by which different cognitive modules talk to each other. If the modules were to directly communicate point to point, there would be no conscious experiences (by that theory). I see no reason in that case why decision making would have to rely on different mechanisms.
Higher order theories suggest that consciousness depends on having representations of our own mental states. A creature could have all sorts of direct concerns that it never reflected on, and these could look a lot like ours.
IIT suggests that you could have a high level duplicate of a conscious system that was unconscious due to the fine grained details.
Etc.
The specific things you need to change in the robots to render them not conscious depends on your theory, but I don’t think you need to go quite so far as to make them a lookup table or an transformer.
My impression was that you like theories that stress the mechanisms behind our judgments of the weirdness of consciousness as critical to conscious experiences. I could imagine a robot just like us but totally non-introspective, lacking phenomenal concepts, etc. Would you think such a thing was conscious? Could it not desire things in something like the way we do?
There's another question about whether I'd actually dissect one, and maybe I still wouldn't, but this could be for indirect or emotional reasons. It could still be very unpleasant or even traumatic for me to dissect something that cries out and against the desperate pleas of its mother. Or, it could be bad to become less sensitive to such responses, when such responses often are good indicators of risk of morally significant harm. People who were confident nonhuman animals don't matter in themselves sometimes condemned animal cruelty for similar reasons.
This supports my main argument. If you value conscious experience these emotional reasons could be concerning for the long term future. It seems like a slippery slope from being nice to them because we find it more pleasant to thinking that they are moral patients, particularly if we frequently interact with them. It is possible that our generation will never stop caring about consciousness, but if we’re not careful, our children might.
FWIW, I meant "How could they not be conscious?" kind of rhetorically, but I appreciate your response. Making it more concrete like this is helpful. My comment here is pretty object-level about the specific views in question, so feel free not to respond to it or any specific points here.
Global workspace theory (...)
There probably still needs to be "workspaces", e.g. working memory (+ voluntary attention?), or else the robots couldn't do many sophisticated things flexibly, and whatever those workspaces are could be global workspaces. Maybe each module has its own workspace, so is "global" to itself, and that's enough. Or, if the workspaces are considered together as one combined system, then it could be a more conventional "global workspace", just distributed. The differences don't seem significant at this level of abstraction. Maybe they are, but I'd want to know why. So, my direct intuitive reaction to "GWT is true and the robots aren't conscious" could be unreliable, because it's hard to entertain.
Higher order theories suggest that consciousness depends on having representations of our own mental states. A creature could have all sorts of direct concerns that it never reflected on, and these could look a lot like ours.
I think this one is more plausible and easier to entertain, although still weird.
I think it means that if you asked the mother robot if she cares about her child, she wouldn't say 'yes' (she might say 'no' or be confused). It seems the robots would all have complete alexithymia, and not just for emotions, but for all mental states, or at least all (the components of) mental states that could matter, e.g. valence, desires, preferences. But they'd still be intelligent and articulate. The mother would have no concept of desire, preference, caring, etc., or she'd be systematically unable to apply such concepts to herself, even though she might apply it to her child, e.g. she distinguishes her child from a "mere thing", and I imagine she recognizes that her child cares about things.
Or, maybe it could depend on the particulars of what's required of a higher order representation according to theory. The mother robot might have and apply a concept of desire, preference, caring, etc. to herself, but it's not the right kind of higher order representation.
IIT suggests that you could have a high level duplicate of a conscious system that was unconscious due to the fine grained details.
IIT is pretty panpsychist in practice, just needing recurrence, IIRC. I don't think you would have a complex society of intelligent robots without recurrence (networks of purely feedforward interactions would end up far too large, but the recurrence might be extended beyond their brains). And at any rate, IIT seems way off track to me as a theory. So, my direct intuitive reaction to "IIT is true and the robots aren't conscious" will probably be unreliable.
My impression was that you like theories that stress the mechanisms behind our judgments of the weirdness of consciousness as critical to conscious experiences. I could imagine a robot just like us but totally non-introspective, lacking phenomenal concepts, etc. Would you think such a thing was conscious? Could it not desire things in something like the way we do?
There are a few "lines" that seem potentially morally significant to me as an illusionist:
As you mention, having and applying phenomenal concepts, or having illusions of phenomenal consciousness, e.g. finding aspects of our perceptions/information processing weird/mysterious/curious/ineffable (or unphysical, private and/or intrinsic, etc., although that's getting more specific, and there's probably more disagreement on this). I agree the robots could fail to matter in this way.
Having states that would lead to illusions of phenomenal consciousness or the application of phenomenal concepts to them, finding them weird/mysterious/curious, etc., if those states were introspected on by a sufficiently sophisticated system in the right way (even if the existing system is incapable of introspection; we consider a hypothetical attaching another system to do it). This is Frankish's and I suspect Dennett's normative interpretation of illusionism, and their views of consciousness are highly graded. Maybe just cognitive impenetrability suffices, if/because the cognitive impenetrability of the things we introspect is what makes them seem weird/mysterious/curious/ineffable to us.[1] I'd guess the robots would matter in this way.
The appearances of something mattering, in causal/functional terms — including desires, pleasure, unpleasantness, preferences, moral intuitions, normative beliefs, etc. — just are phenomenal illusions or (the application of) phenomenal concepts, or parts of phenomenal illusions or phenomenal concepts that matter even on their own. It's not just that consciousness seems weird (etc.), but that part of our phenomenal concepts for (morally relevant) conscious mental states is just that they seem to matter. And, in fact, it's the appearance of mattering that makes the mental states matter morally, not the apparent weirdness (etc.). We wouldn't care (much) about a person's specific experience of red unless they cared about it, too. An experience only matters morally in itself if it seems to matter to the individual, e.g. the individual takes a specific interest in it, or finds it pleasant, unpleasant, attractive, aversive, significant, etc.. Furthermore, it's not important that that "seeming to matter" applies to mental states in a higher-order way rather than "directly" to the intentional objects of mental states, like in the robots' desires; that's an arbitrary line.[2] The robots seem to matter in this way.
1 implies 2, and I suspect 3 implies 2, as well.
I also suspect we can't answer which of 1, 2 or 3 is (objectively, stance-independently) correct. It seems inherently normative and subjective (and I'm not a moral realist), although I've become pretty sympathetic to 3, basically for the reasons I give in 3. We could also go for a graded account of moral status, where each of 1, 2 and 3 ground different degrees of moral status.
Humphrey, another illusionist, said "Consciousness matters because it is its function to matter". However, he's skeptical animals other than mammals and birds are conscious. He thinks consciousness requires finding your own mental states/perceptions/sensations to matter, e.g. engaging in sensation-seeking or sensory play. Such animals find their perceptions themselves interesting, not just the intentional objects of those perceptions. So it's higher order-ish.
I think that would get different responses, but I don't think it is quite as good a thought experiment because it probably doesn't approximate the likely future as well as the other thought experiment.
Future AI is likely to look like nothing at all (in a computer), or look like a non-human entity, perhaps a robot. I agree that an entity looking and acting basically exactly the same as a human would illicit very different emotional and practical responses from us, but it seems very unlikely that is how we encounter early-stage future "non-conscious" AGI or however we want to call it.
My thought experiment was aimed at showing that direct intuitive responses to such thought experiments are irrationally sensitive to framing and how concrete the explanations are.
The asymbolic child is almost identical to a typical child and acts the same way, so you would think people would be less hesitant to dismiss their apparent pain than a robot's. But I would guess people dismiss the asymbolic child's pain more easily.
My explanation for why the asymbolic child's pain doesn't matter (much) actually shouldn't make you more sure of the fact than the explanation given in the robot case. I've explained how and why the child is asymbolic, but in the robot case, we've just said "our best science reveals to us—correctly—that they are not sentient". "correctly" means 100% certainty that they aren't sentient. Making the explanation more concrete makes it more believable, easier to entertain and easier for intuitions to reflect appropriately. But it doesn't make it more probable!
However, on reflection, these probably push the other way and undermine my claim of irrational intuitive responses:
My opportunity cost framing, e.g. thinking it's better to give the painkillers to the typical child doesn't mean you would normally want to perform surgery on the asymbolic child without painkillers, if they're cheap and not very supply-limited and the asymbolic child would protest less (pretend to be in pain less) if given painkillers.
People aren't sure moral patienthood requires sentience, a still vague concept that may evolve into something they don't take to be necessary, but they're pretty sure that the pain responses in the asymbolic child don't indicate something that matters much, whatever the correct account of moral patienthood and value. It can be easier to identify and be confident in specific negative cases than put trust in a rule separating negative and positive cases.
Maybe this is my dualist intuitions speaking, but the suppositions here seem to be in some tension with each other. If there's nothing "all that important" about the identified pattern, whyever would we have identified it as the correct theory of consciousness to begin with? (The idea that "consciousness is just one specific algorithm among many" seems very odd to me. Surely one of the most central platitudes for fixing the concept is that it picks out something that is distinctive, or special in some way.)
If things can matter to us even though they don’t affect how we feel, we may be inclined to think that similar things can matter to systems that feel nothing at all.
One reason to reject this inference is if we accept the phenomenal intentionality thesis that consciousness is necessary for having genuinely representational states (including desires and preferences). I agree that consciousness need not be what's represented as our goal-state; but it may still be a necessary background condition for us to have real goals at all (in contrast to the pseudo-intentionality of mere thermostats and the like).
If there's nothing "all that important" about the identified pattern, whyever would we have identified it as the correct theory of consciousness to begin with?
This particular argument really speaks to the more radical physicalists. I don’t think you should be that moved by it. If I were in your shoes (rather than undecided), I think I’d be more worried that people would come to jettison their concern for consciousness for bad reasons.
One reason to reject this inference is if we accept the phenomenal intentionality thesis that consciousness is necessary for having genuinely representational states (including desires and preferences). I agree that consciousness need not be what's represented as our goal-state; but it may still be a necessary background condition for us to have real goals at all (in contrast to the pseudo-intentionality of mere thermostats and the like).
One case I had in mind while writing this was the matter of unconscious desires in a conscious person. Suppose that we have some desires that shape our motivations but which we never think about. Maybe we have a desire to be near the ocean. We don’t feel any longing, we just find ourselves quickly accepting invitations to the beach. (We also aren’t glad to receive such invitations or any happier when at the beach.) Satisfying that desire seems to me not to count for much in a large part because it has no impact on our conscious states. Would you agree? If so, would you think the intentionality thesis can make sense of this difference? Do you want to withhold intentionality from purely unconscious states in a conscious mind? Or is there a different story you would tell?
That's an interesting case! I am tempted to deny that this (putative unconscious desire to be near the ocean) is really a mental state at all. I get that it can be explanatorily convenient to model it as such, using folk (belief-desire) psychology, but the same is true of computer chess programs. I'd want to draw a pretty sharp distinction between the usefulness of psychological modelling, on the one hand, and grounds for attributing real mental states, on the other. And I think it's pretty natural (at least from a perspective like mine) to take consciousness to be the mark of the mental, such that any unconscious state is best understood as mere information-processing, not meaningful mentality.
That's an initial thought, anyway. It may be completely wrong-headed!
I love the accessible way you wrote this article thanks and I loved that thought experiment. I'm going to test it on more people tomorrow.... I'm interested in this statement
"Whatever you feel about this thought experiment, I believe that most people in that situation would feel compelled to grant the robots basic rights." I would like to actually poll this because I'm not entirely sure its true, I'd be 50/50 on what the general public would think. My instinct was that I'm fine dissecting the robot, and I told the story to my wife and she was all good with it too.
There's an episode in the series "The Good Life" where a similar-ish thing happens and they "kill" a "robot" called Janet several times. The people I watched that with weren't that perturbed at the time.
Good times with anecdata. I could well just be hanging out with edgeish of the bell-curve kind of people as well.
IMO Sentience be Sentience. I'm more compelled by the argument that there's a high chance we won't be able to figure out if its there or not.
I think there is a difference between what people would say about the case and what they would do if actually in it. The question of what people would say is interesting -- I'm curious how your polling goes. But it is easier to support an intellectual stance when you're not confronted by the ramifications of your choice. (Of course, I can also see it going the other way, if we think the ramifications of your choice would harm you instead of the robot.)
That may be true that the real life reaction may be different from what is sad, but I can't test it and like you said it could swing both ways. I think its good as much as possible to test what can be tested and lean towards evidence over theory, even when the evidence isn't great. Maybe this is a bit of a difference between the inclinations of us GHD folks and AI safety folks too.
I agree! I’m used to armchair reflection, but this is really an empirical question. So much of the other discussion this week has focused on sentience. It would be good to get a sense if this wasn’t the crux for the public.
Here is a different thought experiment. Say that I was told that to find the cure to a disease that would kill 1000s of robot children, I had to either dissect the supposedly non-sentient robot or dissect a different, definitely sentient robot. Which do my intuitions point to here?
Executive summary: The moral value we attribute to phenomenal consciousness is a pivotal question that could significantly shape the long-term future, especially if digital minds become prevalent.
Key points:
The value of consciousness is a controversial topic in philosophy, with divided opinions on its necessity for moral value.
Reasons to doubt consciousness is necessary for value include skepticism about its fundamental importance and the fact that we care about things beyond our direct experiences.
Future populations may consist primarily of digital minds, making the question of their moral status crucial for longtermist considerations.
If consciousness isn't the default for digital minds, we risk either neglecting to instill it (potentially losing out on valuable lives) or overemphasizing its importance (potentially overlooking the welfare of unconscious digital beings).
The author is uncertain about the correct stance but emphasizes the importance of making this decision deliberately rather than leaving it to uninformed public opinion.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
We sometimes try to bring about results that we will never know about, such as, for instance, setting our children up to live happy and healthy lives after we are gone. It may matter greatly to us that these results occur and we may make sacrifices here and now to give them a chance.
I urge you to choose a different example, because this one is linked to phenomenal consciousness:
the parents feel good whenever they achieve something that improves the chances of their children
I think we should get rid of the confused term "phenomenal consciousness" and adopt a hard materialist/functionalist/behavioralist perspective, which is nowadays known as illusionism. Instead of consciousness, we should look for systems for pain and pleasure - that's what we really care about! This text by Brian Tomasik explains it well:
It suggests that "we stop thinking in terms of "conscious" and "unconscious" and instead look at physical systems for what they are and what they can do. This perspective dissolves some biases in our usual perspective and shows us that the world is not composed of conscious minds moving through unconscious matter, but rather, the world is a unified whole, with some sub-processes being more fancy and self-reflective than others."
Simply put, "consciousness is as consciousness does".
Longtermists point out that the scale of our potential for impact is far greater if we are able to influence the course of a long future, as we could change the circumstances of a tremendous number of lives.
One potential avenue for long-term influence involves spreading values that persist and shape the futures that our descendants choose to build. There is some reason to expect that future moral values will be stable. Many groups have preferences about the world beyond their backyard. They should work to ensure that their values are shared by those who can help bring them about. Changes in the values that future groups support will lead to changes in the protections for the things we care about. If our values concern how our descendants will act, then we should aim to create institutions that promote those values. If we are successful in promoting those values, we should expect our descendants to appreciate and protect those institutional choices.
What values should we work to shape so that the future is as good as it might be? Many of humanity’s values would be difficult to sway. Some moral questions, however, might be open to change in the coming decades. It is plausible that there are some questions that we haven’t previously faced and for which we have no vested interest. We may be pressed to establish policies and precedents or commit to indifference through inaction. The right policies and precedents could conceivably allow our values to persist indefinitely. These issues are important to get right, even if we’re not yet sure what to think about them.
Controversy
Foremost among important soon-to-be-broached moral questions, I propose, is the moral value that we attribute to phenomenal consciousness (having a ‘what-its-like’ and a subjective perspective). Or, more particularly, whether mental lives can matter in the absence of phenomenal consciousness in anything like the way they do when supplemented with conscious experiences. What we decide about the value of phenomenal consciousness in the coming few centuries may not make a difference to our survival as a species, but it seems likely to have a huge effect on how the future plays out.
To get a grip on the problem, consider the case of an artificial creature that is otherwise like a normal person but who lacks phenomenally conscious experiences. Would it be wrong to cause them harm?
Kagan (2019, 28) offers a thought experiment along these lines:
Imagine that in the distant future, we discover on another planet a civilization composed entirely of machines—robots, if you will—that have evolved naturally over the ages. Although they are made entirely out of metal, they reproduce… and so have families. They are also members of larger social groups–circles of friends, communities, and nations. They have culture (literature, art, music) and they have industry as well as politics. Interestingly enough, however, our best science reveals to us—correctly—that they are not sentient. Although they clearly display agency at a comparable level of our own, they lack qualitative experience: there is nothing that it feels like ('on the inside') to be one of these machines. But for all that, they have goals and preferences, they have complex and sophisticated aims, they make plans and they act on them.
Imagine that you are an Earth scientist, eager to learn more about the makeup of these robots. So you capture a small one—very much against its protests—and you are about to cut it open to examine its insides, when another robot, its mother, comes racing up to you, desperately pleading with you to leave it alone. She begs you not to kill it, mixing angry assertions that you have no right to treat her child as though it were a mere thing, with emotional pleas to let it go before you harm it any further. Would it be wrong to dissect the child?
Whatever you feel about this thought experiment, I believe that most people in that situation would feel compelled to grant the robots basic rights.
The significance of consciousness has become a recentpopulartopicinacademicphilosophy, particularly in the philosophy of AI, and opinions among professionals are divided. It is striking how greatly opinions differ: where some hold that phenomenal consciousness plays little role in explaining why our lives have value, others hold that phenomenal consciousness is absolutely necessary for having any intrinsic value whatsoever.
One reason to doubt that phenomenal consciousness is necessary for value stems from skepticism that proposed analyses of consciousness describe structures of fundamental importance. Suppose that the global workspace theory of consciousness is true – to be conscious is to have a certain information architecture involving a central public repository — why should that structure be so important as to ground value? What about other information architectures that function in modestly different ways? The pattern doesn’t seem all that important when considered by itself. If we set aside our preconceptions of consciousness, we wouldn’t recognize that architecture as having any special significance.
The consciousness-is-special doctrine makes the most sense under dualism, where consciousness is something genuinely different. It is harder to defend a fundamental gap between the moral value of conscious and unconscious minds if we think consciousness is just one specific algorithm among many.
Another reason to doubt that phenomenal consciousness is necessary for value stems from the fact that many of the things that we care about aren’t directly tied to our phenomenal experiences. Few people are willing to limit value entirely to phenomenal experiences. We sometimes try to bring about results that we will never know about, such as, for instance, setting our children up to live happy and healthy lives after we are gone. It may matter greatly to us that these results occur and we may make sacrifices here and now to give them a chance. If this is rational, then these things matter to how well our lives go though they make no difference to how we feel. Desire-satisfactionists raise this observation up to a whole theory of welfare, but we can think that such things are important to the value of our lives even if we are pluralists about value. If things can matter to us even though they don’t affect how we feel, we may be inclined to think that similar things can matter to systems that feel nothing at all.
Phenomenal consciousness still intuitively plays a special role that is difficult to explain. One clue is that the mere presence of some phenomenal experience doesn’t seem to matter. If a creature complemented a rich non-phenomenal life of friendship and virtue and learning accompanied solely by a constant bare experience of phenomenal blueness, that wouldn’t be enough to set it apart from a phenomenal zombie.
When I introspect my own intuitions[1], it seems to me that the best way to explain the essential value of consciousness relates to the fact that it ensures that other important components of value (pleasantness, desires-to-be-satisfied) are witnessed. Pleasures and pains may necessarily be conscious in order to matter, but they aren’t all that matters. Intuitively, a subconscious desire that goes unfulfilled and unfelt doesn’t matter. An unobserved achievement doesn’t matter. These things don’t matter because there is no subject to substantiate their existence in the right way. I think that it is an open question whether it is possible to justify this intuition. How do we make sense of the importance of phenomenal witnessing? Is it plausible that we can give it any non-arbitrary justification under non-dualist theories of consciousness?
Many people, experts and non-experts alike, seem inclined to think that consciousness is important for value and that we should not be particularly concerned about digital minds if they are incapable of consciousness. So while there may be some story to tell about why this matters, I doubt that we can establish it with a compelling logical argument from more basic premises. More likely, I think, it will come down to a gut feeling. It is an intuition that we might build into the foundations of our moral theories or cast aside as an unreliable relic of our evolutionary history. This means that we can’t rely on our epistemically advanced descendants to make the right choice[2]. If we commit to the wrong path now, there probably won’t be any knock-down logical argument that can fix things later.
People have not yet been confronted with charismatic and unquestionably non-conscious beings with human-level intelligence. Widespread complex LLM interfaces have so far avoided any kind of personalization. Chatbot-as-a-friend services exist, but are still primitive. However, if substantial money and talent is directed at them, digital companions may take off. It is also plausible that agentic AIs will be given traits to make them more personable and enjoyable to interact with. Given their potential value, it should not surprise us to find them increasingly integrated into our lives over the course of a few years. We may find ourselves interacting regularly with AI interfaces with their own goals and personalities.
There are incentives for AI services to deny that their systems are conscious and to make their systems disavow their own phenomenal experiences. It is likewise in their interest to find ways to ensure that the systems aren’t actually conscious, to whatever extent that is feasible. They also have incentives for us to engage emotionally with their systems, but not feel as if we’re doing them harm. It is hard to predict where this will end up. While this may help them avoid public pressure or legal interference in the short run, it is also possible that it will influence the public’s feelings about the significance of consciousness and other aspects of welfare.
If the willingness of experts to question the importance of consciousness is suggestive of some deep conceptual flexibility, then it is conceivable that the general public may come to think of consciousness as not a requirement for having welfare. If we do bring personable digital minds into existence that are most likely not conscious, our feelings about consciousness may change. How we think about the moral status of such creations may also influence whether we choose to build them, which traits to give them, and what we let them say about themselves.
Scale
The reason why the significance of consciousness is of such potential importance is that future populations may consist primarily of digital minds. (A significant minority of EAs accept that the majority of expected future value will come from digital minds.) We may choose to live amongst digital minds for a variety of reasons: we may want them as friends, as workers, as the subjects of simulations. Perhaps we may even create them out of benevolence or malevolence. Human beings, with our biological bodies, have complex needs that will surely limit the population size that our planet can support. We may be able to sustain much larger numbers of digital minds.
The most optimistic projections for population growth in the far future see populations consisting largely of digital minds. Space travel is the primary avenue to indefinite population expansion, but the transport of humans promises to be slow and technologically challenging. Even if it is possible, populating alien worlds with biological persons may take a tremendous span of time. Computers are comparatively easy to build rapidly and to sustain in unfriendly environments. Colonization seems to be eased by relaxing the requirements of keeping metabolizing bodies alive in alien environments. Even if we never make it to distant planets, digital minds may have the potential for vast populations on Earth and within our solar system.
There is reason for longtermists to care about the prospects of very large future populations even if they are unlikely to actually result. The numbers that we might conceivably affect if the population continues to grow quickly are vastly higher than if the population grows slowly. If we want to maximize expected value, we may be best off setting aside all cases where population doesn’t rapidly expand and focus our efforts on making sure that such an expansion goes well if it occurs at all.
Digital minds may be likely to be conscious by default. It is possible that consciousness is implied by the most sensible architectures with which to design agentic minds. In this case, assuming most digital minds are conscious, it doesn’t matter what we think about the possibility of value and disvalue in non-conscious minds because we won’t have to make choices that affect such minds[3].
However, it also seems quite likely that consciousness isn’t the default: that consciousness serves some special function in our evolved brains that flexible silicon designs render superfluous. If that is the case, then we face two scenarios that should trouble us. In one scenario, we neglect to instill the digital minds we create with consciousness, thinking that it isn’t important for them to lead valuable lives, when in fact it is. If we rob them of lives of value–even if they behave in a happy and satisfied way–we have failed to achieve their potential. We may have effectively lost out on the value of a large part of the population. If we think that it is good for there to be more good lives, and the majority of potential good lives belong to digital minds, then this seems important to get right.
On the other hand, if our successors decide that consciousness is morally critical, they may not invest in safeguarding the wellbeing of unconscious digital minds or they may avoid creating many minds that would have value. If many digital beings would have poor welfare and if that welfare is overlooked, the consequences could be catastrophic.
I’m not personally sure which way to go on the question. Rethink Priorities’ Worldview Investigations Team is divided. But the potential significance of this decision on the shape of the future suggests it is something that we may not want to leave to the whims of uninformed public opinion to decide. It has such a large potential for impact on the future that we should make the decision deliberatively.
Acknowledgments
The post was written by Derek Shiller. Thanks to Bob Fischer, Hayley Clatterbuck, Arvo Muñoz Morán, David Moss, and Toby Tremlett for helpful feedback. The post is a project of Rethink Priorities, a global priority think-and-do tank, aiming to do good at scale. We research and implement pressing opportunities to make the world better. We act upon these opportunities by developing and implementing strategies, projects, and solutions to key issues. We do this work in close partnership with foundations and impact-focused non-profits or other entities. If you're interested in Rethink Priorities' work, please consider subscribing to our newsletter. You can explore our completed public work here.
Depending on how we think about risk, it may make more sense to take a precautionary attitude and focus on the worst-case scenario. If the potential scale of harm and benefit possible to non-conscious minds vastly outweighs the potential scale to conscious minds, then we should perhaps assume that non-conscious minds matter even if we’re pretty sure they don’t.
However, if we turn out to be mistakenly pessimistic about the distribution of consciousness, we may be better off valuing non-conscious states insofar as that will incidentally provide protections for unrecognized conscious states. Thanks to Toby Tremlett for raising this possibility.
I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences.
Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts:
* Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do
* NEST is an opinionated ideas-fi...
Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job.
My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI.
...
TLDR; To help the effective animal advocacy movement cost-effectively absorb greater amounts of funding in the near future, we are seeking expressions of interest from people who could found a new organization focused on:
* Highly neglected animals: insects, wild animals, shrimp, fish, etc, or
* AI and animals: AI alignment and governance for animal welfare, strategic actions considering transformative AI, AI for wild animals, etc.
* ...
As noted in some of the other comments, I think this is quite debatable. I personally would lean strongly towards dissecting the robot, conditional on the fact that I have the assurance that there is no phenomenal experience going on inside the mother robot. After all, whatever I do to the baby robot, the mental state of the mother robot will not change - because it doesn't exist!
Of course, the question of how I would gain assurance that the mother robot or child robot has no subjective experience is a different issue. If the mother robot pleaded like a human mother would, I would need an extremely strong assurance that these robots have no subjective experience and would probably lean strongly against dissecting, just in case. But (correct me if I'm wrong), this problem is assumed away in the thought experiment.
I stick by my intuition, but it is really just an intuition about how human behavior. Perhaps some people would be completely unbothered in that situation. Perhaps most would. (I actually find that itself worrisome in a different way, because it suggests that people may easily overlook AI wellbeing. Perhaps you have the right reasons for happily ignoring their anguished cries, but not everyone will.) This is an empirical question, really, and I don’t think we’ll know how people will react until it happens.