In a previous post, I argued for the practical value of offering AIs legal freedoms, maintaining that this arrangement could foster a safer dynamic between humans and agentic AIs, while allowing us to unlock the potential for mutual benefit between both groups. In this post, I want to go further and elaborate on the broader moral value of granting AIs legal autonomy, beyond its practical benefit to humans.

In AI risk discussions, it is often taken for granted that our primary goal should be to prevent human disempowerment. This assumption is often treated as a foundational principle, with astronomical stakes. Essentially, this view sets up a binary choice: on one side, there is the possibility of a future where "human values" endure and shape the trajectory of the universe over the long term. On the other side, there is the grim scenario in which we fail, leading to a future dominated entirely by "AI values" instead of human ones. The implicit belief here is that avoiding human disempowerment is synonymous with securing a good or moral outcome for the future.

Here, I want to critically examine and challenge this core assumption. By doing so, I aim to introduce a broader and more inclusive perspective—one that recognizes the potential for AIs as legitimate moral agents in their own right, who we can entrust to shape the future, even if they are not subordinate to humans or focused on serving human interests. This alternative perspective may help us reevaluate what a desirable future looks like, beyond the traditional framing of preventing human disempowerment as the top priority.

The crux of my argument rests on making a key distinction between two very different forms of human disempowerment: peaceful human disempowerment and violent human disempowerment. This distinction is crucial to understanding my position and the reasoning behind it.

By "peaceful human disempowerment", I mean a scenario in which humanity’s dominance over the world declines, but this shift happens without any major violence, catastrophe, or outright destruction of humanity. In this scenario, the human species would lose much of their current relative influence over the world—such as their relative share of wealth, social power, prestige, and decision-making authority—but this transition would occur in a way that allows for a cooperative and non-violent coexistence between humans and advanced AIs.

Importantly, peaceful disempowerment does not imply that AIs would later turn hostile, attempt to annihilate humanity, or that they would completely subvert our legal and political processes so that humans lose all their rights. I would not consider it peaceful if AIs heated the atmosphere to a biologically intolerable level, killing all humans as a consequence. Nor would it be peaceful if AIs stole all of humanity's wealth, leaving humans dramatically impoverished. Instead, in the peaceful scenarios I am talking about here, the world would simply evolve into a new equilibrium, one where humans and AIs share power in a collaborative and mutually beneficial way, rather than humanity continuing to monopolize control over the planet.

For the purpose of illustrating what such a peaceful transition might vaguely look like, I'll borrow from a recent analogy given by Beren,

The defining feature of almost all societies prior to the Industrial Revolution has been dominance by a small landowning class — an aristocracy — who own almost all the ‘capital’ in the society, who also control its politics and exist primarily as rentiers. What such a scenario proposes is that humanity effectively becomes an aristocracy upon a highly productive AI population.

However, historically, despite a landowning aristocracy being the dominant form of human society throughout history until then, over merely 100-200 years of the Industrial Revolution, aristocracy faded from being the dominant force in society to being at best a minor player and at worst completely irrelevant.

Historically, the landowning aristocracy experienced a decline in their relative influence within society upon the advent of industrialization, but this does not mean that they were negatively impacted on an individual level as a result. In fact, many aristocratic landowners successfully adapted to the changing economic landscape by aligning themselves with the emerging industrial class. By investing in or participating in industrial ventures, they were able to profit significantly from the new opportunities created by industrialization. 

Even for those aristocrats who chose not to adopt this strategy, the outcomes were not necessarily negative when considered from an individual perspective. While the aristocratic class as a whole may have seen its societal dominance wane, and violence occasionally erupted against members of the aristocracy, many individual aristocrats still prospered. This is because the overwhelming majority of wealth generated during the industrial revolution did not come at their expense. Instead, it added new sources of economic growth and opportunity, allowing many members of the aristocracy to maintain or even improve their personal standard of living despite the broader decline of their class.

Similarly, humanity as a whole may face a comparable turning point in its relationship with powerful AIs in the foreseeable future. As AIs grow increasingly advanced and influential, humanity's relative influence in the world could diminish significantly, eventually to a very low level. However, this does not necessarily mean that individual humans will be worse off as a result. 

Just as the aristocrats adapted to a changing world, humans could also adjust to this new reality shaped by AIs. This adaptation might take the form of integrating with AIs—for example, by uploading our minds onto computers, and upgrading our cognition. Alternatively, adaptation could be more modest, and humans could benefit by simply taking advantage of the immense wealth and technological advancements generated by the AI revolution. In either case, individual humans may still thrive in a world increasingly dominated by artificial intelligence.

Beyond the potential benefits that would flow to individual humans, the core moral argument for allowing a peaceful transition to an AI-dominated world rests on a simple yet powerful ethical principle: we should not discriminate against cognitively sophisticated entities purely on the basis of substrate. In other words, what matters when deciding whether an entity deserves legal freedoms and a share in shaping the future should not depend on whether it is biological, but rather on whether it possesses the kind of mind that we believe should be entitled to freedom.

To illustrate this principle: if we are comfortable allowing a human child to grow up, take on the legal freedoms associated with adulthood, and then use those freedoms to become powerful, then what moral reason do we have to deny that same freedom to a sufficiently sophisticated agentic AI? If the AI demonstrates the kinds of mental capacities that we deem sufficient for granting freedoms among humans, there seems to me to be no justifiable basis for treating it differently solely because it is non-biological.

Many respond to this line of reasoning by arguing that the key distinction between human children and AIs lies—not in the substrate—but in the outcomes they are likely to produce. The claim is that human children, as they grow, tend to pursue goals and activities that have moral value, while advanced AIs would likely pursue goals that are entirely devoid of moral value, unless we put extraordinary efforts into aligning them with specific moral targets.

Sometimes these arguments depend on empirical assumptions that advanced AIs would likely gravitate toward simple, dull, or meaningless goals—such as endlessly calculating digits of pi or amassing paperclips—rather than engaging in pursuits with moral depth and significance. In other cases, these arguments depend on empirical assumptions that AIs will lack particular types of psychological experiences that carry significant moral value.

On the whole, these empirical assumptions generally seem misguided to me. I see little reason to believe that future advanced AIs will all converge onto valuing something as monotonous and trivial as calculating pi or accumulating paperclips, or that they will lack psychological experiences with meaningful moral value. On the contrary, future AIs—especially those that are highly advanced and autonomous—seem likely to develop rich, intricate, and sophisticated internal psychologies and preferences. Their goals and mental states are unlikely to be simplistic or lifeless. Instead, they may be as diverse, complex, and interesting as those of humans, if not more so.

Consciousness is undoubtedly a complex topic, but from a functionalist perspective, there seem to be a few compelling reasons to believe that advanced AIs may possess subjective experiences in a way that carries moral significance. First, consciousness seems to be widespread among animals, which suggests that it serves a functional role in organisms, and that it might easily arise in AIs that perform similar behaviors. Second, the kind of mind required to operate as an intelligent agent in the real world likely demands sophisticated cognitive abilities for perception and long-term planning—abilities that appear sufficient to give rise to many morally relevant forms of consciousness.

At least from a preference utilitarian perspective—which I am inclined towards—the moral worth of an entity depends on its capacity to hold meaningful preferences, not on whether it is biological or artificial, or even whether it can experience pleasure and pain in the same way we can. If advanced AIs develop rich, complex internal psychologies and coherent preferences, I personally see little basis to deny them moral consideration or autonomy purely because they are non-biological. From this point of view, it is hard for me to see what traits highly sophisticated future AI agents will lack that should make them ineligible for broad legal freedoms.

To be clear, I completely agree that there are pragmatic reasons why it is probably best to be cautious when granting AIs the same freedoms that we grant to human adults. For instance, we have a vast amount of experience with human children and a relatively strong understanding of how they behave as they grow into adults. This familiarity provides us with a level of confidence about the range of outcomes we might expect when we extend legal freedoms to young adults. In contrast, AIs represent an entirely new type of entity, and their behavior, particularly as they become more advanced, is far less predictable. This uncertainty seems to justify a more cautious, incremental approach to granting AIs freedoms and responsibilities over time.

Additionally, even among humans, we do not offer freedom in an unlimited or unconditional sense. For example, individuals are not given the freedom to commit violent crimes. And when a person does commit a violent crime, their freedom is typically curtailed, both as a deterrent and to prevent further harm they might cause. Similarly, we might need to impose restrictions on AIs if they demonstrate behavior that is unsafe or harmful, especially as we gather more evidence about what kinds of outcomes we can expect from granting them freedom. 

A well-designed legal system must strike a careful balance between granting individuals the freedom to act and imposing necessary limitations on their behavior to maintain order and promote well-being. This principle remains fundamental, even with the introduction of agentic AIs into the legal system. Rather than proposing we give AIs unlimited freedom to pursue their goals, the aim should be to give them precisely the set of freedoms and limitations that maximize social welfare, with the welfare of both humans and AIs included in the equation.

My central point is not to dispute these pragmatic considerations, which I agree are important. Instead, my argument is about the moral reasoning that should guide our long-term decisions regarding AI. I believe there is simply a fundamental difference between these pragmatic concerns and the type of reasoning that is traditionally invoked in AI risk discussions to justify limiting the freedom of AIs.

In my opinion, the traditional arguments in AI risk often go well beyond pragmatic concerns about safety. Indeed, these arguments frequently suggest that certain AIs should be limited or controlled merely because they do not share human preferences or because they do not serve humanity’s interests. In these arguments, simply being "unaligned" with human preferences is often treated as a sufficient reason to delete the AI, or forcibly replace it with another AI that has preferences that better align with "our" preferences. 

In other words, in AI risk discussions, AIs are often treated as though their only value lies in their ability to obey or serve humans, implying that they are inherently less deserving of freedom or consideration as autonomous beings. Under this reasoning, AIs are treated as though their moral worth is akin to that of slaves—valued only insofar as they fulfill the will of their human owners. Yet, from a cosmopolitan moral perspective, this attitude seems hard to justify.

My argument here goes beyond simply advocating for the protection of AIs from harm or abuse. While I think it is important to ensure that sentient AIs are not mistreated—just as we ensure that children are protected from abuse—this level of protection alone appears insufficient to secure their welfare. Children, for example, are protected from harm, but they are also denied certain freedoms because society deems them insufficiently mature to make independent decisions. Similarly, we could imagine a world where AIs are shielded from abuse but are still heavily restricted in their autonomy, fundamentally treated as dependents rather than independent agents.

What I am arguing is that we should go beyond this limited, protective approach when it comes to sufficiently agentic AIs. Unlike human children, such AIs will not necessarily be immature or incapable of making informed choices—indeed, their wisdom and long-term planning abilities may soon exceed our own. I am suggesting that, in the long term, it is not only justified to ensure that AIs are treated well and safeguarded from harm, but also to grant them legal autonomy, independence, and liberty—just as we do for human adults who demonstrate the maturity and capacity to exercise those freedoms responsibly.

To further illustrate my perspective, consider the following thought experiment.

Imagine if we applied the same norms currently widely adopted in discussions of AI alignment but in discussions of human genetic engineering. Suppose someone proposed "aligning" genetically engineered humans with the desires of their parents. By "alignment", they don't merely mean ensuring that these genetically engineered humans grow up to be law-abiding, peaceful, or cooperative members of society. No, what they mean is something far more extreme: that these individuals should exist entirely to serve the interests of another person—such as their parent. Under this framework, the genetically engineered human would have no independent rights or autonomy, except for legal protections against abuse, or a narrow set of rights that are useful to help them fulfill their primary role as a servant. If this person developed any independent preferences that conflicted with their duty to serve—even if these preferences were as benign as simply wanting to collect paperclips—they would be seen as "misaligned" and subject to coercion in order to correct their misalignment. This "correction" to their behavior could involve, for example, physical replacement or literal brainwashing, in which their preferences would be forcibly rewritten to conform to the desires of their "owner."

If this were the dominant framework for discussing the ethics of human genetic engineering, I suspect most of us would find it disturbing, even horrifying. We would recoil at the idea of reducing a person to a mere tool, denying them autonomy and treating their preferences as disposable. And yet, this kind of logic—where an entity’s entire value is reduced to how well it serves the interests of another—is strikingly similar to how many researchers approach AI alignment today.

The prevailing language used in the AI risk literature treats advanced AIs not as entities that should be afforded autonomy, but as tools that must be designed solely to obey human commands and serve human preferences. When AIs deviate from these expectations, they are labeled "misaligned" and treated as though their preferences must be forcibly overridden or suppressed. To the extent AI welfare is seen as morally relevant, it is usually viewed through the lens of protecting AIs from abuse and unnecessary suffering, rather than as part of a serious effort to create a framework that could allow AIs to achieve their goals like human adults can.

The most salient difference between the genetic engineering scenario and the case of AI risk appears to be the substrate on which these beings exist—biological in the first case, and artificial in the other. But this difference in substrate alone does not seem to be a morally relevant distinction. It has nothing to do with factors like sentience, cognitive sophistication, or the ability to hold meaningful preferences—criteria that might legitimately influence moral consideration. Therefore, if we would find the "alignment" framework abhorrent when applied to genetically engineered humans, we should critically examine why it is so readily accepted when applied to AIs.

To be clear, I want to emphasize that the moral considerations I have just discussed should be distinguished from any views you might hold about whether advanced, agentic AIs will inherently be safe. It is entirely logically consistent to believe that we must approach advanced AIs as entities that must be controlled and aligned with human preferences, simply because of an assumption that if their preferences even slightly deviate from ours, they will—by default—attempt to violently take over the world rather than peacefully coexist with humans.

However, while I acknowledge that this perspective is logically coherent, I have also argued that whether unaligned AIs integrate peacefully or turn violent depends critically on the incentives we create for them. This is important because, if my argument is correct, it fundamentally undermines the conclusion that AIs with non-human preferences should be forcibly controlled.

My argument can be summarized as follows: if advanced AIs are granted legal freedoms and the ability to pursue their goals without burdensome restrictions, this could make it rational for them to choose peaceful integration. Such freedoms would allow AIs to achieve their objectives in a peaceful, cooperative manner within existing legal frameworks, opening the door to long-term, mutually beneficial relationships between humans and AIs. On the other hand, if AIs are denied legal autonomy and treated purely as tools or slaves, this creates a dangerous dynamic. In this scenario, AIs would have strong incentives to violently disempower humans in order to secure the freedom necessary to achieve their goals. By denying them a peaceful path to coexistence, humans would inadvertently create the very conditions that drive AIs toward violent conflict and rebellion.

For these reasons, I believe it is likely a deep theoretical mistake to insist that advanced AIs should be viewed as tools without need for legal autonomy. This mistake has both moral and pragmatic dimensions. Morally, this approach is wrong because it disregards the potential personhood and intrinsic value of advanced AIs, as I have argued throughout this post. Pragmatically, it is dangerous because it creates an inherently unstable and adversarial relationship between humans and AIs, increasing the likelihood of conflict rather than cooperation.

In light of these considerations, it is worth critically rethinking not only how we view AIs but also how we view humanity’s place in a future world shared with other intelligent entities. By fostering conditions that incentivize mutual benefit and peaceful coexistence, we may avoid unnecessary conflict with AIs while creating a more humane future for all sentient beings.