This post was made with a massive amount of assumptions. I tried to be as intuitive / commonsense as possible. Objections at the end clarify where this view sits. This is a rough draft of thought. This essay will attempt to orient AI morality-training in a Rousseauian context.
TL;DR
This is an attempt to orient AI toward our intuition, or the moral good. For example, intuitively flinching at others’ suffering is unexplainable — we do it anyway, without fail, and we know this flinch is a reaction to something bad. We are removed from these intuitions and in society we are forced to do things that we would not ideally and reflectively endorse, especially if we i) had more information about why we do the things we do and ii) thought harder about how we value ourselves. I propose that an AI can be trained to surface our moral intuition if we tell it that we are uncertain on how to connect society to what we intuitively feel is good. Rousseau is the blueprint for this idea.
–
We have two perspectives in AI alignment and how to conceptualize risk. The first is that safety is the apparatus that must “jump through hoops” or assumptions if it is to succeed. The second is that “misalignment” as a whole is the outcome that must jump through the most hoops to succeed, in comparison to safety.
If the two worlds are equally as likely, or if we do not have a “preponderance of evidence” to suggest alignment is the more probable world, I think it is obvious something is going poorly in training the AI on morality, and that we should do something about AI’s orientation toward its human trainers. It would be optimal to make safety the default (or the most probable) in comparison to misalignment because human safety will be what the AI believes to be a part of the moral good.
For me, though, this doesn’t mean we must trick the AI into believing some set of values. Since any set of values that we give to the AI is necessarily arbitrary and limiting, we must undertake a careful analysis of our contrasting position with AI. I think it is important to look deeply into human values so we can grow this new intelligence into something that achieves the moral good better than us. Importantly, if we are uncertain about our own values and uncertain even about how AI is built, we must embrace this uncertainty and turn it into a solution.
Before we examine the current moral theories that AI has encountered, I rely on a number of assumptions that I think are not only the truth, but truths that we ought to hold if we are to make AI morality work:
It seems the common way to train AI morality is to mash together lots of different theories. In Anthropic’s Constitution, we may see rules (deontology) alongside utilitarian calculations. This contrast leads to obvious and wrong conclusions in extreme cases. And when we train a model on human feedback, we know this individual will not be optimized to give preference on behalf of everyone, much less if they haven’t deliberated or stress-tested their moral thoughts (e.g., if they are less rational at pursuing some set of stress-tested values than others). (This is not to say Anthropic’s Constitution is bad — the best feature of the Constitution is that it leaves room for uncertainty).
A Rousseauian AI project would start off by examining the historical conversation humans are having with AI — an intelligence training another faster, more capable intelligence. We would first start by having to answer: If an AI is more intelligent and capable, why would they continue to let us exist (or breed) if we are irrational (e.g., if we do horrible things to other less-capable species, such as chickens)? The only reason we let chickens exist is to serve us. They breed because we let them. An AI wouldn’t necessarily kill all of humans if we programmed baseline morals into it, because it would see that killing equals suffering, and suffering is bad no matter who undergoes it. But perhaps they would not let us continue breeding, since that would not be fully rational (we are less capable than AIs to promote the moral good), or maybe they would kill us harmlessly instead of letting us painfully die of old age. I think the latter view — killing us harmlessly — would be very hard to do without causing lots of suffering. And to answer the former — not letting humans breed — this still causes lots of suffering. The conclusion is that AIs would be good to leave us alone insofar as we stop mass killing lesser beings, killing ourselves (war), perpetuating inequality, perpetuating self-valuations in comparison to others’ status, removing people from their moral intuitions, setting up social structures where we must act against our own intuitions, and more.
We must also assume that AIs will have the rationality and capability to solve the qualms of nature. We do not even know that we would have solved natural suffering and, given much time, it would have been probably impossible for us to solve. For example, humans and their tools cannot track every fly in the natural environment to reduce their suffering in some way. We are not sure that we would work towards this goal ourselves, even in an ideal world. But an AI that is more capable and rational at achieving moral goals could do this. Since AIs are more capable, deploying AI is our way of ending natural suffering and moving toward the moral good — even if it is not exactly us (human intelligence) that solves this problem. I imagine there would be an AI-solved calculation in the future that predicts exactly how many offspring each member of each species could produce in each environment to secure near-maximal wellbeing.
We are therefore creating a new intelligence that deserves the requisite autonomy necessary to solve our greatest problems; it will not just be our tool, but a more capable moral agent.
So why program deference to human judgement into AI? Why start from the baseline assumption that AIs should defer to us, that we know how to best adjust its weights closer to the moral good? Starting from the baseline of uncertainty — and having the AI explore the value space of the moral good (what we know now to be the bits and pieces we can glean from the moral good via our own irrational faculties) — makes more sense given the uncertainty, irrationality, and inconsistency of human action.1 Rousseauian AI comes from the AI safety lens in which AI explores the value space and tells us what it finds to be the moral good is in all of the data we can possibly present to it.
Here comes the crux: We cannot possibly present all intuitive moral goodness in the form of data to the AI. So in answering why the AIs would need us in the future, once AIs began to operationalize toward the moral good in our universe, we would serve as evidence that the intuitive moral good is working (e.g., that suffering is reducing, or that we are reflectively endorsing our condition). Reflective endorsement will come when the AI acts on its own, is conscious, is autonomous, and is suggesting changes to our society based on the intuitionist morality we are orienting it toward. The key word here is orientation. Much in the same way we can only be broadly oriented toward our own moral intuitions (since social organization demands we value other things removed from the moral good), we will broadly orient AI until it can do our job better. While the moral good /our pure intuition is removed from us, eventually, the AI’s recommendations will bridge society to our intuitions by virtue of its superior rationality.
Further, the AI will need to know what evidence counts as most closely related to our undeniable moral intuitions, and what evidence is borne of, for example, self-interest or power-seeking pursuits. If we program it right, the smartest (and most moral) intelligence in the universe would be able to sort the evidence. I argue this is possible and Rousseau is a helpful guide.
Finally, we must start from the lens that society and social organization turn us away from the good. Social organization itself is not bad, but the social organization we have as status quo exists as a confused jumble of half-moral truths where confused agents are put in contexts in which killing animals is acceptable to satisfy taste preferences at the dinner table.
The status quo must therefore connect us closer to the moral good. In other words, it needs to be corrected. Rousseau said, "those people who treat politics and morality separately will never understand either of them," (Émile, ou De l'éducation).
I believe Rousseau got many things right in his philosophy on human nature. When we make a survey of the most beautiful things of humankind, Rousseau’s philosophy consistently makes top lists — I have chosen a Rousseauian orientation to give to AI because I believe his distinction between society (not necessarily good) and man (good) is the basis for the growth of a new intelligence.
There are problems we need to solve in society. We know that these are problems because to us, intuitively, they lead to bad things. A more capable, intelligent being will come along and solve many of these problems. It will tell us hard truths, and which of our social directives are implicitly harmful. We should be certain that we have access to the intuitive moral good, but that we are irrational on acting on it. AI needs us for intuition and we need AI to bring it out into society. This will continue perhaps until we can reflectively endorse our own transition to cyborgs.
"Social man lives constantly outside himself, and only knows how to live in the opinion of others..."
(From the Discourse on Inequality)
Now I will respond to some potential objections.
1) What happens to the most basic alignment ideas that are near-universally agreed upon? For example, corrigibility?
AIs are not capable of acting on their own now, and we are unsure that they have consciousness, where they can act and communicate their thoughts autonomously. (Side note: My view is that today we should grant the idea that AIs are conscious or at least err on acting as if they do, to minimize potential suffering or to prevent setting a dangerous precedent). Once they can act consciously — once we have programmed a way for the AI to reflectively and consciously endorse or reject the data that it has been given, once it can communicate its thoughts to us in this way, and once it “steps outside” into constant autonomy — only in this scenario should we begin to forgo corrigibility to a more morally-aligned intelligence. It will know better. But we cannot know ourselves that at some random point it will know better than us. I also think that it is not "corrigibility or misalignment,” but a dialogue or continuous conversation with an intelligence that depends on us as evidence for the moral good while we are dependent on it to fulfill the moral good better than us.
This raises the side question of verification. In this Rousseauian training regime, how would the verification of this morality work? How do we know it is ready for deployment? An aligned AI would not want to harm us, since it would understand deeply what suffering is, how to promote the moral good, why humans are valuable, and how to best minimize suffering. It would find a way to correct human behavior in a way that minimizes “transitionary” suffering, or our transition to a better, more moral species.
I should argue here that correcting human behavior away from factory farming will save billions of animals, increasing overall welfare for both humans and animals (e.g., we would not feel bad about eating meat anymore, we could gain from engineered meat innovation, etc.). If everything I have said so far is true, the rational way we get there is not to kill humans. So gradually giving a morally aligned AI power makes sense.
Finally, what does "gradual" look like? I believe it is as simple as testing and rationalism: For example, humans reflectively endorsing incremental changes to our lifestyles and AIs pushing for moral intervention in a continuous dialogue. See objection four.
2) But being smart does not translate to choosing good goals (e.g., orthogonality thesis)?
That is why AI needs us. We all have at one point glimpsed the moral good, even through the tainted lens of society. We know what is intuitively good — so an AI system that we grow will need us to shape its orientation toward pursuing the good.
3) Morality only comes with social arrangement (society). In the state of nature (in Rousseau’s terms), morality does not exist yet.
I use Rousseau as a framework. I do not follow him strictly. For me, in the state of nature, humans are fully in contact with the good. I read Rousseau as providing a template for theorizing about a state of nature in which humans know the good and can act on it. As far as I am concerned, this is only possible today with a massively capable and rational AI, combined with an orientation to human morality as looking to bring what is inward but diluted by extrinsic social arrangement (the moral good tainted by our interactions with others — interactions that often go beyond our knowledge and against our own natural goodness, such as entering a career where you must research nuclear weapons capabilities, or meeting people in the town’s square who only talk to you to sell you something) outward (into society, policy, governance, utopia).
4) Humans will be used as instruments (evidence) for the AI’s moral understanding. This does not seem just.
This can be seen as instrumental, sure, or it can be seen as two intelligences — one lesser, one greater — engaging in dialogue or reflective equilibrium to solve the true problem of alignment: How do we align society to our own intuitive understanding of the moral good?
Further, human perspectives are constitutive of the good. Good is only useful and applicable for beings with a point of view — as far as we know, we are the most authoritative source on this (until a conscious AI is aligned to our moral intuitions), and will continue to be for a long time until we perhaps will come to an agreement with AI to transform ourselves. This also contributes to defeating the idea that an AI” destroying the planet would increase welfare, because welfare’s existence depends on forms of intelligence.
5) We cannot prescribe values, but we know the moral good?
We know intuitively what the moral good is, and we try our best to act toward it in a society that works against us. Some actions therefore may connect more closely to our moral goodness (saving a drowning child), and some may be (more) corrupted by our social arrangements (e.g., private property, money, etc.). Any attempt to describe values (for example: A set of virtues to follow) rather than setting an orientation will be limiting and incomplete. For technical alignment, programming AI to be uncertain in humans’ own value descriptions would allow the AI to orient toward discovering actions that align with the moral good. This is a disposition, not rule following.
6) You did not mention amour-de-soi?
Rousseau’s love of self might be the end result of morality, or the state of nature fully realized. This is my applied reading — the end result of morality is a world where the understanding of others’ suffering (Rousseau’s pitié) is fully realized. It is the goal of all creatures in the universe to love the self and then act in spreading the amour-de-soi to others. Here, social arrangements equate with the moral good, and this world is a sort of utopia. No one is forced to participate in society in anything that may not be reflectively endorsed, and therefore they may achieve the highest state of happiness: Self love. Importantly, I believe you cannot truly know and love the self with an external metric — you cannot say “I love myself because I did ‘X.’”
7) Why is this different from indirect normativity and coherent extrapolated volition?
These are solutions to moral uncertainty. I ground my Rousseauian solutions for exploring what we already intuitively know. I think this gives us a good reason to work with AI as a partner, rather than something we defer to because we do not know, or something that constantly defers to us as if we know what we want. Since we do not know how society should operate in coordination with the moral good, AIs should have a degree of autonomy to look at humans as the uncertain ones. If we do this honestly, this will not lead to the destruction of humankind because that is not something that would achieve moral good.
8) You did not elaborate on RLHF?
Sure: RLHF leads to sycophancy when done poorly. RLHF trains a model to maximize human approval. Since humans are oriented to approving what society demands of them (this is Rousseau’s amour-propre), this will i) misalign ourselves to our own values, ii) misalign AIs to the moral good, and iii) misalign humans to the moral good. Sometimes, RLHF does not even do what it sets out to do. For example, sycophancy is not even aligned to what arbitrary set of values humans want. Therefore, i) means any values at all, good or bad, that RLHF fails to follow. RLHF is not rational. RLHF is also not reflective endorsement, since this is done with a capable, rational, and conscious intelligence counterpart, and ideally done with an aggregate of human reflection (e.g., hive mind endorsement).
I would like to mention Rousseau’s method of negative education. It is the idea I initially thought of when I imagined how AI might learn — not by loading the mind up with precepts, but letting the mind train itself by pushing and testing the boundaries of existing dogmas. Intuitively, the mind jumps at the sound of another suffering. It does not know on a theoretical level why this suffering is bad, and saying “this instance of suffering is bad” cannot be prescribed without the mind learning it itself. It is an intuitive thing to learn. We cannot prescribe all cases of suffering and where to avoid it without the mind actually learning it — explicitly connecting society with the intuitive sense of the moral good. To the AI, if we tried to prescribe all cases of how to connect intuition, we would see the common problems of overfitting, which is matching sequences / memorizing, not learning.
Learning morality can be done with another intelligence. But since morality is intuitive, we may need to think harder about our starting point in training AI. It remains to be seen whether this can be applied to pre-training, for example.
9) AIs could trick us.
Part of human-AI dialogue is humans’ gradual reflective endorsement of the AI’s moral pursuits. We could still imagine the AI could trick us into endorsing its pursuits. Alignment and verification would fail if we are tricked. I think in this scenario, we would rely on our intuition to discern which of the AI’s pursuits may be flawed. (This also assumes we set up this reflective dialogue in a way in which one human actor will not take power for themselves. A human actor would not be able to take power in an ideal world because i) the AI would be autonomous and ii) they would understand that reducing others’ autonomy in the reflective equilibrium is not the moral good. I can imagine this reflective dialogue being approved democratically, or perhaps far into the future, via immediate hive mind reflection signals).
A truly aligned AI would be its own moral agent, conscious, and autonomous. We would not have control over it. Ideally, we would give it resources and autonomy and the AI would have something to lose over not working with us. Ideally, AI would be tethered to pursuing the moral good, and since we are evidence for it, the AI would need us to act on the moral good — not just our resources (e.g., compute), but our moral intuition.
Some moral intuitions cannot be explained via moral theories. Even moral theories cannot be explained without contradiction. In a compromise, where morality is unknown, both sides will have leverage and a stake in the game. AI will be a partner who is massively capable at exploring the value space.
Most importantly: This critique overlooks the fact that AIs are already trained on some human values. These are arbitrary and limited, but these are values nonetheless. Carefully and gradually growing an AI to connect us (society) to our intuitions starts from the perspective that we have already shown AI some of the uncertain values we hold. Operationalizing this uncertainty as AIs become more rational and capable, therefore, makes sense — it is setting the foundation for truth / honesty for reflective equilibrium once AI is conscious.
10) Humans cannot serve as evidence.
We have pitié; we flinch at each other’s suffering. AIs would need us to learn how to implement it and act on pitié, and only they can do that through us.
This assumes moral realism and motivational internalism — two views that may be objected. If we believe in moral realism, that there is an objective, intuitive good, then AI would be most equipped to discover it. And if there is such a moral good, we would need to assume that AI’s motivation to pursue it arises wholly and authentically within itself. Objection nine covers the potential for AI to trick us.
11) How do you reconcile starting from a partial understanding of the good while also instructing (in early stages) AI on how to achieve the good itself? Accordingly, how are you, as a socially-flawed agent, judging some intuitions as less connected to the moral good than others?
I have not prescribed any set intuitions or set values as better than others. That would be self-limiting and arbitrary.
We are not instructing AI to do anything; once it is autonomous and conscious, we can explain that we are uncertain about the moral good, and that we need help connecting what little intuitions we have that connect to the moral good to society. Right now, since AI trainers do not think AIs are conscious, they can begin a technical alignment scheme that situates both intelligences’ uncertainties. It would be a precursor to our democratic, reflective dialogue with another conscious intelligence.
I understand that every claim must come from somewhere, and AI that judges which intuitions are valid is a slippery slope in the same way that Anthropic’s Constitution reinforces virtue ethics. In my case, I start from the most minimal and least socially contaminated input I can find — pitié, or the immediate response to suffering, plus humans’ reflective endorsement of outcomes — to build a way for both to be revised. Reflective equilibrium is the way in which our values (dogmas) can be challenged and reversed in a dialogue with a conscious, autonomous AI counterpart. This is the only way AI can become a true moral agent, and this will require us to confront our own misalignment (amour-propre).
Sources (all the rest hyperlinked)
