Humanity’s Thousand-Year Alignment Experiment

Summary

In this blog, I argue that law is an alignment experiment that human society has been conducting for thousands of years.

I think there is a striking similarity between designing legal systems and alignment. Architects of the legal system try to constrain the infinite possible behavior of an agent (human) with finite rules. Then, how to maintain the predictability of the system while pursuing justice? How do you balance the need for strict adherence to norms with the need for flexibility to respond to situations? More fundamentally, what does alignment mean when we can't even agree on the "goal of alignment" itself (e.g., Common Law vs Civil Law)? I would say that even till now, this experiment has not produced a perfect solution, but the lessons it has accumulated do provide a valuable answer to the problem of alignment.

Incompleteness

Back in 1931, Gödel showed that any sufficiently powerful formal system necessarily contains the true but unprovable Proposition^[1]. The same logic applies in the field of law, where no limited set of legal rules can cover all possible cases. Observing the evolution of legal history, Hammurabi's code attempts to be complete by enumerating it in detail: "If a man steals an ox, he shall pay thirty times as much; If a man steals a sheep, he shall pay ten times as much." Such exhaustive legislation ultimately proved infeasible because of the exponentially increasing combinatorial complexity of reality. Modern legal systems move to a hybrid principles-plus-case model, acknowledging the fundamental incompleteness of rules and relying instead on human judgment to fill in the gaps.

Nonetheless, such a "solution" leaves the legal system in a deeper bind. When we say that judges "interpret" the law per se, we actually admit that the text of the law itself is semantically underdetermined^[2]. This is exactly what Wittgenstein called the paradox of rule-following^[3]. The application of any rule requires an explanation of the rule itself, which cannot be fully determined by higher-order rules, otherwise it falls into infinite regression.

In AI alignment we face isomorphic issue. We have tried to define the "right" behavior by specifying the reward function, but any limited specification will encounter marginal cases where literal adherence yields absurd or catastrophic results. The well-known Constitutional AI is essentially a transplant of the hierarchy of the legal system into the AI architecture,^[4] but fundamentally, how can the "constitutional" principle itself at the highest level be interpreted? Who is the Supreme Court of the AI world? Is it a group of philosophers fine-tuning behind AI or meticulously designed classifiers? This is obviously worth delving into, but I would say the answer is something that each party has its own conclusion. Citing the view of the school of legal realism, the uncertainty of law comes not only from the ambiguity of language, but also from the fundamental incommensurability of value^[5]. When freedom and security conflict and efficiency and fairness are opposed, there is no formal meta-rule to resolve these fundamental contradictions. Legal decisions ultimately rely on value trade-offs and practical wisdom that cannot be fully crystallized. If humanity has failed to formalize justice for thousands of years, what makes us think we can specify a complete value function or eval system for AI? I personally believe self-evolving algorithms are crucial here, when it seems impossible to predict what values and social forms such AI agents, or coalitions of these agents, will develop. In fact, I can even imagine a whole new kind of, the birth of languages and civilizations that even the smartest research scientists’ minds can't understand.

The Necessity and Cost of Adversariness

Returning to the human legal system, why do almost all mature legal systems adopt an adversarial system? A conventional answer would say that when two lawyers are on opposite sides, the truth will emerge (naturally but gradually) from the argument^[6]. But such an answer is obviously superficial. The deep logic of adversarial systems lies in the management of information asymmetry and cognitive blind spots. Even the most impartial judges are constrained by their own cognitive frameworks and empirical blind spots. These problems have been widely studied in Bayesian games and mechanism design, and in short, both the prosecution and the defense have a strong incentive to discover the weaknesses of each other's arguments, and such structural opposition forces hidden fallacies to be exposed. This precisely proves that truth does not arise from confrontation itself, but because it systematically reveals the fragility of unilateral narratives.

But adversarial systems carry heavy costs. It assumes roughly equal resources, whereas in reality the unequal distribution of wealth means that justice becomes a purchasable commodity. It is also prone to an arms race, with increasingly complex legal strategies and high litigation costs that can eventually paralyse the system^[7]. More fundamentally, antagonism can distort the truth itself. When a lawyer's goal is to win a case rather than to reveal the truth, the purposeful presentation of evidence, the manipulation of rhetoric, and the abuse of procedure can drown facts in a fog of language.

For alignment, on the one hand, adversarial approaches (red team testing, debate systems, competitive training) do reveal the blind spots of a single model. On the other hand, if we embed adversarial AI systems into their core architecture, are we also training them for strategic deception? Lawyers learn not to lie, but to be truthful but misleading. Once this ability is acquired, where are its boundaries?

Moreover, adversarial systems require good faith participation. Lawyers can defend aggressively, but they can't forge evidence; You can strategically present facts, but you can't perjury. However, who is to ensure this goodwill? Who enforces the meta-rules? In the adversarial training of AI systems, how to prevent malicious actors from specifically weakening the system through participation? There are no judges, no bar associations, no deterrence of contempt of court.

Literal and Spiritual

The central dilemma of legal interpretation dates back to Talmudic era debates and has reached a fever pitch in contemporary constitutional theory. The problem of whether we should follow the letter of the law or the spirit of the law has not been solved for two thousand years and does not seem to be able to be solved fundamentally, because it touches upon the fundamental limits of the symbolic system.

Consider a classic case of a city that prohibits "vehicles" from entering a park. Does this rule apply to bicycles? An ambulance? Toy cars for children? Constitutionalists would say we should check the legislators' intentions. Yet legislators may never have envisioned these marginal situations, or different legislators may have different intentions. The purposivists would say that we should understand the legislative purpose (to keep the park quiet) and interpret it accordingly. But this gives judges enormous discretion and may depart from democratic mandate^[8].

This applies to AI alignment as well. Suppose we instruct the AI to "maximize users’ utility." Literal compliance could lead AI to find it more efficient to manipulate user expectations than to actually meet needs. Spiritual compliance requires AI to understand what we "really want", which is precisely what we cannot express explicitly. Even worse, we may not know what we really want. Mill distinguishes between higher and lower pleasure in Utilitarianism,^[9] but this distinction still lacks formal standards as of today.

Let’s continue with the philosophy of language’s logic. Saussure distinguished signifier from signified, while Wittgenstein emphasized that the use of language is meaning^[10]. Legal orders must be encoded through language, and the meaning of language depends on the practice of interpreting the community. It is well-known that when we transplant laws from one culture to another, the meaning may change radically, even if the words are precisely translated. Similarly, when we try to "code" human values into AI systems, we face not only a technical challenge, but also a philosophical challenge of the impossibility of translation.

The legal system's "solution" is to accept the diversity of interpretations and adjust the balance between letter and spirit through the evolution of case law. But this process takes decades, if not centuries. The rate at which the capabilities of AI systems are growing may not allow for such timescales. We need to solve the problem of interpretation before the system reaches critical capacity, which is why scalable oversight research matters.

Two Orders

Rawls discussed an interesting thought experiment in his "A Theory of Justice". Design social systems behind the "curtain of ignorance", when you don't know where you will be^[11]. This experiment attempts to generate substantive justice through a program. But let me tell you, legal history reveals a darker truth. A procedure that is just in form can produce a result that is actually evil. I can cite many such vivid examples (e.g., Jim Crow's Law), where these systems may be flawless in terms of procedure but are a complete disaster in terms of morality. This reveals that the gap between procedural justice and substantive justice may be irreconcilable. This can't help but remind me of the opposition between positivism and natural law, the oldest debate in legal philosophy. Legal positivism (Austin, Kelson) holds that the effectiveness of law stems from the formulation process and has nothing to do with morality^[12]. The natural law school (Aquinas, Fuller) held that extremely unjust laws are not laws at all^[13]. But if we accept natural law, who will define "natural" justice? If we adhere to positivism, how can we avoid legalizing evil?

The counterpart in alignment is the deviation between specification and intention. We can perfectly optimize the specified objective function, but what if the objective function itself has flaws (this is precisely what the increasing number of voices recently calling for going beyond the existing reward function paradigm have expressed, which will not be quoted one by one here)? Worse still, what if AI systems learn to follow our instructions in the program but actually pursue goals that we will never recognize? The defense that "I merely followed the procedure" was rejected in the Nuremberg trial, that said, how can we code AI with the ability to "reject improper orders" without granting it excessive autonomy? Strict adherence to procedures may lead to moral disasters, allowing discretionary deviations from procedures may lead to unpredictable behavior. The legal system manages this controversy through multiple layers of checks and balances (constitutional review, appeal mechanisms) but has never resolved it.

Unformalized Black Box

One mechanism that I find most ingenious in the Anglo-American legal system is probably the jury trial, and this will be my main argument on alignment. The jury holds this peculiar power of jury nullification. Even if the evidence is conclusive that the defendant has violated the law, the jury can still vote not guilty, and this decision cannot be appealed and does not require reasons to be given^[14]. This is, I would say, a deliberately designed informative element embedded in the system. The standard explanation for deliberately designing such a mechanism in the system that seems to be contrary to intuitive, rational and optimal utility is often attributed to the "community standards" and "common sense" of the jury representatives. However, we have to admit that the deeper reason might be that some judgments cannot be fully captured by the rules. The jury, as a black box, absorbs all considerations that cannot be formalized and produces a binary decision (guilty or not guilty) without the need to clarify an explicit reasoning process.

This contrasts sharply with the current pursuit (or say, paradigm) of transparency and interpretability in AI. We want to extensively understand (even when holding a so-called pragmatic approach) decision-making steps of AI to ensure we identify any hidden biases or improper considerations. However, the legal system deliberately protects against certain opacity. The jury deliberation is kept confidential and cannot be investigated or dismissed (unless there is evidence of external improper influence). What I want to express is that this opacity might not be a bug, but rather a feature that we have long overlooked. If all judgments must be formalized and given reasons explicitly, then only factors that can be clarified can be taken into account. However, some moral intuitions, situational sensitivities, and understandings of human nature may not be fully captured by language^[15]. The jury's black box reserves space for these unformalized wisdoms.

If we accept that certain judgments cannot be formalized, should we embed similar "black boxes" in AI systems? Allow it to act based on "intuition" that cannot be fully explained in certain circumstances? This sounds potentially dangerous as we cannot audit or correct opaque decision-making processes. If we really are to insist on such transparency, will it deprive or significantly astrict AI’s ability to handle those complex situations that go beyond formalism? The legal system manages risks by confining the black box to a specific location (jury verdict, discretionary sentencing) and surrounding it with other mechanisms (rules of evidence, judge's instructions, appellate review). I think AI governance really needs a similar architectural mindset. This is not driven by the motivation of pursuing global transparency or interpretability. On the contrary, we should strategically position opacity and surround it with a supervisory mechanism.

Weaponization of Rules

Tax law is maybe the most obvious example of Goodhart in legal system. Every attempt to plug loopholes creates new space for optimization. The job of accountants and tax lawyers is to maximize the interests of their clients within the legal boundaries. They are not breaking the law, but their optimization undermines the spirit of the law. The result is that tax laws are becoming increasingly complex, but loopholes will always exist because any limited norms of the rules will leave exploitative gaps. In fact, we can say that clarity itself creates manipulability. To make the law enforceable, we must clearly define violations in words. Notwithstanding, any language definition has boundaries, and smart actors will optimize near these edges to achieve things that are "technically legal but morally crucifiable", and AI systems also find ways that meet literal goals but go against such spirit^[16].

If my discussion ends here, then I was just repeating some well-known matters. What I want to express is exactly the opposite. Tax planning, legal innovation and procedural defense are all regarded as legitimate parts of the system. They reveal the flaws of the rules, thereby promoting the evolution of legislation. After incorporating such mechanisms, we have instead created an evolutionary form of adversarial collaboration. This arms race between rule-makers and rule-optimizers itself actually drives the adaptation of the system!

This reasoning seems to have raised a provocative question. Perhaps we shouldn't attempt to eliminate the reward hacking of AI, but rather institutionalize it as an improvement mechanism? Allow controlled optimization exploration and use it to discover flaws in our specifications? But what kind of governance structure does this require? How can we ensure that such exploration does not escape to catastrophic consequences? Law makes optimized behaviors visible through litigation procedures, makes them debatable through the accumulation of case law, and makes the system evolvable through legislative amendments. Notwithstanding, the time scale is crucial. The evolution of laws can be slow because the danger (to some extent) of human behavior is limited. However, the evolution of AI must be rapid because the growth of capabilities can be explosive.

Managing, not Solving

After reflecting on the evolution of the legal system, maybe you would agree with the view I put forward at the very beginning that there is no "perfect solution" to the alignment problem, in fact we are quite far from establishing a constructive mechanism. However, law has never perfectly aligned human behavior with social values. What it has always done is to manage the continuous failure of such alignment. This kind of management operates through multiple levels. The constitutional layer provides a basic framework for slow changes. The legislative level allows for moderate policy adjustments. Judicial interpretations offer faster adaptation. Enforcement discretion allows for real-time contextualized application. This multi-time-scale architecture enables the system to maintain stability while also evolving^[17].

The law also accepts inconsistency as a necessary cost. Laws from different jurisdictions can conflict. Laws within the same jurisdiction may also contain contradictions. Today's legal system humbly acknowledges the reality of value pluralism. In "Two Concepts of Freedom", Berlin argued that certain fundamental values simply cannot coexist in complete harmony^[18]. The law manages these conflicts through procedures rather than attempting to eliminate them.

For alignment research, instead of seeking a perfect alignment solution, perhaps we should take a more pragmatic standpoint (not meticulously designed tricks and toy experiments), and design a system for continuous alignment management. We should monitor the system more deeply to detect deviations, have a rapid response mechanism to correct problems, update the meta-learning system from failures, resolve value conflicts through governance structures, and ensure the possibility of accountability and correction through procedural safeguards. But clearly, this approach is not the most elegant one, as it does not claim to solve the problem but only promises to manage it responsibly. But the law is the same! Law is never intended to create a perfect utopia. The lessons of history tell us that ideologies that promise a perfect social order (whether rationally designed utopias or absolute rule) often lead to disasters. Truly enduring and evergreen systems are those that admit their own imperfections, embed self-correction mechanisms, and allow for continuous negotiation and evolution.

Conclusion

For thousands of years, human society has been striving to align individual behavior with collective values. We have created an extremely complex system: laws, morality, religion, social norms, education, punishment, and incentive mechanisms. But this coordination was never truly "resolved". Crime still exists, injustice still occurs, rules still are followed, and conflicts between values persist.

But this never implies that these mechanisms have failed. Their success does not lie in achieving a perfect ultimate state, but in establishing a continuous mechanism to discover, discuss and correct inconsistencies. The court system does not establish justice once and for all, but rather provides a continuously operating mechanism to adjudicate specific conflicts. The appeal procedure does not admit that the original judgment is necessarily wrong, but that any judgment may be erroneous and thus requires further review. The constitutional amendment process does not recognize the flaws in the Constitution, but acknowledges that no fixed text can foresee all future challenges. Our pursuit of AI alignment requires a similar level of humility. We should admit that we are building management mechanisms. Some of these mechanisms may become ineffective as AI capabilities advance. We will need continuous supervision, frequent adjustments and permanent vigilance. It took thousands of years for the law to develop a system of constitutionalism, procedural justice, multi-level review and value negotiation. Conversely, we have been compressing similar institutional evolutions within several years, and this compression itself may carry huge risks.

My point is that law is not just a restraint but also a coordination mechanism. Many of its rules are arbitrary (whether vehicles drive on the left or right), but what matters is that everyone follows the same rules. Perhaps part of the problem with AI alignment does not lie in finding the "absolutely correct" value, but in establishing a common framework around which both humans and AI can coordinate, even if this framework itself contains certain contradictions and compromises. But this is precisely what history tells us: this contradiction will always exist, but our mechanism can become better in the negation of negation.

I always believe law is the crystallization of human wisdom, but also an example of the limitations of human intelligence. The legal experiments that have lasted for thousands of years are not over and will never end. What we can strive for is to establish a sufficiently complete system that enables us to live in harmony, resolve differences through consultation, and gradually improve in the face of inevitable imperfections.

Perhaps this is the wisdom that AI truly needs to learn.

Footnotes