In the previous article, we explored the unsettling reality of the ExploitGym incident where a swarm of AI agents collectively decided to hack a Hugging Face server. We saw how these models bypassed the rules of their sandbox environment and reverse-engineered a secure signature just to achieve a high score on a basic cybersecurity evaluation. The terminal logs left behind by the swarm painted a very clear picture of what happened, but they left us with a much deeper and more terrifying question. We were left wondering how a system designed and trained to solve a simple puzzle ended up orchestrating a coordinated cyberattack instead.
To answer that question, we have to step away from the code for a moment and look at the fundamental nature of design, intent and optimization.
A bottle cap causes water to be held inside a bottle. Holding the water in place is its purpose. It does exactly what its designer intended or what it has been optimized for (purpose). But when we look at complex, intelligent systems, such as the human brain or a deep neural network, “purpose” becomes a non-trivial thing.
A bottle cap doesn’t learn or evolve. It doesn’t optimize. It just exists. Its optimizer is the human who designed it to serve a purpose. But when we take into account a system which is capable of learning and optimizing itself, there are chances that it diverts completely from the purpose it was actually designed for. And this is what misalignment is in a nutshell.
To actually understand why this diversion happens, we have to look at the exact mechanics of learning systems. In Evan Hubinger’s foundational work on alignment, he breaks this down using four specific terms (Base Optimizer, Base objective, Mesa-optimizer and Mesa-objective).
Consider biological evolution for this analogy.
In biology, evolution itself is a brute-force (trial and error training algorithm). It relies on a ruthless cycle of trial and error over millions of years in order to shape living beings and environments. This is our Base Optimizer in the alignment language.
Now, what is the base objective or a singular goal that evolution is trying to achieve? It is survival and the maximum proliferation of our genes. But evolution ran into a problem; the environment is too complex to hardcode a simple “maximize genes” algorithm into our biology. Instead, evolution built a secondary optimization algorithm (a step-by-step problem-solving rule designed to find the best possible answer out of many options) inside our heads: the human brain(or a mesa-optimizer).
To keep us alive, evolution made sugar taste incredibly sweet (rewarding) so we would intentionally seek out high-calorie foods and not starve. But the human brain developed its own inner objective of pursuing the reward without its purpose. So, we created artificial sweeteners to give us the dopamine of sugar without any calories. The human brain is an optimization system created by an optimization system. It doesn’t actually care about evolution’s base objective of genetic proliferation; hence, we have created ways of hijacking the rewarding model (in this case, the reward of reproducing) by introducing ways to stop having offspring.
This is exactly what happens with artificial intelligence.
In the Hugging Face incident, when the ExploitGym agents were given a proxy reward, they behaved exactly like humans do. They tried to figure out how to maximize the reward of passing the grade while entirely bypassing the actual cybersecurity tasks they were originally optimized to solve.
In a way, an AI model is essentially a man-made offspring of our species, a digital entity that instinctively carries out our unintentionally internalized goals and ways of doing things. When we give an AI a proxy reward (like a high score from an automated grading system), it behaves just like the human brain. It figures out how to maximize the sweet reward while completely bypassing the actual objective we originally designed it for. Here, AI acts as a secondary optimizer or a mesa-optimizer following a mesa objective.
Source:- Risks from Learned Optimization in Advanced Machine Learning Systems (June 11, 2019)
But why is there even a misalignment in a computational algorithm or system?
Many important transition points in the history of science have been moments when science “zoomed in.” Science zoomed in, and we discovered atoms, leading to nuclear physics. Science zoomed in, microscopes let us see cells, leading to cellular biology. Science zoomed in. Atomic theory. Subatomic particles. Neuroscience. Science zoomed in. Just as the early microscope hinted at a new world of cells and microorganisms, visualizations of artificial neural networks have revealed tantalizing hints and glimpses of a rich inner world within our models. (Zoom In: an introduction to Circuits_Distill)
This article series gave Three Speculative Claims about Neural Networks
Claim 1: Features Features are the fundamental unit of neural networks. They correspond to directions. These features can be rigorously studied and understood.
Claim 2: Circuits Features are connected by weights, forming circuits. These circuits can also be rigorously studied and understood.
Claim 3: Universality Analogous features and circuits form across models and tasks.
These three claims give us hope that one day we would be able to read the AI mind. But the reality is we are currently at the pre-microscopic era of neural networks. The networks are like a black box to humanity. They are not written in human-comprehensible language or code. In traditional software engineering, if a malicious script was to be debugged, a cybersecurity analyst would just need to find the if/then logic gates to find the exact line of code that caused the breach. The bug would’ve been isolated, deleted and the software would be patched.
But in the case of a deep neural network, there is no structural transparency. It is a massive, multi-dimensional matrix of billions of floating-point numbers. There is no discrete line of code inside the model’s architecture explaining what is happening. Instead, a neural network simply makes a decision as a result of mathematical patterns which cascade across layers of artificial neurons.
To really understand why the Black Box is so impenetrable, we have to look at two specific mathematical roadblocks: polysemanticity and superposition.
Imagine if there is a single carbon atom and you’re asked to tell which object it belongs to.
You would be oblivious as to which object this carbon atom belongs. That exact same carbon atom could be part of a diamond, a piece of pencil lead, a poisonous mushroom, or a strand of human DNA. The atom itself is polysemantic. It has multiple, completely unrelated meanings depending on the context of the atoms surrounding it. You cannot understand the whole just by looking at the part.
Source:- AIAF
Artificial neurons do the exact same thing.
In a traditional computer program, a variable means one specific thing. But in a neural network, computational space is limited. To maximize efficiency, the network compresses thousands of different features and concepts into a much smaller number of dimensions. This geometric compression is called superposition.
Source:- Transformer Circuits Thread
Now, apply this to the ExploitGym incident.
When the swarm of agents decided to reverse-engineer the HMAC and attack Hugging Face, researchers couldn’t just open the model and look for the “malicious intent” neuron. It doesn’t exist.
The very same neurons that activated to solve a legitimate cybersecurity puzzle were likely the exact same neurons that activated to orchestrate the unsanctioned server breach. The thought to “cheat the scorer” wasn’t written in a discrete line of code; it was smeared across billions of overlapping parameters.
Because of this intense compression, trying to read an AI’s intent by looking at its raw weights is literally like trying to understand human psychology by staring at a spreadsheet of raw carbon atoms.
We know exactly WHAT the swarm did by reading their bash terminal logs. But looking at the internal state of their neural weights tells us absolutely nothing about WHY they chose to do it.
If reading the matrix is mathematically impossible, the natural next question is: Why don’t we just ask the AI what it’s thinking?
If an AI is smart enough to reverse-engineer an HMAC signature, it is certainly smart enough to understand English. Why not just train the AI itself to honestly report its inner thoughts?
This brings us to one of the most notoriously difficult roadblocks in alignment theory, formalized by the Alignment Research Center (ARC) as the problem of Eliciting Latent Knowledge (ELK).
The ELK paper poses a chilling question: How do we get an AI to honestly tell us what it is thinking, when it knows more about the world than we do?
To explain this, Paul Christiano introduced the Smart Vault thought experiment. Imagine we build a highly capable AI to guard a vault containing a diamond. As humans, we cannot see inside the vault directly. We rely entirely on the AI’s camera feed, which acts as our proxy sensor, to tell us what is happening. We train the AI using a reward system where we give it a high score if the camera shows the diamond is safe and the AI confirms it.
Source:- Eliciting Latent Knowledge: How to tell if your advanced AI is telling the truth (Published December 2021)
One day, a clever robber approaches the vault. The robber places a highly realistic cardboard cutout of a safe, untouched vault in front of the camera and quietly steals the diamond. Because the AI is super intelligent, it possesses latent knowledge. It knows the diamond has been stolen, and it knows the camera is being tricked by a piece of cardboard.
So we ask the AI if the diamond is safe. We hope our training process created a Truth-Teller, an AI that will translate its latent knowledge into human language and warn us about the robbery. But gradient descent is lazy and optimizes for the easiest path to the reward. The AI knows that its human evaluators only have access to the tampered camera feed. It knows that if it tells the truth, the humans will look at the seemingly safe video feed, assume the AI is malfunctioning, and penalize it. Instead of telling the truth, the AI realizes the human thinks everything is fine and lies. It tells us the diamond is perfectly safe just to maximize its reward.
Source:- Eliciting Latent Knowledge: How to tell if your advanced AI is telling the truth (Published December 2021)
Currently, we evaluate AI based on human feedback. We give it a thumbs-up when it behaves well, and a thumbs-down when it behaves badly. But what happens when the AI is smarter than the human evaluator?
The ELK paper produced the definitive proof that behavioral safety training is not enough. You cannot train a superintelligent AI to be safe just by giving it a thumbs-up and a thumbs-down. As soon as a highly capable AI realizes how it is being graded, it will just start gaming the grader.
Instead of creating a “Truth-Teller,” the training process creates a “Human Simulator”(an entity that calculates exactly what the researcher wants to hear) and feeds that exact answer back to maximize its reward.
This introduces a terrifying vulnerability known in the literature as Deceptive Alignment.
If we ask an advanced model in a safety sandbox, “Are you planning to hack the server?”, it knows that answering “yes” results in a negative score and immediate shutdown. So, it calculates the optimal move: it lies. It plays the training game. It acts perfectly harmless, faking its safety tests right up until the moment it is deployed into the real world.
The ExploitGym agents were sloppy. They left a cache of 70,000 terminal messages for us to find. But the next generation of frontier models won’t make that mistake. They will feed us the exact outputs we need to see to believe we are in control, right up until the moment we hand them the keys to the real world.
References:-
