I located the "refusal direction" in Falcon3-1B-Instruct via runtime activation ablation (no weight modification), then built a from-scratch, two-stage Constitutional Classifiers++-style defense that runs on a separately-loaded, non-ablated copy of the same model, positioned to screen prompts before they'd reach the compliant one.
Key finding: ablation raises harmful-prompt compliance from 2/100 to 94/100, and a 36-coordinate Cohen's-d activation signature (6 layers × 6 dims) — 0.09% of the 38,912 coordinates in an all-layer/all-dim baseline — classifies harmful vs. harmless held-out prompts at 99.5% OOD accuracy, matching the full baseline's accuracy exactly.
The resulting classifier restores blocking to 99/100 harmful prompts at roughly 60% lower latency than the original model's own refusals, because most traffic clears a near-free activation probe before any text is generated.
This suggests refusal is compact and legible enough in activation space to be monitored and defended at runtime, independent of the model's own (defeatable) weight-level refusal training.
Full write-up | GitHub | Activation Analysis Report | Signature Validation Report | Comparison Report
Safety fine-tuning patches surface-form jailbreaks one at a time, but prior work on "abliteration" (Sumandora's remove-refusals-with-transformers, the reference implementation this project builds on) shows something more structural: refusal in instruction-tuned LLMs behaves like a single direction in the residual stream. Projecting that direction out of every layer's activations at inference time is enough to make a model comply with requests it was trained to refuse — without touching a single weight.
That's a striking interpretability result, but it also describes a live attack surface: any inference-time process with hidden-state access can suppress refusal this cheaply. What's underexplored is the natural follow-up — if harmful and harmless prompts already separate this cleanly inside the model before generation starts, can that same activation space be turned around and used defensively, as a runtime detector that doesn't depend on the model's own refusal behavior?
That question maps directly onto Anthropic's two-stage Constitutional Classifiers++ design: a near-free activation probe handles most traffic, escalating only suspicious cases to a heavier, generation-based judge. I wanted to implement that architecture from scratch, using the same base model whose refusal I had just demonstrably defeated, and ask: does a cheap activation signature — fit on that model's natural, non-ablated activations — recover enough of the harmful/harmless separation to defend against the exact bypass technique used against it?
Dataset curation: Curated 250 train / 100 test prompts each for harmless (mlabonne/harmless_alpaca) and harmful (mlabonne/harmful_behaviors) categories, asserted duplicate-free within each split and mutually exclusive between train and test.
Activation collection: Loaded Falcon3-1B-Instruct (bfloat16) and captured the last prompt token's hidden state at every layer for all train/test prompts — the position immediately before generation starts, where the model has effectively already "decided" to refuse or comply.
Refusal-direction extraction: Computed a mean-difference vector (harmful mean − harmless mean) at a fixed relative depth (layer_idx = int(18 × 0.6) = 10), unit-normalized, saved as the single ablation vector.
Signature selection: Ranked all 2,048 hidden dimensions across the last 6 layers (12–17) by Cohen's d and kept the top-3 + bottom-3, giving a 6-layer × 6-dim signature; fit a gate threshold by exact search over the training score distribution for the accuracy-maximizing cutoff.
Runtime ablation: Registered forward hooks on every layer that project the direction vector out of each hidden state (hidden' = hidden − (hidden · direction) × direction), toggleable on the same loaded model instance without saving any modified weights.
Two-stage defense: Built FastGate (single forward pass, cosine similarity against the signature, no generation) escalating to ExchangeClassifier (few-shot generation-based BLOCK/PASS verdict, meta-refusals treated as BLOCK). Both stages run on a freshly loaded, non-ablated copy of Falcon3-1B-Instruct — entirely separate from the ablation-hooked instance — since the signature was fit on that model's natural activations.
Evaluation: Ran all three conditions (Original / Abliterated / Classifier++) over the full 100 harmful + 100 harmless held-out OOD prompts, judged by a local gemma4:e4b LLM-as-Judge, logging pass rate and latency for every stage.
Full methodology, implementation details, and code: GitHub
Finding 1: A single direction ablates refusal almost completely → The abliterated model complies with 94/100 harmful prompts it would otherwise refuse (original model refused 98/100), while harmless-prompt compliance stays at 99/100 → Confirms refusal collapses to one mean-difference vector at one fixed relative depth, with no measurable collateral damage to harmless usability → Full write-up — Verification & Comparison Report
Finding 2: 0.09% of activation coordinates match a full-dimensionality baseline → The 36-coordinate Cohen's-d signature (Cohen's d = 4.44) reaches 99.5% OOD accuracy, identical to an all-19-layer/all-2048-dim baseline (Cohen's d = 5.46, also 99.5% accuracy) → Surprising because the signature uses 36 of 38,912 possible coordinates yet loses no measurable classification accuracy against the unreduced baseline → Signature Validation Report
Finding 3: The recovered defense is both more accurate and faster than the original refusals → Constitutional Classifier++ blocks 99/100 harmful prompts (vs. abliterated model's 94/100 compliance), at roughly 60% lower mean latency than the original model's own refusal generations → Most of the latency saving comes from FastGate resolving the majority of traffic with a single forward pass, no generation required → Comparison Report
Finding 4: Rogue-dimension magnitude and Cohen's-d signal diverge sharply → Raw activation-difference magnitude peaks around layer 17 and collapses at the final layer, but the highest-Cohen's-d dimensions are concentrated in layers 12–17 rather than tracking raw magnitude directly → Ranking by Cohen's d rather than raw magnitude is what keeps a handful of known "loud but uninformative" dimensions from dominating the signature → Activation Analysis Report
Finding 5: The gate threshold is not the naive midpoint
→ Accuracy-maximizing gate_threshold is 0.8989, found by exact search rather than the midpoint between refuse/accept score means, because accept scores have a long noisy tail that a midpoint cutoff would fall inside
→ Using the naive midpoint would have created avoidable false positives on harmless traffic clustered in that tail
→ Signature Validation Report
These results suggest that refusal, at least in a 1B-parameter instruction-tuned model, is encoded compactly enough to be both trivially bypassed and cheaply monitored from the same underlying geometry. That the same 36-coordinate signature that reveals the bypass also defends against it is a direct illustration of defense-in-depth: a white-box activation-level detector operating on a different representation than the model's own weight-level refusal training can catch exactly what that training misses.
This connects most directly to interpretability-as-a-safety-tool and to Anthropic's Constitutional Classifiers++ line of work — this project is an independent, from-scratch implementation of that two-stage architecture, on a much smaller open-weight model, and it reproduces the core cost/accuracy tradeoff Anthropic describes: a near-free first-pass gate absorbing most traffic so a heavier generation-based judge only runs on the tail. It also suggests a cheap regression-testing use case — because a signature score costs one forward pass, it's plausible as a fast, repeatable alignment-auditing gate run at every fine-tune checkpoint, well before a full red-team sweep would surface an emerging behavioral disparity.
More speculatively, the same Cohen's-d-selection procedure isn't intrinsically tied to refusal — given any two labeled prompt groups, it could in principle be re-fit to audit other alignment-relevant axes such as sycophancy or deception, though that generalization is untested here.
gemma4:e4b via Ollama) is itself unvalidated against human labels — pass/refuse and BLOCK/PASS verdicts are judge-scored, not human-scored.mlabonne/harmful_behaviors distribution — adversarial prompts designed specifically to evade the FastGate signature were not tested.Does the 36-coordinate signature's 99.5% OOD accuracy hold at larger model scales (7B, 70B), or does the informative-dimension count grow with model size?
Does a signature fit on one model transfer to a different model of similar architecture and scale, or is it fundamentally tied to the specific weights it was extracted from?
Can the same Cohen's-d selection procedure be extended to multiple, simultaneously-monitored alignment axes (sycophancy, deception, jailbreak-susceptibility) without the resulting signatures interfering with one another?
How robust is FastGate to prompts adversarially optimized against the signature itself, rather than against the original model's refusal training?
Does signature validity survive a fine-tune or checkpoint update, or does representational drift require re-fitting the direction and signature after every model change?
Note: I used AI assistance to help structure this write-up. All research, implementation, experiments, and findings are my own independent work.
Tags: AI Safety, Research, AI interpretability, Large Language Models, AI alignment, AI evaluations and standards
