I Ran a Causal Test on Sparse Autoencoders: 77% of "Recovered" Features Were Causally Inert

Epistemic status: empirical, fully reproducible (34 tests, seeded RNG,
zero external model downloads). Toy-model setting with known ground
truth — not a claim about production SAEs, but a controlled test of a
methodology those SAEs rely on.

Cosine similarity between an SAE dictionary atom and a ground-truth
feature direction is the standard evidence used to claim a feature was
"recovered." This is a correlational claim about geometry. It is not a
causal claim about whether that atom's activation actually drives the
behavior attributed to the feature.

I built a from-scratch reproduction of Toy Models of Superposition
(Elhage et al., 2022), trained both L1 and TopK sparse autoencoders on
its activations (multi-seed, 3 seeds per config), and then ran direct
causal interventions (ablation + steering) on every well-represented
feature to check whether correlational recovery predicted causal
specificity.

Result: correlation exists (r=0.657) but diverges sharply in practice.
For a deliberately weaker SAE configuration, 17 of 22 features that
"matched" a ground-truth feature by cosine similarity — including one
at 0.92, comfortably above the standard 0.90 "recovered" threshold —
had a component that never once activated when the corresponding
feature was actually present in isolation. Even the best configuration
tested (TopK, L0=4, precision=1.00) showed this failure mode at a 9%
rate (2 of 22).

Code: https://github.com/mohamed-bal/superposition-to-monosemanticity

Full write-up (theory, math, all plots, production-relevance discussion)
at the link above. Feedback — especially critical — very welcome.