AI disclosure: The concepts were developed jointly by Mark and ChatGPT on 18 November 2026. This post was substantially researched and drafted using ChatGPT and connected retrieval tools under Mark Price's direction. It presents a conditional research proposal, with no experiment or results claimed. It has not been independently peer reviewed or human fact-checked.
Suppose an institution tests an AI on the responsibilities it plans to delegate. The answers are strong, demonstrations succeed, and explanations sound appropriate. Managers expand its authority. Other institutions use similar systems. Over time, independent assessment and the ability to operate without them become harder to maintain.
The concern is not simply that the system might make a mistake. It is that the evidence supporting reliance may leave unresolved a competence gap whose consequences become much larger as reliance grows.
I propose F.U.N.N.Y. — Fabricated Understanding in Neural Network Yields as a name for that problem. “Yields” means outputs or results. The name should stand or fall on whether it helps specify a useful research question.
The defining scenario is deliberately severe: an AI displaying fabricated understanding and an AI that actually understands are indistinguishable through the evidence available before decisive catastrophe. In this strict case, the terminal event permits no timely, reliable human diagnosis or designation. If that first decisive difference manifests through human extinction, nobody survives to identify the difference, designate the apparent understanding as fabricated, or establish what happened or why. The difference remains forever unresolved by humanity.
N.O.T. F.U.N.N.Y. — Nobody Outlived The Fabricated Understanding in Neural Network Yields names that secondary outcome. F.U.N.N.Y. remains the main concept: the evidence problem while investigation and prevention are possible.
This scenario is an assumption to examine, not evidence that all AI lacks understanding. Complete pre-catastrophe indistinguishability also does not, by itself, establish any probability of catastrophe. A causal argument needs more.
“Fabricated” need not mean conscious deception. A system could produce persuasive outputs without possessing some competence its users infer from them. Conversely, a system might understand a task and deliberately behave badly. Those are different mechanisms.
There are concrete reasons to scrutinise inferences from performance. D’Amour and colleagues show that models with similar conventional test performance can differ on deployment-relevant stress tests. Their work on underspecification supplies an empirical analogue of evidence leaving consequential properties unresolved.
McCoy, Pavlick and Linzen’s HANS work shows how correct natural-language-inference answers can be consistent with heuristics that fail carefully chosen counterexamples. Turpin and colleagues demonstrate cases where tested models’ explanations omit prompt features that influence their answers in chain-of-thought experiments.
These studies also constrain the argument. Their investigators exposed differences before an extinction event. They support better discrimination and targeted evaluation, rather than proving discrimination impossible.
There is positive evidence against a blanket “surface imitation only” claim. Li and colleagues find internal world representations with a causal role in a synthetic Othello model. That does not establish dependable competence for civilisation-level responsibilities, but it gives evidence beyond persuasive self-description.
The practical question is therefore responsibility-specific: which consequential capabilities are required, and what evidence establishes their reliability?
A local model failure is not an extinction pathway. Even widespread institutional disruption is not automatically extinction.
A candidate pathway would need to specify at least the following: the unresolved competence gap; why available evaluation misses it; which responsibilities are delegated; how failures propagate across shared dependencies; why independent checks and intervention fail; and why survival or recovery becomes impossible.
The final step carries a particularly heavy burden. A proposed global cascade must explain why geographic diversity, different institutions, residual human expertise, offline systems and surviving refuges do not interrupt it. Calling a system “critical infrastructure” cannot substitute for that explanation.
My current framing identifies a possible assurance failure and a research agenda. It does not establish a complete, probable extinction mechanism. Existing work on gradual disempowerment already examines how incremental automation could erode human influence; permanent disempowerment is broader than literal extinction.
The strict terminal scenario cannot be investigated by waiting to observe its outcome. Safer approximations can test parts of the progression.
Consider simulated institutions allocating authority to AI advisers. Keep the quality of visible demonstrations comparable while independently measuring held-out, task-relevant competence. Vary explanation polish, independent audit access, shared model dependence and the cost of retaining a human fallback.
Measure whether decision-makers expand authority beyond what the independent evidence warrants, whether audits expose gaps before expansion, and whether institutions preserve a workable fallback. Test the intervention under unfamiliar conditions, rather than only the examples used to design it.
That experiment would not reproduce extinction. It could support or weaken claims about the social mechanism connecting persuasive performance to dangerous dependence.
A separate systems study could examine how genuinely independent redundancy changes correlated failures and recovery time. “Several models” should not count as independent redundancy without examining common data, infrastructure and failure modes.
I would want the argument revised if responsibility-specific evaluations consistently revealed the relevant gaps before delegation, if institutions retained effective overrides despite dependence, or if proposed cascades were contained by realistic diversity and recovery measures.
I would also welcome evidence that the terminology obscures more than it clarifies. “Understanding” may bundle multiple properties that should be measured separately. If “unresolved deployment competence” expresses the useful concern more precisely, that is a substantive criticism.
Existing AI safety-case work already organises evidence around explicit claims and assumptions. F.U.N.N.Y. needs to explain what it adds to that approach and to research on distribution shift, automation reliance and systemic risk. New names do not make the underlying concerns unprecedented.
The contribution I most want is a precise objection or discriminating test:
The prevention question is actionable even under uncertainty: can independent verification, bounded authority and preserved recovery capacity prevent an unresolved gap from becoming terminal?
Working paper, sources and revisions are available for readers who want more detail. The purpose of this post is to discover whether the question survives criticism, and how to make it worth investigating.