Epistemic status
The argument that follows is based on empirical work published between 2023 and 2025. Most of that work consists of cross-sectional lab or field studies, typically involving samples of a few dozen to a few hundred participants and follow-up periods measured in hours or days. A handful of longer surveys spanning roughly six months are also presented, but true longitudinal evidence remains scarce. Effect sizes (Cohen’s d, odds ratios, or regression βs) should therefore be taken as order-of-magnitude signals rather than precise forecasts: study designs differ, interventions are heterogeneous, and publication bias almost certainly nudges the headline numbers upward.
External validity is another open question. The best-documented cases come from medicine, education, and software engineering in high-income countries; whether the same magnitudes hold for other domains, cultures, or resource settings is an assumption, not a fact. Causal language is used sparingly: where mediation analyses or natural experiments justify it, we say so; elsewhere, directionality remains suggestive. Finally, our own prior is that high-accuracy automation does erode human error-catching skill, but we’re uncertain about the rate and expect careful tool design to mitigate at least part of the effect.
Why this post exists
In the opening essay, we argued that once people offload enough cognitive effort to an apparently helpful system, their own skills can atrophy without anyone noticing, an effect we called a slow “boiled-frog” decline. We listed four human capacities most exposed to that risk: reasoning, agency, creativity and social bonds. This second instalment focuses on the first of those capacities—reasoning. The central claim is simple: even when an AI assistant delivers answers that are overwhelmingly correct, the act of leaning on those answers changes how people think. Each time the tool supplies a ready-made answer, the marginal value of double-checking falls a little, and the small habit of catching one’s own errors weakens. By tracing that slide—mechanistically and empirically—we set the stage for the rest of the series, which will tackle the downstream losses in agency and creativity that follow once the reasoning muscle has been left to weaken.
Before we state the factual base for the decline, we need a clear picture of what actually counts as “reasoning.”
What counts as reasoning
Reasoning, as used in this series, is the ensemble of mental operations that let a person (i) draw conclusions from evidence, (ii) notice when those conclusions may be wrong, (iii) imagine how the world could be different, and (iv) keep their confidence tethered to reality. We’ll break it down to provide a clearer explanation.
(i) Inference – deduction, induction
Inference is the deliberate act of drawing a justified conclusion from data, observations, or statements you hold true. It comprises two basic moves:
- Deduction — A deductive step is valid when the conclusion is logically entailed by the premises, i.e., it cannot be false if the premises are true. The familiar syllogism “All humans are mortal; Socrates is human; therefore Socrates is mortal” illustrates the form.
- Induction — An inductive step projects beyond the premises, offering a conclusion that is probable rather than guaranteed. The premises give the conclusion some degree of evidential support—stronger samples, for instance, yield stronger support when we infer from observed cases to unobserved ones.
Some authors add abduction as a third mode*,* but for present purposes, deduction and induction are enough for the argument’s sake.
Metacognition is often glossed as “thinking about thinking,” but the recent synthesis by Fleur, Bredeweg & van den Bos (2021) makes the definition more precise: it is both the capacity to be aware of one’s ongoing cognitive processes (metacognitive knowledge) and ****the capacity to regulate those processes in light of that awareness (metacognitive control) (nature.com)
- Metacognitive knowledge (monitoring) — Moment-to-moment insight into the state of your own cognition, e.g, how confident you are in an answer, whether the last paragraph actually sank in, which steps of a proof still feel shaky. Laboratory work measures it with confidence ratings, judgments-of-learning, and other “how sure are you?” probes; higher accuracy in these self-assessments predicts better learning outcomes
- Metacognitive control — The adjustments you make in response to that insight—slowing down, rereading, opening a reference, or choosing to skip a problem until you know more. Researchers track control by asking whether adjustments such as extra study time, strategy switches, actually improve later performance.
Together, knowledge and control form an internal feedback loop: the former tells you where your reasoning stands; the latter lets you correct course. High-quality reasoning depends on both parts working in harmony
(iii) Counterfactual search – imagining alternatives
Counterfactual search is the mental act of positing “what-if” scenarios: If the traffic light had stayed green, I would have caught the train. Philosophers analyse such statements as subjunctive conditionals whose truth hinges on how similar an imagined world is to the actual one. On Judea Pearl’s “ladder of causation,” that operation sits on the third and highest rung. Pearl’s framework shows why they matter for reasoning: counterfactuals are the only queries that let us test competing causal models and choose actions that would work in hindsight.
Psychology shows that people generate such alternatives spontaneously, especially after surprising or negative results, and that it serves several functions: learning and planning, emotional regulation, behavioural adjustment, and neural support. We’ll briefly outline some research behind the two functions most relevant to our concern:
- Learning and planning. A review in the Annual Review of Psychology concludes that counterfactuals help reasoners extract causal rules from past events and apply them to future choices—e.g., “Had I left earlier, I’d have caught the train; next time I’ll leave earlier.” (source)
- Neural support. fMRI work links upward and downward counterfactual simulation to a fronto-parietal network that overlaps with episodic memory and future-planning circuits, underscoring its role in projecting beyond the present. (source)
In line with Pearl’s causal-ladder framework, this evidence shows that counterfactual search is not a philosophical curiosity but a working part of human reasoning: it helps us learn causal structure, adjust behaviour, regulate emotion, and engage the neural circuits used for planning and imagining future events.
(iv) Epistemic virtues in practice
Sound reasoning is not only a matter of having the right algorithms; it also depends on relatively stable character traits that guide how a person handles evidence. Those traits become visible through habitual actions in everyday reasoning. Two of the most studied are calibration and intellectual humility.
Calibration is the habit of matching confidence to reality. Clinical research shows that poorly calibrated risk judgments lead to systematic misdecisions in domains from medicine to finance (source).
Intellectual humility is the disposition to recognise the limits of one’s knowledge and to stay open to revision. Longitudinal studies link the trait to deeper learning and more accurate social judgments (source).
Additional epistemic virtues belong to this adjacent family. Epistemic diligence (sometimes called conscientiousness) sustains evidence search when data are sparse; open-mindedness keeps alternative hypotheses alive long enough to be tested; epistemic courage supports considering unpopular but well-supported views. Virtue theorists treat these excellences as capabilities that persist across situations, shaping the overall reliability of a thinker rather than any single inference step.
Together, calibration, humility, and their allied virtues form the motivational backbone of reasoning: they decide when we pause to check a premise, how seriously we treat counter-evidence, and how quickly we update a belief when the facts change.
Underlying all these virtues is the skill of building a meta-model. The next part unpacks the skills at risk.
Reasoning parts at risk
Beyond the core acts of inference and checking, productive reasoning depends on a set of meta-level habits.
When unstructured notes are passed to an AI, the system produces summary bullets that often misrepresent complex events, because it arranges the text rather than the underlying reality; human structuring, by contrast, invents organising criteria at a meta-model level and therefore yields a truer, more fully understood outline. Research indicates that participants who read LLM-generated bullet summaries believed they understood the source article better, but objective recall and causal reasoning scores did not improve (source). The AI coherently rearranged sentences yet skipped deeper relations, creating a false sense of comprehension and masking gaps a human outline would have exposed. This phenomenon, known as the illusion of explanatory depth, leads individuals to prematurely terminate their search for arguments and impairs reasoning quality. This behavior is also a manifestation of cognitive offloading, where the delegation of complex tasks, such as data analysis, to AI tools can reduce critical thinking by providing ready-made solutions that lower the "deliberation costs" (source). Because search feels effortless, automation bias encourages cognitive offloading during query formulation: people let the system decide which facts matter and rarely debug its answers, so early hallucinations snowball into flawed follow-up questions and mistaken conclusions.
AI queries often return mediocre results; output quality tracks the quality of the meta-model implicit in the prompt, yet the convenience of one-click search discourages the preparatory modelling and question-refinement that would filter out low-value queries. AI-generated content still has errors or "hallucinations”; for instance, a benchmark across 4,000 prompts found hallucination rates of 28.6 % for GPT-4 ****(and higher for other models) when citing evidence for medical reviews. For fresher models, the situation is not great too: o3's hallucination rate is 33%, and o4-mini's hallucination rate is 48% (source). The prevalence of errors or "hallucinations" in AI-generated content means that users frequently encounter plausible but incorrect facts, thereby undermining the deliberate query-refinement process typical of conventional search (source). This reliance is largely driven by automation bias, an excessive trust in AI recommendations that reduces human vigilance in information seeking and processing (source). Users tend to offload control and reduce their critical analysis, especially when the task complexity is high or their subjective cognitive load is low. Consequently, people relying on AI for analysis or information may draw false conclusions or make incorrect decisions due to deceptive or erroneous information provided by the AI. This can result in a fragmented and erroneous collective human reasoning capacity.
AI systems habitually endorse their own outputs and lack real-world grounding, which reduces independent verification and, in the absence of an explicit meta-model, weakens systematic appraisal. This effect is also amplified by ****automation bias (AB). Users tend to offload control and reduce their critical analysis, especially when the task complexity is high or their subjective cognitive load is low. Consequently, people relying on AI for analysis or information may draw false conclusions or make incorrect decisions due to deceptive or erroneous information provided by the AI. Research shows a strong negative correlation between frequent AI use, cognitive offloading, and critical thinking scores (r = -0.75) (source). The availability of ready-made solutions from AI reduces "deliberation costs," subsequently weakening the motivation for thorough verification. This shift in cognitive resources from active information processing to passive monitoring is particularly problematic when the AI itself is biased, as users lose vigilance in identifying and correcting discriminatory patterns. Automation bias manifests even among experts and is not easily mitigated by simple training or instructions (source). Non-specialists appear more susceptible to these errors.
In five lab studies, people followed erroneous AI recommendations 43 % more often than baseline, and adding model explanations made no difference. The system’s self-endorsing format dulled independent verification, illustrating how LLMs weaken systematic appraisal when no explicit meta-model is supplied. (source)
Furthermore, a survey-experiment found that stronger a priori trust in AI tools like Copilot/GPT predicted lower rates of fact-checking and counter-search (source). While self-rated accuracy climbed, real accuracy fell, confirming that AI's self-endorsing format suppresses independent appraisal.
Constructing any domain model implicitly demands prior reflection on the meta-model that will govern it. Prolonged interaction with algorithms can lead to metacognitive degradation, reducing individuals' ability to accurately assess the quality of their own decisions. Users tend to overestimate the "objectivity" of algorithms and underestimate their own capabilities. This bias increases their willingness to accept biased recommendations, even when information about the algorithm's bias is present. This directly impacts model formation by affecting an individual's metacognition—their understanding and assessment of their own thought processes and knowledge. If users overestimate the "objectivity" of algorithms, they are incorporating algorithmic outputs into their mental models with undue confidence, even when those outputs might be biased. This distorts their internal representation of knowledge and decision-making quality. The Performance and Metacognition Disconnect experiments showed that participants using AI prompts over-evaluated their own accuracy and calibrated confidence less effectively (source). This fundamental shift in self-assessment regarding information processing directly influences how one builds and trusts their internal models of reality.
The iterative interaction between algorithmic bias and human thinking produces long-term effects, where biased AI outputs are filtered and amplified by existing human prejudices. People are more likely to accept algorithmic advice when it aligns with their pre-existing biases (selective adherence), creating an illusion of algorithmic objectivity while actually reinforcing discrimination. This point illustrates a bidirectional feedback loop between algorithmic bias and human thinking, leading to long-term effects (source). People processing algorithmic recommendations through existing cognitive schemas means that biased AI outputs are filtered and amplified by pre-existing human prejudices (source).
In a randomised essay-writing study, ChatGPT users earned higher language scores but showed fragmented self-check loops and skipped planning. Authors call this “metacognitive laziness”: when the bot supplies finished prose, students stop reflecting on the meta-model that organises arguments, so durable domain models never form. (source)
fNIRS data showed that chatbots prompting learners to reflect on their plan boosted dorsolateral-PFC activation and transfer scores, whereas bots that supplied full solutions dampened both. Schema-building thus depends on a prior meta-model reflection step that convenience prompts can remove. (source)
For humans to form a robust object-level model, they implicitly engage with meta-models, establishing criteria for object selection. When one aims for a good model, these different levels can be recorded and controllably modified. However, AI interfaces, particularly chat-based ones, make it difficult to display all levels of a model or edit them separately, hindering iterative refinement and collaborative model building.
Conflict resolution within a model
Limited cross-domain input and curtailed search for creative alternatives restrict the discovery of unexpected connections and dampen innovation. The way AI presents information, such as through detailed expositions, links, and structure, can create an illusion that the information is high-quality and exhaustive, even when it often reflects only the "average". This can lead users to mistakenly believe that the answer is satisfactory and that further search is unnecessary, thus curtailing the search for alternative or more diverse information.
AI can easily induce "functional fixedness," a state where an individual begins solving a task using a particular approach and then fails to see that alternative approaches are available. This happens because AI implicitly creates a "meta-model" or "landscape," setting the boundaries for how a problem is perceived and approached. If a user is unaware that this is happening, they are less likely to attempt to change that landscape, effectively restricting the discovery of unexpected connections and alternative solutions.
Both personalized algorithms and sycophant LLMs further reduce the amount of content that contradicts a user's existing views, creating "epistemic bubbles”. These bubbles significantly limit the informational basis for reasoning by changing how information is understood and processed, not just its volume. This algorithmic bias makes unbiased thinking difficult and hinders access to contradictory data, increasing the risk of systematic errors. Constant confirmation of existing views due to personalized AI content reduces motivation to seek alternative viewpoints (source)
In an essay-writing study, writers given ChatGPT prompts produced stories rated as more creative, yet 17% more similar to each other. By offering the same statistically likely connections, the LLM narrows the search space and makes unexpected cross-domain links, and therefore innovative conflict resolution, less likely to appear. (source)
Why accuracy is not safety
A common rejoinder goes like this: “If my assistant is 99 % correct, why worry? I’m more error-prone than that.” It sounds persuasive, but the picture changes when you consider three additional points: every accepted answer is a micro-decision, volume multiplies even small error rates; vigilance collapses when the tool looks flawless..
Volume turns tiny error rates into a steady trickle of bad calls. At 99 % accuracy, the miss rate is 1 %. A worker who relies on an LLM assistant for about 300 micro-decisions a day will meet three wrong answers every day. Bump the load to 1,000 queries, and the tally reaches ten. Multiply further across a ten-person team or a company funneling thousands of requests through the model, and the absolute number of silent missteps scales linearly.
Accepting an AI summary, label, or ranking is itself a commitment: the summary drives what gets followed up, the label routes a task, the ranking shapes a budget. In other words, knowledge work is decision work in miniature, and those micro-decisions cascade into larger choices later on.
Vigilance falls as accuracy rises. Lab data show that people check less when the system is usually right. In the aforementioned wound-care decision-support studies, omission and commission errors climbed 12–17 % precisely because the display showed more than 90% confidence, hence the staff stopped double-checking. A software-debugging experiment echoed the pattern: engineers caught 64 % of bugs when told a model’s past hit-rate was 85 %, but only 38 % when told it was 98 %.
All of it wouldn’t call for attention if not for the fact that errors concentrate in the hardest, least transparent cases. Model accuracy is not uniform; the residual 1 % skews toward edge-cases, sparse data, or adversarial inputs, i.e, the very spots where human oversight matters most. Accepting a wrong answer cements it as a premise for future queries, creating what we can call “latent-error ladder”: initial acceptance leads to downstream decisions that inherit the flaw, making later detection harder.
Safety is thus a product of accuracy × vigilance; If accuracy rises while vigilance sinks, the human-AI partnership is no safer than before, and, because fewer checks are happening, it becomes harder to spot or trace the errors that still slip through.
Yet, low vigilance does more than let an occasional error slip; it steadily transfers the work of thinking to the machine. Over weeks and months, the skipped steps of framing a question or weighing evidence erode the very skills that let a mind map new domains, spot hidden links, and repair its own misconceptions. The danger, then, is not just a future crisis when the system fails; it is a quieter drift toward a population that can no longer form deep models of the world unaided, and so has less to contribute when new problems demand original reasoning.
For the sake of clarity, there is an alternative reading of our data: high-accuracy automation might simply reallocate cognitive effort rather than erode it. Once routine checks are off-loaded, users, driven by a baseline need for cognition, could invest the freed bandwidth in more complex, higher-level problems. Yet we still lack longitudinal or field studies that measure where saved effort is spent, so the displacement hypothesis remains an open question rather than a counter-fact.
That leaves a practical question: how do we build tools that keep the user’s self-check loop alive?
Early design levers
The usability community has learned that a few grams of friction can keep the self-check loop alive without crippling productivity. One option is to slow the hand-off just enough to prompt a micro-reflection: an extra click to unblur the answer, or a short text box that asks the user to jot down their own guess before the model’s suggestion appears. Healthcare studies that tested a one-second reveal delay or a mandatory “enter your diagnosis first” field found that verification rates climbed while overall task time barely moved (source). A parallel line of work in misinformation research shows that forcing a moment of reflection, users rate their confidence or write a one-sentence justification before seeing the AI verdict, cuts blind acceptance roughly in half (source).
These gentle “speed bumps” do not solve automation bias on their own, but they illustrate a broader point: interface tweaks can raise vigilance even when the underlying model is already highly accurate. In later posts, we’ll explore these and other aspects in more detail, from graded uncertainty bands to staggered disclosure of citations, and map each lever onto the specific failure modes it can mitigate.
Where the series goes next
With reasoning decay on the table, the series now turns to the second pillar: agency. The next post will examine how high-accuracy automation can quietly shift the locus of control: first soft suggestions, then default actions, and finally locked-in workflows, and why that matters for long-term alignment between human goals and machine behaviour.
Executive summary: This evidence-informed, cautiously speculative post argues that even highly accurate AI systems can degrade human reasoning over time by weakening inference, metacognition, and other key components of thought—an effect driven not by obvious errors but by subtle shifts in how people offload, verify, and internalize information.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.