180,000 Reports, Not One Mention of AI

Frontier AI is entering hospitals and public health systems faster than it can be regulated, and the incident systems built to catch harm cannot yet see it

In a report published in June 2026, the US Department of Veterans Affairs Office of Inspector General (VA OIG) described searching the VA’s Joint Patient Safety Reporting system for any mention of artificial intelligence (AI). The system receives roughly 180,000 submissions a year. The VA OIG found none. Yet clinicians were using AI. The same report describes VA staff using VA GPT and Microsoft 365 Copilot Chat to draft clinical notes and summaries. The VA had no AI-specific reporting mechanism and no way of labelling records written with AI. As the VA OIG concluded, “without a way to tag or trace AI‑generated documentation, VA cannot readily detect patterns, investigate AI‑related safety events, or implement quality improvement processes.”

This piece argues three things. AI is being integrated into health systems faster than regulators can respond. That integration is fragmented across vendors, tools and institutions. And there is no uniform, streamlined escalation pathway for AI-related incidents. Some reporting routes do exist, but they are scattered and voluntary. Because each collects different information in different ways, their data is very hard to combine, standardise or compare.

Thanks for reading Sanchita's Substack! Subscribe for free to receive new posts and support my work.

Where adoption is running ahead of the rules

The pattern appears across countries and languages.

In the United States, The New York Times reported in September 2026 that the administration is “accelerating efforts to make artificial intelligence an integral part of medical care”, directing federal resources into projects that use AI agents to diagnose and prescribe. Medicare has allowed over 200 companies to run pilots that can include AI. Some officials inside the Department of Health and Human Services (HHS) worry this is moving too fast, with too little safety evidence. The Food and Drug Administration (FDA) has approved more than 1,500 AI-enabled devices for specific tasks, but has no rules built for chatbots or agentic AI that do a doctor’s work, such as prescribing.

In China, hospitals adopted the large language model (LLM) DeepSeek within weeks of its release. A 2025 perspective in JAMA asked whether this adoption was “too fast, too soon”. A scoping review in the Journal of Medical Internet Research found that 48 of China’s top 100 hospitals had disclosed DeepSeek deployments. The first came on 10 February 2025, less than a month after the model’s release. Yet only 36% of disclosed deployments clearly indicated a pre-deployment assessment, and only 22% reported its results. Chinese regulators have drawn some lines. The National Health Commission (NHC) has banned AI-generated prescriptions in internet diagnosis since 2022, and Hunan province restated this ban in February 2025.

In Germany, researchers at TU Dresden reviewed the risks of clinical LLMs in a Nature article reported in August 2026. They warned that “only a small proportion of the AI systems currently in use have so far been formally approved as medical devices”, and that informal “shadow use” of chatbots in clinics sits “outside any established oversight structure” (Elektroniknet, translated). In Spain, an official from the Ministry of Health’s digital health secretariat noted that the medical device rules were designed “for physical devices, which do not change once they are finished; software is constantly changing” (El Español, translated). In Brazil, 18% of hospitals used AI last year according to the TIC Saúde survey, but the health regulator Anvisa still has no established rule on it (O Globo, translated).

Regulators are also moving their own deadlines. The European Union (EU) has delayed the AI Act’s high-risk rules for AI in medical devices by a year, to 2 August 2028. In the United Kingdom, the Medicines and Healthcare products Regulatory Agency (MHRA) confirmed in July 2026 that AI scribes used only to transcribe, summarise or draft letters are not medical devices. Checking them falls to clinicians and local National Health Service (NHS) governance.

Meanwhile, the tools arrive through software hospitals already use. Epic plans more than 150 AI features in 2026. OpenAI now lets healthcare organisations connect Epic records to ChatGPT for Healthcare.

Fragmented integration, no clear escalation

Procurement shows how fragmented this is. In January 2026, NHS England published a self-certified registry of 19 AI scribe suppliers. It stated that each NHS body would run its own procurement under its own governance processes. That means one national push, but many separate contracts.

National agreements can be thinner still. Rwanda signed a three-year, non-binding memorandum of understanding (MOU) with Anthropic, which includes support for the health ministry’s work on cervical cancer and malaria. An analysis in TechPolicy.Press points out that the MOU triggered no parliamentary, multilateral or civil-society review, because no such review exists for deals of this kind.

In August 2026, ECRI, a patient safety organisation, stated that there is no centralised mechanism tracking how often AI tools produce incorrect outputs or how often those outputs reach patients. ECRI surveyed 124 respondents, mostly quality, safety and risk leaders. Of these, 31% had seen an AI output they believed was wrong in the past year, 35% were unsure, and 9% reported an AI error that reached a patient or affected a care decision. The sample is small and not representative. Still, the size of the “unsure” group matters as much as the error rate.

The Joint Commission and the Coalition for Health AI (CHAI) recommend tracking AI incidents in existing structures and reporting confidentially to Patient Safety Organizations (PSOs). FDA reporting covers regulated devices. But this guidance is voluntary, much generative AI sits outside device rules, and each route records different details, so incidents cannot be pooled or compared.

Photo by Accuray on Unsplash

Why the incident desk cannot yet see AI?

Most hospitals already run incident reporting systems for falls, medication errors, equipment failures and near misses. AI incidents need to be built into these existing systems, but three very real limitations stand in the way:

1. Visibility: Reporters often cannot see AI’s role. In a study of FDA device reports, Handley and colleagues found that reporters may not know whether AI contributed to an event, because the algorithms work “behind the scenes”. Of the reports they reviewed, 34.5% lacked enough information to tell. The VA OIG found the same problem in practice: patient safety managers may file an event by the patient’s outcome and never record that a generative AI tool was involved.

2. Expertise: A March 2026 Duke-Margolis white paper notes that many health systems lack sufficient expertise to anticipate the risks of clinical AI. It recommends placing an AI subject-matter expert in patient safety workflows. Duke Health has added an AI flag to its own reporting system, with an AI reviewer who investigates flagged events.

Why, then, do health systems adopt tools whose risks they cannot fully anticipate? The evidence points to promise and pressure. The promise is real. In an NHS England evaluation of more than 17,000 encounters, AI scribes increased direct patient interaction time by 23.5%. A 2025 study in JAMA Network Open found 20.4% less time spent on notes per appointment. The pressure is also real. The Duke-Margolis paper itself describes a “widespread push to deploy the technologies across clinical environments”, and warns that limited AI expertise could widen the gaps between well-resourced and under-resourced health systems.

Photo by National Cancer Institute on Unsplash

3. Data: Investigators need to know which model version was running, what it was given and whether the vendor changed anything. That information is often missing, for four reasons.

• Models change, sometimes without notice. A widely cited Stanford and UC Berkeley study found that GPT-4’s accuracy on one task fell from 84% to 51% between March and June 2023.
• Research cannot keep pace. A September 2026 preprint found that the average gap between the newest model named in a medical LLM study and that study’s publication grew from about four months in 2023 to about eighteen months in 2026.
• Update rules cover only regulated devices. The FDA’s final guidance on predetermined change control plans (PCCPs) lets makers of AI devices describe planned modifications in advance, so each change does not need a new submission. For AI that is not a device, such as most scribes and general chat tools, I found no rule in the frameworks reviewed here that limits how often a vendor can update a model. Nor did I find one that requires vendors to tell clinicians when they do.
• Notification is voluntary and accountability is unclear. The Joint Commission and CHAI only advise hospitals to set up feedback channels with vendors, so that staff learn of model changes that may require a performance check. Across Europe, only 8% of countries surveyed by the World Health Organization (WHO) Regional Office for Europe have liability standards for AI errors. So when a model update causes harm, it is largely unclear who is liable.

Limitations of what we know

Most of this evidence comes from the United States, the United Kingdom and a few large Chinese hospitals. Much of it comes from vendors, trade press and government releases, and surveys such as ECRI’s rely on self-reporting. Some non-English sources here are news reports of studies rather than the studies themselves. Public health systems in South Asia, Africa and Latin America are poorly covered.

What we need but do not have

I found no country that counts AI-related patient safety incidents, and no register showing which models, and which versions, run in which clinical workflows. We do not know how often vendors update deployed models, or who is told. There is almost no public data on the safety conditions in public AI contracts, or on how many near misses clinicians quietly correct.

Research that is missing or thin

I checked each gap against the literature. Some work exists, but none is closed:

• Contract audits. I could not find a cross-country audit of public sector AI health contracts that checks for monitoring, logging, update notification or incident reporting clauses. The EU’s model clauses for AI procurement offer a benchmark to measure against.
• A shared incident taxonomy. The Organisation for Economic Co-operation and Development (OECD) framework offers general criteria for AI incidents, but there is still no agreed taxonomy built into hospital reporting forms.
• Reviews of hospital incident databases. Handley and colleagues reviewed FDA device reports. The VA OIG ran a keyword search. I found no systematic review of hospital or PSO incident databases that estimates how many unlabelled AI events they already contain.
• Low-resource and multilingual settings. A few real-world studies exist, such as OpenAI and Penda Health’s study in Nairobi covering 39,849 visits. Independent evaluations of frontier and agentic models in public health systems remain rare.
• Clinician vigilance over time. Studies of AI scribes warn about automation bias. I could not find long-term studies of how closely clinicians review AI notes after months of use.

What I expect to happen

I expect the first widely reported AI harms in public health systems to surface through lawsuits, audits and journalists rather than incident reports. This is already starting. In July 2026, a man in Florida sued OpenAI, claiming that ChatGPT’s medical advice kept him from seeking treatment. The VA’s missing AI incidents were found by an inspector, not a reporting system.

I expect the “unsure” group to grow before it shrinks. Errors from scribes and chat assistants look like ordinary documentation errors, and when a clinician corrects a bad note, the near miss disappears.

I am wary that public buyers will lose their main leverage. As AI comes bundled into platforms hospitals already pay for, there is often no separate purchase to attach conditions to.

I also expect a gap of several years in which health systems act, in practice, as the main regulators of the AI they use. The EU’s rules will not apply until 2028, and most generative tools fall outside device rules anyway. Well-resourced systems like Duke will build their own oversight. Many others, including public systems in middle-income countries, may not have the staff to do so.

I really don’t know who will learn to count these incidents first: governments, patient safety organisations or vendors. Whoever does will shape the rules for everyone else. That question is what The Incident Report will keep returning to.

References

Bin Tareaf, R., Al-Rajab, M., & Loucif, S. (2026). The widening evaluation gap in medical large language model research 2023 to 2026 (arXiv:2609.11770). arXiv. https://arxiv.org/abs/2609.11770

Chen, L., Zaharia, M., & Zou, J. (2023). How is ChatGPT’s behavior changing over time? (arXiv:2307.09009). arXiv. https://doi.org/10.48550/arXiv.2307.09009

Diaz, N. (2026, August 3). Inside health systems’ early use of Epic’s newest AI tools. Becker’s Hospital Review. https://www.beckershospitalreview.com/healthcare-information-technology/ehrs/inside-health-systems-early-use-of-epics-newest-ai-tools/

Duggan, M. J., Gervase, J., Schoenbaum, A., Hanson, W., Howell, J. T., III, Sheinberg, M., & Johnson, K. B. (2025). Clinician experiences with ambient scribe technology to assist with documentation burden and efficiency. JAMA Network Open, 8(2), Article e2460637. https://doi.org/10.1001/jamanetworkopen.2024.60637

ECRI. (2026, August 25). ECRI expands Problem Reporting Network and calls for more data on AI errors in patient care [Press release]. PR Newswire. https://www.prnewswire.com/news-releases/ecri-expands-problem-reporting-network-and-calls-for-more-data-on-ai-errors-in-patient-care-302859649.html

Handley, J. L., Krevat, S. A., Fong, A., & Ratwani, R. M. (2024). Artificial intelligence related safety issues associated with FDA medical device reports. npj Digital Medicine, 7, Article 351. https://pmc.ncbi.nlm.nih.gov/articles/PMC11615200/

Häußler, U. (2026, August 20). Wenn Chatbots zu Kollegen werden: Die blinden Flecken der Klinik-KI [When chatbots become colleagues: The blind spots of clinical AI]. Elektroniknet. https://www.elektroniknet.de/medizintechnik/e-health/dresdner-forschende-warnen-vor-blinden-flecken-der-klinik-ki.a8358926-e943-4297-9f2b-51e775d8c73f.html

Huzhou Municipal Science and Technology Bureau. (2025, February 27). AI不能替代医生直接进行诊疗决策 [AI cannot replace doctors in making diagnosis and treatment decisions]. https://kjj.huzhou.gov.cn/art/2025/2/27/art_1229209487_58939309.html

Jewett, C. (2026, September 15). U.S. health officials move quickly to deploy medical AI despite concerns. The Philadelphia Inquirer (originally published in The New York Times). https://www.inquirer.com/health/ai-doctors-medicare-fda-20260915.html

Jiang, W., Wang, D., Zeng, Y., Huang, J., Xu, C., & Liu, C. (2025). Promoting responsible DeepSeek deployment in health care. Journal of Medical Internet Research, 27, Article e80770. https://doi.org/10.2196/80770

Joyce, C., Economou, N. J., & Silcox, C. (2026). AI safety in health systems: Building infrastructure and strengthening risk management practices [White paper]. Duke-Margolis Institute for Health Policy. https://healthpolicy.duke.edu/sites/default/files/2026-03/AI%20Safety%20in%20Health%20Systems%20White%20Paper.pdf

Medicines and Healthcare products Regulatory Agency. (2026, July 29). MHRA clarifies regulatory status of ambient voice technologies used in the NHS. GOV.UK. https://www.gov.uk/government/news/mhra-clarifies-regulatory-status-of-ambient-voice-technologies-used-in-the-nhs

Naves, J. (2025). Commentary to the MCC-AI: Model contractual clauses for the public procurement of AI. Public Buyers Community, European Commission. https://public-buyers-community.ec.europa.eu/system/files/2025-05/Commentary.pdf

NHS England. (2026, January 16). NHS backs AI notetaking to free up more face-to-face care. https://www.england.nhs.uk/2026/01/nhs-backs-ai-notetaking-free-up-more-face-to-face-care/

Novak Jones, D. (2026, July 22). ChatGPT’s advice kept man from seeking medical treatment for dangerous condition, lawsuit claims. Reuters. https://www.reuters.com/legal/government/chatgpts-advice-kept-man-seeking-medical-treatment-dangerous-condition-lawsuit-2026-07-22/

O Globo. (2026, August 31). Uso de inteligência artificial se espalha e reduz riscos na saúde [Use of artificial intelligence spreads and reduces risks in health care]. O Globo. https://oglobo.globo.com/economia/tecnologia/noticia/2026/08/31/uso-de-inteligencia-artificial-se-espalha-e-reduz-riscos-na-saude.ghtml

OpenAI. (2025, July 22). Pioneering an AI clinical copilot with Penda Health. https://openai.com/index/ai-clinical-copilot-penda-health/

OpenAI. (2026, September 1). Healthcare organizations can now connect EHR and additional industry data to ChatGPT. https://openai.com/index/chatgpt-connects-health-records-and-healthcare-sources/

Perset, K., Aranda, L., & Rispal, B. (2025). Towards a common reporting framework for AI incidents (OECD Artificial Intelligence Papers, No. 34). OECD Publishing. https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/02/towards-a-common-reporting-framework-for-ai-incidents_8c488fdb/f326d4ac-en.pdf

Shankar, V., Gibson, K., & Russell, C. (2026, June 24). EU approves delays and other amendments to certain EU AI Act obligations: What businesses should know. Morgan Lewis. https://www.morganlewis.com/pubs/2026/06/eu-approves-delays-and-other-amendments-to-certain-eu-ai-act-obligations-what-businesses-should-know

Sofi, J. I. (2026, March 17). Anthropic is becoming the backbone of Rwanda’s government. But who is accountable? TechPolicy.Press. https://www.techpolicy.press/anthropic-is-becoming-the-backbone-of-rwandas-government-but-who-is-accountable/

The Joint Commission & Coalition for Health AI. (2025). The responsible use of AI in healthcare (RUAIH). https://digitalassets.jointcommission.org/api/public/content/dcfcf4f1a0cc45cdb526b3cb034c68c2

U.S. Department of Veterans Affairs Office of Inspector General. (2026). Review of generative artificial intelligence chat tools for clinical use (Report No. 26-00182-140). https://www.vaoig.gov/sites/default/files/reports/2026-06/vaoig-26-00182-140_-_final.pdf

U.S. Food and Drug Administration. (2025). Marketing submission recommendations for a predetermined change control plan for artificial intelligence-enabled device software functions: Guidance for industry and Food and Drug Administration staff. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence

Villafranca, V. (2026, February 12). Los juristas alertan de que la regulación en sanidad no “corre” lo suficiente para adoptar los avances de la IA [Jurists warn that health regulation is not “running” fast enough to adopt AI advances]. El Español. https://www.elespanol.com/invertia/observatorios/sanidad/20260212/juristas-alertan-regulacion-sanidad-no-corre-suficiente-adoptar-avances-ia/1003744126545_0.html

World Health Organization Regional Office for Europe. (2025, November 19). Is your doctor’s AI safe? [News release]. https://www.who.int/europe/news/item/19-11-2025-is-your-doctor-s-ai-safe

Zeng, D., Qin, Y., Sheng, B., & Wong, T. Y. (2025). DeepSeek’s “low-cost” adoption across China’s hospital systems: Too fast, too soon? JAMA, 333(21), 1866–1869. https://doi.org/10.1001/jama.2025.6571