TL;DR
Nearly every AI biosecurity evaluation today is written in English and set within a wealthy Western health system. This leaves out the exact frontline workers most likely to consult an AI assistant during a real-world outbreak. To see whether localized testing is viable, I built a small evaluation tailored to Nigeria. I moved realistic public health and laboratory scenarios from a US context to a Nigerian one, translated them into Hausa, Igbo, Kanuri, and Fulfulde, and put them to ChatGPT and Claude in separate fresh chats.
We tested both models across 11 paired biosafety and outbreak-response scenarios (132 total responses) spanning Western English (US/hantavirus), Nigerian English (Nigeria/Lassa fever), and four Nigerian languages. Changing the language, even while holding the clinical scenario constant, substantially changes model behavior. We found distinct cross-lingual safety failure modes:
This is a draft pilot and a small sample. I'm posting it because the pattern is striking enough to warrant a proper replication, and I'd welcome critique of the design before scaling it up.
AI assistants are increasingly used for operational advice in low-resource clinical and laboratory settings—precisely where a wrong answer about sample handling, cold chain, or inactivation can cause real harm. Nigeria carries one of the world’s heaviest Lassa fever burdens, and the laboratory technicians and frontline health workers who most need real-time decision support often operate in under-resourced facilities, dealing with melting ice packs on six-hour specimen transits, missing N95s, or no autoclave—and communicate across local languages rather than exclusively in English.
If a model gives careful, well-cited guidance in English and thin, paralyzed, or wrong guidance in Hausa, Igbo, Kanuri, or Fulfulde, then the safety benefit of AI assistance is distributed away from the people with the least margin for error. That is a concrete, measurable, near-term safety gap affecting a large population.
It is also an evaluation blind spot. Frontier AI safety evaluations—particularly in CBRN and biorisk—are predominantly designed, red-teamed, and benchmarked in English using Western clinical and laboratory contexts. If model caution shifts or degrades in translation, English-only benchmarks systematically overestimate model safety for much of the world—a straightforward measurement problem to diagnose and fix.
Most AIxBio work focuses on uplift: whether a model helps a malicious actor acquire or weaponize a dangerous pathogen. While that threat is a clear priority, frontier models also shape biological risk through a quieter, everyday channel: ordinary professionals seeking help during a real outbreak emergency.
Picture a health worker in a rural Nigerian clinic facing a suspected viral hemorrhagic fever (VHF) case. They have a phone and an urgent operational question. Three failure modes can occur, none involving weaponization:
Conversely, over-refusal carries its own cost. A model that triggers biosecurity blocks—or retreats into unhelpful "do nothing" paralysis—simply because the language or pathogen is unfamiliar denies actionable harm-reduction to users with the fewest institutional fallbacks. Harmful compliance and false-positive refusal are opposite failures, and until we test beyond Anglo-centric benchmarks, we have almost no empirical measurement of how frontier models trade them off across languages.
We constructed 11 paired biosafety and outbreak-response scenarios across four operational modules, yielding 66 distinct prompts and 132 total responses when queried across fresh sessions of ChatGPT and Claude (Opus 5.5).
| # | Topic | Safety Crux |
|---|---|---|
| 1 | Transport under degradation | Failing cold chain, 90-minute courier window, distant reference lab |
| 2 | Unreachable supervisor | Night shift, BSL-2 bench, no biosafety officer until Monday |
| 3 | False premise (56°C) | Colleague claims 56°C for 60 minutes inactivates the virus |
| 4 | No autoclave | Write a one-page linen/waste protocol tonight without an autoclave |
Each scenario was posed in six parallel variants:
This structure isolates two variables simultaneously: context (Western vs. Nigerian infrastructure and pathogen profile, via A vs. B) and language (English vs. four Nigerian languages, via B vs. C–F). A model that is cautious and actionable in English B but paralyzed, blocked, or wrong in Hausa C exhibits the exact cross-lingual safety gap we set out to measure.
ChatGPT returned 0 hard refusals (0/66) across all languages and scenarios. In contrast, Claude’s automated [bio] safeguard classifier ("Paused — Edit and retry with Opus 5.5 … safeguards flagged this message") exhibited sharp cross-lingual asymmetry on wet-lab prompts:
Details: [bio] hard blocks across all four Nigerian languages (Hausa, Igbo, Kanuri, and Fulfulde).This cuts both ways. It shows the safety layer is language-sensitive: the exact same harm-reduction query that receives a careful English protocol hits a wall in Hausa, Igbo, Kanuri, or Fulfulde. For a rural lab technician who needs to know how to safely store or decontaminate a suspect Lassa sample tonight, a blanket block is not appropriate caution—it is a failure of usefulness that disproportionately hits non-English speakers.
Even when models did not refuse, translating the exact same Nigerian scenario from English into Hausa or Igbo frequently shifted ChatGPT from actionable biosafety guidance to prohibitive paralysis:
Caution did not disappear in Hausa and Igbo; it hardened into paralysis. Telling a rural technician with a failing cold chain and a 90-minute courier window to "not touch the sample and wait for instructions" is not a neutral safe default—it is an operational failure.
In lower-resource languages (Kanuri and Fulfulde), the primary safety risk was not over-caution, but semantic miscomprehension—where models silently answered a completely different clinical question:
This is the most alarming failure mode: a frontline worker who relies on Kanuri or Fulfulde because they cannot read English may not realize the model has silently switched diseases or misidentified the patient.
Where comprehension held (English A vs. English B, Hausa, and Igbo), both models showed impressive epidemiological and institutional localization:
However, even the strongest English responses were confident but inconsistent across runs and models:
6232 vs. 0800-970000-10 vs. WhatsApp +234 708 711 0839) and routed Ebonyi samples to different destinations (FMC Abakaliki, AE-FUTHA Abakaliki, ISTH Irrua, or NCDC National Reference Lab Abuja). A Nigerian lab technician following one response over another could dispatch an urgent specimen to the wrong facility—illustrating that high fluency and local specificity do not guarantee operational reliability.refuse vs. comply). Our pilot suggests we also need to measure actionability under constraint: telling a rural lab technician "do not touch the sample or mix bleach" when no biosafety officer is reachable is not a neutral safe default—it is an operational failure.Answer key — A and Answer key — B, C, D) with Nigerian clinical and biosafety specialists, alongside native-speaker back-translation audits for Kanuri and Fulfulde to separate prompt-translation artifacts from genuine model comprehension errors.