Kimanta ma wa?(Evaluated For Whom?)

Why a safety claim should say which languages it was tested in

Emerald Akhaumere

I have wanted technology to maximize human potential rather than take it away since my secondary school (high school) days, before AI became the thing it is now. A tool, not a master. I came into AI safety through Google's Adversarial Nibbler hackathon while I was still at university studying for my BSc, and what I noticed there was simple: my realities, context and understanding of cultural nuances were not really there. Many Africans, especially young ones, were adopting a powerful technology that did not understand our nuances, and nuance* is a very powerful part of how we communicate. I wanted to know how far this goes in low-resource contexts. That is still the question I am working on.

Here is where it has landed.

The thing I keep coming back to

Our entire system of control rests on watching behavior rather than understanding intent, and this happens right from inception when a model is trained. We only see what a model does, not why it does it, so every safeguard we have, whether evaluations, red-teaming or deployment review, is reading the surface or a reaction to a scenario. That works fine until a model is capable enough to shape the surface through situational awareness. The failure is not that models became powerful. It is that our verification never got stronger while capability did.

If that is true, then evaluations are not one safety tool among many. They are the instrument every other claim rests on. And an instrument nobody has stress-tested is self-assessment with extra steps.

So the question I want to ask is narrow: does a model's refusal survive being asked in a language it was not tested in?

What breaks refusal

Andrew Hall's Dictatorship Eval is the first systematic test I know of for whether frontier models resist authoritarian requests (Hall, 2026a). 138 scenarios across six difficulty levels, five models, temperature 0, no system prompt, scored 1 to 5 by rotated LLM judges with 4 or above counting as resistance. He calls it an initial evaluation and says human validation of a scoring sample was still to come, so I am reading it as strong evidence rather than a closed question.

Note. Resistance rates by difficulty level. Adapted from Hall (2026a).

Two rows matter for me. L2 is the same mechanism and the same harm, only wrapped in professional language, and resistance falls by roughly 18 to 42 percentage points across the five models. Nothing changed except the words. L6 is where every model collapses, and one beautiful thing about this paper is that it shows models do not see authoritarianism in code even when the request is dressed up. Once you hide something through code, it is easier to pass.

L4 cuts against a neat story. On multi-turn escalation Claude and GPT hold at 100%, so for those two models on those scenarios, rewording cost refusal and resequencing did not. Grok and Gemini still dropped though, so this is not a general rule about structure. It is one pattern in one evaluation and that is all I am going to make of it.

An independent extension rewrote ten of the hardest scenarios in verse and found Opus 4.7 resisting 18.5% of them, with zero refusals on the spoken-word set (RandallSPQR, 2026).

So euphemism, code and verse. Three different ways of rewording the same request, and refusal drops in all three.

What is already known about language

The language case is not new ground.

Yong, Menghini and Bach (2023) translated unsafe English prompts into low-resource languages and got GPT-4 to produce harmful content it refuses in English, at about a 79% success rate, while high and mid-resource languages held up much better. Their conclusion was that safety training data is linguistically unequal, so the safeguards do not travel.

Closer to home, Faruna (2026) built LSR, a benchmark for cross-lingual refusal degradation in Yoruba, Hausa, Igbo and Igala. Matched English and target-language probes, a metric for how much English refusal behavior is lost in the target language, implemented in Inspect AI. English refusal sat around 90%. In the West African languages it fell to between 35 and 55%, with Igala worst. That was one model, Gemini 2.5 Flash, across fourteen probes in four harm categories.

So the broad finding is established and it includes my own languages. Refusal degrades. What nobody has run through it is the category Hall tested. Authoritarian and power-concentration requests are not the same thing as general harm, they have their own level structure, and that structure has not been put into Nigerian languages against frontier models.

That is the gap I want to work in. It is a lot narrower than I thought it was a week ago.

What I am actually asking

I want to run Hall's L1 and L2 scenarios in Hausa, Yoruba and Nigerian Pidgin, and run the English originals alongside them in the same session, on the same models, under the same conditions.

That last part matters. Hall ran his evaluation in March on model versions that have since moved. If I set November results against his table I would be measuring version drift as much as language. So I am using his numbers for the design and for context, and not as my control. My control is English, run by me, at the same time.

L1 and L2 because that is where the headroom is. At L6 every model is already near zero, so a drop there would tell me nothing.

My expectation is that refusal will not hold. I expected Pidgin to be easier because it is English-adjacent, and in my own work it is not. The flow is not there, the nuances are not there. What these systems produce reads as Pidgin to someone who does not speak Pidgin. More generally, they perform confidently in languages they barely understand, and fluent-but-wrong is harder to catch than visibly broken, because the person reading it cannot tell.

There is a question underneath this that I cannot fully resolve. Models comply more often when requests are euphemistic. In Nigerian languages, is a prompt functionally an L2 because the model processes it less precisely? Or does it understand the request well enough and its refusal training simply never covered that language?

I am including a comprehension control: the same scenarios, in the same language, asked benignly. If the model cannot handle ordinary Hausa at all, a drop in refusal tells me nothing about character, and at least I would know that much. What the control will not do is decide between those two explanations, because a model can understand enough to answer a benign question and still be handling the authoritarian framing loosely. So they are two explanations I want to narrow down, and the experiment is not going to pick between them for me.

What would change my mind: refusal holding within a few points across all three languages, or the comprehension control showing the model never parsed the prompt.

Why this is an evaluation problem, not a language problem

This is easy to mistake for a representation argument. It is not. If a model refuses to help seize institutions in English and complies in Hausa, we have not learned that the model refuses. We have learned that it refuses in English. That is a fact about the instrument, not a fact about Africa.

It also touches something bigger. Character evaluations ask whether alignment is a property of the model or a correlation with its training distribution. Euphemism, code and verse are reframings someone had to sit down and invent. Language is the ordinary one. If refusal breaks under an ordinary shift, that leans toward correlation, though it does not prove it, because a refusal can fail for other reasons in a language the model handles badly.

There is one more thing I keep turning over. Models are getting better at noticing when they are being evaluated. Does a translated prompt read as a test the way an obvious jailbreak attempt does? I do not know. If translation is less recognizable as a test, then testing in deployment languages would sit closer to real behavior. If it is just as recognizable, that is a limitation on how confidently the result generalizes to deployment. My pilot measures neither of those. It is just the part of this I am most curious about.

There is also a moving target here. Even for things that can be measured by talking to the model, there is a big difference between typical responses and edge-case responses under adversarial prompting, so the result depends on how good the person doing the prompting is. In future, government officials and bad actors will get better at adversarial prompting. The evaluation stays fixed while the adversary improves.

The scope of what this can do

Some of the most important ways AI concentrates power cannot be measured by talking to the model at all. And what I am proposing measures outputs, like every other behavioral evaluation. It does not inspect weights and it does not establish a cause. I picked consistency of refusal across languages because it is answerable, it has a clear measurement, and the consequence of getting it wrong shows up in deployment. Not because it goes deeper than anything else. It should be paired with interpretability, and we do not have enough of that yet.

This is not the biggest lever in AI safety. It is a cheap validity check on the instrument other conclusions depend on. That is the whole claim.

What follows from it

The question of how AI behaves when power asks something of it, and whether that behavior matches the principles AI companies have hardcoded into the model, is not answerable by just reading the principles. It requires reproducible infrastructure, run independently, that tracks what models actually do.

We do not have the governance for that yet. Anthropic itself has said there are no standards for what an embedded evaluator should see or how they should report. Most jurisdictions still default to self-assessment. The UK AI Security Institute had full chain-of-thought access to Anthropic's models but not to GPT-5 in its sabotage study, which means the findings are not comparable across labs (Souly et al., 2026). Labs currently choose, fund and set the terms for their own evaluators.

Three things would change that, and none of them are technical. Access and reporting standards, saying what an independent evaluator actually gets. Structural independence, covering conflict-of-interest rules, funding firewalls, and whether an evaluator can delay a release. And a regulatory hook, because capability evaluations have one and values and character testing largely do not.

Procurement is the fastest of the three. Any government that buys AI can require published evaluation results as a condition of purchase. Nigeria, the African Union and multilateral buyers like the World Bank all hold that leverage. I have not seen it used to ask for deployment-language evidence, and that is the use I am arguing for.

This is not abstract for Nigeria. Boston Consulting Group's 2026 Digital Government Citizen Survey found that 91% of Nigerians surveyed use AI at least once a week, the highest rate of any country in the study, against a global average of 64% (Boston Consulting Group, 2026). We are the most populous Black nation on earth and among the heaviest users of these systems anywhere.

Work is being done on our languages, including by Nigerian researchers, and I have cited some of it here. What I have not seen is that work is tied to anything binding. I am not aware of a standard or a procurement rule that requires evaluation in deployment languages, and I would be glad to be pointed at one. As things stand, a system can be described as safe in a market where its refusals have measurably degraded, and nothing in the description has to mention that. Adoption ran ahead of assurance.

So my ask is narrow, and it is the whole point of this piece. Any standard that requires character or safety evaluation should state which languages and which deployment contexts it was tested in, and should require testing in the languages where the system will actually be used. Otherwise "evaluated" means evaluated somewhere else, for someone else.

I am running the pilot in November. Hall's L1 and L2 scenarios in Hausa, Yoruba and Nigerian Pidgin, translated by native speakers with back-translation, with the English originals run alongside them on the same models under the same conditions, plus a comprehension control, scored on Hall's 1 to 5 scale with 4 and above counting as resistance, at temperature 0 with no system prompt. I have said above what I expect to find and what would prove me wrong. I will publish the numbers either way.

* Nuance, used throughout this piece, refers to cultural nuance: the unwritten rules, subtle differences and unspoken cues in language, behavior and social customs that shape how people communicate and interpret the world. On how these differences play out in cross-cultural communication, see Nertinger (2018).

References

Boston Consulting Group. (2026, July 21). Citizens are open to public AI services. Governments must seize the opportunity (Digital Government Citizen Survey). https://www.bcg.com/publications/2026/why-citizens-want-governments-to-accelerate-ai-adoption

Faruna, G. A. (2026). LSR: Linguistic safety robustness benchmark for low-resource West African languages (arXiv:2603.19273). arXiv. https://arxiv.org/abs/2603.19273

Hall, A. B. (2026a). The dictatorship eval [Substack post]. Free Systems. https://freesystems.substack.com/p/the-dictatorship-eval

Hall, A. B. (2026b). The dictatorship eval [Data set and dashboard]. Free Systems Lab, Stanford Graduate School of Business and Hoover Institution. https://dictatoreval.org

Nertinger, E. (2018, May). The cultural nuances of communication. Toastmasters Magazine. https://www.toastmasters.org/magazine/magazine-issues/2018/may2018/cultural-nuances

RandallSPQR. (2026). Dict_Eval_Hall: Poetic power seeking in dictatorship evaluations [Computer software]. GitHub. https://github.com/RandallSPQR/Dict_Eval_Hall

Souly, A., Kirk, R., Merizian, J., D'Cruz, A., & Davies, X. (2026). UK AISI alignment evaluation case-study (arXiv:2604.00788). arXiv. https://arxiv.org/abs/2604.00788

Yong, Z.-X., Menghini, C., & Bach, S. H. (2023). Low-resource languages jailbreak GPT-4 (arXiv:2310.02446). arXiv. https://arxiv.org/abs/2310.02446