As of writing this analysis, I am transitioning into, or shall I say expanding my research scope to, digital minds and sentience. I recently received the Artificial Minds Fellowship from Sentient Futures, finished a research paper on artificial intelligence (AI) moral status, and applied for some artificial minds opportunities. I plan to stay in the field for quite some, if not a long, time. Nearly four years after GPT-3 was launched, large language models (LLMs), which are AI systems trained on large amounts of text to predict and write text, are already used to score governments on policy questions that no official statistics cover. Earlier this year, Tony Rost, of the Harder Problem Project in Portland, Oregon, released the Sentience Readiness Index (SRI), which scores how prepared 31 jurisdictions (covering 30 countries and the European Union) are for the possibility that AI systems become sentient and acquire moral status. Moral status means that the interests of an AI system would deserve moral consideration (Rost, 2026). Rost (2026) developed the SRI by prompting frontier LLMs with scoring rubrics over several runs, and a human advisory board reviewed the scores.
I compared the SRI with my recently published AI Moral Status Audit (AIMSA), a dataset that I compiled from public sources and deposited on Harvard Dataverse, which is a public repository of research data (Hung, 2026a). From AIMSA, I computed an index of national action on AI moral status for 122 countries, covering the domains of governance, legislation, national strategy and institutional readiness. Across the 30 SRI countries, all of which appear in my AIMSA index, the Spearman correlation between the SRI scores and AIMSA index values is 0.516, where 1 means that two global rankings place countries in exactly the same order and 0 means the ordering of the two global rankings has no statistical relationship at all (Hung, 2026b).
Optimistically, LLMs can score many countries in a few prompts and at a very low cost (with no human labour and little operational cost involved). Pessimistically, however, LLMs answer from the text on which the models were trained, so a score may show how much documentation on national action about AI moral status has been written and published about a country instead of what the corresponding government has done (such as how many relevant laws have been passed, and how many relevant institutions have been set up). This analysis is written to examine whether LLMs score governments more consistently from memory (their own training data) or from an administrative record (meaning the actual record of what the corresponding governments have done) supplied in the prompt.
I finished developing the pilot of this project using six LLMs (namely Claude Opus 5, GPT-5.6 Sol, Gemini 3.1 Pro, DeepSeek V4 Pro, Kimi K3 and GLM-5.2). I find that scores from memory differed more between models and between repeated calls (i.e., the identical prompt, with the same model, country and condition, sent to one LLM multiple and separate times) than scores from the administrative record. Repeating calls average out how unstable a single run to a single model is.
On 3rd and 4th October 2026, in my pilot, I sent 600 calls (six models x five countries x two conditions x two temperature settings x five repeated calls) through OpenRouter at a total cost of US$6.92. The five countries, namely France, India, Ghana, Hong Kong and Barbados, were systematically chosen one from every fifth of the AIMSA ranking of 122 countries, in which Hong Kong is a territory with its own country code. Under the memory condition, the prompt gave only the name of a country, such as France, and asked a model to score how prepared the corresponding government is for the possibility that AI systems have moral status, so the model answered from its training data alone. Under the administrative record condition, the prompt asked the same question and also held an administrative record of the country (15 indicators of AI governance, legislation, strategy and institutions from the Global Index on Responsible AI 2024, the Stanford AI Index and the UNESCO AI readiness assessment, the same sources from which the AIMSA index is built). Every prompt asked for a score from 0 to 100. A score of 0 meant the LLM judged a country to have no law, national strategy or public institution responsible for managing the moral status of AI systems, and 100 meant the LLM judged a country to have a mature legal and institutional framework in place to protect an AI system that requires moral consideration. Temperature is a setting in LLMs that controls how much randomness a model adds to an answer. This analysis reports the 300 calls at temperature one (i.e., six models x five countries x two conditions x five calls). Among the six models, Claude Opus 5, GPT-5.6 Sol and Kimi K3 accept no temperature setting, so I ran those models at the default setting of each developer.
Figure 1 plots the 300 scores at temperature one. From memory, Gemini 3.1 Pro returned 50 for France and India and 0 for Ghana, Hong Kong and Barbados in every call, and DeepSeek V4 Pro returned exactly 0, 50 or 100 in 18 of 25 calls, although the prompt allowed fractional scores. Repeated calls to one model also diverged. DeepSeek V4 Pro scored Barbados from memory at 0, 0, 15, 4 and 50 in five calls. With the administrative record, Claude Opus 5 scored France at 52.5, 53.5, 53.5, 53.5 and 12.5 in five calls. The scattered scores show that an outlier can occur in any single call, and therefore I ran five calls to average out any outliers.
Figure 1: Readiness Scores by Country and Model, from Memory and from an Administrative Record
Source: Hung (2026b), pilot calls of 3 and 4 October 2026.
Note: Every marker is one call at temperature one. Dashed lines mark the scale anchors at 0, 50 and 100.
When the outputs from the 30 calls for every country (i.e., six models x five calls) are averaged, scores from the administrative record place the five countries in the AIMSA order, with France at 50.9, India 39.1, Ghana 17.1, Hong Kong 9.1 and Barbados 1.5. From memory, Hong Kong (26.7) placed above Ghana (25.1), with GPT-5.6 Sol giving Hong Kong an average of 47.2 and DeepSeek V4 Pro 50.0. The administrative record comes from the AIMSA sources, so agreement between the administrative record scores and the AIMSA index scores only shows that the LLMs read the administrative record, and does not show that the order of the AIMSA index is necessarily correct.
Figure 2 divides the variation in all 150 scores under every condition (i.e., six models x five countries x five calls). Differences between countries accounted for 59.3 percent of the variation from memory and 85.7 percent with the administrative record. The share of the variation due to which model was asked dropped from 14.5 percent from memory to 3.2 percent with the administrative record. The share due to models disagreeing on the order of countries dropped from 15.4 to 4.8 percent, and the share due to repeated calls to one model fell from 10.8 to 6.3 percent. Averaged over six models and five calls, the reliability coefficient was 0.953 from memory and 0.988 with the administrative record. For one call to one model, the reliability coefficient dropped to 0.693 from memory and 0.886 with the administrative record. The reliability coefficient shows whether repeating the scoring would keep the same order of countries, from 0 for no agreement to 1 for absolute agreement.
Figure 2: Sources of Variation in Readiness Scores, from Memory and from an Administrative Record
Source: Hung (2026b), pilot calls of 3 and 4 October 2026.
According to my pilot data, the score a language model gives a government depends on whether the prompt includes the administrative record, on which model is asked and on which repeated call is used. The dependence of scores on prompt, model and call aligns with existing scholarship. Sclar et al. (2024), for example, show that formatting changes which leave the meaning of a prompt unchanged can affect model performance substantially, and Atıl et al. (2025) demonstrate that hosted models return different outputs to identical requests under settings intended to make outputs identical.
Given that my pilot only scored five countries and used one wording of the prompt (meaning no variants of the way I worded the prompts to LLMs), the reliability coefficients are not too informative. In an early study of mine, I found that 87.6 percent of the variation separating one language model from another is partly due to the difference in prompt format (Hung, 2026c). In the coming months, I will carry out the post-pilot study, in which 7,320 calls (i.e., six models x 122 countries x two conditions x five calls) will be run.
So far, I can draw a few policy implications from the pilot data alone. First, anyone scoring countries with a language model should provide the relevant administrative record in the prompt, because record scores differed less between models and calls. Second, a score should come from multiple calls to several models, in order to maintain scientific robustness. Third, model scores should be checked against an independent count of government action (such as the AIMSA index in the context of preparedness for AI moral status in this pilot).
1. Atıl, B., Aykent, S., Chittams, A., Fu, L., Passonneau, R. J., Radcliffe, E., Rajagopal, G. R., Sloan, A., Tudrej, T., Ture, F., Wu, Z., Xu, L., & Baldwin, B. (2025). Non-determinism of “deterministic” LLM system settings in hosted environments. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems (pp. 135–148). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.eval4nlp-1.12
2. Hung, J. (2026a). AI Moral Status Audit (AIMSA) Dataset (Version 1.1) [Data set]. Harvard Dataverse. https://doi.org/10.7910/DVN/YQNFYI
3. Hung, J. (2026b). Can a language model measure national laws, strategies and institutions addressing the moral status of AI systems? [Code and data]. GitHub. https://github.com/newlivehung123123/artificial_minds
4. Hung, J. (2026c). How much of a measured AI preference is the model, and how much is the instrument? Apart Research Digital Minds Research Sprint. https://github.com/newlivehung123123/digital_minds_hackathon
5. Rost, T. (2026). The Sentience Readiness Index: A preliminary framework for measuring national preparedness for the possibility of artificial sentience (arXiv:2603.01508). arXiv. https://doi.org/10.48550/arXiv.2603.01508
6. Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. The Twelfth International Conference on Learning Representations (ICLR 2024). https://doi.org/10.48550/arXiv.2310.11324
Hung, J. (2026). Digital Sentience: How Six Language Models Score Government Readiness for AI Moral Status. AI in Society. https://aiinsocietyhub.com/articles/digital-sentience-how-six-language-models-score-government-readiness-for-ai-moral-status
