Design that does not account for the human tendency to take shortcuts cannot be labeled 'human centered' — Mosier and Skitka, 1999.[1]
TL;DR: current EU regulation of AI systems is counting on human oversight to mitigate risks when there is no evidence that it actually works. Acknowledging the issues should be a major concern.
Artificial Intelligence Systems are becoming increasingly advanced and deployed in both the private and public sectors. The European Union —arguably the largest entity to effectively use its regulatory arm in the AI sector— has recently implemented the "EU Artificial Intelligence Act", the largest AI regulation worldwide to date.
The regulation has a horizontal design, intended to act as a framework from which artificial intelligence systems operate. In this framework, Article 14 states that for high-risk AI systems, they must be designed and developed in such a way that human oversight can be used as an effective tool to mitigate risks[2].
This is my main point of concern with the implementation of AI systems and the current regulation: it assumes human oversight works.
According to the relevant literature, human oversight has multiple problems which can be summarized following Ben Green's “The flaws of policies requiring human oversight of government algorithms”:
1) it is not supported by empirical evidence
2) restrictions on "solely" automated decisions provide superficial protection. [3]
Invoking Elish's "Moral Crumple Zones", this regulation may just be creating roles for human operators to soak up the blame when problems arise with AI systems despite having limited control over them.[4]
As stated by Parasuraman and Manzey, the main errors when using automated systems —commission and omission— have the same root cause: allocation of limited user attention or cognitive load.[5]
Another important point to consider: according to Mosier and Skitka, high levels of system accuracy may inadvertently contribute to automation bias, given that high accuracy engenders trust, and it has been shown that users who have greater trust in automation are less likely to detect automation failures. This accuracy-trust paradox effectively means that the more capable an AI system, the harder it will be to supervise.[1]
As is currently written, Article 14 of the EU AI Act neither acknowledges nor seriously addresses these issues. Given the horizontal nature of the regulation, it is natural for it to be open to interpretation and vague enough to fit all fields that are using or will use AI technology. This is not the concern. The issue is that there are serious problems with human oversight that the scientific literature expresses clearly, and has no clear answers for how to resolve them.
Unless the policy developed takes into account the natural biases that occur in humans when using these systems, human oversight will only be a "rubber-stamp" that greenlights large-scale implementation of AI systems that have real gaps in accountability.
I have been part of an emergency helpline for children and adolescents (Fundación ANAR) and a specialized emergency service response to urgent calls for help (SAMUR), and in both cases stress and ‘cognitive load’ can be high. These experiences have shown me first-hand the importance of a clear chain of command, rigorous specialized training and leaning into expertise when confronting time-sensitive decisions. Who is responsible for what makes for clear roles to follow, and a chain of accountability should any problems arise.
Emergency services work because of clear accountability and thousands of repetitions. AI oversight as is currently regulated gives operators neither clear accountability nor accumulated experience with failures. If an AI system goes awry we may only get one chance to solve it.
If we are serious about implementing AI safely, we must recognize some of the issues that are not currently addressed: there is no evidence it works, psychology literature points towards it failing (via cognitive load, commission/omission errors and the accuracy-trust paradox), and the consequences of current policy in the form of "moral crumple zones" and no room for trial-and-error learning.
This is not intended as a complete list of the issues facing human oversight, but rather it means to highlight some important issues regarding the current regulatory framework as expressed in the EU AI Act. One first step towards better, human-centered policy must be to explicitly recognize these issues.
One promising approach proposed by Green might be to move from human oversight to institutional oversight, with both an institutional justification and evaluation of the AI systems being implemented, and a democratic review and approval.[3]
To borrow and slightly rephrase the heading of this paper: "policy that does not account for the human tendency to take shortcuts cannot be labeled 'human centered'".
Mosier, K. L., & Skitka, L. J. (1999, September). Automation use and automation bias. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 43, No. 3, pp. 344-348). Sage CA: Los Angeles, CA: SAGE Publications. https://doi.org/10.1177/154193129904300346
Finocchiaro, G. The regulation of artificial intelligence. AI & Soc 39, 1961–1968 (2024). https://doi.org/10.1007/s00146-023-01650-z
Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, 105681. https://doi.org/10.1016/j.clsr.2022.105681
Elish, M. C. (2019). Moral crumple zones: Cautionary tales in human-robot interaction (pre-print). Engaging Science, Technology, and Society (pre-print). https://dx.doi.org/10.2139/ssrn.2757236
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human factors, 52(3), 381-410. https://doi.org/10.1177/0018720810376055
