Comments
Translated by Daniela Tiznado.
Summary: The European Union is designing a regulatory framework for artificial intelligence (AI) that could be approved by the end of 2023. This regulation prohibits unacceptable practices and stipulates requirements for AI systems in critical sectors. These obligations consist of a risk management system, a quality management system, and post-market monitoring. The legislation enforcement will be tested for the first time in Spain, in a regulatory sandbox of approximately three years. This will be a great opportunity to prepare the national ecosystem and influence the development of AI governance internationally. In this context, we present several policies to consider, including third-party auditing, the detection and evaluation of frontier AI models, red teaming exercises, and creating an incident database.
Everything indicates that the European Union will become the first major political entity to approve a comprehensive regulatory framework for artificial intelligence (AI). On April 21, 2021, The European Commission presented the Regulation laying down harmonised rules on AI –henceforth AI Act or Act–. This legislative proposal covers all types of AI systems in all sectors except the military, making it the most ambitious plan to regulate AI.
As we will explain below, Spain will lead the implementation of this regulation in the context of a testing ground or sandbox. This is an opportunity for the Spanish Government to contribute to establishing good auditing and regulatory practices that can be adopted by other member states.
This article is divided into six sections. Firstly, we provide a brief history of the Act. The second part summarizes the legislative proposal of the European Commission. The third section details the first sandbox of this regulation, carried out in Spain. The fourth lists the public bodies involved in the testing environment. The fifth part explains the relevance of this exercise. Finally, we present proposals to improve the governance of risks associated with AI in this context. We conclude that this project provides an excellent opportunity to develop a culture of responsible AI and determine the effectiveness of various policies.
The foundations of the text date back to 2020, when the European Commission published the White Paper on Artificial Intelligence. This was the beginning of a consultation process and a subsequent roadmap that involved the participation of hundreds of stakeholders, resulting in the aforementioned proposal.
After its publication, the Commission received feedback from 304 actors and initiated a review process involving the European Parliament and the Council of the European Union as legislative bodies. In December 2022, the Council adopted a common approach. In the case of the Parliament, the vote to agree on a joint position is scheduled for May (Bertuzzi, 2023). The trilogue will begin immediately afterward, and the final version could be approved by the end of 2023, entering into force at the beginning of 2024.
The main starting point of the proposed law[1] is the classification of AI systems according to the level of risk they entail. Specifically, the proposal is based on a hierarchy distinguishing between unacceptable, high, limited, and minimal risks. The first two are the main focus of the regulation.
As part of the category of unacceptable risks, practices that pose a clear threat to the safety, livelihoods, and rights of people will be banned. Currently, three practices have been deemed unacceptable as they go against European values: distorting human behavior to cause harm; evaluating and classifying individuals based on their social behavior; and using real-time remote biometric identification systems in public spaces, except in emergency situations.
On the other hand, high-risk systems are those with the potential to cause greater impact by being deployed in critical sectors, including essential infrastructure, education, employment, essential public and private services, law enforcement, and border management. In this case, several requirements fall on the development and implementation of all products.
Providers of high-risk systems are required to establish, implement, document, and maintain a two-phase risk management system. Firstly, known and foreseeable risks must be identified and assessed before and after commercialization. Risks may be considered "known" or "foreseeable" if the AI system developer is aware of them by adopting a reasonable level of diligence. However, at the moment, the regulation does not clearly explain what constitutes "a reasonable level of diligence." (Schuett, 2023).
The second phase involves reducing the detected risks to an acceptable level: providers must completely eliminate risks as far as possible or, failing that, implement mitigation and control measures along with training users to make responsible use of the system. In this way, the risk management system will be a process to be repeated until all identified risks are acceptable. The identification of unacceptable risks that cannot be reduced will result in the immediate cessation of the development and/or deployment of the AI system in question (Schuett, 2023).
In parallel, providers will develop a quality management system to ensure that the development and verification of the AI system comply with the Regulation. Before going to market, developers must provide technical documentation that includes details about the design and architecture of the system. In addition, training datasets must have followed governance guidelines regarding design choice, data collection and processing, and the examination of possible deficiencies and biases.
Additionally, providers will present technical efforts aimed at strengthening accuracy, robustness, and cybersecurity. At the same time, transparency measures will be required, such as providing accessible user instructions and, when applicable, informing the user that they are interacting with an AI. Based on the documentation, conformity assessments, mostly internal, will be carried out. If the system passes this examination, it will be endorsed by a declaration of conformity written by the provider and made available to the authorities.
Throughout the entire period of use, the systems must be supervised by humans who understand the capabilities and limitations of the model and can intervene in its operation if necessary. In parallel, events (logs) occurring throughout the life cycle will be automatically recorded to ensure traceability. In post-market monitoring, any serious incident or failure must be reported. In this case, European market surveillance authorities are granted access to data, documentation, and source code. When the operator is unable to take corrective action, these authorities will also possess the power to prohibit or restrict the marketing of the system.
For the implementation of the regulation, the EU bets on the creation of controlled testing environments or sandboxes, which aim to identify and solve potential problems in the application of the Act. These environments will be available through a call for participation so that companies and organizations that wish to test new AI solutions can participate in them. The projects selected to integrate the sandboxes will be able to share information and knowledge, thus promoting collaboration and the exchange of experiences and best practices. In addition, they will have access to advice and guidance from experts, having a secure and controlled environment to test AI solutions before their launch onto the market.
The results of the tests carried out will contribute to the efforts of the European Commission in the effective implementation of the new regulation and will facilitate the flexibility and adaptation of the rules to the real needs demanded by this technology (Rodríguez, 2022).
In this context, the Act orders the assignment of national supervisory authorities and introduces the European Artificial Intelligence Board as a link between all of them. During the sandbox, national authorities must submit annual reports to the Board and the Commission, including results, lessons learned, and recommendations.
In June 2022, the Government of Spain and the European Commission jointly presented the first regulatory sandbox for the development of artificial intelligence in the European Union (MINECO, 2022). Since the second half of 2022, the first phase of this sandbox has been underway, as guidelines and a national legal framework are being developed to enable its operation. This involves determining aspects such as the selection process for companies that will participate, how data will be managed and protected, and the participation of other countries that wish to join the pilot (Rodríguez, 2022). As of April 2023, the government is finalizing the law that will officially launch the project and has already begun inviting companies to participate in it (Aguilar, 2023). The tests are expected to last approximately three years.
The sandbox will seek an iterative learning process through experience, allowing guidelines to be adjusted as testing progresses. In addition, reports will be carried out to evaluate the effectiveness and costs of different strategies for implementing AI. Also, synergies in other markets related to their operation in the sandbox will be analyzed. That will be beneficial not only for Spain but also for all European Union countries.
The estimated budget for its development is 4.3 million euros. This funding was allocated through the Recovery, Transformation, and Resilience Plan, which channels Next Generation EU recovery funds. In general, the plan assigns a total of 500 million to the National Artificial Intelligence Strategy (ENIA), which includes among its objectives "the development of the necessary regulations to enable regulatory sandboxes." The project has been included in the Spain Digital 2026 agenda among the measures for a regulatory and ethical framework for the deployment of AI.
The main reasons for the decision to implement the first sandbox in Spain have been the proactivity and willingness shown by the Spanish authorities. Spain is the first country to create a supervisory agency for the Act and the one that most firmly decided to invest in the initiative. The national authorities consider it an excellent opportunity to promote the entire Spanish ecosystem in the responsible development of AI.
Another strong point of Spain may have been its recent experience with a sandbox in the fintech sector. The initiative was considered an international success after receiving 67 projects that integrated technologies such as biometrics, blockchain, cloud computing, AI, or the Internet of Things (IoT) (MINECO, 2021).
Finally, the international prestige of the visible faces of the Spanish government may have helped to gain trust in Spain for this challenge. Nadia Calviño, First Vice President and Minister of Economic Affairs and Digital Transformation, worked at the European Commission between 2006 and 2018, reaching the position of Director-General. In turn, the Secretary of State for Digitalization and Artificial Intelligence, Carme Artigas, is a recognized international expert in Big Data and AI.
The Spanish Agency for the Supervision of Artificial Intelligence (AESIA) is the designated actor to ensure the correct application of the Regulation. A Coruña will be the headquarters of this future State Agency that must act as an attraction pole and mitigator of the risks associated with AI, both during the sandbox and in the continuous implementation of the ENIA. This recently created body, approved in the 2022 Government Budget, has its legal personality and is attached to the Secretary of State for Digitalization and Artificial Intelligence (SEDIA). The latter is a superior body of the Ministry of Economic Affairs and Digital Transformation (MINECO), responsible for promoting the digitalization of society and the economy in a way that respects individual and collective rights and the Spanish legal system. At the same time, the Advisory Council on AI, constituted of twenty experts who provide independent recommendations to the government, will also be collaborating with these agencies. Finally, the National Institute of Cybersecurity (INCIBE) will support cybersecurity efforts.
At the European level, the European Artificial Intelligence Board will coordinate national agencies, share technical knowledge and best practices, standardize administrative processes, and issue recommendations. The European Commission, for its part, will continue to develop and implement policies, mainly through the Directorate-General for Communications Networks, Content and Technology (DG-CNECT). In this case, the Joint Research Centre (JRC) will provide scientific knowledge and make proposals. Finally, the European Data Protection Supervisor will be the competent authority for market surveillance when institutions, agencies, and bodies of the European Union fall within the scope of the regulation.
The sandbox conducted in Spain will be an excellent opportunity to gain experience in the governance of AI. Recent advances in this technology have attracted much public interest, and part of this attention has been directed toward the risks associated with its development and implementation. Society is now fueling a debate that had barely transcended academic circles and is urging the actors involved to assume responsibilities. This process of awareness-raising must be consolidated.
AI governance is a young discipline without academic consensus. It is difficult to determine which solutions will help channel AI development in a beneficial way. However, various promising proposals have been successfully implemented in other industries and have the approval of most international experts.
Currently, the priority is to effectively implement the European AI Act and evaluate the adequacy of its requirements. But the sandbox in Spain, a limited and controlled space, is also an ideal environment to test the feasibility of complementary policies reinforcing European demands. In this sense, the tests carried out could significantly influence the rest of the European Union and, indirectly, the rest of the world (Siegmann & Anderljung, 2022). Additionally, successful management of minor risks is the basis for addressing more complex cases, including catastrophic risks.
This section presents some measures to consider for AI governance in the European context, which could be practiced in the Spanish sandbox. We suggest exploring independent audits, detecting and evaluating frontier AI systems, simulating attacks to detect vulnerabilities (red teaming), and creating an incident database.
One of the most endorsed practices to increase trust in AI systems is using third-party audits to examine company procedures (Brundage et al., 2020, section 2.1). The Act already requires developers to be accountable and to collaborate with authorities, but implementing independent audits can be an additional step to strengthen verification mechanisms and ensure compliance with responsibilities (Floridi et al., 2022). There is also a need to optimize these procedures to adapt them to the peculiarities of AI, something that could be tested in the sandbox. Mökander et al. (2023) propose a three-layer approach that integrates governance structures, the legality and impact of its applications, and the technical capabilities and limitations of the model. Regarding the latter, it would be positive for auditors to have access to execute the model on the company's hardware, facilitating the logistics of the audit and minimizing the risk of confidential information leaks (Shevlane, 2022). Including these practices in the Spanish experience would constitute a fundamental experience to reveal possible practical and strategic obstacles, especially regarding reluctance due to confidentiality concerns.
At the public level, we recommend that the state develop its capabilities to detect and evaluate frontier AI models. Risks related to AI are concentrated in the training and deployment of general systems with new capabilities. To define which systems should be considered cutting-edge, we suggest using a measure based on computational resources for training the model. The computation used to train the models, measured in terms of floating-point operations (FLOP), is a predictive variable of the resulting capabilities (Sevilla et al, 2022). In Annex IV, the EU AI Act requires that technical documentation describes the computational resources used to develop, train, test, and validate the AI system. While technical documents are internal, informing authorities of developing systems that exceed a certain amount of FLOP would be advisable. More stringent external auditing could be implemented in these cases, focusing on determining dangerous capabilities such as manipulating vulnerable users, autonomous resource acquisition, or assisting in crimes (OpenAI, 2023a). The commercialization of cutting-edge systems would be conditioned on the auditor's endorsement.
The participation of the National Cybersecurity Institute (INCIBE) can also help to incorporate standard practices in cybersecurity. An important exercise is red teaming, i.e., simulating attacks to detect vulnerabilities. Some of the major AI developers are adopting this practice as a mechanism to identify and correct dangerous behaviors. In the case of GPT-4, for example, OpenAI had a team of researchers and professionals from various industries who tried to incite harmful content such as hate speech, information to manufacture weapons, or subversive inclinations (OpenAI, 2023b). Consolidating this practice, together with audits, would be positive to increase the level of robustness of the systems before they are released to the public. This is especially important for high-risk systems, particularly those in which AI significantly influences critical decisions –medical diagnosis, financial analysis, etc. INCIBE and AESIA could coordinate to institutionalize these processes in Spain, creating a network of independent professionals focused on identifying risks and testing responses. This would be especially beneficial for distributing costs and sharing information among the various actors in the ecosystem (Brundage et al., 2020), as well as ensuring that the practice is standardized across the board regardless of the interests and possibilities of each developer.
Finally, authorities could also cooperate to create an incident database. A compilation of serious incidents and malfunctions, which Article 62 already orders to report, would be useful for facilitating joint learning by the involved actors. This solution could be tested in the sandbox and later scaled at the European level, something that the Future of Life Institute and a group of experts from the University of Cambridge have proposed in their respective feedback to the Act. In this case, the Partnership on AI initiative could be a source of inspiration.
The European AI Act is a crucial step in the collective effort to align AI development with human interests. The imminent regulation presents a comprehensive legal framework that should serve as a basis for successfully addressing the risks associated with current systems and more complex future scenarios. Procedures such as the risk management system, quality management system, or post-market monitoring constitute pioneering governance guidelines that will be a reference in Europe and, presumably, the rest of the world.
In this context, Spain has ensured, through a regulatory sandbox, a privileged position to influence the definition and execution of the legislative text. Given this opportunity, we present several policies that can be applied nationally to complement the regulation and expand its scope: conducting independent audits; the public evaluation of the state-of-the-art based on monitoring of computational power; establishing professional teams for red teaming exercises; and the creation of incident sharing databases among developers.
Implementing all these recommendations is feasible in the short term and should not be counterproductive. Although the proposals are exploratory, we estimate that the tangible and potential costs would be relatively low, offset by the expected benefits. In this sense, the sandbox is an excellent opportunity to test policies because the possibility of experimentation provides the principal value.
We thank Javier Prieto, Pablo Villalobos, and Pablo Moreno for their comments on this article. Fernando Irarrázaval and Claudette Salinas contributed to its edition. Daniela Tiznado translated it into English.
Aguilar, A. R. (2023, March 6). El Gobierno ya está invitando a grandes empresas al ensayo del Reglamento de la IA: quiere resultados para noviembre. Business Insider. https://www.businessinsider.es/gobierno-ya-invita-empresas-ensayos-reglamento-ia-1209262
Bertuzzi, L. (2023, March 30). AI Act: MEPs close in on rules for general purpose AI, foundation models. EURACTIV. https://www.euractiv.com/section/artificial-intelligence/news/ai-act-meps-close-in-on-rules-for-general-purpose-ai-foundation-models/
Brundage, M. et al. (2020). Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. https://doi.org/10.48550/arXiv.2004.07213
Floridi, L.; Holweg, M.; Taddeo, M.; Amaya Silva, J.; Mökander, J.; Wen, Y. (2022). capAI - A Procedure for Conducting Conformity Assessment of AI Systems in Line with the EU Artificial Intelligence Act. http://dx.doi.org/10.2139/ssrn.4064091
MINECO (2021, February 24). El Sandbox financiero recibe en su primera convocatoria un total de 67 proyectos. Ministerio de Asuntos Económicos y Transformación Digital. https://portal.mineco.gob.es/es-es/comunicacion/Paginas/210224_np_-Sandbox.aspx
MINECO (2022, June 27). El Gobierno de España presenta, en colaboración con la Comisión Europea, el primer piloto del sandbox de regulación de Inteligencia Artificial en la UE. Ministerio de Asuntos Económicos y Transformación Digital. https://portal.mineco.gob.es/es-es/comunicacion/Paginas/20220627-PR_AI_Sandbox.aspx
Mökander, J.; Schuett, J.; Kirk, H. R. & Floridi, L. (2023). Auditing large language models: a three-layered approach. https://doi.org/10.48550/arXiv.2302.08500
OpenAI (2023a). GPT-4 System Card. https://doi.org/10.48550/arXiv.2303.08774
OpenAI (2023b). GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774
Rodríguez, B. (2022, 29 de septiembre). España Lanza el primer piloto de “AI regulatory sandbox” de la Unión Europea. Observatorio IA de AMETIC. https://observatorio-ametic.ai/regulacion-de-la-inteligencia-artificial/espana-lanza-el-primer-piloto-de-ai-regulatory-sandbox-de
Schuett, J. (2023). Risk Management in the Artificial Intelligence Act. European Journal of Risk Regulation, 1-19. https://doi.org/10.1017/err.2023.1
Sevilla, J.; Heim, L.; Ho, A.; Besiroglu, T.; Hobbhahn, M. & Villalobos, P. (2022). Compute Trends Across Three Eras of Machine Learning. https://doi.org/10.48550/arXiv.2202.05924
Shevlane, T. (2022). Structured Access. An Emerging Paradigm for Safe AI Deployment. https://doi.org/10.48550/arXiv.2201.05159
Siegmann, C. & Anderljung, M. (2022). The Brussels Effect and Artificial Intelligence. Centre for the Governance of AI.
To narrow down the article and facilitate its understanding, the European Parliament's amendments and the Council's position are not taken into account. In any case, it is expected that the provisions here discussed will not change too much.
This seems exciting.
I think the EU AI act is somewhat irrelevant to AGI stuff given it is predominantly usage based so seems to skirt around the challenges of more general AI.
But I expect it will prompt member states (in this case Spain) to set up AI regulators and if that is done well the process put in place could form useful models for wider adoption (e.g. in the US) which would act as a layer of defence in depth against AGI risk.
Will you be working on inputting into this. If not why not / what barriers do you face?
I agree! I think this is an excellent opportunity to shape how AI regulation will happen in practice.
We are currently working on a more extensive report with recommendations to execute the EU AI act sandbox in Spain. As part of this process, we are engaging some relevant public stakeholders in Spain with whom we hope to collaborate.
So far, the most significant barrier to our work is that we are running out of funding. Other than that, access to the relevant stakeholders is proving more challenging than in previous projects, though we are still early in the engagement process. Lastly, we would love to come across more auditing desiderata and proposals to inform our work (the document you shared it's helpful in that regard! Can you share more context around it?).