On behalf of my firm, I recently wrote a short paper (around 2000 words) on fundamental challenges in AI governance, emerging trends and themes in different jurisdictions, and what issues the recently-formed UN High-Level Advisory Board on AI ought to consider, in response to a call for submissions.
I want to share an edited version of the paper here, to get a sense of what the community thinks of how I've condensed the issues. My goal in this piece is to take some key EA insights - like that policymakers should focused on what these systems may be capable of in the near-future rather than on what they can do at present - and to make them legible to a broad audience, using as little jargon as possible.
I think I've done a decent job of condensing what I take to be the fundamental challenges from a governance perspective. The other sections of the paper - on emerging trends and issues to consider - are perhaps less well thought out. And of course, this is really just a sketch of the problem - more impressionism than photorealism.
In any event, my sense is that while there is currently immense interest from policymakers and civil society groups (both globally and in Africa, where I'm currently based) in understanding issues of AI governance, there is a lack of pithy and accessible material that speaks to this audience from a perspective that takes existential risk seriously. This paper is one attempt to close this gap; and is part of my broader efforts to build bridges between the EA and human rights communities - as I think these groups often talk past each other, and are worse off for it. I may write more about this latter point in a separate post.
Below is an adapted version of the paper, with some references omitted. For the full version, please see the link above.
There are three reasons that governing AI poses a distinct challenge – the nature of the technology, the sociopolitical context in which it is being created, and the nature of its attendant risks.
Modern AI systems have three key components: the computing power necessary to train and run the system, the data used to train the system, and the model architecture and machine-learning algorithms that produce the system’s behaviour. In the present paradigm, scaling the amount of computing power and data available to train systems has led to exponential improvements in their performance. While it is not guaranteed that these exponential improvements will continue indefinitely, at present they show no sign of slowing down.
The rapid advances in AI models in recent years, and particularly the existence of their “emergent capabilities” – where AI models demonstrate abilities which their creators neither foresaw nor explicitly encoded, such as the ability to reason or to do algebra – make the governance of AI a distinct challenge. It is difficult for technologists, let alone policymakers, to predict what cutting-edge AI systems will be capable of in the next two, five, or fifteen years. Even so, it seems likely that AI technologies will become increasingly capable across an increasingly wide range of domains.
A related challenge here is that at present, cutting-edge AI systems are “black boxes” – their creators are typically unable to explain why a given input produces an associated output. This makes demands that these systems be transparent and explainable not just a political challenge, but a technical one.
AI governance is made more complex by the sociopolitical context in which AI is being developed. Two factors are particularly salient.
AI technologies can be used both to produce immense benefits (for example by advancing scientific research or increasing access to quality healthcare and education) and grave harms. With their current capabilities, AI systems can be used to generate disinformation, perpetuate prejudices, and violate copyright law. As capabilities scale, future systems could be used by malicious actors to create bioweapons, execute advanced cyber-attacks, and otherwise cause catastrophic harm.
Some have argued this puts the risks posed by advanced AI systems in the same category as those posed by pandemics and nuclear war.8 And this is before one accounts for the possibility of an “artificial general intelligence”: an AI system that is as competent as humans at most economically valuable tasks.9 Such a system may not be aligned to human values, may be capable of deception, and may be able to act agentically (operating beyond human control) unless appropriately constrained.
These risks are compounded by the fact that many AI models are being released in an open-source fashion, allowing anyone in any region to access them. Even if 99,9% of people who use these models have no malicious intentions, the 0,01% of people who do could cause enormous harm.
[This section is particularly sparse. Its analysis draws on a report I prepared for a client which I can't yet make public, but I'm excited to produce more on this front in the near future.]
AI governance is a nascent field. As of September 2023, no country has comprehensive AI legislation, and existing governance efforts largely take the form of international statements and declarations, national policy and strategy documents, and draft laws.
There is emerging consensus across the world on the key principles in AI governance. For example, the need for systems to be transparent and explainable, and to actively counter algorithmic discrimination, are referenced in policies, strategies, and draft laws emerging from the European Union, the United States, the United Kingdom, and China. Other cross-cutting themes include provisions on safety and security (particularly on the need to conduct both internal and external audits of advanced AI systems); data privacy; and the need for human oversight and accountability.
However, there is substantial divergence in how these principles should be implemented – as if each region were using virtually the same ingredients to create different dishes. National and regional governance regimes are being built atop pre-existing regulatory cultures. Thus, the United States’ approach to date has been piecemeal, sector-specific, and distributed across federal agencies; while the European Union – through its draft AI Act – seeks to create comprehensive risk-based regulation that would require each member state to establish new national authorities to administer.
Given this divergence of approaches, and that the most appropriate national governance regime will likely vary based on the local context of a given region, UN bodies can play an important role in fostering international collaboration and in clarifying how emerging principles should be interpreted.
[The original paper then includes a brief discussion of the state of AI governance in Africa, which I've omitted from this post.]
Global guidance from international bodies such as the UN is necessary to ensure that all states can benefit from the boons offered by increasingly advanced AI systems, while constraining these technologies’ risks. Acknowledging the complexity of this topic, we invite the High-Level Advisory Board to consider the following points in its work:
In laying out the above case, I have tried to rely on as few premises as possible. For example, there are clearly compelling reasons for policymakers the world over to take seriously risks from misuse without them needing to accept the concept of AGI (particularly if one thinks of AGI as emerging gradually rather than at a discrete moment in time). I've done this because I think it makes the work of persuasion easier by keeping the focus at a level of abstraction more easily understandable to a general audience (so it sticks within the Overton window); and because I think many of the intermediate steps necessary to establish effective global governance regimes are the same regardless of precisely how the problem is framed. But I might be mistaken on this.
As I've had the opportunity to lay out arguments like this to a range of audiences this year (particularly those in the media and in civil society), I continue to workshop the material. So I'd welcome constructive feedback of any form.
Executive summary: The paper argues that AI governance poses unique challenges due to rapid advances in capabilities, global power dynamics, and risks of misuse.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.