Hide table of contents

On December 8, EU policymakers announced an agreement on the AI Act. This post aims to briefly explain the context and implications for the governance of global catastrophic risks from advanced AI. My portfolio on Open Philanthropy’s AI Governance and Policy Team includes EU matters (among other jurisdictions), but I am not an expert on EU policy or politics and could be getting some things in this post wrong, so please feel free to correct it or add more context or opinions in the comments!

If you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via this short Google form.

Context

The AI Act was first introduced in April 2021, and for the last ~8 months, it has been in the “trilogue” stage. The EU Commission, which is roughly analogous to the executive branch (White House or 10 Downing Street), drafted the bill; then, the European Parliament (sort of like the U.S. House of Representatives, with seats assigned to each country by a population-based formula) and the Council of the EU (sort of like the pre-17th-Amendment U.S. Senate, with each country's government getting one vote in a complicated voting system)[1] each submitted proposed revisions; then, representatives from each body negotiated to land on a final version (analogous to conference committees in the US Congress).

In my understanding, AI policy folks who are worried about catastrophic risk were hoping that the Act would include regulations on all sufficiently capable GPAI (general-purpose AI) systems, with no exemptions for open-source models (at least for the most important regulations from a safety perspective), and ideally additional restrictions on “very capable foundation models” (those above a certain compute threshold), an idea floated by some negotiators in October. In terms of the substance of the hoped-for regulations, my sense is that the main hope was that the legislation would give the newly-formed AI Office substantial leeway to require things like threat assessments/dangerous capabilities evaluations and cybersecurity measures, with a lot of the details to be figured out later by that Office and by standard-setting bodies like CEN-CENELEC’s JTC-21

GPAI regulations appeared in danger of being excluded after Mistral, Aleph Alpha, and the national governments of France, Germany, and Italy objected to what they perceived as regulatory overreach and threatened to derail the Act in November. There was also some reporting that the Act would totally exempt open-source models from regulation.

What’s in it?

Sabrina Küspert, an AI policy expert working at the EU Commission, summarized the results on some of these questions in a thread on X:

  • The agreement does indeed include regulations on “general-purpose AI,” or GPAI. 
  • There does appear to be a version of the “very capable foundation models” idea in the form of “GPAI models with systemic risks,” which are based on capabilities and “reach,” which I think means how widely deployed they are. 
  • It looks like GPAI models are presumed to have these capabilities if they’re trained on 10^25 FLOP, which is one order of magnitude smaller than the October 30 Biden executive order’s cutoff for reporting requirements (and which would probably include GPT-4 and maybe Gemini, but no other current models as far as I know).
  • Küspert also says “no exemptions,” which I interpret to mean “no exemptions to the systemic-risk rules for open-source systems.” Other reporting suggests there are wide exemptions for open-source models, but the requirements kick back in if the models pose systemic risks. However, Yann LeCun is celebrating based on this part of a Washington Post article: "The legislation ultimately included restrictions for foundation models but gave broad exemptions to “open-source models,” which are developed using code that’s freely available for developers to alter for their own products and tools. The move could benefit open-source AI companies in Europe that lobbied against the law, including France’s Mistral and Germany’s Aleph Alpha, as well as Meta, which released the open-source model LLaMA." So it’s currently unclear to me where the Act lands on this question, and I think a close review by someone with legal or deep EU policy expertise might help illuminate.

The Commission’s blog post says: “For very powerful models that could pose systemic risks, there will be additional binding obligations related to managing risks and monitoring serious incidents, performing model evaluation and adversarial testing. These new obligations will be operationalised through codes of practices developed by industry, the scientific community, civil society and other stakeholders together with the Commission.” (I’m guessing this means JTC-21 and similar, but if people with more European context can better read the tea leaves, let me know.)

Parliament’s announcement notes that GPAI systems and models will “have to adhere to transparency requirements” including “technical documentation, complying with EU copyright law and disseminating detailed summaries about the content used for training.” I think these transparency requirements are the main opportunity to develop strong requirements for evaluations.

Enforcement will be up to both national regulators and the new European AI Office, which, as the Commission post notes, will be “the first body globally that enforces binding rules on AI and is therefore expected to become an international reference point.” Companies that fail to comply with these rules face fines up to “35 million euro or 7% of global revenue,” whichever is higher. (Not sure whether this would mean 7% of e.g. Alphabet’s global revenue or DeepMind’s).

The Act also does what some people have called the obvious thing of requiring that AI-generated content be labeled as such in a machine-readable format, with fines for noncompliance. (Seems easy to do for video/audio, much harder for text, but at least requiring that AI chatbots notify users that they’re AI systems rather than humans would be a first step.)

This post focuses on the most relevant parts of the Act to frontier models and catastrophic risk, but most of the Act is focused on the application layer. It bans the use of AI for:

  • “biometric categorisation systems that use sensitive characteristics (e.g. political, religious, philosophical beliefs, sexual orientation, race);
  • untargeted scraping of facial images from the internet or CCTV footage to create facial recognition databases;
  • emotion recognition in the workplace and educational institutions;
  • social scoring based on social behaviour or personal characteristics;
  • AI systems that manipulate human behaviour to circumvent their free will;
  • AI used to exploit the vulnerabilities of people (due to their age, disability, social or economic situation).”

The Act will start being enforced at the end of “a transitional period,” which the NYT says will be 12-24 months. In the meantime, the Commission is launching the cleverly titled “AI Pact,” which seeks voluntary commitments to start implementing the Act’s requirements before the legal deadline. EU Commission president Ursula von der Leyen says “around 100 companies have already expressed their interest to join” the Pact.

How big of a deal is this?

A few takeaways for me so far:

  • Despite the frontier AI companies being American and English, the EU has what I’d describe as a moderate amount of leverage on AI companies by being a large market (~17% of global GDP in dollar terms). 
    • If the regulations they impose aren’t terribly costly and make enough sense, companies will likely comply with them. If they’re poorly executed or super costly, companies may fight them in court (as they did with the GDPR) or pull out of the EU market. So, European regulators will have a not-tiny budget of cost imposition that they could spend wisely and get a decent amount of safety (but of course can’t unilaterally pause AI development).
    • This effect will be especially important insofar as the EU’s regulations apply to training, rather than applications: AI companies might release EU-compliant versions of chatbots within the EU while deploying non-EU-compliant versions elsewhere, but due to the costs of training foundation models, they’re unlikely to train entirely separate models.
  • The EU and this GovAI paper (summarized here) are both very fond of the “de jure” Brussels Effect, where other jurisdictions imitate EU regulation. (They contrast this with the “de facto” Brussels Effect, which includes the kind of direct effects in the previous bullet.) So far, I haven’t seen many signs of the US or UK imitating the EU, but it’s possible that China’s approach will be informed by the EU. Other countries with less leverage over frontier AI might also be influenced, but this is less of a big deal.
  • There’s also an effect that Markus Anderljung pointed out to me, which is that even with no imitation by policymakers, regulators themselves might be influenced by the object-level outputs of the AI Office: if Europe rules that a particular AI system was not sufficiently evaluated or secured before release, some regulators might defer, as many countries’ pharma regulators apparently defer to the FDA.
  • The EU adopting pretty strong regulations even after industry and their allies in government were seriously activated is a good sign for the politics of AI regulation in similar polities. (This is less true to the extent that even very powerful/risky/expensive-to-train models are exempt if their weights get released.)
  • The multilateral EU regulating like this is also a step towards international agreements; a robust international regime needs to include the US, China, EU, and UK at bare minimum, and the EU might be important connective tissue between a mutually distrustful China and US/UK.

Making the AI Act effective for catastrophic risk reduction

The Act appears to stake out a high-level approach to Europe’s AI policy, but will very likely task the AI Office, standard-setting organizations (SSOs) like JTC-21, and EU member states with fleshing out a lot of detail and implementing the policies. Depending on the standardization and implementation phases over the next few years, the Act could wind up strongly incentivizing AI developers to act more safely, or it could wind up insufficiently detailed, captured by industry, bogged down in legal challenges, or so onerous that AI companies withdraw from the EU market and ignore the law. 

To achieve outcomes more like the former, people who would like to reduce global catastrophic risks from future AI systems could consider doing the following:

  • Joining the SSOs. These bodies tend to include a mix of industry lobbyists and civil society representatives, and the civil society folks have a huge range of priorities, so there are very few people at these who are focused on frontier model safety, and you could make a big difference by joining.
  • Working for the European AI Office or member state implementation bodies. By default, I think these offices (like most tech regulators) will have difficulty recruiting technically knowledgeable people. (The bar for “technically knowledgeable” in government tends to be pretty low; if you’re familiar with lots of the material on the AI Safety Fundamentals governance syllabus I think you’re in decent shape.) Having a few more such people in these offices could make them more informed about the risks and governance opportunities, both specifically regarding models with “systemic risks” and in general.
  • Policy research that aims to inform catastrophic-risk-focused people in these groups or other European institutions. Think tanks and research institutions tend to be on a spectrum from “policy/strategy development,” where they write reports about what policymakers should be aiming for, and advocacy, where they mostly take ideas developed by others and talk a lot with people in governments to turn them into policy outcomes. The people I’ve spoken with who are currently working closely with the EU (in the Commission, SSOs, or think tanks on the latter side of that spectrum) said that additional “upstream” (i.e. on the former side of the spectrum) policy work like that done by e.g. IAPS and GovAI would be really useful.
  • Generally, I think AI policy and governance folks should invest some time in understanding what’s going on in the EU (though some people might have strong comparative advantage reasons not to), and relatedly probably encourage catastrophic-risk-focused Europeans to try to do useful work in the EU rather than coming to the US. At the very least, in my view, the picture has changed in an EU-favoring direction in the last year (despite lots of progress in US AI policy), and this should prompt a re-evaluation of the conventional wisdom (in my understanding) that the US has enough leverage over AI development such that policy careers in DC are more impactful even for Europeans.

And once again: if you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via this short Google form.

  1. ^

    Thanks to the commenter Sherrinford for correcting me on these.

Comments13
Sorted by Click to highlight new comments since: Today at 11:37 PM

Thanks for sharing! Speaking as a European I think this is a pretty good summary of the latest state of events.

I currently expect the full text of the Act as agreed on in the trilogue to be published by the EU some time in January or February.

Thanks for sharing!

Küspert also says “no exemptions,” which I interpret to mean “no exemptions to the systemic-risk rules for open-source systems.” Other reporting suggests there are wide exemptions for open-source models, but the requirements kick back in if the models pose systemic risks. However, Yann LeCun is celebrating based on this part of a Washington Post article: "The legislation ultimately included restrictions for foundation models but gave broad exemptions to “open-source models,” which are developed using code that’s freely available for developers to alter for their own products and tools. The move could benefit open-source AI companies in Europe that lobbied against the law, including France’s Mistral and Germany’s Aleph Alpha, as well as Meta, which released the open-source model LLaMA." So it’s currently unclear to me where the Act lands on this question, and I think a close review by someone with legal or deep EU policy expertise might help illuminate.

It's a shame this is so unclear. To me this is basically the most important part of the act, and intuitively seems like it makes the difference between 'the law is net bad because it gives only the appearance of safety while adding a lots of regulatory overhead' and 'the law is good'.

Thanks for this overview, Trevor. I expect it'll be helpful– I also agree with your recommendations for people to consider working at standard-setting organizations and other relevant EU offices.

One perspective that I see missing from this post is what I'll call the advocacy/comms/politics perspective. Some examples of this with the EU AI Act:

  • Foundation models were going to be included in the EU AI Act, until France and Germany (with lobbying pressure from Mistral and Aleph Alpha) changed their position.
  • This initiated a political/comms battle between those who wanted to exclude foundation models (led by France and Germany) and those who wanted to keep it in (led by Spain).
  • This political fight rallied lots of notable figures, including folks like Gary Marcus and Max Tegmark, to publicly and privately fight to keep foundation models in the act.
  • There were open letters, op-eds, and certainly many private attempts at advocacy.
  • There were attempts to influence public opinion, pieces that accused key lobbyists of lying, and a lot of discourse on Twitter.

It's difficult to know the impact of any given public comms campaign, but it seems quite plausible to me that many readers would have more marginal impact by focusing on advocacy/comms than focusing on research/policy development.

More broadly, I worry that many segments of the AI governance/policy community might be neglecting to think seriously about what ambitious comms/advocacy could look like in the space.

I'll note that I might be particularly primed to bring this up now that you work for Open Philanthropy. I think many folks (rightfully) critique Open Phil for being too wary of advocacy, campaigns, lobbying, and other policymaker-focused activities. I'm guessing that Open Phil has played an important role in shaping both the financial and cultural incentives that (in my view) leads to an overinvestment into research and an underinvestment into policy/advocacy/comms. 

(I'll acknowledge these critiques are pretty high-level and I don't claim that this comment provides compelling evidence for them. Also, you only recently joined Open Phil, so I'm of course not trying to suggest that you created this culture, though I guess now that you work there you might have some opportunities to change it).

I'll now briefly try to do a Very Hard thing which is like "put myself in Trevor's shoes and ask what I actually want him to do." One concrete recommendation I have is something like "try to spend at least 5 minutes thinking about ways in which you or others around you might be embedded in a culture that has blind spots to some of the comms/advocacy stuff." Another is "make a list of people you read actively or talked to when writing this post. Then ask if there were any other people/orgs you could've reached out, particularly those that might focus more on comms+adovacy". (Also, to be clear, you might do both of these things and conclude "yea, actually I think my approach was very solid and I just had Good Reasons for writing the post the way I did.")

I'll stop here since this comment is getting long, but I'd be happy to chat further about this stuff. Thanks again for writing the post and kudos to OP for any of the work they supported/will support that ends up increasing P(good EU AI Act goes through & gets implemented). 

(Cross-posting from LW)

Thanks for these thoughts! I agree that advocacy and communications is an important part of the story here, and I'm glad for you to have added some detail on that with your comment. I’m also sympathetic to the claim that serious thought about “ambitious comms/advocacy” is especially neglected within the community, though I think it’s far from clear that the effort that went into the policy research that identified these solutions or work on the ground in Brussels should have been shifted at the margin to the kinds of public communications you mention.

I also think Open Phil’s strategy is pretty bullish on supporting comms and advocacy work, but it has taken us a while to acquire the staff capacity to gain context on those opportunities and begin funding them, and perhaps there are specific opportunities that you're more excited about than we are. 

For what it’s worth, I didn’t seek significant outside input while writing this post and think that's fine (given the alternative of writing it quickly, posting it here, disclaiming my non-expertise, and getting additional perspectives and context from commenters like yourself). However, I have spoken with about a dozen people working on AI policy in Europe over the last couple months (including one of the people whose public comms efforts are linked in your comment) and would love to chat with more people with experience doing policy/politics/comms work in the EU.

We could definitely use more help thinking about this stuff, and I encourage readers who are interested in contributing to OP’s thinking on advocacy and comms to do any of the following:

  • Write up these critiques (we do read the forums!); 
  • Join our team (our latest hiring round specifically mentioned US policy advocacy as a specialization we'd be excited about, but people with advocacy/politics/comms backgrounds more generally could also be very useful, and while the round is now closed, we may still review general applications); and/or 
  • Introduce yourself via the form mentioned in this post.

I appreciate the comment, though I think there's a lack of specificity that makes it hard to figure out where we agree/disagree (or more generally what you believe).

If you want to engage further, here are some things I'd be excited to hear from you:

  • What are a few specific comms/advocacy opportunities you're excited about//have funded?
  • What are a few specific comms/advocacy opportunities you view as net negative//have actively decided not to fund?
  • What are a few examples of hypothetical comms/advocacy opportunities you've been excited about?
  • What do you think about EG Max Tegmark/FLI, Andrea Miotti/Control AI, The Future Society, the Center for AI Policy, Holly Elmore, PauseAI, and other specific individuals or groups that are engaging in AI comms or advocacy? 

I think if you (and others at OP) are interested in receiving more critiques or overall feedback on your approach, one thing that would be helpful is writing up your current models/reasoning on comms/advocacy topics.

In the absence of this, people simply notice that OP doesn't seem to be funding some of the main existing examples of comms/advocacy efforts, but they don't really know why, and they don't really know what kinds of comms/advocacy efforts you'd be excited about.

Great summary. 

I was pleasantly surprised at how good this turned out to be, despite it having to be re-evaluated when Chat-GPT came along, despite the objections of major governments. 

The EU Commission is a fantastic organisation. Yes, massive levels of bureaucracy, but the people there tend to be extremely smart and very committed to doing what's best. Just being accepted to work in the Commission requires finishing in the top 1% or less of a very tough evaluation process and then passing a series of in-person evaluations. 

So normally when they produce a proposal, it has been thought through very carefully. 

Of course it's not perfect, and I especially appreciate that the post ends with tangible ideas for how to help make it more impactful. 

Obviously this is an area where we'll need to keep working all the time as AI evolves, as regulations elsewhere evolve. But good to see someone taking the lead and actually putting something tangible in place which seems to be 80/20 what's needed. Maybe this can be the starting point for an even better US legislation ??

Executive summary: The EU has reached an agreement on regulations for AI systems, including requirements for general-purpose AI systems that could reduce risks.

Key points:

  1. The EU's AI Act will regulate general-purpose AI systems and "very capable" models.
  2. It requires threat assessments, model evaluations, transparency, and addressing systemic risks.
  3. There are questions around exemptions for open-source models.
  4. The Act could influence companies due to the size of the EU market.
  5. Effective implementation requires expertise in standards bodies and regulators.
  6. More policy research could inform catastrophic risk reduction.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

It uses the language of "models that present systemic risks" rather than "very capable," but otherwise, a decent summary, bot.

At the very least, in my view, the picture has changed in an EU-favoring direction in the last year (despite lots of progress in US AI policy), and this should prompt a re-evaluation of the conventional wisdom (in my understanding) that the US has enough leverage over AI development such that policy careers in DC are more impactful even for Europeans.

Interesting! I don't quite understand what updated you. To me, it looks like, given the EU AI Act is mostly determined at this stage, there is less leverage in the EU, not more. Meanwhile, the approach the US takes to AI regulation still remains uncertain, indicating many more opportunities for impact.

The text of the Act is mostly determined, but it delegates tons of very important detail to standard-setting organizations and implementation bodies at the member-state level.

And your update is that this process will be more globally impactful than you initially expected? Would be curious to learn why.

The shape of my updates has been something like:

Q2 2023: Woah, looks like the AI Act might have a lot more stuff aimed at the future AI systems I'm most worried about than I thought! Making that go well now seems a lot more important than it did when it looked like it would mostly be focused on pre-foundation model AI. I hope this passes!

Q3 2023: As I learn more about this, it seems like a lot of the value is going to come from the implementation process, since it seems like the same text in the actual Act could wind up either specifically requiring things that could meaningfully reduce the risks or just imposing a lot of costs at a lot of points in the process without actually aiming at the most important parts, based on how the standard-setting orgs and member states operationalize it. But still, for that to happen at all it needs to pass and not have the general-purpose AI stuff removed.

November 2023: Oh no, France and Germany want to take out the stuff I was excited about in Q2. Maybe this will not be very impactful after all.

December 2023: Oh good, actually it seems like they've figured out a way to focus the costs France/Germany were worried about on the very most dangerous AIs and this will wind up being more like what I was hoping for pre-November, and now highly likely to pass!

More from tlevin
Curated and popular this week
Relevant opportunities