Should you work in the European Union to do AGI governance?

19

If anyone reading this post thinks that the arguments in favor outweigh the arguments against working on EU AI governance, then consider applying for the EU Policy Analyst role that we are hiring for at the Future of Life Institute: https://futureoflife.org/2022/01/31/eu-policy-analyst/. If you have any questions about the role, you can participate in the AMA we are running: https://forum.effectivealtruism.org/posts/j5xhPbj7ywdv6aEJc/ama-future-of-life-institute-s-eu-team.

richard_ngo

4y

14

Intuitive reaction: I think these are all valuable arguments to have explicitly laid out, thanks for doing so. I think they don't quite capture my main intuitions about the value of EU-directed governance work, though; let me try explain those below.

One intuition draws from the classic distinction between realism and liberalism in international relations. Broadly speaking, I see the EU as being most relevant from a liberalist perspective; whereas it's much less relevant from a realist perspective. And although I think of both sides as having important perspectives, the dynamics surrounding catastrophic AGI development feel much better-described by realism than by liberalism - it feels like that's the default that things will likely fall back to if the world gets much more chaotic and scary, and there are potential big shifts in the global balance of power.

Second intuition: when it comes to governing AGI, I expect that acting quickly and decisively will be crucial. I can kinda see the US govt. being able to do this (especially by spinning off new agencies, or by presidential power). I have a lot more trouble seeing the EU being able to do this, even in a best-case scenario (does the EU even have the ability in theory, let alone in practice, to empower fast-moving organisations with specific mandates)?

Compared with your arguments, I think these two intuitions are more focused on working backwards from a "theory of victory" to figure out what's useful today (as opposed to working forwards towards gaining more influence). Our overall thinking about theories of victory is still so nascent, though, that it feels like there's a lot of option value in having people going down a bunch of different pathways. Plus I have a few other intuitions in favour of the value of EU-directed governance research: firstly, I think people often overestimate the predictability of AGI development. E.g. a European DeepMind popping up within the next decade or two doesn't seem that much less plausible than the original DeepMind popping up in England. Might just take a few outlier founders to make that happen. Secondly, separate from progress on AI itself, it does seem plausible that the EU will have significant influence over the chip supply chain going forward (right now most notably via ASML, as you mention).

Overall I do think people with a strong comparative advantage should do EU-governance-related things, I'm just very uncertain how strong the comparative advantage needs to be for that to be one of the best career pathways (although I do know at least a few people whose comparative advantage does seem strong enough for that to be the correct move).

tamgent

4y

7

I think liberalism vs realism is an interesting lens but the conclusion doesn't seem right to me. You say you're working backwards from a theory of victory, but at least that argument was working backwards from a theory of catastrophe. I think this is an is-ought problem, and if we want things to go well then we might want to actively encourage more cooperative IR, whilst also not ignoring the powerful forces.

[anonymous]

4y

3

I believe “Victory” here means avoiding catastrophic AGI, which could require encouraging more cooperative international relations.

As for your point on Liberalism vs Realism Richard, I think it is captured by at least 6 arguments listed in the post (Brussels effect, AI development/governance collaborations, growing the political capital, AI superpower, US-China differential, military pathways). Indeed when I read:

"the dynamics surrounding catastrophic AGI development feel much better-described by realism than by liberalism - it feels like that's the default that things will likely fall back to …”

I decompose “dynamics” and “things” ultimately as actions taken by relevant actors (if necessary, here is my post-length explanation of what I mean by that). At the risk of frustrating generations of IR schools of thought, I'd push the framework further by saying that “Realism” could very roughly be reduced to “the way relevant actors think when they have a low level of trust in relevant actors from different nations”. And to make sure the other half of IR scholars also shriek, "Liberalism" would be reduced to the "the way relevant actors think when they have a high level of trust in relevant actors from different nations" (I would also reduce "nations" to "groups of actors", but that's not necessary to the main point.)

The implications of that lower level of international trust (aka Realism) for the debate on whether the EU is relevant, is that the validity of these 6 arguments changes (e.g. in a more Realist world, the Brussels effect is weaker than in a Liberal world). Let me know, Richard, if you think that there is separate, independent causal link between international trust levels and the relevance of the EU that doesn’t rely on these 6 arguments/factors, and I can add it to the main post.

For the concrete impact of governance on AGI, which you tamgent seem to allude to, the implications of Realism are deeper: mistrust definitely alters all the “game theoretical” results. This reduces the range of options that an AGI-concerned decision-maker has. A plausible example is his.her inability to establish a joint alignment testing protocol/international standard. Is there anything that can be achieved with mistrust/realism that cannot be achieved with trust/liberalism? I don’t think so. (That doesn’t mean that increasing this level of international trust is a cost-effective intervention though).

[anonymous]

4y

4

Thank you Richard – I think your second intuition is a great point. Does this rephrasing capture your point? I included the notion of decisiveness as well, which is related. If so, and with your permission, I would add to the main post.

+++
7. EU-level policymaking is slower to react to events like AGI and/or less decisive
Compared to the US or China, the EU has little centralized power and resources. All 27 EU member states preserve significant control over EU decision-making, as embodied by veto powers and “executive” procedures that still require member states’ collective approval. This is occasionally made more difficult by the European Parliament, which includes 7 political parties and 705 members and whose power has been growing over the years. As a result, most decisions from the European Commission require consultations with many stakeholders and therefore take time. Moreover, the EU-level public budget represents ~1.5% of GDP, compared to ~20% of GDP in the US – so even when there is agreement, it is unclear whether it can garner the resources to whip up a decisive response. This structure also prevents the EU to produce the equivalent of US executive orders.

How does it affect the EU’s relevance?

If the development of AGI requires a quick or well-resourced policy response from government, EU-level policymaking might not be as influential as American or Chinese policymaking.

+++

My opinion:
This factor is definitely relevant, so thanks for bringing it up; I think it crucially depends on takeoff speeds though. (Anyone with a better understanding of US policymaking should correct me if I am wrong here:) If AGI requires strong and one-off government interventions within a 3-6 month window, US policymaking offers a considerable advantage through its strong and quick executive powers. However, for interventions with time horizons of over 6 months where emergency is not as important, the case for executive orders fades as far as I understand. Decision-making therefore falls onto the legislative power, which has been deadlocked for the past 12 years. On the EU’s side however, legislative power has been particularly strong – spurning tech policies with relative ease. Even though the EU procedure is slow, I have the intuition its impact is more structural than US executive orders (please correct me?). In the cases where AGI safety requires a government intervention within 3-6 months, my hope would be that institutions are already in place to guarantee that this quick intervention takes place – e.g. a regulatory agency that has been mandated to do AI code probing and auditing for >20 years before takeoff, for whom AI labs alignment would be common practice.

This conversation makes me realize a more constructive version of this post would list “important factors” for the relevance of the EU rather than arguments in favour and against. Ah well.

Jan-Willem

4y

14

I think there is one argument I really want to back, but I also want to provide a different angle: “Growing the political capital of AGI-concerned people”

I think that even when you think there are substantial odds that the EU doesn’t play an important role in regulation of AGI, having political capital could still be useful for other (tech-related) topics. Quite often I think there is a “halo-effect” related to being perceived as an tech-expert in government. That means that if you are perceived as a tech expert in government because you know a lot about AI, people will also perceive you as an expert on other technologies (where the EU might be more relevant).

This is also one of the reasons that I advise people to work in AI/tech regulation, even when it’s not (solely) on the long term consequences of AI we care about, but e.g. on short term risks or even more on the side of economic stimulus of AI development. Often it will provide EAs with the political capital and credibility to deal with long term / x-risk relevant risks later on when there is an opportunity to switch roles.

Jan-Willem

4y

6

If you believe however that the EU becomes irrelevant at all (argument 5 against), all policy careers for EAs in mainland Europe become quite unappealing suddenly. This makes me think: if you believe the EU market and political environment favor AGI safety (argument 4 in favor), shouldn’t it be a priority for European EAs to keep the EU a relevant political force?

[anonymous]

4y

2

I think that's a very indirect intervention whose cost-effectiveness is probably lower than many other priorities (given the political forces and the many other stakeholders' strong interests, neglectedness is quite low) - but maybe I am missing something? It sounds definitely relevant as a "byproduct" of one's career though. Would it therefore be a good principle to, ceteris paribus, push for the intervention that strengthen the EU, while working on one's own priority?

[anonymous]

4y

2

Agreed on the halo-effect, but besides one's prestige, I think one's region-specific knowledge and network matter a lot and does not transfer. As a result, if the EU is less relevant, building up prestige in the EU might not be as efficient as building up prestige in China or the US, given that in parallel you'd be developing a network and region-specific knowledge that will be more helpful to be impactful overall.

(That being said, even though I wanted to avoid anchoring the reader by expressing my opinion in the post, I expect the EU to be most relevant right now for AGI governance given the institutional precedents it sets. I believe the lack of investment in EA time & money there is an unfortunate mistake. So the "if the EU is less relevant" scenario should be disregarded in my opinion.)

Mau

4y

13

Thanks for this! I appreciate the clear outline of arguments around an important question.

It sounds like this post might be centered around the question, "Should people who want to improve AGI governance work in the EU?", rather than the question of the title, "Is the EU Relevant for AGI Governance?". The focus on the former question seems right to me, since that's the more decision-relevant question. So I wonder if explicitly reframing (e.g., retitling) this post so that it's centered on the former question would make this post more clear?

After all, a bunch of the arguments presented seem to address the value of EU AI governance work, rather than its relevance to AI. I'll try to show that in a subcomment.

Mau

4y

6

Here's the in-the-weeds subcomment:

4.The EU market and political environment favor AGI safety

I agree this magnifies the importance of the EU, but just if we assume that the EU is relevant.

5.Direct influence from inside relevant AI labs is limited

This seems like an argument about relative tractability, rather than relevance.

6.Growing the political capital of AGI-concerned people

This point only seems like an argument for the EU's relevance if we assume (a) that the EU is relevant, or (b) that political capital transfers well across policy communities. (And that seems like a nontrivial assumption, since (a) is the desired conclusion and (b) seems overstated at best.) The point is much more plausible on its own as an argument for EU AI governance work in the near term being valuable.

Personal fit [...] Low-regret career pathway [...] High personal career capital [...] Neglectedness

These all seem like arguments for EU AI governance work being in some way valuable; I don't see how any of these are arguments for the EU being relevant to AI's trajectory.

EU governance would slow down US research towards AGI more than it would Chinese research

This seems to me like an argument for the EU's relevance to AI (and also for the value of working in EU AI governance--if the EU might take some actions that would be harmful for AI's trajectory, then working in the EU and preventing these actions would be a way to positively contribute to AI governance).

Higher returns on EA investment in the US and China AI governance space than in the EU’s [...] The EA EU AI governance space is not mature enough for personal career progression and impact

These also strike me as arguments about career value rather than institutional relevance.

[anonymous]

4y

1

I think there is a bit of confusion on several fronts here and in your main comment.
On title/framing:
The post aims to be descriptive of factors relevant for decision-making, rather than prescribing a decision (ergo absence of "should", absence of judgment of the arguments). As mentioned in the intro of the post, I am hoping to inform or trigger a conversation/further research, rather than to directly presume that people should make a decision about it. I trust readers would derive decision-relevant aspects themselves (is that what I am wrong about?)

Relevance vs value & importance:

I agree this magnifies the importance of the EU, but just if we assume that the EU is relevant.

I am not sure I understand. The way I see this, the value of EU governance work and the importance of the EU is a function of the relevance of the EU for AGI governance.

The concept of relevance:
I am confused by "relevance" in your comment:

This point only seems like an argument for the EU's relevance if we assume (a) that the EU is relevant

[...]

These all seem like arguments for EU AI governance work being in some way valuable; I don't see how any of these are arguments for the EU being relevant to AI's trajectory.

Given your comment mentioning AI's trajectory, is it possible you understood the post as being about AI trajectory rather than the way we govern AI trajectory? Also, to be sure we are on the same page, in the post, relevance is not a binary concept and is directly related to the actions leading up to AGI governance.

If you have time, please let me know what I misunderstood.

Mau

4y

5

Hm, now I'm also a little confused. I agree with a bunch of the points you clarify (and I think they're in line with how I had originally interpreted this post). Specifically, I think we're on the same page about all of these things, among others:

This post aims to outline decision-relevant considerations rather than to make an overall prescription
The value of EU governance work and the importance of the EU is a function of [among other variables] the relevance of the EU for AGI governance
This post is about the way people govern the trajectory of AI
Relevance is not a binary concept

--

I'll try to clarify my earlier point. I'm trying to draw a distinction between these two things:

(a) the relevance of EU policy to the trajectory of AI, and
(b) the value of pursuing (AI-related) work in EU policymaking.

I agree that arguments about (a) are, by extension, arguments about (b). For example, the Brussels Effect (as the post notes) is an argument for the relevance of EU AI policy to AI's trajectory, and this makes it an argument for the value of EU AI policy work.

But I don't think it necessarily goes the other way; some argument can be an argument about (b) without being an argument about (a). For example, this post raises the point that EU policy matters for animal welfare. This option value may be a reason why someone should work in EU policy, but it tells us nothing about whether the EU's AI policy will influence the global trajectory of AI. So it is an argument about (b), but not about (a).

More broadly, my understanding is that roughly all of this post's arguments are arguments regarding (b), but they are not all arguments regarding (a). So it may be clearer to frame the post as a collection of considerations about (b), rather than a collection of considerations about (a), even if the post doesn't propose an overall judgement.

Or maybe I've misunderstood?

[anonymous]

4y

3

Thank you for taking the time for explaining this so clearly, I understand now √ I will edit the title and link to this comment to disclaim on framing.

MichaelA🔸

4y

7

Thanks for this post!

Prompted in part by this post, I've now made a Collection of work on whether/how much people should focus on the EU if they’re interested in AI governance for longtermist/x-risk reasons, which might be of interest to the author and/or to readers. I'd also be keen for people to note other relevant work in comments on that shortform so the collection can become more comprehensive (which I hope will in turn make it easier for other people to get up to speed, make decisions, and make novel contributions building on the existing work).

[anonymous]

4y

1

Thanks Michael, that's a great idea!

MaxRa

4y

5

Thanks, this is great!

1. The EU is not an AI superpower

When I was thinking about how important this argument is, I thought that I still expect any generically influential international body to be able to have a lot of influence if it wants to and is seen as being a productive party to negotiations. What do you think? Maybe there are similar situations today where parties who don't have much relevant economic power can influence negotiations by showing interest and ability to contribute to finding better solutions?

Though this would only hold in a cooperative atmosphere I suppose... would also be interesting how cooperative we expect the atmosphere of international politics around AI to be in the coming years and decades.

[anonymous]

4y

3

Interesting - that seems to converge towards the discussion on liberalism vs realism atmosphere above. On your specific point, if "productive party" = "party able/motivated to contribute to finding better solutions", I would agree. In political negotiations at many level, from my limited experience there is this odd phenomenon about the relevance of expertise/motivation: expertise/motivation doesn't matter if everyone has roughly the same amount, but then matters significantly as soon as a subset of parties are having a lot of it (because motivation c an be converted into further expertise or proactive control of the negotiations procedure such as sending reminders to comment to other parties, etc.).

I have observed that phenomenon between nations (e.g. economically small Estonia mattering much more on e-government or digital discussions than you would expect) and between policymakers (e.g. for multiple dossiers, the opinion of the Member of European Parliament (MEP) or diplomat that cares/knows most about the issue matters more and is more respected by other MEPs/diplomats).

I imagine low international trust environments would make this difficult though (as knowledge/expertise can be manipulated for one's own interest)

MaxRa

4y

5

Yes, that was also what I intended to communicate with "productive party". Thanks for the response and especially the examples about Estonia and expert parliamentarians.

A related thought: the more that parties are able to productively contribute to discussions, the more productive the discussions will be and feel, and the more likely a cooperative atmosphere can be maintained. Might be another reason why increasing AI governance expertise of the EU could be really helpful, as it seems very likely to me that the EU will be involved in many important discussions on AI governance.

[anonymous]

4y

1

That's a good point

Should you work in the European Union to do AGI governance?

Should you work in the European Union to do AGI governance?

Arguments in favor

1. The Brussels Effect

2. The EU is taking AI governance decisions now

3. The EU is part of many AI development and governance collaborations

4. The EU market and political environment favor AGI safety

5. Direct influence from inside relevant AI labs is limited

6. Growing the political capital of AGI-concerned people

Other arguments in favor

Arguments against

1. The EU is not an AI superpower

2. EU legislation does not matter enough

3. EU governance would slow down US research towards AGI more than it would Chinese research

4. The EU is irrelevant to military and national security AGI pathways

5. The EU is fragile and could become irrelevant

6. Policy-related AI governance negatively affects AGI safety

Other arguments against