Several months ago, I summarized Open Philanthropy's work on AI governance (which I lead) for a general audience here. In this post, I elaborate my thinking on AI governance in more detail for people who are familiar with effective altruism, longtermism, existential risk, and related topics. Without that context, much of what I say below may be hard to understand, and easy to misunderstand.
These are my personal views, and don't necessarily reflect Open Phil's views, or the views of other individuals at Open Phil.
In this post, I
- briefly recap the key points of my previous post,
- explain what I see as the key bottlenecks in the space, and
- share my current opinions about how people sympathetic to longtermism and/or AI existential risk mitigation can best contribute today.
Open Phil's AI governance work so far (a recap)
First, some key points from my previous post:
-
In practice, Open Phil's grantmaking in Potential Risks from Advanced Artificial Intelligence is split in two:
-
One part is our grantmaking in "AI alignment," defined here as "the problem of creating AI systems that will reliably do what their designers want them to do even when AI systems become much more capable than their designers across a broad range of tasks."
-
The second part, which I lead, is our grantmaking in "AI governance," defined here as "local and global norms, policies, laws, processes, politics, and institutions (not just governments) that will affect social outcomes from the development and deployment of AI systems."
-
Our AI focus area is part of our longtermism-motivated portfolio of grants, and we focus on AI alignment and AI governance grantmaking that seems especially helpful from a longtermist perspective. On the governance side, I sometimes refer to this longtermism-motivated subset of work as "transformative AI governance" for relative concreteness, but a more precise concept for this subset of work is "longtermist AI governance."
-
It's difficult to know which “intermediate goals” we could pursue that, if achieved, would clearly increase the odds of eventual good outcomes from transformative AI (from a longtermist perspective). As such, our grantmaking so far tends to focus on:
-
…research that can help clarify how AI technologies may develop over time, and which intermediate goals are worth pursuing.
-
…research and advocacy aimed at the few intermediate goals we've come to think are clearly worth pursuing, such as particular lines of AI alignment research, and creating greater awareness of the difficulty of achieving high assurance in the safety and security of increasingly complex and capable AI systems.
-
…broad field-building activities, for example scholarships, career advice for people interested in the space, professional networks, etc.
-
…training, advice, and other support for actors with plausible future impact on transformative AI outcomes, as always with a focus on work that seems most helpful from a longtermist perspective.
Key bottlenecks
Since our AI governance grantmaking began in ~2015, we have struggled to find high-ROI grantmaking opportunities that would allow us to move grant money into this space as quickly as we'd like to. As I see it, there are three key bottlenecks to our AI governance grantmaking.
Bottleneck #1: There are very few longtermism-sympathetic people in the world, and even fewer with the specific interests, skills, and experience to contribute to longtermist AI governance issues.
As a result, the vast majority of our AI governance grantmaking has supported work by people who are (as far as I know) not sympathetic to longtermism (and may have never heard of it). However, it's been difficult to find high-ROI grantmaking opportunities of this sort, too, because, as mentioned above:
Bottleneck #2: We lack the strategic clarity and forecasting ability to know which "intermediate goals" are high-ROI or even net-positive to pursue (from a longtermist perspective). If we had more clarity on intermediate goals, we could fund more people who are effectively pursuing those goals, whether they are sympathetic to longtermism or not.
In the past few years, I've spent hundreds of hours discussing possible high-value intermediate goals with other "veterans" of the longtermist AI governance space. Thus I can say with some confidence that there is very little consensus on which intermediate goals are net-positive to pursue, or even on more fundamental questions such as "Is AI x-risk lower if AI progress is faster or slower?" or "Could a broadly superhuman 'prosaic' AGI system be robustly aligned with human values at all?" or "What are the most likely paths by which AI technology could lead to existential catastrophe or eucatastrophe?" In fact, there might be even less consensus on such questions than there was a few years ago.
To some extent, such lack of consensus should be expected for almost any "wicked problem," especially those requiring long-range forecasting (which it's not clear anyone can do successfully). However, my sense is that the challenge of reaching strategic consensus (or at least strategic clarity) has been even harder in longtermist AI governance than it has been in, for example, longtermism-motivated biosecurity and pandemic preparedness (another Open Phil focus area).
So, can we fund work that will help to clarify which intermediate goals are worth pursuing (from a longtermist perspective)? We've done some of this, but (per bottleneck #1) there aren't many longtermists who can be funded to do this work, and also:
Bottleneck #3: It's difficult to define and scope research projects that can help clarify which intermediate goals are worth pursuing from a longtermist perspective even if those research projects are done by people who are not themselves thinking about the issues from a longtermist perspective.
My advice to longtermists interested in AI governance
To be clear, I think we're making progress on these bottlenecks, and we may see major breakthroughs on one or more of them in the next few years. But given current bottlenecks, what should longtermists do today? My intuitions about this have evolved considerably over the last few years, and they will no doubt continue to evolve. In general, the advice below is still fairly tentative, and I hope that people interested to help mitigate AI x-risk will try to think through the situation themselves and (in some cases) pursue strategies that are quite different from what I advise below (while taking care to avoid causing accidental harm).
That said, as of today, here is some of my advice:
-
Given the lack of strategic clarity/consensus in longtermist AI governance, project and career path decisions should be especially influenced by factors like experimentation / learning about yourself, aptitude development, and other career capital development.
-
In the absence of greater strategic clarity/consensus (via strategy-relevant research and by just seeing how things play out in the world, and thus which scenarios we're most likely headed toward), one high-EV option is to prioritize future impact, e.g. by building credibility both deserved and perceived over many years of dedicated service — while continuing to study and think through how you can best have a positive impact in potential future scenarios. Once there is more strategic clarity/consensus, you may then be in a position to help increase the degree to which wise and broadly beneficial AI governance ideas are actually implemented (by governments, companies, etc.). Also, your experience in positions of generally increasing credibility and responsibility will help you think more about the details of the incentives and constraints facing people in and near such roles, thus improving your ability to contribute to the quest for greater strategic clarity (see #4 below).
-
It might be especially valuable to focus on potential impact during a plausible "AI crunch time," i.e. a period lasting 1-20 years when the decisions most impactful on TAI outcomes might be made. Of course, it's difficult to know when such an AI crunch time might occur, how long it might last, and which kinds of credibility and roles might lend themselves to helpful impact, but I think we have enough information now to make some reasonable guesses. Personally, it seems to me that if there is an AI crunch time before 2100, it is most likely to begin between 2025 and 2060, it is most likely to last 2-15 years, and many of the most impactful decisions will be made by (i) technology and national security policymakers in the US, China, and a few other jurisdictions, (ii) AI-leading tech firms with access to very large compute resources, and perhaps (iii) a few key semiconductor firms.
-
I hope to write more in the future about how best to pursue these particular career paths and more generally optimize for AI crunch time impact. In the meantime, see e.g. 80,000 Hours on AI Policy and US AI Policy, two articles on working in Congress, Science and Technology Policy Opportunities, AI policy careers in the EU, and AI Governance Career Paths for Europeans (but I don't endorse everything in those posts). I wish there were similar guides on how to work toward impactful roles outside the US/European government and think tank space, but I don't yet have any to recommend.
-
If you might be interested in a US AI policy path, consider applying to the Open Philanthropy Technology Policy Fellowship. It was designed to accommodate people who don't yet know much about the policy process or which paths might interest them.
-
Another thing you could do in the absence of greater strategic clarity is to help increase the number of thoughtful longtermists with particular aptitudes and other career capital that is most relevant to working on AI governance, e.g. by promoting longtermism to key audiences and by helping longtermists develop relevant aptitudes and career capital. (Prioritizing future impact can also help with this, because e.g. it might get you into roles with hiring authority.) Some ideas for how to do this are in the post Open Philanthropy is seeking proposals for outreach projects.
-
You can also contribute to research that may provide greater strategic clarity in the future. However, I caution that several people have tried this (including me) over the past several years, and it's not clear we have much better strategic clarity or consensus than we did a few years ago. I think some recent work has produced small amounts of added strategic clarity, but it's not yet sufficient to make it much clearer which intermediate goals are robustly good to pursue — in part because answers to that question are dependent on many factors, only a few of which have been studied in depth so far. Such work is very hard to do in a way that is likely correct, convincing to others, and thorough enough to be action-guiding.
- If you want to try this kind of work, in most cases I recommend that you first (1) study many different relevant topics, (2) discuss the issues in depth with "veterans" of the topic, and (3) gain experience working in relevant parts of key governments and/or a top AI lab (ideally both) so that you acquire a detailed picture of the opportunities and constraints those actors operate with.
- Or, you can try to help answer one or more narrowly-scoped questions that an AI x-risk motivated person who is closer to having those advantages has identified as especially action-informing (even if you don't have the full context as to why).
- I hope to write more in the future about which research projects I personally think might be the most action-informing on the current margin. In the meantime, if you're interested in this path, you could work on learning as much as you can about longtermism-motivated AI alignment and AI governance issues, and also study broader relevant topics such as machine learning, security studies, and how other high-stakes technology industries use a "defense in depth" approach to avoiding catastrophic failure.
Notes
Thank you Luke for sharing your views. I just want to pick up one thing you said where your experience of the longtermist space seems sharply contrary to mine.
You said: "We lack the strategic clarity ... [about] intermediate goals". Which is a great point and I fully agree. Also I am super pleased to hear you have been working on this. You then said:
This surprised me when I read it. In fact my intuition is that such work is highly neglected, almost no one has done any of this and I expect it is reasonably tractable. Upon reflection I came up with three reasons for my intuition on this.
1. Reading longtermist research and not seeing much work of this type.
I have seem some really impressive forecasting and trend analysis focused but if anyone had worked on setting intermediate goals I would expect to see some evidence of basic steps such as listing out a range of plausible intermediate goals or consensus building exercises to set viable short and mid term visions of what AI governance progress looks like (maybe it's there and I've just not seen it). If anyone had made a serious stab at this I would expect to have seen thorough exploration exercises to map out and describe possible near-term futures, assumption based planning, scenario based planning, strategic analysis of a variety of options, tabletop exercises, etc. I have seen very little of this.
2. Talking to key people in the longtermist space and being told this research is not happening.
For a policy research project I was considering recently I went and talked to a bunch of longtermists about research gaps (eg at GovAI, CSET, FLI, CSER, etc). I was told time and time again that policy research (which I would see as a combination of setting intermediate goals and working out what policies are needed to get there) was not happening, was a task for another organisation, was a key bottleneck that no-one was working on, etc.
3. I have found it fairly easy to make progress on identifying intermediate goals and short-term policy goals that seem net-positive for long-run AI governance
I have an intermediate goal of: key actors in positions of influence over AI governance are well equipped to make good decisions if needed (at an AI crunch time). This leads to specific policies such as: Ensuring clear lines of responsibility exist in military procurement of software /AI or, if regulation happens it should be expert driven outcome based regulation or some of the ideas here. I would be surprised if longtermists looking into this (or other intermediate goals I routinely use) would disagree with the above intermediate goal or that the policy suggestions move us towards that goal. I would say this work has not been difficult.
– –
So why is our experience of the longtermist space so different. One hunch I have is that we are thinking of different things when we consider "strategic clarity on intermediate goals".
In supporting governments to make long-term decisions and has given me a sense of what long-term decision making and "intermediate goal setting" and long-term decision making involves. This colours the things I would expect to see if the longtermist community was really trying to do this kind of work and I compare longtermists' work to what I understand to be best practice in other long-term fields (from forestry to tech policy to risk management). This approach leaves me thinking that there is almost no longtermist "intermediate goal setting" happening. Yet maybe you have a very different idea of what "intermediate goal setting involves" based on other fields you have worked in.
It might also be that we read different materials and talk to different people. It might be that this work has happened I've just missed it or not read the right stuff.
– –
Does this matter? I guess I would be much more encouraging about someone doing this work than you are and much more positive about how tractable such work is. I would advise that anyone doing this work should have a really good grasp of how wicked problems are addressed and how long-term decision making works in a range of non-EA fields and the various tools that can be used.
As far as I know it's true that there isn't much of this sort of work happening at any given time, though over the years there has been a fair amount of non-public work of this sort, and it has usually failed to convince people who weren't already sympathetic to the work's conclusions (about which intermediate goals are vs. aren't worth aiming for, or about the worldview cruxes underlying those disagreements). There isn't even consensus about intermediate goals such as the "make government generically smarter about AI policy" goals you suggested, though in some (not all) cases the objection to that category is less "it's net harmful" and more "it won't be that important / decisive."
Thank you Luke – great to hear this work is happening but still surprised by the lack of progress and would be keen to see more such work out in public!
(FWIW Minor point but I am not sure I would phrase a goal as "make government generically smarter about AI policy" just being "smart" is not good. Ideally want a combination of smart + has good incentives + has space to take action. To be more precise when planning I often use COM-B models, as used in international development governance reform work, to ensure all three factors are captured and balanced.)