Also no governance course from AGI Safety Fundamentals in a while

My independent impression here, having facilitated in this course and in other virtual programs, is that the curriculum provides ~90% of the value of the AGISF Governance course.^[1] Therefore, I'd encourage those looking to skill up to simply get started working through the curriculum independently, rather than wait for the next round of the course.^[2]

1. ^{^}

   Caveat: The discussion-and-support aspects of the course may have evolved since early 2022, when I facilitated, in ways that'd change my ~90% estimate.

2. ^{^}

   This “get started independently” conclusion follows, in my view, even with a much weaker premise: that the curriculum provides ~50% of the course's value, say. And I'd be very surprised if many AGISF alumni believe that less than half of the course's value comes from the curriculum.

To help with the talent pipeline, GovAI currently runs twice-a-year three-month fellowships. We've also started offering one-year Research Scholar positions. We're also now experimenting with a new policy program. Supporting the AI governance talent pipeline is one of our key priorities as an organization.

That being said, we're very very far from filling the community's needs in this regard. We're currently getting far more strong applications than we have open slots. (I believe our acceptance rate for the Summer Fellowship is something like 5% and will probably keep getting lower. We now need to reject people who actually seem really promising.) We'd like to scale our programs up more, but even then there will still be an enormous unmet need. I would definitely welcome more programs in this space!

Rethink Priorities unfortunately wasn't able to offer our own Fellowship this year due to capacity reasons and financial constraints (especially post-FTX), but we would be excited to potentially run one next year.

Instead this year we put a lot of our ops capacity behind the Existential Risks Alliance via our Special Projects program and helped them run a large fellowship this year. I hope that was helpful to the community.

I think I'm also just generally more excited to create more permanent jobs in AI governance and strategy work than do fellowships (though of course we can and will do both) as I think a bigger bottleneck right now is what people do after they do a fellowship and making sure we have enough permanent opportunities for people. I think the "what do I do after SERI MATS?" is also a problem too.

This seems like a useful topic to raise. Here's a pretty quickly written & unsatisfactory little comment: 

• I agree that there's room to expand and improve the pipeline to valuable work in AI strategy/governance/policy. 
• I spend a decent amount of time on that (e.g. via co-leading RP AI Governance & Strategy team, some grantmaking with EA Infrastructure Fund, advising some talent pipeline projects, and giving lots of career advice).
• If a reader thinks they could benefit from me pointing you to some links or people to talk to, or via us having a chat (e.g. if you're running a talent pipeline project or strongly considering doing so), feel free to DM me. 
  • (But heads up that I'm pretty busy so may reply slowly or just with links or suggested people to talk to, depending on how much value I could provide via a chat but not via those other quicker options.)

I totally agree there's a gap here. At BlueDot Impact  (/ AGI safety fundamentals), we're currently working on understanding the pipeline for ourselves.

We'll be launching another governance course in the next week, and in the longer term we will publish more info on governance careers on our website, as and when we establish the information for ourselves.

In the meantime, there's great advice on this account, mostly targeted at people in the US, but there might be some transferrable lessons:

https://forum.effectivealtruism.org/users/us-policy-careers

One specific thing I'll mention in case it's relevant to some people looking at this post: The AI Governance & Strategy team at Rethink Priorities (which I co-lead) is hiring for a Compute Governance Researcher or Research Assistant. The first application stage takes 1hr, and the deadline is June 11. @readers: Please consider applying and/or sharing the role! 

We're hoping to open additional roles sometime around September. One way to be sure you'd be alerted if and when we do is filling in our registration of interest form.  

It's worth mentioning the Horizon Fellowship and RAND Fellowship.

Nice post, and I appreciate you noticing something that bugged you and posting about it in a pretty constructive manner.

In AI alignment, I think SERI MATS has improved the early-career pipeline dramatically-- kudos to them. Maybe I should ask them why they haven't expanded to AI strategy or if they have takes on that pipeline.

I know that around the start of this year, the SERI SRF (not MATS) leadership was thinking seriously about launching a MATS-styled program for strategy/governance. I'm not sure if the idea is still alive, though.

Also, CBAI ran a pilot AI strategy research fellowship this past winter, which I participated in and found worthwhile. At the time they were, I think, planning on running a bigger version of the fellowship in the summer, though it appears that's no longer happening.

no summer research programs this year from [...] SERI SRF

On the other hand, ERA, formerly known as CERI, and CHERI are running fellowships this summer, and I expect they'll both have several AI governance fellows. (Though I do also expect, from what I know of these programs, that their AI governance focus will be more on applied governance than on strategy/theoretical governance. I don't have a strong stance on whether this is overall positive or negative, but it does mean there's less of an AI strategy pipeline.)

FYI I prefer "AI governance" over "AI strategy" because I think the latter pushes people towards trying to just sit down and think through arbitrarily abstract questions, which is very hard (especially for junior people). Better to zoom in more, as I discuss in this post.

This situation was somewhat predictable and avoidable, in my view. I’ve lamented the early-career problem in the past but did not get many ideas for how to solve it. My impression has been that many mid-career people in relevant organizations put really high premiums on “mentorship,” to the point that they are dismissive of proposals that don’t provide such mentorship. 

There are merits to emphasizing mentorship, but the fact has been that there are major bottlenecks on mentorship capacity and this does little good for people who are struggling to get good internships. The result for me personally was at least ~4 internships that were not very relevant to AI governance, were not paid, and did not provide substantial career benefits (E.g., mentorship).

In summary, people should not let the perfect be the enemy of the good: I would have gladly taken an internship working on AI governance topics, even if I had almost no mentorship (and even if I had little or no compensation). I also think there are ways of substituting this with peer feedback/engagement.

I have multiple ideas for AI governance projects that are not so mentorship-dependent, including one pilot idea that, if it worked, could scale to >15 interns and entry-level researchers with <1 FTE experienced researcher in oversight. But I recognize that the ideas may not all be great (or at least their merits are not very legible). Unfortunately, we don’t seem to have a great ecosystem for sharing and discussing project ideas, at least if you aren’t well connected with people to provide feedback through your job or through HAIST/MAIA or other university groups.

Ultimately, I might recommend that someone aggregate a list of past programs and potential proposals, evaluate the importance of various goals and characteristics (e.g., mentorship, skill development, topic education, networks, credentials/CVs), and identify the key constraints/bottlenecks (e.g., funding vs. mentorship).

I also have the impression that there's a gap and would be interested in whether funders are not prioritizing it too much, or whether there's a lack of (sufficiently strong) proposals.

Another AI governance program which just started its second round is Training For Good's EU Tech Policy fellowship, where I think the reading and discussion group part has significant overlap with the AGISF program. (Besides that it has policy trainings in Brussels plus for some fellows also a 4-6 months placement at an EU think tank.)

This is a timely post. It feels like funding is a critical obstacle for many organisations. 

One idea: Given the recent calls by many tech industry leaders for rapid work on AI governance, is there an opportunity to request direct funding from them for independent work in this area. 

To be very specific: Has someone contacted OpenAI and said: "Hey, we read with great interest your recent article about the need for governance of superintelligence. We have some very specific work (list specific items)  in that area which we believe can contribute to making this happen. But we're massively understaffed and underfunded. With $1m from you, we could put 10 researchers working on these questions for 1 year. Would you be willing to fund this work?"

What's in it for them? Two things:

1. If they are sincere (as I believe they are), then they will want this work to happen, and some groups in the EA sphere are probably better placed to make it happen than they themselves are.
2. We can offer independence (any results will be from the EA group, not from OpenAI and not influenced or edited by OpenAI) but at the same time we can openly credit them with funding this work, which would be good PR and a show of good faith on their part. 

Forgive me if this is something that everyone is already doing all the time! I'm still quite new to EA!