The very short version: If you are interested in an introduction to transformative AI governance, apply here to be a participant or here to be a (compensated) facilitator by December 15th.

Summary & Information About Applying

While many people are interested in transformative AI governance, there is currently no scalable introduction to the field that offers substantial breadth, depth, context, accountability, and information about relevant career opportunities. Anecdotally, finding even a few of those things can be tough. Aiming to improve this state of affairs, I’m excited to introduce this AI governance course. This program seeks to efficiently bring people up to speed on issues in transformative AI governance through an 11-week virtual course. It consists of 8 weeks of readings, facilitated group discussions, speaker sessions, and a 4-week capstone project, for a total of ~3-4 hours per week.

Collaborators, advisors, and acknowledgements:

  • In creating the curriculum, I drew gratefully and extensively from eight older AI governance reading lists,[1] collaboration with Richard Ngo (a former machine learning research engineer at DeepMind, now working on the policy team at OpenAI), generously detailed feedback from Ben Garfinkel (The Centre for the Governance of AI’s Acting Director) and Luke Muehlhauser (Open Philanthropy’s AI Governance and Policy Program Officer), and additional useful advice from Jenny Xiao (Columbia University), Sam Clarke (CSER), a researcher from CSET/FHI, and two affiliates of Concordia Consulting.[2] (This does not constitute endorsements from any of these advisors or their organizations.)

  • Professionals with relevant experience in the Future of Humanity Institute, the UK Office for AI, the Center for Security and Emerging Technology, the Centre for the Study of Existential Risk, the Center for Human-Compatible Artificial Intelligence, and the Stanford Institute for Human-Centered Artificial Intelligence have already expressed interest in facilitating course discussions, as have some students with relevant context.[3]

  • Logistically, the course will be run as another track alongside the technical AI alignment track of the AGI Safety Fundamentals Programme (after which this program and post are modeled, if the name didn’t give it away), in collaboration with Dewi Erwan and Jamie Bernardi, with additional support from Vince Huang and Will Aldred.

If you're interested in joining the next version of the course (taking place January - March 2022) apply here to be a participant or here to be a (compensated) facilitator. Applications are open to anyone and close December 15th. Note that:

  • This is the same application form as the one used for the technical AI alignment track of the AGI Safety Fundamentals programme; you can select to apply for the governance track or the technical track. (The AGI Safety Fundamentals Programme is now what we’re currently calling the umbrella program that is hosting both tracks.)
  • If you've already applied to the AGI Safety Fundamentals programme and selected the governance track, then you've already applied for this program—no need to do anything else.
  • We will offer honoraria of £800 (~$1,070) to facilitators for their time, through Cambridge EA CIC.
  • The curriculum is intended to be accessible to people with a wide range of backgrounds—backgrounds in computer science, AI, or social sciences are not necessary.

I encourage you to apply if:

  • You are potentially interested in eventually doing work aimed at improving the trajectory of AI through AI governance research/policy (or full-time work that could indirectly help a lot, like technical AI safety research or recruiting people to work on pressing problems).
  • And you’d like to learn more about whether to pursue a career in this field, and/or you’d like to get background context that would be a useful step toward contributing.
  • And the syllabus seems interesting to you, including its focus on large-scale risks.

This post contains an overview of the course and an abbreviated version of the curriculum; the full version (which also contains optional readings and notes/context on each core reading, and future discussion prompts and project ideas) can be found here. Comments and feedback are very welcome, either on this post or in the full curriculum document; suggestions of new exercises, prompts, or readings would be particularly helpful. I'll continue to make updates until shortly before the program starts.

See the final section of this post for an FAQ.

Course overview

Participants are divided into groups of 4-6 people, matched based on their prior knowledge of transformative AI risks and governance. From weeks 1 to 7, each group and their discussion facilitator will meet for 1.5 hours to discuss the readings and exercises. The course consists of 8 weeks of readings, plus a final project. Broadly speaking, the first half of the course explores potential risks, while the second half focuses on strategic considerations and possibilities for good governance. After Week 7, participants will have several weeks to work on projects of their choice, to present at the final session. Each week (apart from week 0) each group and their discussion facilitator will meet for 1.5 hours to discuss the readings and exercises.

Each week's curriculum contains:

  • Key ideas for that week
  • Core readings
  • Optional readings
  • Two exercises (participants should pick one to do each week)
  • Further notes on the readings
  • Discussion prompts for the weekly session
  • Week 0 replaces the small group discussions with a lecture plus live group exercises, since it's aimed at getting people with little machine learning knowledge up to speed quickly.

Some high-level approaches that informed the syllabus design:

  • Focus on large-scale risks: A focus on especially large-scale risks that future AI systems may pose
  • Problem-first: An emphasis on understanding relevant problems and risk scenarios, for better generating and prioritizing among paths to impact
  • Pluralistic: (Attempted) Inclusion of a range of the (very different) views that are prominent in the transformative AI governance community
  • Foundational: An emphasis on relevant concepts, context, and big-picture ideas, to help prepare participants to themselves discover new strategic considerations and promising policy options

Topics for each week:

This syllabus is structured into three parts:

  • Before the main readings, there is a recommended week of background context:
    • Week 0 (Recommended Background): AI, Machine Learning, and the Importance of Their Long-Term Impacts
  • Part 1 dives into the risks, with the motivating idea that thoroughly understanding problems is very helpful for both identifying potential solutions and prioritizing among them.
    • Week 1: Introduction to Governing World-Transforming AI
    • Week 2: Deep Dive - The Alignment Problem
    • Week 3: Potential Sources of AI Existential Risk
  • Part 2 dives into the questions of how governance decisions can help address AI risks.
    • Week 4: Avoiding a Race to the Bottom
    • Week 5: Corporate Actors & Levers
    • Week 6: (Inter)Governmental Actors & Levers, With Historical Case Studies
    • Week 7: Career Advice & Opportunities for People Who Are Interested in Helping

Some limitations of this program and suggested mindsets:

  • In general, transformative AI governance remains at least partly pre-paradigmatic, and explanation of existing ideas has been limited. Somewhat more concretely:
    • Many ideas in transformative AI strategy and governance have never been properly written up and published.[4]
    • Viewpoints on important questions (e.g. timelines) often change pretty rapidly.
    • Many arguments and ideas have yet to be proposed and scrutinized in much depth.
    • There is much disagreement and uncertainty about:
      • What abstractions are useful[5]

      • What topics are useful to think about

      • How large different risks are

      • Other critical object-level questions

    • In other words, borrowing from Olah and Carter’s essay on “Research Debt,” the transformative AI governance field currently has substantial limitations from limited exposition of important ideas, undigested ideas, and (debatably) bad abstractions.
    • Some transformative AI governance research focuses on preparing for advances in AI, and some strategies involve drawing on advances in AI safety technical research. However, we don’t know precisely what any of these advances will be, or in precisely what (e.g. political) context they will take place. So some strategic clarity and precision might only come with time. More generally though, big-picture strategic research has much worse feedback loops than some other fields.
  • While the above state of affairs need not be a reason to despair (the field is still very young and has arguably made significant progress), it means that it can be useful to keep these things in mind (much more so than in an average course):
    • Unfortunately, public materials often:
      • Give impressions of views that are outdated by a few years.
      • Include little of the thinking of researchers who write and publish less often.
      • Include little material on some important topics.
    • Facilitators/mentors (and organizers) are still in the process of learning, especially if they're new to the area.
    • Overall, this content is not a settled map of the AI governance landscape.
    • There is very little “established wisdom” or informed consensus.
    • Skeptical and questioning mindsets are especially valuable.

Click here for the full version of the curriculum, which contains additional readings, notes/context on each core reading, and project ideas, and will include exercises and discussion prompts.

This is the current version of the curriculum, as of writing. It will be further edited leading up to the start of the program.


Why is this valuable if there's a mentorship bottleneck in AI governance?

A view that seems common in the transformative AI governance field is that the research field’s growth is bottlenecked by capacity to train/mentor junior researchers. We might worry that, because of this, introducing more people to the field is of little value (or maybe even harmful, if it means more people will be frustrated by the difficulty of breaking into the field). I think there are several reasons why this program is valuable anyway:

  • There is a good chance the mentorship bottleneck will soon be eased.
    • From talking with a few people at relevant organizations, there seems to be significant interest in easing the mentorship bottleneck, as well as some promising, scalable ideas for doing so.
  • Mentorship is not a bottleneck in all important areas of the field.
    • My sense is that mentorship (more specifically, mentorship from members of this community) is not as severe of a bottleneck for ladder-climbing in government, which also seems important.
  • Even under mentorship bottlenecks, efficient introductions to the field are useful.
    • It’s convenient for people to be efficiently introduced to the field, both for saving their time and for lowering mentorship costs.

As an “outside view” consideration, relevant professionals have generally been encouraging.

What should I do if I already did the AGI safety fundamentals program and am interested in this program?

If you’re interested after checking out this program’s syllabus, we’d encourage you to apply! The governance track is sufficiently different in focus that you're still likely to learn a lot of new content, and the foundational ideas within AGI safety are complex and confusing enough to warrant more discussion and evaluation.

What if I want to do both the technical alignment and the governance tracks?

Since this would mean a roughly doubled time commitment, we mainly just recommend this to people who have a lot of free time or are at a particularly pivotal point in their career (such that learning more about these fields soon would inform time-sensitive decisions). If you’d like to apply to do both tracks simultaneously, there’s an option to indicate that preference in the “choose a track” question of the application form.

I didn’t receive a confirmation email—did you receive my application?

We likely did—our version of the software used to process applications doesn’t send confirmation emails.

What if I already applied to the upcoming AGI Safety Fundamentals program and indicated an interest in the governance track?

Then you’re all set! No need to submit an additional application here.

What if I already did an AI governance reading group?

Take a look at the syllabus and decide from there! If you previously participated in one of the smaller AI governance reading groups I helped run, I think you’ll find lots of new content in this syllabus.

What if I have feedback on the syllabus, or ideas for exercises/discussion prompts/etc?

I’d love to hear your ideas! Please let me know by commenting below or on the linked syllabus, messaging me through the forum, or emailing stanfordeateam [at] gmail [dot] com (although that is checked less frequently).

I’ll be editing the syllabus and adding exercises and discussion prompts until the program starts in January.


  1. These, as well as an incomplete private list: [1] [2] [3] [4] [5] [6] [7] ↩︎

  2. For context on my own background, among other hats, I currently run the Stanford Existential Risks Initiative’s AI governance and US policy programming. I’ve only been learning about this field for about a year, so I’m especially thankful for the 20+ years of collective expertise our advisors brought to this curriculum. ↩︎

  3. If you are listed here and would prefer a different description/additional caveats/etc., please let me know! ↩︎

  4. Some specific content gaps in current public materials, as flagged by one reviewer:

    • A good taxonomy of plausible sources of existential risk from AI
    • More thorough analysis of some potential AI-related existential risks to which misalignment is non-central
    • A good overview of different views on “theories of victory” and cruxes between people who disagree on them
      • (I don’t have a very precise sense of what this reviewer has in mind with this term.)
    • (There are probably many more things.)
  5. As an example of researchers having different views on which abstractions to use, researchers have proposed and used a slew of different terms and concepts for thinking about future AI capabilities: AGI, human-level AI, advanced AI, strong AI, superintelligence (collective, speed, quality), transformative AI, oracle AI, genie AI, sovereign AI, tool AI, agentic AI, optimizer, mesa-optimizer, comprehensive AI services, and probably others. As another example, definitions/framings of “outer alignment,” “inner alignment,” and related terms are contested (you can find relevant in-the-weeds discussion e.g. here, here, here, and here) (thanks to Sam Clarke for flagging these to me). ↩︎


Sorted by Click to highlight new comments since: Today at 2:58 AM

This is fantastic! Thank you for making this.

Would you like us to convert the readings into audio to make it easier for people to participate? This would be pretty easy on our end.

I'm not involved in this program, but I would like to see that happen. Though note that some of the readings are copyrighted.

Thank you! That would be great.

(I'm not sure I'd have capacity to manage hosting of the audio files, so no worries if that would be a requirement (although my sense was that it generally isn't, with Nonlinear?))

Fantastic! You're right, we'd just put it into podcast form so people could listen on their podcast players, so no need to host the audio files or anything. I'll DM you with more details.

Thanks for your work on this, and thanks to all other people who helped as well! From a fairly quick look, this seems very exciting and useful, and I've already recommended a few people apply.

I came back to this post to try to find a link to the actual latest non-abbreviated curriculum, but wasn't able to find that via skimming. Is this the right link? If so, maybe you should edit the post to more clearly link to that near the top? (I found it via a link in a link in one of the footnotes, so kind-of just got lucky.)

Thanks! Good to know that wasn't easy enough to find--I've now added links in several additional spots in the post, including near the top. (The link you had was also right.)

This sounds amazing. I would love to apply. Are you planning to do another one next year? Is there somewhere that I can be notified when the applications open up again?

Thanks so much! My sense is we'll probably run this regularly (on a yearly basis or more often than that), although it feels a bit early to commit to that before we've seen how the current version goes.

Updates will likely go out to at least the SERI mailing list.

Depending on your situation and interests, I might tentatively suggest diving in to whatever parts seem most interesting to you rather than (only) waiting for next time (since if you're considering relevant career changes, that extra 6 months or so might be pretty impactful).

This is helpful, thanks! I will check out the curriculum.

Thanks for organising :)

When do you expect decisions on applications will be made by?

Thanks! I expect we'll get them out by the end of the month (hopefully on the week after the application deadline), although there's enough potential bottlenecks that I can't promise that.