AGI safety career advice

People often ask me for career advice related to AGI safety. This post (now also translated into Spanish) summarizes the advice I most commonly give. I’ve split it into three sections: general mindset, alignment research and governance work. For each of the latter two, I start with high-level advice aimed primarily at students and those early in their careers, then dig into more details of the field. See also this post I wrote two years ago, containing a bunch of fairly general career advice.

General mindset

In order to have a big impact on the world you need to find a big lever. This document assumes that you think, as I do, that AGI safety is the biggest such lever. There are many ways to pull on that lever, though—from research and engineering to operations and field-building to politics and communications. I encourage you to choose between these based primarily on your personal fit—a combination of what you're really good at and what you really enjoy. In my opinion the difference between being a great versus a mediocre fit swamps other differences in the impactfulness of most pairs of AGI-safety-related jobs.

How should you find your personal fit? To start, you should focus on finding work where you can get fast feedback loops. That will typically involve getting hands-on or doing some kind of concrete project (rather than just reading and learning) and seeing how quickly you can make progress. Eventually, once you've had a bunch of experience, you might notice a feeling of confusion or frustration: why is everyone else missing the point, or doing so badly at this? (Though note that a few top researchers commented on a draft to say that they didn't have this experience.) For some people that involves investigating a specific topic (for me, the question “what’s the best argument that AGI will be misaligned?“); for others it's about applying skills like conscientiousness (e.g. "why can't others just go through all the obvious steps?") Being excellent seldom feels like you’re excellent, because your own abilities set your baseline for what feels normal.

What if you have that experience for something you don't enjoy doing? I expect that this is fairly rare, because being good at something is often very enjoyable. But in those cases, I'd suggest trying it until you observe that even a string of successes doesn't make you excited about what you're doing; and at that point, probably trying to pivot (although this is pretty dependent on the specific details).

Lastly: AGI safety is a young and small field; there’s a lot to be done, and still very few people to do it. I encourage you to have agency when it comes to making things happen: most of the time the answer to “why isn’t this seemingly-good thing happening?” or “why aren’t we 10x better at this particular thing?” is “because nobody’s gotten around to it yet”. And the most important qualifications for being able to solve a problem are typically the ability to notice it and the willingness to try. One anecdote to help drive this point home: a friend of mine has had four jobs at four top alignment research organizations; none of those jobs existed before she reached out to the relevant groups to suggest that they should hire someone with her skillset. And this is just what’s possible within existing organizations—if you’re launching your own project, there are far more opportunities to do totally novel things. (The main exception is when it comes to outreach and political advocacy. Alignment is an unusual field because the base of fans and supporters is much larger than the number of researchers, and so we should be careful to avoid alignment discourse being dominated by advocates who have little familiarity with the technical details, and come across as overconfident. See the discussion here for more on this.)

Alignment research

I’ll start with some high-level recommendations, then give a brief overview of how I see the field.

1. Alignment is mentorship-constrained. If you have little research experience, your main priority should be finding the best mentor possible to help you gain research skills—e.g. via doing research in a professor’s lab, or internships at AI labs. Most of the best researchers and mentors aren't (yet) working on alignment, so the best option for mentorship may be outside of alignment—but PhDs are long enough, and timelines short enough, that you should make sure that your mentor would be excited about supervising some kind of alignment-relevant research. People can occasionally start doing great work without any mentorship; if you’re excited about this, feel free to try it, but focus on the types of research where you have fast feedback loops.
2. You’ll need to get hands-on. The best ML and alignment research engages heavily with neural networks (with only a few exceptions). Even if you’re more theoretically-minded, you should plan to be interacting with models regularly, and gain the relevant coding skills. In particular, I see a lot of junior researchers who want to do “conceptual research”. But you should assume that such research is useless until it cashes out in writing code or proving theorems, and that you’ll need to do the cashing out yourself (with threat modeling being the main exception, although even then I think most threat modeling is not concrete enough to be useful). Perhaps once you’re a senior researcher with intuitions gained from hands-on experience you’ll be able to step back and primarily think about potential solutions at a high level, but that can’t be your plan as a junior researcher—it’ll predictably steer you away from doing useful work.
3. You can get started quickly. People coming from fields like physics and mathematics often don’t realize how much shallower deep learning is as a field, and so think they need to spend a long time understanding the theoretical foundations first. You don’t—you can get started doing deep learning research with nothing more than first-year undergrad math, and pick up things you’re missing as you go along. (Coding skill is a much more important prerequisite, though.) You can also pick up many of the conceptual foundations of alignment as you go along, especially in more engineering heavy roles. While I recommend that all alignment researchers eventually become familiar with the ideas covered in the Alignment Fundamentals curriculum, upskilling at empirical research should be a bigger priority for most people who have already decided to pursue a career in alignment research and who aren't already ML researchers.
   Some recommended ways to upskill at empirical research (roughly in order):
   1. MLAB
   2. ARENA
   3. Jacob Hilton’s deep learning curriculum
   4. Neel Nanda's guide to getting started with mechanistic interpretability
   5. Replicating papers
      Each of these teaches you important skills for good research: how to implement algorithms, how to debug code and experiments, how to interpret results, etc. Once you’ve implemented an algorithm or replicated a paper, you can then try to extend the results by improving the techniques somehow.
4. Most research won’t succeed. This is true both on the level of individual projects, and also on the level of whole research directions: research is a very heavy-tailed domain. You should be looking hard for the core intuitions for why a given research direction will succeed, the absence of which may be hidden under mathematics or complicated algorithms (as I argue here). (You can think of this as a type of conceptual research, but intended to steer your own empirical or theoretical work, rather than intended as a research output in its own right.) In the next section I outline some of my views on which research directions are and aren't promising.

Alignment research directions

From my perspective, the most promising alignment research falls into three primary categories. I outline those below, as well as three secondary categories I think are valuable. Note that I expect the boundaries between all of these to blur over time as research on them progresses, and as we automate more and more things.

1. Scalable oversight: finding ways to leverage more powerful models to produce better reward signals. Scalable oversight research may be particularly high-leverage if it ends up being adopted widely, e.g. as a tool for preventing hallucinations (like how alignment teams’ work on RLHF has now been adopted very widely).
   1. The theoretical paper I most often point people to is Irving et al.’s debate paper.
   2. The empirical paper I most often point people to is Saunders et al.’s critiques paper, which can be seen as the simplest case of the debate algorithm; Bowman et al. (2022) is also useful from a methodological perspective.
   3. The two other well-known algorithms in this area are iterated amplification and recursive reward modeling. My opinion is that people often overestimate the differences between these algorithms, and that standard presentations of them obfuscate the ways in which they’re structurally similar. I personally find debate the easiest to reason about (and it seems like others agree, since more papers build on it than on the others), hence why I most often recommend people work on that.
   4. Will scalable oversight just lead to more capabilities advances? This is an important question; one way I think about it is in terms of the generator-discriminator-critique gap from Saunders et al.’s critiques paper. Specifically, while I expect that closing the generator-discriminator gap is a dual-purpose advance (and could be good or bad depending on your other views), closing the discriminator-critique gap by producing correct human-comprehensible explanations should definitely be seen as an alignment advance.
2. Mechanistic interpretability: finding ways to understand how networks function internally. While still only a small subfield of ML, I think of it as a way of pushing the whole field of ML from a “behaviorist” perspective that only focuses on inputs and outputs towards a “cognitivist” framework that studies what’s going on inside neural networks. It’s also much easier to do outside industry labs than scalable oversight work. To get started, check out Nanda's 200 Concrete Open Problems in Mechanistic Interpretability.
   1. Three strands of mechanistic interpretability work:
      1. Case studies: finding algorithms inside networks that implement specific capabilities. My favorite papers here are Olsson et al. (2022), Nanda et al. (2023), Wang et al. (2022) and Li et al. (2022); I’m excited to see more work which builds on the last in particular to find world-models and internally-represented goals within networks.
      2. Solving superposition: finding ways to train networks to have fewer overlapping concepts within individual neurons. The key resource here is Elhage (2022) (as well as other work in the Transformer Circuits thread).
      3. Scalable interpretability: finding algorithms to automatically identify or modify internal representations. My favorite papers: Meng et al. (2022) and Burns et al. (2023) (although some consider the latter to be closer to scalable oversight work).
3. Alignment theory: finding formal frameworks we can use to reason about advanced AI. I want to flag that success at this type of research is even more heavy-tailed than the other research directions I’ve described—it seems to requires exceptional mathematical skills, a deep understanding of ML theory, and nuanced philosophical intuitions. I'm not optimistic that any of the research directions listed here will work out, but they are attempting to address such fundamental problems that even partial successes could be a big deal.
   1. I’m most excited about Christiano’s work on formalizing heuristic arguments, Kosoy’s learning-theoretic agenda (particularly infra-bayesianism), and various work by Scott Garrabrant (e.g. geometric rationality, finite factored sets, and Cartesian frames).
   2. Historically most of the work in this category has been done by MIRI (e.g. work on functional decision theory and Garrabrant induction). Their output has dropped significantly lately, though; so I mainly think of them as having a handful of researchers pursuing their individual interests, rather than a unified research agenda.
   3. Why do I think alignment theory is worth pursuing? In large part because scientific knowledge is typically very interconnected. Alignment theory often seems disconnected from modern ML—but the motions of the stars once seemed totally disconnected from events on earth. And who could have guessed that understanding variation in the beaks of finches would advance our understanding of...well, basically everything in biology? In many domains there are key principles that explain a huge range of phenomena, and the main difficulty is finding a tractable angle of attack. That's why asking the right questions is often more important than actually getting concrete results. For example, asking "what is the optimal strategy in this specific formalization of a 2-player game?" is a large chunk of the work of inventing game theory.

Three other research areas that seem important, but less central:

1. Evaluations: finding ways to measure how dangerous and/or misaligned models are.
   1. There’s been little published on this so far; the main thing to look at is the ARC evals (also discussed in section 2.9 of the GPT-4 system card). In general it seems like alignment evals are very difficult, so most people are focusing on evals for measuring dangerous capabilities instead.
   2. My own opinion is that evaluations will live or die by how simple and scalable they are. The best evals would be easily implementable even by people without any alignment background, and would meaningfully track improvements all the way from current systems up to superintelligences. In short, this is because the primary purpose of evals is to facilitate decision-making and coordination, and both of these benefit hugely from legible and predictable metrics.
2. Unrestricted adversarial training: finding ways to generate inputs on which misaligned systems will misbehave.
   1. It seems like there are strong principled reasons to expect this to be difficult—in general you can only generate fake data which fools one model using a much more powerful model. But it may be possible to find unrestricted adversarial examples by leveraging mechanistic interpretability, as explored in this post by Christiano.
   2. The empirical paper I point people to most often is Ziegler et al. (2022) (see also the other papers they cite).
3. Threat modeling: understanding and forecasting how AGI might lead to catastrophic outcomes.
   1. I most often point people to my own recent paper (Ngo et al., 2022). Other good work includes reports by Joe Carlsmith and Ajeya Cotra. (Cohen et al. (2022) make a peer-reviewed case for existential risk from AGI, but it’s too focused on outer alignment for me to buy into it.)
   2. One threat modeling research direction that seems valuable is understanding gradient hacking (and understanding cooperation between different models more generally). Another is to explore the specific ways that AGIs are most likely to be deployed in the real world, and what sorts of vulnerabilities they may be able to exploit.

By contrast, some lines of research which I think are overrated by many newcomers to the field, along with some critiques of them:

1. Cooperative inverse reinforcement learning (the direction that Stuart Russell defends in his book Human Compatible); critiques here and here.
2. John Wentworth’s work on natural abstractions; exposition and critique here, and another here.
3. Work which relies on agents acting myopically, including by only making next-timestep predictions (e.g. work on the simulators abstraction, or on conditioning predictive models); critique here.

Governance work

I mentally split this into three categories: governance research, lab governance, and policy jobs. A few high-level takeaways for each:

1. Governance research
   1. The main advice I give people who want to enter this field: pick one relevant topic and try to become an expert on it. There are about two dozen topics where I wish there were a world expert on applying this topic to making AGI go well, and no such person exists; I’ve made a list of those topics below. To learn about them I strongly recommend not just reading and absorbing ideas, but also writing about them. It’s very plausible that, starting off with no background in the field, within six months you could write a post or paper which pushes forward the frontier of our knowledge on how one of those topics is relevant to AGI governance.
   2. You don’t necessarily need to stick with your choice longer-term; my claim is mainly that it’s important to have some concrete topic to investigate. As you do so, you’ll gradually branch out to other topics which are tangentially relevant, and pick up a broader knowledge of the field (the Governance Fundamentals course is one good way of doing so). Eventually you’ll be able to do “strategy research” with much wider implications. But trying to do that from the beginning is a bad plan—it’ll go much better with a base of detailed expertise to work from.
   3. In general I think people overrate “analysis” and underrate “proposals”. There are many high-level factors which will affect AGI governance, and we could spend the rest of our lives trying to analyze them. But ultimately what we need is concrete mechanisms which actually move the needle, which are currently in short supply. Of course you need to do analysis in order to understand the factors which will influence proposals’ success, but you should always keep in mind the goal of trying to ground it out in something useful.
   4. Relatedly, I personally don’t think that quantitative modeling is very valuable. I have yet to see such a model of a big-picture question (e.g. compute projections, takeoff speed, timelines) whose conclusions substantively change my opinions about what the best governance proposals are. If such a model is a strong success it may shift my credences from, say, 25% to 75% in a given proposition. But that’s only a factor of 3 difference, whereas one plan for how to solve governance could be one or two orders of magnitude more effective than another. And in general models rarely move me that much, because even a few free parameters allow people to dramatically overfit to their intuitions; I’d typically prefer having a short summary of the core insights that the person doing the modeling learned during that process. So prioritize plans first, insights second and models last.
   5. Don’t be constrained too much by political feasibility, especially when formulating early versions of a plan. Almost nobody in the world has both good intuitions for how politics really works, and good intuitions for how crazy progress towards AGI will be. All sorts of possibilities will open up in the future—we just need to be ready with concrete proposals when they do. However, a deep understanding of the fundamental drivers of today’s policy decisions will be helpful in navigating when things start changing much faster.
2. AI lab governance
   1. Leading labs are often amenable to carrying out proposals which don’t strongly trade off against their core capabilities work; the bottleneck is usually the agency and work required to actually implement the proposal. Thus interventions of the form “tell labs to care more about safety” generally don’t work very well, whereas interventions of the form “here is a concrete ask, here are the specific steps you’d need to take, here’s a person who’s agreed to lead the effort” tend to go well. This post conveys that idea particularly well.
   2. It’s hard for people outside labs to know enough details about what’s going on inside labs to be able to make concrete proposals, but I expect there are a few important cases where it’s possible. This probably looks fairly similar to the path I outlined in the section on governance research, of first gaining expertise on a specific topic, then generating specific proposals.
   3. There is a specific skill of getting things done inside large organizations that most EAs lack (due to lack of corporate experience, plus lack of people-orientedness), but which is particularly useful when pushing for lab governance proposals. If you have it, lab governance work may be a good fit for you.
3. Policy-related jobs
   1. By this I mean going to work in government-related positions, with the goal of trying to get into a position where you can help make government regulation go well. I don’t have too much to say here, since it’s not my area of expertise. You should probably take fairly general advice (e.g. the advice here) about how to have a successful career in this area, and then figure out how to go faster under the assumption that people will get increasingly stressed about AI. Short masters degrees and policy fellowships are quick ways to fast-track towards mid-career policy roles; getting even a small amount of legible AI expertise (e.g. any CS/AI-related degree or job) is also helpful.

List of governance topics

Here are some topics where I wish we had a world expert on applying it to AGI safety. One example of what great work on one of these topics might look like: Baker’s paper on lessons from nuclear arms control (a topic which would have been on this list if he hadn’t written that).

One cluster of topics can be described roughly as “anything mentioned in Yonadav Shavit’s compute governance paper”, in particular:

1. Tamper-evident logging in GPUs
2. Global tracking of GPUs
3. Proof-of-learning algorithms
4. On-site inspections of models
5. Detecting datacenters
6. Building a suite for verifiable inference
7. Measuring effective compute use (e.g. by measuring and controlling for algorithmic progress)
8. Regulating large-scale decentralized training (if it becomes competitive with centralized training)

Another cluster: security-related topics such as

1. Preventing neural network weight exfiltration (by third parties or an AI itself)
2. Evaluating the possibility of autonomous replication across the internet
3. Privilege escalation from within secure systems (e.g. if your coding assistant is misaligned, what could it achieve?)
4. Datacenter monitoring (e.g. if unauthorized copies of a model were running on your servers, how would you know?)
5. Detecting unauthorized communication channels between different copies of a model.
6. Detecting tampering (e.g. if your training run had been modified, how would you know?)
7. How vulnerable are nuclear command and control systems?
8. Scalable behavior monitoring (e.g. how can we aggregate information across monitoring logs from millions of AIs?)

And a more miscellaneous (and less technical) third category:

1. What regulatory apparatus within the US government would be most effective at regulating large training runs?
2. What tools and methods does the US government have for auditing tech companies?
3. What are the biggest gaps in the US export controls to China, and how might they be closed?
4. What AI applications or demonstrations will society react to most strongly?
5. What interfaces will humans use to interact with AIs in the future?
6. How will AI most likely be deployed for sensitive tasks (e.g. advising world leaders) given concerns about privacy?
7. How might political discourse around AI polarize, and what could mitigate that?
8. What would it take to automate crucial infrastructure (factories, weapons, etc)?