Thanks JP!

(Post 6/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

Some heuristics for prioritising between talent pipeline interventions

Explicit backchaining is one way to do prioritisation. I sometimes forget that there are other useful heuristics, like:

• Cheap to pilot
  • E.g. doesn't require new infrastructure or making a new hire
• Cost is easier to estimate than benefit, so lower cost things tend to be more likely to actually happen
• Visualise some person or org has been actually convinced to trial the thing. Imagine the conversation with that decision-maker. What considerations actually matter for them?
• Is there someone else who would do most of the heavy lifting?

(Post 5/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

Laundry list of talent pipeline interventions

• More AI governance groups/programs at universities
• Run workshops on the most marginally valuable aptitudes
  • E.g. a macrostrategy workshop could look like: people tell stories about, concretely, things could go badly, and backchaining to what we should do
• Run bootcamps on particularly important topics, e.g. compute
  • Help bring more people up to speed in most important areas
• Create better resources for teaching low-hanging fruit skills
  • E.g. resources for learning how to do reasoning transparency—currently this is pretty hard to learn, but seems like one of the more teachable skills
  • E.g. compile zero-shot tips that occasionally people just miss, like "don't read books from cover to cover"
  • Tag those resources onto existing programs
• AI-governance-careers.com
• "Ambitious talent search"
• The best way of predicting happiness and success on the job is to actually do the job -> hackathon type stuff. “This is the problem. Here’s the internet. Here’s some resources. Go nuts.”

Note to self: more detailed but less structured version of these notes here.

(Post 4/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

Some exercises for developing good judgement

I’ve spent a bit of time over the last year trying to form better judgement. Dumping some notes here on things I tried or considered trying, for future reference.

• Jump into the mindset of “the buck stops at me” for working out whether some project takes place, as if you were the grantmaker having to make the decision. Ask yourself: “wait, should this actually happen?”^[1]
  • (Rather than “does anything jump out as incorrect” or “do I have any random comments/ideas”—which are often helpful mindsets to be in when giving feedback to people, but don’t really train the core skill of good judgement.)
• I think forecasting trains a similar skill to this. I got some value from making some forecasts in the Metaculus Beginners’ Tournament.
• Find Google Docs where people (whose judgement you respect) have left comments and an overall take on the promisingness of the idea. Hide their comments and form your own take. Compare. (To make this a faster process, pick a doc/idea where you have enough background knowledge to answer without looking up loads of things).
• Ask people/orgs for things along the lines of [minimal trust investigations | grant reports | etc.] that they’ve written up. Do it yourself. Compare.
• Do any of the above with a friend; write your timeboxed answers then compare reasoning.

1. ^{^}

   I think this framing of the exercise might have been mentioned to me by Michael Aird.

(Post 3/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

Some hot takes on AI governance field-building strategy

• More people should consciously upskill as ‘founders’, i.e. people who form and lead new teams/centres/etc. focused on making AI go well
  • A case for more founders: plausibly in crunch time there will be many more people/teams within labs/govs/think-tanks/etc. that will matter for how AI goes. Would be good if those teams were staffed with thoughtful and risk-conscious people.
  • What I think is required to be a successful founder:
    • Strong in strategy (to steer their team in useful directions), management (for obvious reasons) and whatever object level work their team is doing
    • Especially for teams within existing institutions, starting a new team requires skill in stakeholder management and consensus building.
  • Concrete thing you might consider doing: if you think you might want to be a founder, and you agree with the above list of skills, think above how to close your skill gaps
• More people should consciously upskill for the “AI endgame” (aka “acute risk period” aka “crunch time”). What might be different in the endgame and what does this imply about what people should do now?
  • Lots of ‘task force-style advising’ work
    • → people should practise it now
  • Everyone will be very busy, especially senior people, so it won’t work as well to just defer
    • → build your own models
  • More possible to mess things up real bad
    • → start thinking harder about worst-case scenarios, red-teaming, etc. now, even if it seems a bit silly to e.g. spend time tightening up your personal infosec
  • The world may well be changing scarily fast
    • → practice decision-making under pressure and uncertainty. Strategy might get even harder in the endgame
  • Being able to juggle 6 different kinds of things might be more valuable than being able to do one thing really well, because there might just be lots of different kinds of things to do (cf. ‘task force-style advising’)
    • → specialise less? But specialisation tends to be pretty valuable, so I’m not sure this carries much weight overall
  • Relationships and reputation matter
    • → build them now

(Post 2/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

Misc things it seems useful to do/find out

• To inform talent development activities: talk with relevant people who have skilled up. How did they do it? What could be replicated via talent pipeline infrastructure? Generally talk through their experience.
  • Kinds of people to prioritise: those who are doing exceptionally well; those who have grown quite recently (might have better memory of what they did)
• To inform talent search activities: talk with relevant people—especially senior folks—about what got them involved. This could feed into earlier stage talent pipeline activities
• Case studies of important AI governance ideas (e.g. model evals, importance of infosec) and/or pipeline wins. How did they come about? What could be replicated?
• How much excess demand is there for fellowship programs? Look into the strength of applications over time. This would inform how much value there is in scaling fellowships.
• Figure out whether there is a mentorship bottleneck.
  • More concretely: would it be overall better if some of the more established AI governance folk spent a few more hours per month on mentorship?
  • Thing to do: very short survey asking established AI governance people how many hours per month they spend on mentorship.
  • Benefits of mentorship:
    • For the mentee: fairly light touch involvement can go a long way towards bringing them up to speed and giving them encouragement.
    • For the mentor: learn about fit for mentorship/management. Can be helpful for making object-level progress on work.
  • These benefits are often illegible and delayed in time, so a priori likely to be undersupplied.
  • If there’s a mentorship bottleneck, it might be important to solve ~now. The number of AI governance jobs is likely going to rise dramatically over the coming years—so having more thoughtful, risk-conscious people who are better placed to land those roles is more urgent than you might think if only considering having enough people by the acute risk period.
  • If there is a mentorship bottleneck how might one actually go about solving that? Obvious idea is to nudge potential mentors to consider: 
    • Asking GovAI, who have been collecting expressions of interest for RAs, whether there’s anyone who might be a good fit as your mentee
    • Posting on the forum or otherwise broadcasting what kind of thing a potential mentee could do that might make you excited about mentoring them (e.g. write a review of report X, or write a memo on topic Y), and that (e.g.) if they do that they should send it to you, and you'll at least take a 30 minute call with them
    • Mentoring someone on a summer program (e.g. GovAI Fellowship, ERIs, SERI MAGS, HAIST/MAIA programs, AI Safety Camp, …)

(Post 1/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

Some key uncertainties in AI governance field-building

According to me, these are some of the key uncertainties in AI governance field-building—questions which, if we had better answers to them, might significantly influence decisions about how field-building should be done.

How best to find/upskill more people to do policy development work?

• I think there are three main skillsets involved in policy development work:
  • Macrostrategy
  • “Traditional” policy development work (e.g. detailed understanding of how policymaking works in institution, to devise feasible policy action)
  • Impact focus (i.e. working to improve lasting impacts of AI in a scope-sensitive way)
• A more concrete question is: which pillars to prioritise in talent search/selection vs upskilling. E.g. do you take people already skilled in X and Y and give them Z; X and Z and give them Y; etc. etc. ?

What are the most important profiles that aren’t currently being hired for, but nonetheless might matter?

Reasons why this seems important to get clarity on:

• Focus on neglected aspects of the talent pipeline. People want to get hired, so will be trying to skill up for positions that are currently being hired for. Whereas for future positions—especially for positions that will never be “hired for” per se (e.g. leading a policy team that wouldn’t exist unless you pitched it), and “positions with a deadline”^[1]—the standard career incentives to skill up for them aren’t as strong. Also, some people currently hiring are already trying to solve their own bottlenecks (e.g. designing efficient hiring processes to identify talent) whereas future people aren’t.^[2]
• Avoid myopically optimising the talent pipeline. The world will probably change a lot in the run up to advanced AI. This will affect the value of different talent pipeline interventions in three ways:
  
  • There will likely be more people interested in AI (governance). So, more people who want to do things, and hence more value in work that usefully leverages a large amount of labour.
  • The people who become interested may have different skills and inclinations, compared to the current talent pool. This will change the future comparative advantage of people we can currently find/upskill.
    
    • More concretely, you might think that people currently working on AI governance are disproportionately inclined towards macro/strategy, relative to the talent pool in, say, 5 years’ time. Optimising for ticking all the talent boxes by the end of the year might look like finding/upskilling people with deep knowledge in certain areas that we’re lacking (e.g. more lawyers). But if you instead think these people will just be drawn to the field once there are important questions they can answer, and that the community can usefully leverage their knowledge, this could suggest instead doubling down on building a community that’s excellent at strategy [I’m very uncertain about this particular line of reasoning, but think there might be some important thread to pull on here, more generally]
  • The nature of important work changes as we move into the AI endgame. E.g. probably less field-building, more public comms, more founding of new institutions, more policy development, etc.

To what extent should talent pipeline efforts treat AI governance as a (pre-)paradigmatic field?

• More concrete versions of this question, that are all trying to get at the same thing:
  
  • On the current margin (now and in the future), how much of the most valuable work is crank-turn-y? By “crank-turn-y” I mean “work which can be delegated to sensible people even if they aren’t deeply integrated into the existing field”.
  • On the current margin (now and in the future), how high a premium should talent search/development efforts put on the macro/strategy aptitude?
  • On the current margin (now and in the future), how much of the most valuable work looks like contributing to an intellectual project that has been laid out (rather than doing the initial charting out of that intellectual project)?
• Answers to these questions seem like they should affect: how quickly the field scales up, and how much we are trying to attract people who are excellent at crank-turn-y work vs strategy work. I lightly hold the intuition that erring on this question is a main way that this field could mess up.

1. ^{^}

   Re: “positions with a deadline”: it seems plausible to me that there will be these windows of opportunity when important positions come up, and if you haven’t built the traits you need by that time, it’s too late. E.g. more talent very skilled at public comms would probably have been pretty useful in Q1-2 2023.

2. ^{^}

   Counterpoint: the strongest version of this consideration assumes a kind of “efficient market hypothesis” for people building up their own skills. If people aren’t building up their own skills efficiently, then there could still be significant gains from helping them to do so, even for positions that are currently being hired for. Still, I think this consideration carries some weight.

Thanks, fixed!

Thanks for this, I won't use "bet" in this context in the future

Finally posted