Thanks for writing this; I agree there is a lot of uncertainty about how TAI will affect neartermist philanthropy (I take it longtermist/AI safety philanthropy is out of scope here). I probably lean more towards MacAskill than you on much of this (although we would need to get quantitative re how much to save for post-TAI for that), but I look forward to seeing the full model to play around with my numbers.

I'm not sure how many TAI-pilled neartermist donors there are (maybe a lot of Ants? Not sure) but insofar as that is an important constituency, redoing a bunch of GiveWell's analyses in light of TAI could be interesting. I assume the main implication is that saving lives now starts looking a lot better while deworming style interventions that pay off over many decades of income gains take a big impact hit from expecting the world to be transformed (and either everyone to be dead, or rich, or at least medical R&D progress skyrocketing). Not sure for animal welfare.

(Minor: probably you should make this set of posts into an EAF sequence.)

Thanks for writing this! I agree it is a bit odd that both 1) funders say they are short on for grantakers, and 2) lots of people would love to do AI safety grantmaking. I basically think funders are right to keep a high bar, and have strong AI safety knowledge/experience as an important hiring desideratum. I think the more logistical/operational aspects of investigating and making grants can be learned quite quickly on the job, whereas developing taste and judgement about AI strategy and safety is harder. So I think the current strategy of taking existing research-y/policy/technical AI safety people and training them up to be grantmakers seems better than taking existing non-AI grantmakers and training them up in AI safety (though of course running career transition programs for people getting into AI safety is also valuable!).

(I am biased by being one of those EA/AI safety research-y people who transitioned into grantmaking.)

Wow, that's great, I had missed that! Thanks for pointing it out :) 

I think it is simultaneously true that:

1. EAs are unusually trustworthy, virtuous, high-integrity people, who are unusually unlikely to be (as you rightly note) "motivated by personal financial gain".
2. Despite 1, there will no doubt be some existing bad actors within EA, and the influx of money may attrat more.
3. More importantly, it is still beneficial to have sound governance even when the people involved are well-intentioned and virtuous. Partly for optics, and partly so that it is someone's job to spot and correct mistakes.

I definitely don't want to lose sight of 1, and how wonderful that is. But I agree that as EA grows in money and power, good internal governance will become more important.

One minor point of hope is that as more funding comes online, from more different sources, orgs may well feel more able to criticise any single funder (e.g. cG) when previously that would have been even more costly. I'm not sure how large that effect is. I also expect that as money becomes less scarce, founders and org-builders will increase more in status, as they will be (seen as) more of the limiting factors.

Thanks, very interesting!

Thanks :)

That makes sense, yes I think there is something of a rigour-importance tradeoff where some of the most important questions are hard to address as rigorously. Probably even in global development, where things you can do RCTs on are great, but improving economic growth is far harder to rigorously measure and analyse.

One of my worries with articles like the one we wrote is that most of this can just be left to figure out after we have solved alignment and are in some sort of long (or short) reflection period. I still find it useful to think about these big picture strategy futurism things sometimes even if any answers we come to will need to be somewhat tentative and to-be-revised.

I think that much of Forethought's work is a concerning departure from the very evidence-based roots of the movement and the significantly researched concerns about AI, and I think their work is indicative of an extreme echo chamber.

I agree that GHD and neartermist interventions are far easier to collect evidence about (e.g. you can just run RCTs!) but I think that doesn't mean we should avoid thinking about speculative and future-oriented things.

Re being an echo-chamber, it is true that there is a very small (but growing!) set of people thinking about the intelligence explosion and what it might entail. I'm glad to see more and more people taking AI seriously and thinking about what RSI would mean and what risks it poses. The alternative to us publishing work like this is just leaving it in google docs that 'echo chamber' people comment on. So I think where possible and not infohazardous to do so, sharing work like this publicly is useful. I'd be happy to see thoughtful critiques of the piece! Though I expect we have fundamental enough disagreements (e.g. about the legacy of FHI - to me it seems like it was a prescient and very generative place) that it would be hard to double-crux.

(I co-wrote the Nightwatchmen article.)

Haha, this is great!

Wow, great!

Yes, I expect people to be quite bad at estimating counterfactuals, and to (perhaps subconsciously) be generous. I'd be excited for you to do the more direct approaches you suggest comparing to people who were waitlisted or couldn't come for exogenous reasons. Probably relying on AIs searching the internet for what each person is doing now would be almost as good as the survey and get more data, so I am maybe less worried about low response rates for people who didn't participate.

Also, most of the impact probably comes from a small minority of alumni, so maybe it would be good to have a higher bar, or try to have some rough metric beyond a binary 'working in AI safety' (probably something Clauded is fine, like specifying someone later doing MATS as 1 unit, and then having conversions where e.g. a fulltime job at a decent org is worth 6 MATS fellows, or whatever).

I also expect that a bunch of this work can just better be done in a centralised way, notably by cG (which I expect is happening behind the scenes). E.g. if cG asks people in a survey 'how did you get into AI safety - what were the few counterfactually most important things?' and people name ML4Good that seems like stronger evidence.

But still, regardless of the details, it seems like time to scale up given this track record? What is the main bottleneck? I suppose if you rank applicants rather than just accept/reject you will be able to see whether your ratings when deciding whether to admit someone is predictive of their career trajectory, which would also be interesting.

Interesting, I would have expected the additional effort of including error correction would be very small compared to the energy costs of sending out probes.