That makes sense, yes I think there is something of a rigour-importance tradeoff where some of the most important questions are hard to address as rigorously. Probably even in global development, where things you can do RCTs on are great, but improving economic growth is far harder to rigorously measure and analyse.
One of my worries with articles like the one we wrote is that most of this can just be left to figure out after we have solved alignment and are in some sort of long (or short) reflection period. I still find it useful to think about these big picture strategy futurism things sometimes even if any answers we come to will need to be somewhat tentative and to-be-revised.
I think that much of Forethought's work is a concerning departure from the very evidence-based roots of the movement and the significantly researched concerns about AI, and I think their work is indicative of an extreme echo chamber.
I agree that GHD and neartermist interventions are far easier to collect evidence about (e.g. you can just run RCTs!) but I think that doesn't mean we should avoid thinking about speculative and future-oriented things.
Re being an echo-chamber, it is true that there is a very small (but growing!) set of people thinking about the intelligence explosion and what it might entail. I'm glad to see more and more people taking AI seriously and thinking about what RSI would mean and what risks it poses. The alternative to us publishing work like this is just leaving it in google docs that 'echo chamber' people comment on. So I think where possible and not infohazardous to do so, sharing work like this publicly is useful. I'd be happy to see thoughtful critiques of the piece! Though I expect we have fundamental enough disagreements (e.g. about the legacy of FHI - to me it seems like it was a prescient and very generative place) that it would be hard to double-crux.
Yes, I expect people to be quite bad at estimating counterfactuals, and to (perhaps subconsciously) be generous. I'd be excited for you to do the more direct approaches you suggest comparing to people who were waitlisted or couldn't come for exogenous reasons. Probably relying on AIs searching the internet for what each person is doing now would be almost as good as the survey and get more data, so I am maybe less worried about low response rates for people who didn't participate.
Also, most of the impact probably comes from a small minority of alumni, so maybe it would be good to have a higher bar, or try to have some rough metric beyond a binary 'working in AI safety' (probably something Clauded is fine, like specifying someone later doing MATS as 1 unit, and then having conversions where e.g. a fulltime job at a decent org is worth 6 MATS fellows, or whatever).
I also expect that a bunch of this work can just better be done in a centralised way, notably by cG (which I expect is happening behind the scenes). E.g. if cG asks people in a survey 'how did you get into AI safety - what were the few counterfactually most important things?' and people name ML4Good that seems like stronger evidence.
But still, regardless of the details, it seems like time to scale up given this track record? What is the main bottleneck? I suppose if you rank applicants rather than just accept/reject you will be able to see whether your ratings when deciding whether to admit someone is predictive of their career trajectory, which would also be interesting.
Interesting, I would have expected the additional effort of including error correction would be very small compared to the energy costs of sending out probes.
Thanks, this is useful to hear. I think intuitively I lean towards your view of this being helpful and good to do. But the received wisdom among grantmakers as far as I can tell (that I have mostly been deferring to) is that it is bad to do this due to grantees overupdating. I think if grantees/applicants were perfectly even-keeled and rational this would be a great idea. And many applicants probably would benefit from it. But there will likely be a minority of cases where people (unreasonably) get upset if they don't get funded after seeing an initial high probability. Not sure what to do about this, maybe we can be just like 'tough luck, we put a disclaimer saying not to rely on this probability estimate overmuch' but phrased more nicely.
Substack isn't perfect, but I feel like for getting more people to read these great posts that would be a clear winner? I would also personally prefer it as it has native audio functionality and fits into my broader reading diet more seamlessly. Also crossposting to EAF seems good for key posts.
Idea for thinking about the future of EAGs and whether to keep them cause-neutral or have separate cause-themed events:
Get (anonymised) swapcard data and look at who had meetings with whom to work out how strong the clustering is. If there are clear groups that had lots of meetings within the group, but few meetings with people from other groups, that is a sign that there should just be a separate event for that group. Whereas if there is no group that can be cleanly split into its own event, that is a sign to keep it together.
One simple metric for this could be: if you split EAG in two (e.g. longtermist vs neartermist or whatever) how many fewer total connections would there be.
A key consideration for me is that earth-originating civilisation first spreading to other galaxies seems likely to be a lock-in, where if the values/organising structures of those early space missions are bad, the future seems quite bad in expectation to me.
And it seems likely that such space colonisation will become possible soon.
Urgent-watch: allows me to close Slack and email, and have Claude run a loop in the background to check every 30 mins if there is anything time-sensitive that means I should exit deep work and reply to things.
Gdoc2md: I work mainly in Google Docs, often with many comments, and the native ‘download as markdown’ feature loses the comments. So I have a skill that take a doc url and downloads it as a word file, which Claude can parse to extract the comments and format them as footnotes in a clean md file for other Claudes to read and edit.
Memo: When I want Claude to do some macrostrategy research/writing, it reads a detailed summary of my org’s beliefs, values, plans, etc. The summary links to many more markdown files with detailed takes on relevant topics. Interestingly, telling Claude to read all the memos and then write something on a new topic produces worse results than just reading the summary and the few most relevant memos.
Thanks :)
That makes sense, yes I think there is something of a rigour-importance tradeoff where some of the most important questions are hard to address as rigorously. Probably even in global development, where things you can do RCTs on are great, but improving economic growth is far harder to rigorously measure and analyse.
One of my worries with articles like the one we wrote is that most of this can just be left to figure out after we have solved alignment and are in some sort of long (or short) reflection period. I still find it useful to think about these big picture strategy futurism things sometimes even if any answers we come to will need to be somewhat tentative and to-be-revised.
I agree that GHD and neartermist interventions are far easier to collect evidence about (e.g. you can just run RCTs!) but I think that doesn't mean we should avoid thinking about speculative and future-oriented things.
Re being an echo-chamber, it is true that there is a very small (but growing!) set of people thinking about the intelligence explosion and what it might entail. I'm glad to see more and more people taking AI seriously and thinking about what RSI would mean and what risks it poses. The alternative to us publishing work like this is just leaving it in google docs that 'echo chamber' people comment on. So I think where possible and not infohazardous to do so, sharing work like this publicly is useful. I'd be happy to see thoughtful critiques of the piece! Though I expect we have fundamental enough disagreements (e.g. about the legacy of FHI - to me it seems like it was a prescient and very generative place) that it would be hard to double-crux.
(I co-wrote the Nightwatchmen article.)
Haha, this is great!
Wow, great!
Yes, I expect people to be quite bad at estimating counterfactuals, and to (perhaps subconsciously) be generous. I'd be excited for you to do the more direct approaches you suggest comparing to people who were waitlisted or couldn't come for exogenous reasons. Probably relying on AIs searching the internet for what each person is doing now would be almost as good as the survey and get more data, so I am maybe less worried about low response rates for people who didn't participate.
Also, most of the impact probably comes from a small minority of alumni, so maybe it would be good to have a higher bar, or try to have some rough metric beyond a binary 'working in AI safety' (probably something Clauded is fine, like specifying someone later doing MATS as 1 unit, and then having conversions where e.g. a fulltime job at a decent org is worth 6 MATS fellows, or whatever).
I also expect that a bunch of this work can just better be done in a centralised way, notably by cG (which I expect is happening behind the scenes). E.g. if cG asks people in a survey 'how did you get into AI safety - what were the few counterfactually most important things?' and people name ML4Good that seems like stronger evidence.
But still, regardless of the details, it seems like time to scale up given this track record? What is the main bottleneck? I suppose if you rank applicants rather than just accept/reject you will be able to see whether your ratings when deciding whether to admit someone is predictive of their career trajectory, which would also be interesting.
Interesting, I would have expected the additional effort of including error correction would be very small compared to the energy costs of sending out probes.
(I am a new-ish AI policy grantmaker)
Thanks, this is useful to hear. I think intuitively I lean towards your view of this being helpful and good to do. But the received wisdom among grantmakers as far as I can tell (that I have mostly been deferring to) is that it is bad to do this due to grantees overupdating. I think if grantees/applicants were perfectly even-keeled and rational this would be a great idea. And many applicants probably would benefit from it. But there will likely be a minority of cases where people (unreasonably) get upset if they don't get funded after seeing an initial high probability. Not sure what to do about this, maybe we can be just like 'tough luck, we put a disclaimer saying not to rely on this probability estimate overmuch' but phrased more nicely.
Substack isn't perfect, but I feel like for getting more people to read these great posts that would be a clear winner? I would also personally prefer it as it has native audio functionality and fits into my broader reading diet more seamlessly. Also crossposting to EAF seems good for key posts.
Idea for thinking about the future of EAGs and whether to keep them cause-neutral or have separate cause-themed events:
(Maybe this analysis already happens!)
A key consideration for me is that earth-originating civilisation first spreading to other galaxies seems likely to be a lock-in, where if the values/organising structures of those early space missions are bad, the future seems quite bad in expectation to me.
And it seems likely that such space colonisation will become possible soon.
Three Claude Code skills I find helpful: