I think AI x-risk reduction is largely bottlenecked on a lack of strategic clarity, more than e.g. bio x-risk reduction is, such that it's very hard to tell which intermediate goals we could aim for that would reduce rather than increase AI x-risk. Even just among (say) EA-motivated longtermists focused on AI x-risk reduction, there is a very wide spread of views on on AI timelines, takeoff speeds, alignment difficulty, theories of victory, sources of doom, and basically everything else (example), and I think this reflects how genuinely confusing the strategic situation is.
One important way to improve strategic clarity is via what we've come to call AI "worldview investigations," i.e. reports that:
-
Aim to provide an initial tentative all-things-considered best-guess quantitative answer to a big, important, fundamental ("worldview") AI strategy question such as "When will we likely build TAI/AGI?" or "How hard should we expect AI alignment to be?" or "What will AI takeoff look like?" or "Which AI theories of victory are most desirable and feasible?" Or, focus on a consideration that should provide a noticeable update on some such question.
-
Are relatively thorough in how they approach the question.
Past example "worldview investigations" include:
Worldview investigations are difficult to do well: they tend to be large in scope, open-ended, "wicked," and require modeling and drawing conclusions about many different phenomena with very little hard evidence to work with. Here is some advice that may help researchers to succeed at completing a worldview investigation:
-
Be bold! Attack the big important action-relevant question directly, and try to come to a bottom-line quantitative answer, even though it's unjustified in many ways and will be revised later.
-
Reach out to others early for advice and feedback, especially people who have succeeded at this kind of work before. Share early, scrappy drafts for feedback on substance and direction.
-
On the research process itself, see Minimal-trust investigations, Learning by writing, The wicked problem experience, and Useful Vices for Wicked Problems.
-
Reasoning transparency has advice for how to communicate what you know, with what confidence, and how you know it, despite the fact that for many sub-questions you won't have enough time or evidence to come to a well-justified conclusion.
-
How to Measure Anything is full of strategies for quantifying very uncertain quantities. (Summary here.)
-
Superforecasting has good advice about how to quantify your expectations about the future. (Summary here.)
Finally, here are some key traits of people who might succeed at this work:
- Ability to carve up poorly-scoped big-picture questions into the most important parts, operationalize concepts that seem predictive, and turn a fuzzy mess of lots of factors into a series of structured arguments connected to (usually necessarily weak / of limited relevance) evidence.
- At least moderate quantitative/technical chops, enough to relatively quickly learn topics like the basics of machine learning scaling laws or endogenous growth theory, while of course still significantly relying on conversations with subject matter experts.
- Ability to work quickly and iteratively, limiting their time on polish/completeness and on "rabbit holes" that could be better-explored or better-argued or more evidence-based but that aren't likely to change the approximate bottom line.
- Motivation to focus on work that is decision-relevant (in our case, for intervening on AI x-risk) and engage in regular thought and dialogue about what's most important to analyze next; not too attached to "academic freedom" expectations about following whatever is intellectually stimulating to them at the moment.
- Reasoning transparency, and "superforecaster"-style reasoning processes for evidence assessment and prediction/judgment calibration.
Notes
I'm skeptical of work like this and what you write in note 2. makes me think of some comments. You write:
If I'm honest, this seems to me like a large - and slightly odd - claim. Does it really say that research work that can be written up in fewer than one hundred pages cannot be persuasive and cannot be solid enough to build on? Do you really mean to suggest that?
You go on to write:
But I don't really see how all this is much different from saying something a bit like "more research is better than less research" or that although an individual short paper cannot provide what you call 'strategic' insight, it may be gained by essentially synthesizing the results contained in lots of smaller contributions. And this latter description sounds a lot like how I would describe the 'normal' way that research papers have worked for ages, i.e. something like the fact that each individual contribution tends to be narrow and then as time goes by, the bigger picture emerges from some kind of aggregation and synthesis that the research community somehow effects. (I find it particularly hard to square the example of Saif Khan's papers with your overall point: Is it not the case that many of Khan's earlier outputs were much shorter? But each was high-quality enough that he could build on them, even after having put them out into the community? And then later, once the bigger picture had genuinely emerged for him to see, he drew on his experience to produce the longer report?)
I remain unconvinced that it is better for someone to set out from the start to write a single long report, as opposed to - at the very least - an approach similar to Khan's in which one starts out with shorter, concrete 'pieces of the puzzle' and actually puts each of them out in the community where they can be criticized and scrutinized. Then along the way there can be more public discourse, more feedback, and more public variety of opinion. One will be better able to judge which threads really are promising or important and which ideas really are robust and any decision to write a longer, expanded report that draws on the lessons of the many shorter papers, need not be made until later.
It's a fair question. Technically speaking, of course progress can be more incremental, and some small pieces can be built on with other small pieces. Ultimately that's what happened with Khan's series of papers on the semiconductor supply chain and export control options. But in my opinion that kind of thing almost never really happens successfully when it's different authors building on each other's MVPs (minimum viable papers) rather than a single author or team building out a sorta-comprehensive picture of the question they're studying, with all the context and tacit knowledge they've built up from the earlier papers carrying over to how they approach the later papers.