I think AI x-risk reduction is largely bottlenecked on a lack of strategic clarity, more than e.g. bio x-risk reduction is, such that it's very hard to tell which intermediate goals we could aim for that would reduce rather than increase AI x-risk. Even just among (say) EA-motivated longtermists focused on AI x-risk reduction, there is a very wide spread of views on on AI timelines, takeoff speeds, alignment difficulty, theories of victory, sources of doom, and basically everything else (example), and I think this reflects how genuinely confusing the strategic situation is.
Aim to provide an initial tentative all-things-considered best-guess quantitative answer to a big, important, fundamental ("worldview") AI strategy question such as "When will we likely build TAI/AGI?" or "How hard should we expect AI alignment to be?" or "What will AI takeoff look like?" or "Which AI theories of victory are most desirable and feasible?" Or, focus on a consideration that should provide a noticeable update on some such question.
Are relatively thorough in how they approach the question.
Past example "worldview investigations" include:
- Open Phil's series on "When will we likely build TAI?", summarized here, and comprised mainly by Bio Anchors, Brain Computation, Semi-Informative Priors, Explosive Growth, and Human Trajectory. (~1185pp)
- Carlsmith's Is power-seeking AI an existential risk? (~90pp)
- My 2017 Report on Consciousness and Moral Patienthood (~485pp)
Worldview investigations are difficult to do well: they tend to be large in scope, open-ended, "wicked," and require modeling and drawing conclusions about many different phenomena with very little hard evidence to work with. Here is some advice that may help researchers to succeed at completing a worldview investigation:
Be bold! Attack the big important action-relevant question directly, and try to come to a bottom-line quantitative answer, even though it's unjustified in many ways and will be revised later.
Reach out to others early for advice and feedback, especially people who have succeeded at this kind of work before. Share early, scrappy drafts for feedback on substance and direction.
Reasoning transparency has advice for how to communicate what you know, with what confidence, and how you know it, despite the fact that for many sub-questions you won't have enough time or evidence to come to a well-justified conclusion.
Finally, here are some key traits of people who might succeed at this work:
- Ability to carve up poorly-scoped big-picture questions into the most important parts, operationalize concepts that seem predictive, and turn a fuzzy mess of lots of factors into a series of structured arguments connected to (usually necessarily weak / of limited relevance) evidence.
- At least moderate quantitative/technical chops, enough to relatively quickly learn topics like the basics of machine learning scaling laws or endogenous growth theory, while of course still significantly relying on conversations with subject matter experts.
- Ability to work quickly and iteratively, limiting their time on polish/completeness and on "rabbit holes" that could be better-explored or better-argued or more evidence-based but that aren't likely to change the approximate bottom line.
- Motivation to focus on work that is decision-relevant (in our case, for intervening on AI x-risk) and engage in regular thought and dialogue about what's most important to analyze next; not too attached to "academic freedom" expectations about following whatever is intellectually stimulating to them at the moment.
- Reasoning transparency, and "superforecaster"-style reasoning processes for evidence assessment and prediction/judgment calibration.
See e.g. Our AI governance grantmaking so far, A personal take on longtermist AI governance, How to make the best of the most important century?, Important, actionable research questions for the most important century. ↩︎
Other types of research can also contribute to strategic clarity, e.g. studies of much narrower empirical questions. However, I've come to think that <100pp one-off papers and blog posts typically contribute little to our strategic understanding (though they may contribute to more granular "tactical" understanding), because even when they're well done, they can't be thorough or deep enough on their own to be persuasive, or solid enough to use as a foundation for later work. Instead, I learn more from unusually detailed and thorough "extended effort" analyses that often require 1-5 FTE years of effort, e.g. OpenAI's series of "scaling laws" papers. Another example is Saif Khan's series of papers on semiconductor supply chains and policy options: I couldn't tell from just one paper, or even the first 5 papers, whether genuine strategic progress was being made or not, but around paper #8 (~2 years in) I could see how most of the key premises and potential defeaters had been investigated pretty thoroughly, and how it all made a pretty robustly-considered (but still uncertain) case for a particular set of policy interventions — culminating roughly in The Semiconductor Supply Chain: Assessing National Competitiveness and Securing Semiconductor Supply Chains. ↩︎
E.g. with explicit probabilities or probability distributions. ↩︎
The TAI timelines reports add up to ~355,000 words at 300 words per page, counting each main report and its appendices but not associated blog posts or supplemental materials (e.g. conversation notes or spreadsheets). The report on power-seeking AI is ~27,000 words, and my consciousness report is ~145,000 words. ↩︎
This report is on animal welfare strategy rather than AI strategy. In particular, the report helped us decide to expand our animal welfare grantmaking to help fish. ↩︎
Some people won't have time to comment, but it doesn't cost much for them to reply "Sorry, I don't have time." ↩︎