Edit: I was told privately that the grand strategy was unclear. The proposed grand strategy is to:
- Convince everyone. Recruit all AI capabilities researchers to transition into AI safety until the alignment problem is solved.
- Scale 1-on-1 outreach to AI capabilities researchers (what seems to be working best so far), and learn from experience to optimize outreach strategies.
- Exponentially grow the AI safety movement by mentoring newcomers to mentor newcomers to mentor newcomers (especially AI capabilities researchers).
- Maximize the social prestige of the AI safety movement (especially from the perspective of AI capabilities researchers).
Original post: AI capabilities research seems to be substantially outpacing AI safety research. It is most likely true that successfully solving the AI alignment problem before the successful development of AGI is critical for the continued survival and thriving of humanity.
Assuming that AI capabilities research continues to outpace AI safety research, the former will eventually result in the most negative externality in history: a significant risk of human extinction. Despite this, a free-rider problem causes AI capabilities research to myopically push forward, both because of market competition and great power competition (e.g., U.S. and China). AI capabilities research is thus analogous to the societal production and usage of fossil fuels, and AI safety research is analogous to green-energy research. We want to scale up and accelerate green-energy research as soon as possible, so that we can halt the negative externalities of fossil fuel use.
My claim: A task that seems extremely effective on expectation (and potentially, maximally effective for a significant number of people) is to recruit AI capabilities researchers into AI safety research.
More generally, the goal is to convince people (especially, but not limited to, high-impact decision-makers like politicians, public intellectuals, and leaders in relevant companies) of why it is so important that AI safety research outpace AI capabilities research.
Logan has written an excellent post on the upside of pursuing trial-and-error experience in convincing others of the importance of AI alignment. The upside includes, but is not limited to: the development of an optimized curriculum for how to convince people, and the use of this curriculum to help numerous and widely located movement-builders pitch the importance of AI alignment to high-impact individuals. Logan has recently convinced me to collaborate on this plan. Please let us know if you are interested in collaborating as well! You can email me at [email protected].
I would also like to propose the following additional suggestions:
1) For behavioral-science researchers like myself, prioritize research that is laser-focused on (a) practical persuasion, (b) movement-building, (c) how to solve coordination problems (e.g., market competition, U.S.-China competition) in high-risk domains, (d) how to scale up the AI safety research community, (e) how to recruit AI capabilities researchers into AI safety, and (f) how to help them transition. Examples include RCTs (and data collection in general; see Logan’s post!) that investigate effective ways to scale up AI safety research at the expense of AI capabilities research. More generally, scientific knowledge on how to help transition specialists of negative-externality or obsolete occupations would be broadly applicable.
2) For meta-movement-builders, brainstorm and implement practical ways for decentralized efforts around the world to scale up the AI safety research community and to recruit talented people from AI capabilities research. Examples of potential contributions include the development of optimized curricula for training AI safety researchers or for persuading AI capabilities researchers, a well-thought-out grand strategy, and coordination mechanisms.
3) For marketing specialists, brainstorm and implement practical ways to make the field of AI safety prestigious among the AI researcher community. Potential methods to recruit AI capabilities researchers include providing fellowships and other financial opportunities, organizing enjoyable and well-attended AI-safety social events like retreats and conferences, and inviting prestigious experts to said social events. Wining-and-dining AI researchers might help too, to the extent that effective altruists feel comfortable doing so. Note that a barrier to AI safety researchers’ enthusiasm is the lack of traditional empirical feedback loops. (“I’m proud of this tangible contribution, so I’m going to do this more!”) So, keeping the AI safety research community enthusiastic, social, and prestigious will be in many ways a unique challenge.
An especially strong priority for the AI safety research community is to attract talented AI researchers from all over the world. What really matters is that AI safety research has a monopoly on the relevant talent globally, not just in the San Francisco Bay Area. Ideally, this means continually communicating with AI research communities in other countries, facilitating the immigration of AI researchers, persuading them to pursue AI safety research rather than capabilities research, and creating an environment conducive for immigrant AI researchers to stay (in, for example, the San Francisco Bay Area’s thriving AI safety community).
We should also seek to maximize the long-term prestige differential between AI safety research and AI capabilities research, especially in Silicon Valley (but also among the general public). Building this prestige differential would be difficult but potentially very impactful, given that the strategy of offering financial incentives to poach AI researchers may hit a wall at some point. This is because many of the industry-leading research teams are employed by deep-pocketed companies who are enthusiastic about AI capabilities research. But given that working at a fossil fuel company is not prestigious despite its high pay, a long-term prestige differential between AI safety research and AI capabilities research might be feasible even without a large pay increase in the favor of the former.
Finally, we really need to scale up the AI safety mentoring pipeline. I’ve heard anecdotally that many enthusiastic and talented people interested in transitioning into AI safety are “put on hold” due to a lack of mentors. They have to wait for a long time to be trained, and an even longer time to make any contributions that they are proud of. Sooner or later, they are probably going to lose their enthusiasm. This seems like a substantial loss of value. We should instead aim to be supercharging the growth of the AI safety community. This means the streamlining and widening the availability of AI safety training opportunities, especially to potential and current AI capabilities researchers, should be an absolutely top priority. Please feel free to email me at [email protected] if you have any ideas on how to do this!
tl;dr. It is extremely important on expectation that AI safety research outpace AI capabilities research. Practically speaking, this means that AI safety research needs to scale up to the point of having a monopoly on talented AI researchers, and this must be true globally rather than just in the San Francisco Bay Area. We should laser-focus our efforts of persuasion, movement-building, and mentoring on this goal. Our grand strategy should be to continue doing so until the bulk of global AI research is on AI safety rather than AI capabilities, or until the AI alignment problem is solved. To put it bluntly, we should—on all fronts—scale up efforts to recruit talented AI capabilities researchers into AI safety research, in order to slow down the former in comparison to the latter.
Thanks for sharing. – I love the spirit of aiming to come up with a strategy that, if successful, would have a shot at significantly moving the needle on prospects for AI alignment. I think this is an important but (perhaps surprisingly) hard challenge, and that a lot of work labeled as AI governance or AI safety is not as impactful as it could be in virtue of not being tied into an overall strategy that aims to attack the full problem.
That being said, I feel fairly strongly that this strategy as stated is not viable, and that you aiming to implement the strategy would come with sufficiently severe risks of harming our prospects for achieving aligned AI that I would strongly advise against moving forward with this.
I know that you have emailed several people (including me) asking for their opinion on your plan. I want to share my advice and reasoning publicly so other people you may be talking to can save time by referring to my comment and indicating where they agree or disagree.
--
Here is why I'm very skeptical that you moving forward with your plan is a good idea:
(I also think that, in fact, there is not a clean division between "AI capabilities" and "AI safety" research. For instance, work on interpretability or learning from human feedback arguably significantly contributes to both capabilities and safety. I have bracketed this point because I don't think it is that relevant for the viability of your plan, except perhaps indirectly by providing evidence about your understanding of the AI field.)
--
To be clear, I think that some of the specific ideas you mention are very good if implemented well. For instance, I do think that better AI safety curricula are very valuable.
However, these viable ideas are usually things that are already happening. There are AI alignment curricula, there are events aimed at scaling up the field, there are efforts to make AI safety seem prestigious to mainstream AI talent, and there even are efforts that are partly aimed at increasing the credibility of AI risk ideas to AI researchers, such as TED talks or books by reputable AI professors or high-profile conferences at which AI researchers can mingle with people more concerned about AI risk.
If you wanted to figure out to which of these efforts you are best placed to contribute, or whether there might be any gaps among current activities, then I'm all for it! I just don't think that trying to tie them into a grand strategy that to me seems flawed in all those places at which it is new and specific, and not new in all those places where it makes sense to me, will be a productive approach.
Thank you very much for the constructive criticisms, Max! I appreciate your honest response, and agree with many of your points.
I am in the process of preparing a (hopefully) well-thought-out response to your comment.
Thanks for the in depth response!
The most valuable part of this project I’m interested in personally is a document with the best arguments for alignment and how to effectively go about these conversations (ie finding cruxes).
You made a logarithmic claim of improving capabilities, but my model is that 80% of progress is made by a few companies and top universities. Less than 1000 people are pushing general capabilities, so convincing these people to pivot (or the people in charge of these people’s research direction) is high impact.
You linked the debate between AI researchers, and I remember being extremely disappointed in the way the debate was handled (eg why is Stuart using metaphors? Though I did appreciate Yoshua’s responses). The ideal product I’m thinking of says obvious things like “don’t use metaphors as arguments” and “don’t have a 10 person debate” and “be kind”, along with the actual arguments to present and the most common counter arguments.
This could have negative effects if done wrong, so the next step is to practice on lower stakes people while building the argument-doc. Then, higher stakes people can be approached.
Additionally, a list of why certain “obvious solutions to alignment” fails is useful for pointing out dead-ends in research. For example, any project that relies on orthogonality thesis being wrong is doomed to fail imo.
This is a tangent: The links for scaling alignment are very inadequate, (though I’m very glad they exist!). MLAB had what? 30/500 applicants accepted. AISC had 40/200 accepted (and I talked to one rejected who was very high quality!) Richards course is scaling much faster though and I’m excited about that. I do believe that none of the courses handle “how to do great research” unless you do a mentorship, but I think we can work around that.
I agree that this would be valuable, and I'd be excited about empirically informed work on this.
You are most likely aware of this, but in case not I highly recommend reaching out to Vael Gates who has done some highly relevant research on this topic.
I do think it is important to keep in mind that (at least according to my model) what matters is not just the content of the arguments themselves but also the context in which they are made and even the identity of the person making them.
(I also expect significant variation in which arguments will be most convincing to whom.)
Yes, I agree that the number of people who are making significant progress toward AGI/TAI specifically is much smaller, and that this makes the project of convincing all of those more feasible.
For the reasons mentioned in my original comment (incentives, failed past attempts, etc.) I suspect I'm still much more pessimistic than you might be that it is possible to convince them if only we found the right arguments, but for all I know it could be worth a shot. I certainly agree that we have not tried to do this as hard as possible (at least not that I'm aware of), and that it's at least possible that a more deliberate strategy could succeed where past efforts have failed.
(This is less directly relevant, but fwiw I don't think that this point counts against expected research returns being logarithmic per se. I think instead it is a point about what counts as research inputs – we should look at doublings of 'AGI-weighted AI research hours', whatever that is exactly.)
That being said, my guess is that in addition to 'trying harder' and empirically testing which arguments work in what contexts, it would be critical to have any new strategy to be informed by an analysis of why past efforts have not been successful (I expect there are useful lessons here) and by close coordination with those in the AI alignment and governance communities who have experience interacting with AI researchers and who care about their relationships with them, how they and AI safety/governance are being perceived as fields by mainstream AI researchers, etc. - both to learn from their experience trying to engage AI researchers and to mitigate risks.
FWIW my intuition is that the best version of a persuasion strategy would also include a significant component of preparing to exploit windows of opportunity – i.e., capitalizing on people being more receptive to AI risk arguments after certain external events like intuitively impressive capability advances, accident 'warning shots', high-profile endorsements of AI risk worries, etc.
I really like the window of opportunity idea.
I am talking to Vael currently thanks to a recommendation from someone else. If there’s other people you know or sources of failed attempts in the past, I’d also appreciate those!
I also agree that a set of really good arguments is great to have but not always sufficient.
Although convincing the top few researchers is important, also convincing the bottom 10,000’s is also important for movement building. The counter argument of “we can’t handle that many people switching careers” is to scale our programs.
Another is just trusting them to figure it out themselves (I want to compare with COVID research, but I’m not sure how well that research went or what incentives there were to make it better or worse), but this isn’t my argument but another’s intuition. I think an additional structure of “we can give quick feedback on your alignment proposal”would help with this.