This essay was submitted to Open Philanthropy's Cause Exploration Prizes contest (before the deadline). We are uploading some back entries late, but all good-faith entries were considered for prizes
Author's note: This piece assumes unaligned AGI is a serious threat to humanity, and does not spend time laying out this argument.
Since it might be impossible to solve technical AGI alignment in time, AGI governance is an incredibly important area to work on. Moreover, there are other serious risks pertaining to AGI in addition to alignment, such as weaponization or the concentration of extreme power, that will most likely require governance solutions as opposed to technical ones.
While the field of AGI governance is still largely uncertain about what specific policies would be beneficial, we do have enough information to foresee some of the bottlenecks AGI safety policies are likely to face.
One fairly common bottleneck that AGI governance may be particularly susceptible to is lack of public support. The status quo of a total absence of public awareness about serious AGI risks could be far from the optimal situation. If the only people who care about AGI risk are a niche group of a few thousand people across the globe, the views of AI/CS experts are mixed, and there are multi-billion-dollar companies who actively oppose meaningful safety policies, it will be very difficult to implement large-scale reforms.
We could wait until we have a list of good AGI governance policies that we want people to rally behind before trying to build public support around them. The downside of this approach is that outreach will probably be harder to do effectively later. A version of AGI accelerationism may beat us to the punch and become the dominant narrative, especially as programs like DALL-E 2 become popularized, enter the lives of everyday people, and grow increasingly flashier, cooler, and more useful. If this happens, the task will go from persuading people to form an opinion about a subject they had not previously considered to persuading people to change their minds, which is much harder. It is probably preferable—or even essential—for the public, relevant experts, and policymakers to be on “our side”, and delaying efforts to communicate our concerns about AGI makes this harder to achieve.
A responsible version of bringing AGI risks to the public awareness probably involves lots of focus groups, pilots, feedback, and gradual scaling. There are many ways in which this project could go wrong, and it is very important that we get it right. Among other concerns, we want to make sure the arguments sound compelling to laypeople, the safety measures the public/key stakeholders support are in line with those that are actually helpful, and that AGI safety does not become politically polarized in a counterproductive way.
There are three main groups that we could engage in outreach to, outlined below.
Working directly with government may be the best strategy in some cases, such as when a policy is very technical or the asks we are making are very specific, such that we want technically-versed people in direct contact with policymakers. This can include elected officials, ministers and department heads, policy advisors, and other civil servants.
Most of this work right now seems to be happening in the US and in Europe at an EU level. It might be worthwhile to do similar types of government outreach in Europe at a national level, as well as in other countries with major technology sectors, such as high-income Asian nations. While they are not currently the main AI hubs, successful outreach there could set a precedent for governments of current AI hubs to take AGI safety seriously.
However, communication with governments may be insufficient in some cases. Some national or cross-national policies may require broader public support in order to be ratified, especially given the likely corporate resistance they will face. If your constituents do not care about an issue, the media is not covering it, and multiple powerful companies are giving you a hard time or showcasing the short-term economic benefits that AI innovation could bring to your country, you have far less of an incentive to put effort into getting that policy passed than if you would get more political praise for it.
One challenge for outreach through credible figures is that a lot of the researchers and engineers in the field of AI are doing work that shortens the timeline for the arrival of AGI, and thus have an obvious conflict of interest that makes it much harder for them to want to advocate for measures restricting it.
For this reason, persuading experts in other fields (e.g. neuroscience, psychology, social sciences, or philosophy) to take AGI risk seriously may be more tractable. Having them on board would add credibility to our concerns and help convince both policymakers and the public that this is a real risk. Public figures with a strong track record for outreach on a different issue or who have previously worked with governments seem especially promising.
It is also possible that doing outreach to people in CS/AI may still be worthwhile. There have been instances of Google employees having ethical objections to some of Google’s work that other employees presumably thought was fine, such as working with the Pentagon or developing a censored search engine for China, and the protest of employees was able to hamper the project. Stopping or slowing down AGI capabilities research in the near term via employee protests is unlikely to work, but employees advocating for more safety research or higher safety standards might nevertheless be constructive and lay the groundwork for making the case to potentially slow down or stop capability research in the medium-term.
Public support also seems helpful, and in some cases is pivotal. There have been instances of arguments about technology risks that lacked expert consensus (e.g. GMOs or nuclear power) where public concern was enough to lead to bans or shutdowns, even when significant profit motives were at stake. An additional advantage of targeting the public is that it overlaps with the clientele of many of the companies that are working on AGI, and having them become aware of the risks of this technology would put pressure on these companies to increase the resources they put towards safety (if people e.g. boycott Facebook Messenger, then Facebook loses a lot of ad money). It also overlaps with the people AI developers are surrounded by in their personal lives; changing the public sentiment around AGI may make being an AGI developer lower-status and reputationally destructive, which may encourage some people currently working on this to pivot their work to something they would get more praise and social capital for.
The obvious concern is that AGI alignment may be too weird and sci-fi-sounding for the general public to become invested in it. However, this may be more tractable than it initially appears— there are experts that specialize in message testing and reframing issues to be more compelling, and some of the lessons from this field might have applications for the case of communicating about AGI risks. As a speculative proof-of-concept, one example of how this could be done is by outlining catastrophic examples of specification gaming that could happen due to misaligned AGI other than paperclip maximizers, which sound more plausible to a layperson and will probably be more effective at getting people to seriously consider them. There are also different potential framings (e.g. “AGI will be to humans what humans are to monkeys”, “AGI is a genie that cannot be put back in the bottle”, “AGI could be the next nuclear bomb”, “specification gaming already happens and will get worse”, etc.) that may work best for different audiences.
A second concern is that solutions that sound helpful to lay-people may be different from the solutions that are actually helpful. It is possible that an AGI development lab would push for a “safety policy” that doesn’t actually reduce existential risk but gives people the illusion that the issue is solved over the short term, and the public interest that was generated as a result of these efforts would rally behind it.
In order to prevent this, outreach efforts would need to provide interested citizens with a reliable way of distinguishing probably-good from probably-bad AGI safety policies, such that they will support the ones most likely to be helpful. This could look something like setting up an official board of renowned AGI safety experts to independently and publicly rate different safety policies, while encouraging people to place huge value on these ratings.
Getting public outreach right may be very difficult, but given the potential value, seems worth at least exploring. It is possible that, with short notice, we will realize we need to resort to a drastic AGI governance solution that requires public support. It would be helpful if we spent resources now figuring out how or whether we could deploy public support effectively and responsibly in case it turns out to be a useful tool. As of right now, it seems like no one has seriously looked into this or rigorously evaluated it as a strategy.
An AGI safety communications initiative should probably involve an iterative loop between research and implementation, i.e. research involving practical pilots to test out preliminary conclusions and gather feedback, and then scaling and implementation of the more promising approaches. An approach involving addressing all the questions and uncertainties first could take a decade, and not doing any outreach work for that long could be very damaging. Therefore, the two efforts should go hand in hand and improve each other.
Some areas that would be important to research further are:
This research would hopefully provide some clarity about what the outreach should look like. There are a wide range of different models and theories of change for this, such as bottom-up grassroots campaigns versus top-down campaigns led by experts, or “shout it from the rooftops and tell everyone who will listen” models versus funnels with higher barriers of entry for citizens to take action. Some of the forms the public outreach could take are:
If we make arguments about AGI risk that sound ridiculous, it may make people become convinced of the opposite of what we want—that AGI risks are a ridiculous issue to care about. This is why it is especially important to test in focus groups which framings sound compelling and resonate the most with laypeople.
If we lack transparency, it is also possible that this work could be suspected of being a bad-faith disinformation campaign by a malicious actor designed to slow down AI development in the West/the United States/Europe/etc.
If this becomes an issue that appeals only to one political orientation, it may lead to the other side of the aisle opposing it instinctively. If that side has more political power at a given time, this could be net negative. There could be ways to counter this, such as studying the properties of issues with bipartisan support and trying to adopt them. It may also be the case that this is less of a risk in countries without a bi-party political system, e.g. Netherlands or Germany, in which case the outreach could initially be piloted there.
Talking about the dangers of AGI or how powerful and versatile this technology can be could further fuel an arms race. It may also encourage some people to attempt to develop it for nefarious purposes.
There are three scenarios in which this could be harmful:
There may be case studies of this happening to advocacy about other issues that we can learn from, as well as tools we can implement to help citizens parse probably-good and probably-bad safety policies (such as the aforementioned board of AGI safety experts that independently rate them).
We want citizens to be concerned—the main reason anything got done about climate change or denuclearization is because everyday citizens started caring. That said, citizen concern could escalate into levels where it is counterproductive.
Extinction-level events can be difficult to picture and emotionally process, leading to overwhelm and inaction (see climate paralysis). In other cases, it can result in disproportionate and unwise policies—if we lack nuance, it can turn people against the field of AI as a whole, blocking the progress of really valuable technology.
When evaluating the probability of this risk, it is worth highlighting that it is very difficult to internalize existential risk from unaligned AGI—even many current AI safety researchers took years of convincing in order to believe this was a problem they should work on—so it is not clear that this would easily escalate into everyone becoming convinced that the end of the world is around the corner. If this turns out to indeed be a risk, it can be mitigated by talking about examples of really bad things that misaligned AGI could do that fall short of causing the extinction of humanity, such that we aren’t advertising that this is going to happen.
Over the past few months, I have had informal conversations about this topic with researchers working on alignment, as well as individuals from organizations like GovAI, FLI, FHI, and Google. While their levels of optimism varied, they agreed that if we can figure out how to do it safely and responsibly, this seems like a broadly good idea—from “worth looking into” to “essential”. It would give policymakers an incentive to care about this issue, which is currently lacking, as well as possibly put pressure on those working on AI capabilities to halt their work or increase standards for safety.
However, the resources the EA community is currently devoting to this are minimal, as it falls outside the scope of most existing AI safety organizations. There is currently an absence of rigorous research looking into the potential value, risks, and strategies for communicating with the public and key stakeholders about AGI risks. While there are important ways this could go wrong, if it goes right, work in this area could create a lot of value and meaningfully increase the probability that humanity survives the alignment problem.