(Note: This post is probably old news for most readers here, but I find myself repeating this surprisingly often in conversation, so I decided to turn it into a post.)
I don't usually go around saying that I care about AI "safety". I go around saying that I care about "alignment" (although that word is slowly sliding backwards on the semantic treadmill, and I may need a new one soon).
But people often describe me as an “AI safety” researcher to others. This seems like a mistake to me, since it's treating one part of the problem (making an AGI "safe") as though it were the whole problem, and since “AI safety” is often misunderstood as meaning “we win if we can build a useless-but-safe AGI”, or “safety means never having to take on any risks”.
Following Eliezer, I think of an AGI as "safe" if deploying it carries no more than a 50% chance of killing more than a billion people:
When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. [...] Practically all of the difficulty is in getting to "less than certainty of killing literally everyone". Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don't care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI 'this will not kill literally everyone'.
Notably absent from this definition is any notion of “certainty” or "proof". I doubt we're going to be able to prove much about the relevant AI systems, and pushing for proofs does not seem to me to be a particularly fruitful approach (and never has; the idea that this was a key part of MIRI’s strategy is a common misconception about MIRI).
On my models, making an AGI "safe" in this sense is a bit like finding a probabilistic circuit: if some probabilistic circuit gives you the right answer with 51% probability, then it's probably not that hard to drive the success probability significantly higher than that.
If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they've probably... well, they've probably found a way to keep their AGI weak enough that it isn’t very useful. But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
The condition that the AI actually be useful for pivotal acts is an important one. We can already build AI systems that are “safe” in the sense that they won’t destroy the world. The hard part is creating a system that is safe and relevant.
Another concern with the term “safety” (in anything like the colloquial sense) is that the sort of people who use it often endorse the "precautionary principle" or other such nonsense that advocates never taking on risks even when the benefits clearly dominate.
In ordinary engineering, we recognize that safety isn’t infinitely more important than everything else. The goal here is not "prevent all harms from AI", the goal here is "let's use AI to produce long-term near-optimal outcomes (without slaughtering literally everybody as a side-effect)".
Currently, what I expect to happen is that humanity destroys itself with misaligned AGI. And I think we’re nowhere near knowing how to avoid that outcome. So the threat of “unsafe” AI indeed looms extremely large—indeed, this seems to be rather understating the point!—and I endorse researchers doing less capabilities work and publishing less, in the hope that this gives humanity enough time to figure out how to do alignment before it’s too late.
But I view this strategic situation as part of the larger project “cause AI to produce optimal long-term outcomes”. I continue to think it's critically important for humanity to build superintelligences eventually, because whether or not the vast resources of the universe are put towards something wonderful depends on the quality and quantity of cognition that is put to this task.
If using the label “AI safety” for this problem causes us to confuse a proxy goal (“safety”) for the actual goal “things go great in the long run”, then we should ditch the label. And likewise, we should ditch the term if it causes researchers to mistake a hard problem (“build an AGI that can safely end the acute risk period and give humanity breathing-room to make things go great in the long run”) for a far easier one (“build a safe-but-useless AI that I can argue counts as an ‘AGI’”).
A lot of people misunderstand "existential risk" as meaning something like "extinction risk", rather than as meaning 'anything that would make the future go way worse than it optimally could have'. Tacking on "safety" might contribute to that impression; we're still making it sound like the goal is just to prevent bad things (be it at the level of an individual AGI project, or at the level of the world), leaving out the "cause good things" part. That said, "existential safety" seems better than "safety" to me.
(Nate's thoughts here.)
I don't know what you mean by "governance". The EA Forum wiki currently defines it as:
... which makes it sound like governance ignores plans like "just build a really good company and save the world". If I had to guess, I'd guess that the world is likeliest to be saved because an adequate organization existed, with excellent internal norms, policies, talent, and insight. Shaping external incentives, regulations, etc. can help on the margin, but it's a sufficiently blunt instrument that it can't carve orgs into the exact right shape required for the problem structure.
It's possible that the adequate organization is a government, but this seems less likely to me given the absolute number of exceptionally competent governments in history, and given that govs seem to play little role in ML progress today.
Open Phil's definition is a bit different:
Open Phil goes out of its way to say "not just governments", but its list ("norms, policies, laws, processes, politics, and institutions") still makes it sound like the problem is shaped more like 'design a nuclear nonproliferation treaty' and less like 'figure out how to build an adequate organization', 'cause there to exist such an organization', or the various activities involved in actually running such an organization and steering it to an existential success.
Both sorts of activities seem useful to me, but dividing the problem into "alignment" and "governance" seems weird to me on the above framings—like we're going out of our way to cut skew to reality.
On my model, the proliferation of AGI tech destroys the world, as a very strong default. We need some way to prevent this proliferation, even though AGI is easily-copied software. The strategies seem to be:
1 sounds the most like "AGI governance" to my ear, and seems impossible to me, though there might be more modest ways to improve coordination and slow progress (thus, e.g., buying a little more time for researchers to figure out how to do 2 or 3). 2 and 3 both seem promising to me, and seem more like tech that could enable a long (or short) reflection, since e.g. they could also help ensure that humanity never blows itself up with other technologies, such as bio-weapons.
Within 2, it seems to me that there are three direct inputs to things going well:
There's then a larger pool of enabling work that helps with one or more of those inputs: figuring out what sorts of organizations to build; building and running those organizations; recruiting, networking, propagating information; prioritizing and allocating resources; understanding key features of the world at large, like tech forecasting, social dynamics, and the current makeup of the field; etc.
"In addition to alignment, you also need to figure out target selection, capabilities, and (list of enabling activities)" seems clear to me. And also, you might be able to side-step alignment if 3 (or 1) is viable. "Moreover, you need a way to hand back the steering wheel and hand things off to a reasonable decision-making process" seems clear to me as well. "In addition to alignment, you also need governance" is a more opaque-to-me statement, so I'd want to hear more concrete details about what that means before saying "yeah, of course you need governance too".