In a recent New Yorker article, Will McAskill ponders
“My No. 1 worry is: what if we’re focussed on entirely the wrong things?” he said. “What if we’re just wrong? What if A.I. is just a distraction?”
In order to determine that, one question we might want to ask ourselves is:
Out of all the possible universes where human extinction occurs this century, what percentage of extinction events will be proximally caused by X?
Your answer may look something like this:
But what if instead it looks like this?
("Other" meaning both "unlisted" and "unknown".)
Is the chart above plausible? One reason why it may be - think of what would have happened if you asked experts for their Doom Pie in 1930 (less than a century ago).
In 1930, the nuclear bomb hadn't yet been invented. The modern computer was still in the future, and science fiction was barely emerging as a genre. Leading lights of the day would probably have mentioned pandemics (with the Spanish Flu in recent memory) and war (though without the specifics of the nuclear weapon in mind) as the main risks to worry about.
Specific scenarios of a rogue AI or engineered pandemic would have been categorized as "Other" on their chart. And yet, here we are today, with many of us arguing that we should drop everything and single-mindedly focus on addressing them. Knowing this, how confident should we feel in our current judgment? What do you think the typical Doom Pie might look like in 2110 (should we make it this far?)
Assuming our premise is plausible, let us make a few further assumptions for the sake of argument, and work through the consequences.
Imagine you are playing a new game. In this game:
1. the pie is an accurate representation of reality.
2. the total of possible extinction events is 100. The number can decrease but never increase.
3. each turn, you will roll a dice, and if you get a particular unlucky combination, 1 of the possible extinction events will randomly occur.
4. if you are still alive, each turn allows you to either "uncover" or "eliminate" 1 category of extinction event (for example, eliminate all 5 pandemic events at once). Uncovering a category costs 2 turns. Eliminating it only costs 1. You do not know the size of unknown categories in advance - only that regardless of their size, you can eliminate them in 1 turn once uncovered.
What should your strategy be? I am not a mathematician but presumably you would have a strong incentive to use your early turns (=time) uncovering unknown risks in case they represent a large percentage of the sample. Most of the expected value would be found in "Uncover" moves.
Also note that, if your hope turns out to be unfounded, most of the moves required to beat the game will be "Uncover" moves. Take the extreme scenario in which the "Other" territory comprises 82 categories each representing an event with a 1% chance of occurring. In that scenario, beating the game requires 264 moves. Of those, 164, or 62%, will be "Uncover" moves.
The model is of course both crude and highly flawed, but I would argue that most plausible refinements would increase the value of exploration even further. For instance, we might question why the total number of extinction events would be fixed. Perhaps it would be more realistic to add new unidentified risks at certain points in time. However that would only make exploration even more valuable.
If more existential risks than we think are curently unknown to us, what should we do about it?
Here is a tentative list to get the conversation started:
What do you think? Am I overestimating the degree of our unknowledge? Am I misreading the consequences?
And perhaps more practically - what does your Doom Pie look like, and why?
