This competition entry has been selected for publication by the Forum team.
My soft critique of the unawareness argument, in a nutshell
One reason to doubt DiGiovanni's unawareness argument (specifically, his empirical premise) is that it is unclear where the line is between A) my understanding of a decision problem is fine-grained enough for me to trust my best guess, and B) it is too coarse-grained (and I must suspend judgment). While DiGiovanni does not object to this and recognizes some version of the problem, he suggests that the line must come before “promoting the impartial good”.
I object to the most radical interpretation of his claim (cf., “unconditional cluelessness”). There are at least some contrived decision situations where someone is justified in believing some action promotes the impartial good better than another—consider, e.g., an unrealistic scenario where you could press a button with guaranteed overall catastrophic consequences or one that would get the universe rid of all unwanted suffering forever while keeping everything else equal. (More on this in §1.)
Of course, this is presumably not how DiGiovanni intended the argument, and does not help us figure out what to do, or not do, in our actual present situations. However, this shows that it is not “promoting the impartial good” that poses intractability, per se. It is our coarse awareness of the total consequences in our specific real-world situations. But how coarse is too coarse? In unrealistically convenient decision situations, our understanding of the total consequences may be somewhat coarse without yet forcing suspension of judgment. And slightly changing the situation so it is barely coarser may not change this, still. But there is a threshold of change above which suspending judgment becomes mandatory. While this threshold may be reached long before any real-world decision dilemma between two available actions, this may not be obvious. Once some c-preferences are admitted in clean cases, the empirical premise needs a domain restriction: why should the line fall before every real-world-relevant dilemma? Is there really not a single actual available-to-us action that we are justified in believing is c-better than one single other? While I do not attempt to find any in the present piece, I show how one might try to move outward from, e.g., the clean certain catastrophe button case, gradually removing idealizations and adding new sources of uncertainty and backfire risks. (See §§2-3.) This may help us locate where, if anywhere, the switch from justified c-preference to arbitrary c-preference actually happens, and double-check the claim that it necessarily occurs before any real-world-relevant dilemma between two actions is reached.
This leaves the intended version of the unawareness argument (cf., “expansive cluelessness”) untouched, but points to a potential weakness in its empirical premise—a weakness that may deserve further investigation. In fact, given the harder-to-contest problems with i) the other objections to DiGiovanni's case, and ii) ad hoc proposals such as consequentialist bracketing, this appears to me as the most promising way forward.
Clarifications, illustrations, and (modest) extensions of some of these points are given through a dialogue between the two main protagonists of the following short story.
Chapter 1: If you can kill everyone, don't do it
You have just boarded a train to Arbitraryland, a place where people hold c-preferences (i.e., beliefs about how to do good impartially) that are no more justified than the hypothetical exact opposite c-preferences. Your goal is to get off the train as late as possible before the Arbitrarylandic border. However, there will be no announcement. In fact, passengers disagree about where the border is, and all believe they get off early enough. The people living in Arbitraryland do not know that they are.
As you enter the train, you see a woman sitting in the first row, in the aisle seat closest to the door. She had not taken off her coat, as if she were ready to jump out at any second. She looked rather tranquil, however. The seat next to her seems to be the only one free in this wagon. You sit there and take off your jacket. She smiles at you.
“Hey, I'm Tony. First time?”
“Yes,” you say. “I'm a bit nervous.”
“Don't worry,” Tony says. “It won't be long before we should get off. We aren't justified in c-preferring any action over any oth—”
An onboard announcement cuts her off:
“Next station is 'Do not press the guaranteed extinction button.' Passengers who suspend judgment on whether human extinction would be a good or bad outcome, all things considered, should leave the train before the doors close.”
You turn your head towards Tony.
“We are still at the departure station, right?” you ask. “Why get off already?”
“We are,” Tony says. “But this does not mean the Arbitrarylandic border isn't right in front of us, before even the first station on our way.”
You see one person standing in front of the door, hesitating for a moment before getting off, right before the automatic door closes.
“Bye, Jacy!” Tony calls.
“Crap,” you say. “Should I have left, too? I didn't get the time to think! I mean, I disvalue human extinction, but is that truly an impartial belief that accounts for all possible consequences?”
“It's okay,” Tony says, with her soft, reassuring voice. “All the stations after the next one assume human extinction is outcome-robustly bad, so it is all about implementation robustness (i.e., does the action actually increase or reduce the chance of extinction). But if you feel too uncertain about outcome robustness, here, just replace 'human extinction' with whatever outcome you actually find bad, in your head. This will prepare you for the other trains.”
You nod, relax, and lie back in your seat.
“What is the 'guaranteed extinction button,' anyway?” you ask.
“It's a hypothetical button,” Tony says, “that, if you press it, makes humanity instantly vanish, with 100% certainty.”
“But... that's so unrealistic. No one actually faces this choice in the real world.”
“That's right,” Tony says. “The first stations always assume hypothetical clean situations. It's to make sure that people like me stay a bit for the ride. Such hypotheticals are relevant. They show that the most radical interpretations of DiGiovanni's cluelessness thesis are implausible.”
She unfolds a piece of paper and shows you a printed copy of DiGiovanni's unawareness argument. In particular, she points to these last lines:
P3. Empirical premise: Due to unawareness (at least), our understanding of any pair of actions' possible consequences is indeed very coarse-grained — enough that the conclusion of (P2) follows (i.e., these actions' “EVs” are incomparable). In particular, the actions' “EVs” are too severely imprecise to compare them, regardless of whether we (a) formally model these “EVs” or (b) appeal to informal/heuristic arguments.
Conclusion: We aren't justified in c-preferring any action over any other.
Tony lowers the paper.
“DiGiovanni did not specify who 'we' is,” she says. “Who should feel concerned? Anyone plausibly reading this in the real world? Anyone who could possibly read this in the real world? Any agent in any hypothetical context?”
“Right,” you say. “So if we buy that the empirical premise is false in the situation where someone is literally in front of the guaranteed extinction button, we have already found a counterexample to the most radical interpretation of his conclusion.”
“Exactly,” Tony says. “I like to call this conclusion unconditional cluelessness: the view according to which no one, in any hypothetical situation, is justified in c-preferring any action over any other. No one buys this view, except perhaps some Pyrrhonian skeptics who suspend judgment on everything. And this is clearly not the conclusion DiGiovanni meant to defend.”
The train gathers speed. The only sound is the rhythm of the wheels. Outside, it is dark. But as the tracks curve, you can already see the lamps of the next stop glowing somewhere ahead.
Chapter 2: If you might kill everyone, you might make them stronger
About ten minutes later, the train stops again.
The speakers crackle. “We have arrived at 'Do not press the guaranteed extinction button.' Next station: 'Stop a misanthropic superintelligent AI from trying to wipe out humanity.'”
You look around you to see if anyone wants to get off. No one does. You look at Tony:
“Isn't the next station basically the same as this one?” you ask.
“Not quite,” Tony says, her eyes closed, as if she were meditating. “At this station, absolute outcome certainty is gone. A superintelligent AI is very likely to wipe out humanity, but it—”
You cut her off. “Sure, it might fail. But either it succeeds, and the outcome is bad, or it doesn't, and then we're back to business as usual, right?”
Tony opens her eyes and turns her head towards you, smiling, as the train is already starting again.
“Do you think that a humanity that somehow survives a misanthropic superintelligence trying to wipe it out amounts to business as usual?”
You think for a few seconds.
“Right... It might learn valuable lessons and reduce future AI risks, or something like that. 'What doesn't kill you makes you stronger,' I guess. Or it may, at least.”
She nods, then settles back into her meditative posture.
“So if you prevent the superintelligence from trying to annihilate humans,” you add, “there's some tiny chance you actually increase extinction risks.”
“Yes,” Tony says. “It's a variant of the 'Smokey Bear effect': suppressing fires can make forests less resilient, making later fires worse. Except extinction risks are the fire here.”
“Still, in our case, surely this is unlikely enough for us to prefer stopping the superintelligence, right? This Smokey Bear effect is not so relevant here. We were definitely right to stay on the train.”
“Yes. It seems negligible here. But don't dismiss it too fast. It might not be as negligible at the later stations.”
As you wonder what the next stations could be, the two of you stay silent until the train stops again.
The speakers crackle again. “We have arrived at 'Stop a misanthropic superintelligent AI from trying to wipe out humanity.' Next station: 'Stop a misanthropic coalition of the world's best bioengineers from releasing an extinction-seeking pandemic.'”
You look around. No one gets off.
“Okay, I see,” you say. “A bioengineer coalition is less likely than a superintelligent AI to succeed in killing everyone, so the Smokey Bear risk is bigger.”
Tony nods, smiling.
“Is it going to be like that at every station?” you ask. “A slightly trickier thought experiment, up until it becomes really unclear what the right choice is?”
“You got it,” Tony says. “And whoever gets off before the first real-world-relevant dilemma agrees with the most natural interpretation of DiGiovanni's thesis: expansive cluelessness, the view that no present-day human, in (nearly) any situation they may face, is justified in c-preferring any action over any other.”
“I see. That's not as radical as unconditional cluelessness, but just as damning for any practical purpose.”
“Yes, it is.”
The door closes, and the train starts again.
“Surely there's at least one realistic station before the train reaches Arbitraryland, though,” you say. “There's no way no single action is c-preferable over any other. DiGiovanni's empirical premise cannot be true for all pairs of actions I have available.”
“Do you want to take a look at what the next stations will be?” Tony asks, smiling.
She unfolds another piece of paper with a surprisingly short list of stations. Unlike her previous note, this one is fully handwritten.
Do not press the guaranteed extinction button
Stop a misanthropic superintelligent AI from trying to wipe out humanity
Stop a misanthropic coalition of the world's best bioengineers from releasing an extinction-seeking pandemic
Stop a misanthropic coalition of all nuclear state leaders from trying to wipe out humanity
Stop a misanthropic Trump from trying to wipe out humanity
Stop Trump from first-striking Russia with American nuclear weapons
Convince the U.S. government not to withdraw from the Nuclear Non-Proliferation Treaty
Convince the U.S. government to increase the budget dedicated to the NNSA's Defense Nuclear Nonproliferation
Try to convince the U.S. government to take x-risks from nuclear weapons more seriously (Note: First real-world-relevant one?)
???
Do untargeted nuclear x-risk reduction community building
???
Do untargeted AI safety community building
???
“Why the question marks?” you ask.
“There are many more stations I don't know,” Tony says. “I always get off before them. I know about many of the stations written on this sheet only from conversations with other passengers.”
“You have never gone further than...”
You look at the list again.
“...than 'Try to convince the U.S. government to take x-risks from nuclear weapons more seriously'?”
“No,” Tony says. “In fact, I have never been to this station. Or the one before that. And I'm not quite sure about the one before that. Maybe also the one before that, actually. And... well. You get it.”
She looks at you and smiles, noticing your incredulous stare. After a short silence, you finally ask:
“Why?” you ask. “Why do you get off so early?”
“Remember Smokey Bear?” Tony says. “Humanity is decently likely to survive Trump launching nuclear weapons, even if he were trying to kill everyone. It is very plausible that at least some groups, in some isolated locations, survive and end up rebuilding civilization: a civilization that would have survived a nuclear strike and a nuclear winter, and might therefore be far more likely than we are to avoid future ones, or survive them.”
She glances back at the handwritten list before continuing.
“Plus, if the U.S. attacks Russia, we also need to consider asymmetries in who gets affected. Russian citizens and their government would be much more likely than the rest of humanity to be wiped off the map, which may have important implications for future x-risks.”
She taps the next line on the paper.
“But say you'd still c-prefer preventing this and want to convince the U.S. government to increase the budget dedicated to the NNSA's Defense Nuclear Nonproliferation. In addition to Smokey Bear and these actor-asymmetry worries, you now have additional worries. For example, your action might take attention away from potentially more serious AI x-risks.”
She looks up from the list.
“Also, rivals may interpret expanded U.S. nuclear-nonproliferation efforts as part of a broader counterforce or regime-pressure strategy, which reduces trust and makes nuclear command postures more brittle. Finally, making the U.S. government take x-risks from nuclear weapons more seriously may, in addition to all the above, make the U.S. more worried about Russia and more ready to retaliate, in a way that overall increases nuclear x-risks.”
She shrugs.
“At this point, you may believe whatever you want. The train will already be well within Arbitraryland, in my humble opinion.”
You stay quiet for a moment, looking at your feet.
Eventually, you ask, “But... when does it enter Arbitraryland, then? Where do you draw the line? At what station are we supposed to get off?”
“Honestly, I don't really know,” Tony says. “But well before the first real-world-relevant one, I'm afraid.”
She notices your apparent despair and looks at you with empathy.
“So... what do we do?” you ask.
“Let's talk about this once we're off this train, shall we?” Tony says. “You should rest while you can.”
You take a deep breath and lie back in your seat. After a few minutes of rumination, you start slowly dozing off.
Chapter 3: The other trains
“Wake up, my friend, wake up!” Tony shouts. “We need to get off!”
You open your eyes and see her blocking the door a few meters away. You have no idea how long you've been asleep. You grab your coat and hurry towards her. She lets the door go. You are now both on the platform. Completely alone. No one else got off. As the train leaves the station, you see the passengers crowding against the windows, looking at you as if you were animals in a zoo.
“It cancels out!” one passenger calls.
“Heuristiiiiiics!” another shouts.
“You're just risk-averse!” yet another yaps.
The accelerating engine then becomes too loud for you to hear anything more from the passengers still on board. You turn your head towards Tony.
“What station is this?”
“We got off way before the first real-world-relevant one,” Tony says.
“I figured. But where, exactly?”
“I'm not sure,” Tony says. “I didn't pay much attention, honestly. It's always a bit arbitrary which exact station to get off at. It's unclear where the border precisely is. Come. We should move.”
The two of you leave the platform to enter a surprisingly busy train station. You see screens filled with information about train departures. Everyone seems in a hurry to get somewhere. Some are running. Some are shouting “Timelines!”
“You might want to take one of these trains,” Tony says, pointing at the biggest screen in front of you. “Not all of them are about the implementation robustness of strategies to prevent human extinction, or lack thereof. There are many others, even though effects on the likelihood of human extinction may be relevant for all of them.”
You take a moment to think, looking at the screen above your head. What is written on there is hard to decipher, however.
“Which of these trains do not go to Arbitraryland?” you ask.
“All trains go to Arbitraryland,” Tony says. “Some might serve at least a few more legit stations before reaching it, though. Bye, my friend. I hope you find action-guidance.”
“Wait, which ones?” you ask, still squinting at the screen. “And do these stations represent choices in realistic scenarios that matter for what I should do, right now, in the real w—”
You turn your head towards her as you are finishing your sentence. But she is gone.
Acknowledgments
Thanks to Anthonin Broi for helpful conversations.
I don't have any very insightful immediate thoughts on where to get off the train (and which train to take in the first place), but I wanted to leave a brief comment to say that I found this short story, and the main character's ceaseless demand for a clear answer, helpfully relatable. And on first reflection, the idea of using the "train to Arbitrariland" as an analogy for thinking about DiGiovanni's empirical premise seems like a promising route, thanks for spelling it out and sharing it here!
Thanks Sarah, very helpful and pleasant to have your reaction :)