This was originally written by Eliezer Yudkowsky and posted on his Facebook wall. It is reposted here with permission from the author:
In August 2015, renowned statistician and predictor Nate Silver wrote "Trump's Six Stages of Doom" in which he gave Donald Trump a 2% chance of getting the Republican nomination (not the presidency, the nomination).
It's too late now to register an advance disagreement, but now that I've seen this article, I do think that Nate Silver's argument was a clear instance of something that I say in general you shouldn't do - what I used to call the Conjunction Fallacy Fallacy and have now renamed to the Multiple-Stage Fallacy (thanks to Graehl for pointing out the naming problem).
The Multiple-Stage Fallacy is when you list multiple 'stages' that need to happen on the way to some final outcome, assign probabilities to each 'stage', multiply the probabilities together, and end up with a small final answer. In other words, you take an arbitrary event, break it down into apparent stages, and say "But you should avoid the Conjunction Fallacy!" to make it seem very low-probability.
In his original writing, Nate listed 6 things that he thought Trump needed to do to get the nomination - "Trump's Six Stages of Doom" - and assigned 50% probability to each of them.
(Original link here: http://fivethirtyeight.com/…/donald-trumps-six-stages-of-d…/)
We're now past the first four stages Nate listed, and prediction markets give Trump a 74% chance of taking the nomination. On Nate's logic, that should have been a 25% probability. So while a low prior probability might not have been crazy and Trump came as a surprise to many of us, the specific logic that Nate used is definitely not holding up in light of current events.
On a probability-theoretic level, the three problems at work in the usual Multiple-Stage Fallacy are as follows:
1. First and foremost, you need to multiply *conditional* probabilities rather than the absolute probabilities. When you're considering a later stage, you need to assume that the world was such that every prior stage went through. Nate Silver was probably - though I here critique a man of some statistical sophistication - Nate Silver was probably trying to simulate his prior model of Trump accumulating enough delegates in March through June, not imagining his *updated* beliefs about Trump and the world after seeing Trump be victorious up to March.
1a. Even if you're aware in principle that you need to use conjunctive probabilities, it's hard to update far *enough* when you imagine the pure hypothetical possibility that Trump wins stages 1-4 for some reason - compared to how much you actually update when you actually see Trump winning! (Some sort of reverse hindsight bias or something? We don't realize how much we'd need to update our current model if we were already that surprised?)
2. Often, people neglect to consider disjunctive alternatives - there may be more than one way to reach a stage, so that not *all* the listed things need to happen. This doesn't appear to have played a critical role in Nate's prediction here, but I've often seen it in other cases of the Multiple-Stage Fallacy.
3. People have tendencies to assign middle-tending probabilities. So if you list enough stages, you can drive the apparent probability of anything down to zero, even if you solicit probabilities from the reader.
3a. If you're a motivated skeptic, you will be tempted to list more 'stages'.
Fallacy #3 is particularly dangerous for people who've read a lot about the dangers of overconfidence. Right now, we're down to two remaining stages in Nate's "six stages of doom" for Trump, accumulating the remaining delegates and winning the convention. The prediction markets assign 74% probability that Trump passes through both of them. So the conditional probabilities must look something like 90% probability that Trump accumulates enough delegates, and then 80% probability that Trump wins the convention given those delegates.
Imagine how overconfident this would sound without the prediction market! Oh, haven't you heard that what people assign 90% probability usually doesn't happen 9 times out of 10?
But if you're not willing to make "overconfident" probability assignments like those, then you can drive the apparent probability of anything down to zero by breaking it down into enough 'stages'. In fact, even if someone hasn't heard about overconfidence, people's probability assignments often trend toward the middle, so you can drive down their "personally assigned" probability of anything just by breaking it down into more stages.
For an absolutely ridiculous and egregious example of the Multiple-Stage Fallacy, see e.g. this page which commits both of the first two fallacies at great length and invites the third fallacy as much as possible: http://www.jefftk.com/p/breaking-down-cryonics-probabilities. To be sure, Nate Silver didn't do anything remotely on THAT order, but it does put all three subfallacies on clearer display than they appear in "Trump's Six Stages of Doom".
From beginning to end, I've never used this style of reasoning and I don't recommend that you do so either. Beware the Multiple-Stage Fallacy!
Edit (3/16): Reply from Jeff Kaufman below: