Theres a very interesting passage in here, showing that asking the public extinction questions in terms of odds rather than percentages made the estimates of risk six orders of magnitude lower:

Participants in our public survey of 912 college graduates estimated a higher median probability of extinction by 2100 (5%) than superforecasters (1%) but lower than that of experts (6%). A similar pattern also emerged for AI-caused extinction (public survey participants gave a 2% probability, and superforecasters and domain experts gave 0.38% and 3%, respectively).

However, respondents of the same sample estimated much lower chances of both extinction and catastrophe by 2100 when presented with an alternative elicitation method. In a follow-up survey, we gave participants examples of low probability events—for example, that there is a 1-in-300,000 chance of being killed by lightning. We then asked them to fill in a value for “X” such that there was a “1-in-X” chance of a given risk—like human extinction by 2100.69 69 The set of reference classes we gave to participants had ten examples, including: 1 in 2: Probability a flip of a fair coin will be Tails 1 in 300,000: Lifetime probability of dying from lightning 1 in 10,000,000: Probability a random newborn becomes a U.S. president

Using that method, the median probability of humanity’s extinction before 2100 was 1 in 15 million. The median probability of AI-caused extinction before 2100 was 1 in 30 million. 

This seems like incredibly strong evidence of anchoring bias in answers to this question, and it goes both ways (if you see percentages, you anchor to the 1-99 range). What I want to know is whether this effect would show up for superforecasters and domain experts as well. Are the high risk estimates of AI extinction merely an anchoring bias artifact? 

One potential reason for the observed difference in expert and superforecaster estimates: even though they're nominally participating in the same tournament, for the experts, this is a much stranger choice than it is for the superforecasters, who presumably have already built up an identity where it makes sense to spend a ton of time and deep thought on a forecasting tournament, on top of your day job and other life commitments. I think there's some evidence for this in the dropout rates, which were 19% for the superforecasters but 51% (!) for the experts, suggesting that experts were especially likely to second-guess their decision to participate. (Also, see the discussion in Appendix 1 of the difficulties in recruiting experts - it seems like it was pretty hard to find non-superforecasters who were willing to commit to a project like this.)

So, the subset of experts who take the leap and participate in the study anyway are selected for something like "openness to unorthodox decisions/beliefs," roughly equivalent to the Big Five personality trait of openness (or other related traits). I'd guess that each participant's level of openness is a major driver (maybe even the largest driver?) of whether they accept or dismiss arguments for 21st-century x-risk, especially from AI.

Ways you could test this:

• Test the big five personality traits of all participants. My guess is that the experts would have a higher average openness than the superforcasters - but the difference would be even greater if comparing the average openness of the "AI-concerned" group (highest) to the "AI skeptics" (lowest). These personality-level differences seem to match well with the groups' object-level disagreements on AI risk, which mostly didn't center on timelines and instead centered on disagreements about whether to take the inside or outside view on AI.
  • I'd also expect the "AI-concerned" to have higher neuroticism than the "AI skeptics," since I think high/low neuroticism maps closely to something like a strong global prior that the world is/isn't dangerous. This might explain the otherwise strange finding that "although the biggest area of long-run disagreement was the probability of extinction due to AI, there were surprisingly high levels of agreement on 45 shorter-run indicators when comparing forecasters most and least concerned about AI risk." 
• When trying to compare the experts and superforecasters to the general population, don't rely on a poll of random people, since completing a poll is much less weird than participating in a forecasting tournament. Instead, try to recruit a third group of "normal" people who are neither experts nor superforecasters, but have a similar opportunity cost for their time, to participate in the tournament. For example, you might target faculty and PhD candidates at US universities working on non-x-risk topics. My guess is that the subset of people in this population who decide "sure, why not, I'll sign up to spend many hours of my life rigorously arguing with strangers about the end of the world" would be pretty high on openness, and thus pretty likely to predict high rates of x-risk.

I bring all this up in part because, although Appendix 1 includes a caveat that "those who signed up cannot be claimed to be a representative of [x-risk] experts in each of these fields," I don't think there was discussion of specific ways they are likely to be non-representative. I expect most people to forget about this caveat when drawing conclusions from this work, and instead conclude there must be generalizable differences between superforecaster and expert views on x-risk.

Also, I think it would be genuinely valuable to learn the extent to which personality differences do or don't drive differences in long-term x-risk assessments in such a highly analytical environment with strong incentives for accuracy. If personality differences really are a large part of the picture, it might help resolve the questions presented at the end of the abstract:

"The most pressing practical question for future work is: why were superforecasters so unmoved by experts’ much higher estimates of AI extinction risk, and why were experts so unmoved by the superforecasters’ lower estimates? The most puzzling scientific question is: why did rational forecasters, incentivized by the XPT to persuade each other, not converge after months of debate and the exchange of millions of words and thousands of forecasts?"

Why do you think participants largely didn't change their minds?

Thanks so much for writing this. I'm really looking forward to the area by area reports.

"track record of predictive accuracy over short time horizons" 

Out of interest, how long do you consider "short time horizons" to be?

Thanks for sharing!

For reference, here are the predictions for human extinction by 2100:

The 95 % confidence intervals for the total extinction risk by 2100 seem quite narrow. For superforecasters, domain and non-domain experts, and general x-risk experts, the upper bound is only 2, 3 and 5 times the lower bound.

It is interesting to note that, according to the superforecasters, the extinction risk from nuclear war is predicted to be 7.4 (= 0.074/0.01) times that of engineered pathogens, and 19.5 % (= 0.074/0.38) that of AI. In contrast, Toby Ord guessed in The Precipice that the existential risk from 2021 to 2120 from nuclear war is 3.33 % (= 0.1/3) that of engineered pathogens, and 1 % (= 0.1/10) that of AI.

Where would man-made global warming/climate change rank? I'd have thought, from all the informations we receive daily, it would have been number one? I assume non-anthropogenic means climate change happening naturally, and it's ranked number 5.