"Humanity has seen many claims of this form." What exactly is your reference class here? Are you referring just to religious claims of impending apocalypse (plus EA claims about AI technology? Or are you referring more broadly to any claim of transformative near-term change?

I agree with you that claims of supernatural apocalypse have a bad track record, but such a narrow reference class doesn't (IMO) include the pretty technically-grounded concerns about AI. Meanwhile, I think that a wider reference class including other seemingly-unbelievable claims of impending transformation would include a couple of important hits. Consider:

• It's 1942. A physicist tells you, "Listen, this is a really technical subject that most people don't know about, but atomic weapons are really coming. I don't know when -- could be 10 years or 100 -- but if we don't prepare now, humanity might go extinct."

• It's January 2020 (or the beginning of any pandemic in history). A random doctor tells you "Hey, I don't know if this new disease will have 1% mortality or 10% or 0.1%. But if we don't lock down this entire province today, it could spread to the entire world and cause millions of deaths."

• It's 1519. One of your empire's scouts tells you that a bunch of white-skinned people have arrived on the eastern coast in giant boats, and a few priests think maybe it's the return of Quetzalcoatl or something. You decide that this is obviously crazy -- religious-based forecasting has a terrible track record, I mean these priests have LITERALLY been telling you for years that maybe the sun won't come up tomorrow, and they've been wrong every single time. But sure enough, soon the European invaders have slaughtered their way to your capital and destroyed your civilization.

Although the Aztec case is particularly dramatic, many non-European cultures have the experience of suddenly being invaded by a technologically superior foe powered by an exponentially self-improving economic engine -- this sounds at least as similar to AI worries as your claim that Christianity and AI worry are in the same class. There might even be more stories of sudden European invasion than predictions of religious apocalypse, which would tilt your base-rate prediction decisively towards believing that transformational changes do sometimes happen.

Thanks for writing this.

I thought about why I buy the AI risk arguments despite the low base rate, and I think the reason touches on some pretty important and nontrivial concepts.

When most people encounter a complicated argument like the ones for working on AI risk, they are in a state of epistemic learned helplessness: that is, they have heard many convincing arguments of a similar form be wrong, or many convincing arguments for both sides. The fact that an argument sounds convincing fails to be much evidence that it's true.

Epistemic learned helplessness is often good, because in real life arguments are tricky and people are taken in by false arguments. But when taken to an extreme, it becomes overly modest epistemology: the idea that you shouldn't trust your models or reasoning just because other people whose beliefs are similar on the surface level disagree. Modest epistemology would lead you to believe that there's a 1/3 chance you're currently asleep, or that the correct religion is 31.1% likely to be Christianity, 24.9% to be Islam, and 15.2% to be Hinduism.

I think that EA does have something in common with religious fundamentalists: an orientation away from modest epistemology and towards taking weird ideas seriously. (I think the number of senior EAs who used to be religious fundamentalists or take other weird ideas seriously is well above the base rate.) So why do I think I'm justified in spending my career either doing AI safety research or field-building? Because I think the community has better epistemic processes than average. 

Whether it's through calibration, smarter people, people thinking for longer or more carefully, or more encouragement of skepticism, you have to have a thinking process that results in truth more often than average, if you want to reject modest epistemology and still believe true things. From the inside, the EA/rationalist subcommunity working on AI risk is clearly better than most millenarians (you should be well-calibrated about this claim, but you can't just say "but what about from the outside?"-- that's modest epistemology). If I think about something for long enough, talk about it with my colleagues, post it on the EA forum, invite red-teaming, and so on, I expect to reach the correct conclusion eventually, or at least decide that the argument is too tricky and remain unsure (rather than end up being irreversibly convinced of the wrong conclusion). I'm very worried about this ceasing to be the case.

Taking weird ideas seriously is crucial for our impact: I think of there being a train to crazy town which multiplies our impact by >2x at every successive stop, has increasingly weird ideas at every stop, and at some point the weird ideas cease to be correct. Thus, good epistemics are also crucial for our impact.

Hey! I liked certain parts of this post and not other parts of this post. I appreciate the thoughtfulness by which you critique EA through this post.

On your first point about the AI messiah: 

I think the key distinction is that there are many reasons to believe this argument about the dangers of an AGI are correct, though. Even if many claims with a similar form are wrong, that doesn't exclude this specific claim from being right. 

"Climate scientists keep telling us about how climate change is going to be so disastrous and we need to be prepared. But humanity has seen so many claims of this form and they've all been so wrong!"

The key distinction is that there is a lot of reason to believe that AGI will be dangerous. There is also a lot of reason to support the claim that we are not prepared currently. Without addressing that chain of logic directly, I don't think I'm convinced by this argument.

On your second point about the EA religious tendencies:

Because religious communities are one of the most common communities we see, there's obviously going to be parallels that exist between religious communities and EA. 

Some of these analogies hold, others not so much. We, too, want to community build, network, and learn from each other. I'd love for you to point at specific examples of things EA do, from conferences to holding EA university groups, that are ineffective or unnecessary.

To perhaps a greater point of EA perhaps becoming too groupthink-y, which I think may be warranted:

I think a key distinction is that EA has a healthy level of debate, disagreement, and skepticism - while religions tend to demand blind faith in believing something unprovable. This ongoing debate on how to do the most good I personally find the most valuable in the community - and I hope this spirit never dies.

Keep on critiquing EA; I think such critiques are extremely valuable. Thanks for writing this.

In brief: writing that is kind, relevant to the discussion at hand, and honest.

https://forum.effectivealtruism.org/posts/yND9aGJgobm5dEXqF/guide-to-norms-on-the-forum

Obvious point, but you could assign significant credence to this being the right take, and still think working on A.I. risk is very good in expectation, given exceptional neglectedness and how bad an A.I. takeover could be. Something feels sleazy and motivated about this line of defence to me, but I find it hard to see where it goes wrong. 

Hmm, do you have close friends who grew up in non-Abrahamic cultures? I feel like a lot of the stuff in this post seems to presume, though does not state outright, that something like religion (or even a very specific form of religion??) is approximately a human universal. This feels very alien/surprising compared to my upbringing (which is not particularly atheist by Chinese standards, like my mom literally converted to Christianity), so I'm wondering if one of us is just confused here.

(Looping back here because I thought the post never made sense to me but it did to a lot of people. At first I attributed it to just pro- vs anti- LT tribalism, but wanted to advance the alternative hypothesis that we just have different expectations for the appeal of religiosity or how probable different ideas* map to Christianity in particular).

*I also thought the mapping of "simulation hypothesis -> Christianity" that seems popular in some circles recently to be kind of silly pattern-matching.

It does seem like an optimistic expectation that there will be an arrival of entities that are amazingly superior to us. This is not far-fetched though. Computers already surpass humans' capacities on several thought processes, and therefore have already demonstrated that they are better in some aspects of intelligence. And we've created robots that can outperform humans in virtually all physical tasks. So, the expectation is backed by evidence.

Expecting super AGI differs from expecting the arrival of a messiah-like figure in that instead of expecting a future in which an entity will come on its own and end all our suffering and improve our lives immeasurably, we are doing the work to make AI improve our lives. Also, the expectations differ in how we prepare for them. In the case of the messiah, it seems like acting moral so we can get into heaven might be vague, unchanging, and random. On the other hand, in the case of super AGI,  AI safety work is constantly changing and learning new things. However, it is still interesting that the two expectations bear a resemblance.