why are humans moral but our moralities vary so much across different communities? Why do people sometimes abandon or fail to act according to their beliefs/values without epistemic or philosophical reasons to do so? Why do communities sometimes collectively become very extreme in their beliefs/values, again without apparent epistemic or philosophical justification?

I think "status" plays some part in the answers to these, but only a fairly small one. 

Why do moralities vary across different communities? Primarily because they are raised in different cultures with different prevalent beliefs. We then modify those beliefs from the baseline as we encounter new ideas and new events, and often end up seeking out other people with shared values to be friends with. But the majority of people aren't just pretending to hold those beliefs to fit in (although that does happen), the majority legitimately believe what they say. 

Why do communities get extreme? Well, consult the literature on radicalisation, there are a ton of factors. A vivid or horrible event or ongoing trauma sometimes triggers an extreme response. Less radical members of groups might leave, making the average more radical, so even more moderates leave or split, until the group is just radicals. 

As to why we fail to act according to their values, people generally have competing values, including self-preservation and instincts, and are not perfectly rational. Sometimes the primal urge to eat a juicy burger overcomes the calculated belief that eating meat is wrong. 

These are all amateur takes, a sociologist could probably answer better. 

I define “optimism” as the tendency to weigh positive outcomes higher than their expected value and negative outcomes lower

I don't think this is the definition that Pope & Belrose are using. I think they are using it in the same sense as "I'm optimistic about my relationship": A genuine belief that something will go well. 

I think they claim to be optimistic because they genuinely believe that the development of AI will have good effects and that significant harms are unlikely, and they want policies such as open sourcing to reflect that. 

Can you go into more detail about this? Utilitarians and other people with logically/intellectually precise worldviews seem to be pretty consistently against human extinction; whereas average people with foggy worldviews tend to randomly flip in various directions depending on what hot takes they've recently read.

Foggy worldviews tend to flip people around based on raw emotions, tribalism, nationalism, etc. None of these are likely to get you to the position "I should implement a long term machievellian scheme to kill every human being on the planet". The obvious point being that "every human on the planet" includes ones family, friends, and country, so almost anyone operating on emotions will not pursue such a goal. 

On the other hand, utilitarian math can get to "kill all humans" in several ways, just by messing around with different assumptions and factual beliefs. Of course, I don't agree with those calculations, but someone else might. If we convince everyone on earth that the correct thing to do is "follow the math", or "shut up and calculate", then some subset of them will have the wrong assumptions, or incorrect beliefs, or just mess up the math, and conclude that they have a moral obligation to kill everyone. 

One interesting implication of this theory is that the spread of strict utilitarian philosophies would be a contributing factor to existential risk. The more people are willing to bite utilitarian bullets, the more likely it is that one will bite the "kill everyone" bullet. 

This would make the EA movement potentially existentially dangerous. Even if we don't agree with the human extinction radicals, people might split off from the movement and end up supporting it. One interpretation of the FTX affair was that it was a case of seemingly EA aligned people splitting off to do unethical things justified by utilitarian math.  

This is very vague. What other threats? It seems like a virus wiping out most of humanity would decrease the likelihood of other threats. It would put an end to climate change, reduce the motivation for nuclear attacks and ability to maintain a nuclear arsenal, reduce the likelihood of people developing AGI, etc.  

A virus that killed everyone except humans in extremely remote locations might well destroy humanity’s long-term potential

How?

I see it delaying things while the numbers recover, but it's not like humans will suddenly become unable to learn to read. Why would humanity not simply pick itself up and recover? 

The list of achievements in the post contains items like "played a big part in creating the YIMBY movement" (A bit of an overstatement imo), and "sparked a renaissance in forecasting" (made a bunch of fun online prediction markets). To be clear, I like both these things! 

But if you are going to include relatively minor things like this in the "pros" column, it's disingenuous to leave out things like " accidently created a cult" out of the "cons" column (plus the worse stuff I'm trying to avoid mentioning directly here). 

Either we can list all the benefits and all the harms caused by EA, or we can just list the very seriously good stuff and the very seriously bad stuff. If you list all of the benefits, and only the super mega bad harms, you aren't making an honest argument. Although I wouldn't have even had a problem if he'd just stated upfront that he was writing an unbalanced defence. 

I'm allowing for the possibility that we hit another AI winter, and the new powerful technology just doesn't arrive in our lifetime. Or that the technology is powerful for some things, but remains too unreliable for use in life-critical situations and is kept out of them. 

I think it's likely that AI will have at least an order or magnitude or two greater body count than it has now, but I don't know how high it will be. 

Hey, welcome to the EA forum! I hope you stick around. 

I pretty much agree with this post. The argument put forward by AI risk doomers is generally flimsy and weak, with core weaknesses involving unrealistic assumptions about what AGI would actually be capable of, given limitations of computational complexity and the physical difficulty of technological advancements, and also a lack of justification for assuming AI will be fanatical utility function maximisers. I think the chances of human extinction from AI are extremely low, and that estimates around here are inflated by subtle groupthink, poor probabilistic treatment of speculative events, and a few just straight up wrong ideas that were made up a long time ago and not updated sufficiently for the latest events in AI. 

That being said, AI advancements could have a significant effect on the world. I think it's fairly likely that if AI is misused, there may be a body count, perhaps a significant one. I don't think it's a bad idea to be proactive and think ahead about how to manage the risks involved. There is a middle ground between no regulation and bombing data centers. 

This might be stating the obvious, but this article is not a balanced accounting of the positive and negative effects of the effective altruism movement. It's purely a list of "the good stuff", with only FTX and the openAI mess being mentioned as "the bad stuff". 

For example, the article leaves out EA's part in helping get openAI off the ground, which many in AI think is a big mistake, and I believe has caused a notable amount of real world harm already. 

It also leaves out the unhealthy cultish experiences at leverage, the alleged abuse of power at nonlinear, and the various miniature cults of personality that lead to extremely serious bad outcomes, as well as the recent scandals over sexual harrassment and racism. 

It's also worth pointing out that in a counterfactual world without EA, a lot of people would still be donating and doing good work. Perhaps a pure Givewellian movement would have formed, focusing on evidence based global health alone, without the extreme utilitarianism and other weird stuff, and would have saved even more lives. 

This is not to say that EA has not been overall good for the world. I think EA has done a lot of good, and we should be proud of our achievements.  But EA is not as good as it could be, and fixing that starts with honest and good-faith critique of it's flaws. I'll admit you won't find a lot of that on twitter though.