How x-risk projects are different from startups

I don't think risk of this type is given too much weight now. In my model, considerations like this got at some point in the past rounded of to some over-simplified meme like "do not start projects, they fail and it is dangerous". This is wrong and led to some counterfactual value getting lost.

This was to some extent reaction to the previous mood, which was more like "bring in new people; seed groups; start projects; grow everything". Which was also problematic.

In my view we are looking at something like pendulum swings, where we were somewhere at the extreme position of not many projects started recently, but the momentum is in direction of more projects, and the second derivative is high. So I expect many projects will actually get started. In such situation the important thing is to start good projects, and avoid anti-unicorns.

IMO the risk was maybe given too much weight before, but is given too little weight now, by many people. Just look at many of the recent discussions, where security mindset seem rare, and many want to move fast forward.

Discussing specific examples seems very tricky - I can probably come up with a list of maybe 10 projects or actions which come with large downside/risks, but I would expect listing them would not be that useful and can cause controversy.

Few hypothetical examples

• influencing mayor international regulatory organisation in a way leading to creating some sort of "AI safety certification" in a situation where we don’t have the basic research yet, creating false sense of security/fake sense of understanding
• creating a highly distorted version of effective altruism in a mayor country e.g. by bad public outreach
• coordinating effective altruism community in a way which leads to increased tension and possibly splits in the community
• producing and releasing some infohazard research
• influencing important players in AI or AI safety in a harmful leveraged way, e.g. by bad strategic advice

A few examples are mentioned in the resources linked above. The most well-known and commonly accepted one is Intentional Insights, but I think there are quite a few more.

I generally prefer not to make negative public statements about well-intentioned EA projects. I think this is probably the reason why the examples might not be salient to everyone.