They know that American tech companies are legally obligated (by 'fiduciary duty' to shareholders) to prioritize quarterly profits over long-term human survival.

Fwiw I read here that there actually is not such a legal duty to prioritize profits.

Good question. I'm still relatively new to thinking about AI governance, but would guess that two puzzle pieces are 

a) broader public advocacy has not been particularly prioritized so far

• there's uncertainty what concretely to advocate for and still a lot of (perceived) need for nuance for the more concrete ideas that do exist
• there are other ways of more targeted advocacy, such as talking to policy makers directly, or talking to leaders in ML about risk concerns

b) there are not enough people working on AI governance issues to be so prepared for things

• not sure what the numbers are, but e.g. it seems like at least a few key sub-topics in AI governance rely on the work of like 1-2 extremely busy people

Also, the letter just came out. I'd not be very surprised if a few more experienced people would publish responses in which they lay out their thinking a bit, especially if the letter seems to gather a lot of attention.

I expect there will be much more public discussion on regulating AI and much more political willingness to do ambitious things about AI in the coming years when economic and cultural impacts become more apparent, so I'm spontaneously wary of investing significant reputation on something (potentially) not sufficiently well thought through. 

Also, it's not a binary of signing vs. not-signing. E.g. risk reducers can also enter the discussion caused by the letter and make constructive suggestions what will contribute more to longterm safety.

Nice post, found this pretty well written and convincing (though I already shared the bottom line, just less firmly). 

Random thoughts:

A severe extreme geopolitical tail event, such as a great power conflict between the US and China, may occur.

What type of great power conflict do you have in mind here? "Extreme tail event" makes it sound like you're thinking of a fairly large scale war, but great power conflict seems to refer to any military confrontation. E.g. I haven't at all wrapped my head around a military confrontation between China and the US over Taiwan yet, and Metaculus is at ~20% for 

Will armed conflicts between the Republic of China (Taiwan) and the People's Republic of China (PRC) lead to at least 100 deaths before 2026?

Also, I wonder if you have considered any potential craziness that happens after conditional on development of TAI before 2030. E.g. say TAI is developed in 2027, maybe the plausible set of scenarios for 2028 and 2029 include sufficiently many scenarios where we see a >50% decrease in AI funding such that I might want to increase your bottom line forecast?

Glad you're thinking about improving research and forecasting! My very quick take after only skimming your post is that it just happens too rarely that an EA forum post significantly informs specific Metaculus questions. Consequently investing in such a feature seems not sufficiently useful. (Though that might change of course if there were many more Metaculus questions that relate to topics in EA.) Maybe you could list some concrete examples for such EA forum - Metaculus connections? (Sorry if I missed you listing examples!)

Some other random ideas about forecasting on the forum:

• Display relative Brier Scores from Metaculus in EAF profiles -> incentivize forecasting, enable forum readers to roughly weigh comments by epistemic track records
• Hook up the forum to the GPT-4 API and allow forum writers to generate forecasting questions at the end of their posts.
  • Scott Alexander does this regularly and I find it useful to get a concrete bottom line of his uncertainties and concrete things he's forecasting based on the post, e.g. see here.

Relatedly, was pretty surprised by the results from this Twitter poll by Lex Friedman from yesterday:

Thanks for laying out these points! After having engaged with many people's thoughts on these issues I'm similarly unconvinced about the very unfavourable odds many people seem to assign, so I really look forward to the discussion here.

I'm particularly curious about this point, because when I think about AI risk scenarious I put quite some stock into the potential for very direct government interventions when the risks are more obvious and more clearly a near term problem:

Some decisive demonstration of danger is achieved, and AIs also help to create a successful campaign to persuade key policymakers to aggressively work toward a standards and monitoring regime. (This could be a very aggressive regime if some particular government, coalition or other actor has a lead in AI development that it can leverage into a lot of power to stop others’ AI development.) 

AI seems to me to already clearly be among the top priorities for geopolitical considerations for the US, and it seems like when this is the case the space of options is fairly unrestricted. 

Thanks for sharing this, I already knew about the phenomena and had vague thoughts this might be a significant contributor, but appreciate you spelling it out. 

I'm suspecting that the comeuppance-related behavior is not only about EA being a movement emphasizing ethical innovation, but also about EA recently gaining a lot in more visible influence and social status, e.g. via more public outreach campaigns leading to features in cover stories. My  impression is that the distribution of social status of public actors and social movements is fairly zero sum (actually probably even negative sum because of the incentives to invest in and stick to defensive PR). This, combined with the public discourse being relatively scatterbrained and not very optimized for truth-seeking leads to a lot of distorted publications that aim more at lowering the social standing of an actor than giving a clear impression of what is really going on.

I think the idea is that when you (even unintentionally) signal to others (and to yourself) that you are/want to be a more mainstream respectable institution (e.g. via having a modern respectable looking website), then this causes people from inside and outside the institution to have expectations of the institution to be more mainstream respectable. This includes things like the application of more mainstream Overton windows, which e.g. leads to people complaining about discussions on LW that touch on mainstream-taboo topics even if they seem fine from the previous LW norms.

Thanks for writing this Johannes, I really appreciate when people seriously engage with good faith criticism.