The AI bubble popping would be a strong signal that this [capabilities] optimism has been misplaced.

Are you presupposing that good practical reasoning involves (i) trying to picture the most-likely future, and then (ii) doing what would be best in that event (while ignoring other credible possibilities, no matter their higher stakes)?

It would be interesting to read a post where someone tries to explicitly argue for a general principle of ignoring credible risks in order to slightly improve most-probable outcomes. Seems like such a principle would be pretty disastrous if applied universally (e.g. to aviation safety, nuclear safety, and all kinds of insurance), but maybe there's more to be said? But it's a bit frustrating to read takes where people just seem to presuppose some such anti-precautionary principle in the background.

To be clear: I take the decision-relevant background question here to not be the binary question Is AGI imminent? but rather something more degreed, like Is there a sufficient chance of imminent AGI to warrant precautionary measures? And I don't see how the AI bubble popping would imply that answering 'Yes' to the latter was in any way unreasonable. (A bit like how you can't say an election forecaster did a bad job just because their 40% candidate won rather than the one they gave a 60% chance to. Sometimes seeing the actual outcome seems to make people worse at evaluating others' forecasts.)

Some supporters of AI Safety may overestimate the imminence of AGI. It's not clear to me how much of a problem that is? (Many people overestimate risks from climate change. That seems important to correct if it leads them to, e.g., anti-natalism, or to misallocate their resources. But if it just leads them to pollute less, then it doesn't seem so bad, and I'd be inclined to worry more about climate change denialism. Similarly, I think, for AI risk.) There are a lot more people who persist in dismissing AI risk in a way that strikes me as outrageously reckless and unreasonable, and so that seems by far the more important epistemic error to guard against?

That said, I'd like to see more people with conflicting views about AGI imminence arrange public bets on the topic. (Better calibration efforts are welcome. I'm just very dubious of the OP's apparent assumption that losing such a bet ought to trigger deep "soul-searching". It's just not that easy to resolve deep disagreements about what priors / epistemic practices are reasonable.)

Quick clarification: My target here is not so much people with radically different empirical beliefs (such that they regard vaccines as net-negative), but rather the particular form of status quo bias that I discuss in the original post.

My guess is that for relatively elite audiences (like those who read philosophy blogs), they're unlikely to feel attached to this status quo bias as part of their identity, but their default patterns of thought may lead them to (accidentally, as it were) give it more weight than it deserves. So a bit of heated rhetoric and stigmatization of the thought-pattern in question may help to better inoculate them against it.

(Just a guess though — I could be wrong!)

I think if some people are importantly right about something big, and others (esp. with more power) are importantly wrong, it's worth cheerleading getting things right even if it happens to correlate with your in-group!

Interesting post! Re: "how spotlight sizes should be chosen", I think a natural approach is to think about the relative priorities of representatives in a moral parliament. Take the meat eater problem, for example. Suppose you have some mental representatives of human interests, and some representatives of factory farmed animal interests. Then we can ask each representative: "How high a priority is it for you to get your way on whether or not to prevent this child from dying of malaria?" The human representatives will naturally see this as a very high priority—we don't have many better options for saving human lives. But the animal representatives, even if they aren't thrilled by retaining another omnivore, have more pressing priorities than trying to help animals by eliminating meat-eaters one by one. Given how incredibly cost-effective animal-focused charities can be, it will make sense for them to make the moral trade: "OK, save this life, but then let's donate more to the Animal Welfare Fund."

Of course, for spotlighting to work out well for all representatives, it's going to be important to actually follow through on supporting the (otherwise unopposed) top priorities of neglected representatives (like those for wild animal welfare). But I think the basic approach here does a decent job of capturing why it isn't intuitively appropriate to take animal interests into account when deciding whether to save a person's life. In short: insofar as we want to take animal interests into account, there are better ways to do it, that don't require creating conflict with another representative's top priorities. Avoiding such suboptimal conflict, and instead being open to moral trade, seems an important part of being a "good moral colleague".

Funnily enough, the main example that springs to mind is the excessive self-flagellation post-FTX. Many distanced themselves from the community and its optimizing norms/mindset—for understandable reasons, but ones more closely tied to "expressing" (and personal reputation management) than to actually "helping", IMO.

I'd be curious to hear if others think of further candidate examples.

I think it's more like he disagrees with you about the relative strengths of the objections and responses. (fwiw, I'm inclined to agree with him, and I don't have any personal stake in the matter.)

Any intellectual community will have (at least implicit) norms surrounding which assumptions / approaches are regarded as:

(i) presumptively correct or eligible to treat as a starting premise for further argument; this is the community "orthodoxy".

(ii) most plausibly mistaken, but reasonable enough to be worth further consideration (i.e. valued critiques, welcomed "heterodoxy")

(iii) too misguided to be worth serious engagement.

It would obviously be a problem for an intellectual community if class (ii) were too narrow. Claims like "dissent isn't welcome" imply that (ii) is non-existent: your impression is that the only categories within EA culture are (i) and (iii). If that were true, I agree it would be bad. But reasoning from the mere existence of class (iii) to negative conclusions about community epistemics is far too hasty. Any intellectual community will have some things they regard as not worth engaging with. (Classic examples include, e.g., biologists' attitudes towards theistic alternatives to Darwinian evolution, or historians' attitudes towards various conspiracy theories.)

People with different views will naturally dispute which of these three categories any given contribution ideally ought to fall into. People don't tend to regard their own contributions as lacking intellectual worth, so if they experience a lack of engagement it's very tempting to leap to the conclusion that others must be dogmatically dismissing them. Sometimes they're right! But not always. So it's worth being aware of the "outside view" that (a) some contributions may be reasonably ignored, and (b) anyone on the receiving end of this will subjectively experience it just as the OP describes, as seeming like dogmatic/unreasonable dismissal.

Given the unreliability of personal subjective impressions on this issue, it's an interesting question what more-reliable evidence one could look for to try to determine whether any given instance of non-engagement (and/or wider community patterns of dis/engagement) is objectively reasonable or not. Seems like quite a tricky issue in social epistemology!

Thanks for looking into it more!

I'm not seeing the barrier to Person A's thinking there's a 1/1000 chance, conditional on reaching the 50th century, of going extinct in that century. We could easily expect to survive 50 centuries at that rate, and then have the risk consistently decay (halving each century, or something like that) beyond that point, right?

If you instead mean to invoke, say, the 50 millionth century, then I'd think it's crazy on its face to suddenly expect a 1/1000 chance of extinction after surviving so long. That would no longer "seem, on the face of it, credible".

Am I missing something?