Ok, thanks for expanding upon your view! It sounds broadly akin to how I'm inclined to address Pascal's Mugging cases (treat the astronomical stakes as implying proportionately negligible probability). Astronomical stakes from x-risk mitigation seems much more substantively credible to me, but I don't have much to add at this point if you don't share that substantive judgment!

You keep describing possible scenarios in which the actual value of averting extinction would be low. Do you not understand the scenario conditions under which the actual value would be high? I'm suggesting that those kinds of scenario (in which astronomically valuable futures are reliably securable so long as we avoid extinction this century) seem reasonably credible to me, which is what grounds my background belief in x-risk mitigation having high expected value.

Edited to add: You may in a sense be talking about "expected value", but insofar as it is based on a single model of how you expect the future to go ("rapid diminution"), it seems problematically analogous to actual value — like the fallacy I diagnose in 'Rule High Stakes In, Not Out'. We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.

The point of RHSINO is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldn't establish that our all-models-considered EV must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/scenarios.

I really want to stress the difference between saying something will (definitely) happen versus saying there's a credible chance (>1%) that it will happen. They're very different claims!

Lovelace and Menabrea probably should have regarded their time as disproportionately likely (compared to arbitrary decades) to see continued rapid progress. That's compatible with thinking it overwhelmingly likely (~99%) that they'd soon hit a hurdle.

As a heuristic, ask: if one were, at the end of history, to plot the 100 (or even just the 50) greatest breakthrough periods in computer science prior to ASI (of course there could always be more breakthroughs after that), should we expect our current period to make the cut? I think it would be incredible to deny it.

Thanks for explaining your view! 

On the first point: I think we should view ASI as disproportionately likely in decades that already feature (i) recent extraordinary progress in AI capabilities that surprises almost everyone, and (ii) a fair number of experts in the field who appear to take seriously the possibility that continued progress in this vein could soon result in ASI.

I'd then think we should view it as disproportionately unlikely that ASI will either be (a) achieved before any such initial signs of impressive progress, OR (b) achieved centuries after such initial progress. (If not achieved within a century, I'd think it more likely to be outright unachievable.)

I don't really know enough about AI safety research to comment on the latter disagreement. I'm curious to hear others' views.

I'm a bit confused by this response. Are you just saying that high expected value is not sufficient for actual value because we might get unlucky?

Some possible extinction-averting events merely extend life for 1 second, and so provide little value. Other possibilities offer far greater extensions. Obviously the latter possibilities are the ones that ground high expected value estimates.

I think Shulman's points give us reason to think there's a non-negligible chance of averting extinction (extending civilization) for a long time. Pointing out that other possibilities are also possible doesn't undermine this claim.

Distinguish pro tanto vs all-things-considered (or "net") high stakes. The statement is literally true of pro tanto high stakes: the 1% chance of extremely high stakes is by itself, as far as it goes, an expected high stake. But it's possible that this high stake might be outweighed or cancelled out by other sufficiently high stakes among the remaining probability space (hence the subsequent parenthetical about "unless one inverts the high stakes in a way that cancels out...").

The general lesson of my post is that saying "there's a 99% chance there's nothing to see here" has surprisingly little influence on the overall expected value. You can't show the expected stakes are low by showing that it's extremely likely that the actual stakes are low. You have to focus on the higher-stakes portions of probability space, even if small (but non-negligible).

The responses to my comment have provided a real object lesson to me about how a rough throwaway remark (in this case: my attempt to very briefly indicate what my other post was about) can badly distract readers from one's actual point! Perhaps I would have done better to entirely leave out any positive attempt to here describe the content of my other post, and merely offer the negative claim that it wasn't about asserting specific probabilities.

My brief characterization was not especially well optimized for conveying the complex dialectic in the other post. Nor was it asserting that my conclusion was logically unassailable. I keep saying that if anyone wants to engage with my old post, I'd prefer that they did so in the comments to that post—ensuring that they engage with the real post rather than the inadequate summary I gave here. My ultra-brief summary is not an adequate substitute, and was never intended to be engaged with as such.

On the substantive point: Of course, ideally one would like to be able to "model the entire space of possibilities". But as finite creatures, we need heuristics. If you think my other post was offering a bad heuristic for approximating EV, I'm happy to discuss that more over there.

On (what I take to be) the key substantive claim of the post:

I think that nontrivial probability assignments to strong and antecedently implausible claims should be supported by extensive argument rather than manufactured probabilities.

There seems room for people to disagree on priors about which claims are "strong and antecedently implausible". For example, I think Carl Shulman offers a reasonably plausible case for existential stability if we survive the next few centuries. By contrast, I find a lot of David's apparent assumptions about which propositions warrant negligible credence to be extremely strong and antecedently implausible. As I wrote in x-risk agnosticism:

David Thorstad seems to assume that interstellar colonization could not possibly happen within the next two millennia. This strikes me as a massive failure to properly account for model uncertainty. I can’t imagine being so confident about our technological limitations even a few centuries from now, let alone millennia. He also holds the suggestion that superintelligent AI might radically improve safety to be “gag-inducingly counterintuitive”, which again just seems a failure of imagination. You don’t have to find it the most likely possibility in order to appreciate the possibility as worth including in your range of models.

I think it's important to recognize that reasonable people can disagree about what they find antecedently plausible or implausible, and to what extent. (Also: some events—like your home burning down in a fire—may be "implausible" in the sense that you don't regard them as outright likely to happen, while still regarding them as sufficiently probable as to be worth insuring against.)

Such disagreements may be hard to resolve. One can't simply assume that one's own priors are objectively justified by default whereas one's interlocutor is necessarily unjustified by default until "supported by extensive argument". That's just stacking the deck.

I think a healthier dialectical approach involves stepping back to more neutral ground, and recognizing that if you want to persuade someone who disagrees with you, you will need to offer them some argument to change their mind. Of course, it's fine to just report one's difference in view. But insisting, "You must agree with my priors unless you can provide extensive argument to support a different view, otherwise I'll accuse you of bad epistemics!" is not really a reasonable dialectical stance.

If the suggestion is instead that one shouldn't attempt to assign probabilities at all then I think this gets into the problems I explore in Good Judgment with Numbers and (especially) Refusing to Quantify is Refusing to Think, that it effectively implies giving zero weight. But we can often be in a position to know that a non-zero (and indeed non-trivially positive) estimate is better than zero, even if we can't be highly confident of precisely what the ideal estimate would be.

Hi David, I'm afraid you might have gotten caught up in a tangent here! The main point of my comment was that your post criticizes me on the basis of a misrepresentation. You claim that my "primary argumentative move is to assign nontrivial probabilities without substantial new evidence," but actually that's false. That's just not what my blog post was about.

In retrospect, I think my attempt to briefly summarize what my post was about was too breezy, and misled many into thinking that its point was trivial. But it really isn't. (In fact, I'd say that my core point there about taking higher-order uncertainty into account is far more substantial and widely neglected than the "naming game" fallacy that you discuss in the present post!) I mention in another comment how it applied to Schwitzgebel's "negligibility argument" against longtermism, for example, where he very explicitly relies on a single constant probability model in order to make his case. Failing to adequately take model uncertainty into account is a subtle and easily-overlooked mistake!

A lot of your comment here seems to misunderstand my criticism of your earlier paper. I'm not objecting that you failed to share your personal probabilities. I'm objecting that your paper gives the impression that longtermism is undermined so long as the time of perils hypothesis is judged to be likely false. But actually the key question is whether its probability is negligible. Your paper fails to make clear what the key question to assess is, and the point of my 'Rule High Stakes In' post is to explain why it's really the question of negligibility that matters.

To keep discussions clean and clear, I'd prefer to continue discussion of my other post over on that post rather than here. Again, my objection to this post is simply that it misrepresented me.

It's not a psychological question. I wrote a blog post offering a philosophical critique of some published academic papers that, it seemed to me, involved an interesting and important error of reasoning. Anyone who thinks my critique goes awry is welcome to comment on it there. But whether my philosophical critique is ultimately correct or not, I don't think that the attempt is aptly described as "personal insult", "ridiculous on [its] face", or "corrosive to productive, charitable discussion". It's literally just doing philosophy.

I'd like it if people read my linked post before passing judgment on it.