Yeah, that's an interesting challenge. One idea that makes sense to me is to split into different sub-agents when you feel like there's an in-principle tension or tradeoff between different kinds of values. For example, in a previous post I suggested different "buckets" for the broad goals of (i) pure suffering reduction, (ii) reliable global capacity growth,  and (iii) high-impact long-shots. Lots of different concrete "causes" might compete within each philosophical bucket, but I wouldn't be tempted to break out a fourth bucket unless I felt like there was a deeper principle at stake (e.g. if I thought that human vs animal pure suffering reduction might be meaningfully different).

(There's a separate question of how to decide what to investigate within each bucket, which is also tricky and I'm afraid I don't have anything helpful to add there. In practice I mostly just defer to people I at least vaguely trust who seem to have looked into the area in greater depth. But as a community, we obviously benefit from people who are willing to do the hard work of novel investigations!)

What are the central virtues a committed utilitarian should cultivate? I would kill for a really great answer to this question

I think Stefan Schubert & Lucius Caviola's Virtues for Real-World Utilitarians is pretty good!

I'm happy to see someone raising these challenges (and it's plausible to me that "whether EA is functioning well as a truth-seeking community deserves more attention than it’s getting"), but fwiw I remain glad that CEA in particular are leading from a position of greater confidence in EA.

I agree that "Growth is only good if EA is functioning well," but note also that missed growth would be extremely bad if EA is functioning well, so it's hardly costless to forsake potential growth. My personal view is optimistic: it seems like EA does a lot of good, and more people learning about the ideas and getting involved (perhaps taking the 🔶 10% pledge, or identifying and working on a neglected high-impact problem) would also be good. But if I'm wrong about that, I hope others would be able to bring to my attention the reasons why. So I certainly agree that maintaining a good epistemic environment and truth-seeking culture is also important!

Not really. Rule consequentialism as an ethical theory has various (implausible) implications about which acts are objectively right or wrong, that can be assessed independently of the question of whether it would be a good thing for (some, many, or all) people to use a rule-based decision procedure.

Ok, thanks for expanding upon your view! It sounds broadly akin to how I'm inclined to address Pascal's Mugging cases (treat the astronomical stakes as implying proportionately negligible probability). Astronomical stakes from x-risk mitigation seems much more substantively credible to me, but I don't have much to add at this point if you don't share that substantive judgment!

You keep describing possible scenarios in which the actual value of averting extinction would be low. Do you not understand the scenario conditions under which the actual value would be high? I'm suggesting that those kinds of scenario (in which astronomically valuable futures are reliably securable so long as we avoid extinction this century) seem reasonably credible to me, which is what grounds my background belief in x-risk mitigation having high expected value.

Edited to add: You may in a sense be talking about "expected value", but insofar as it is based on a single model of how you expect the future to go ("rapid diminution"), it seems problematically analogous to actual value — like the fallacy I diagnose in 'Rule High Stakes In, Not Out'. We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.

The point of RHSINO is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldn't establish that our all-models-considered EV must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/scenarios.

I really want to stress the difference between saying something will (definitely) happen versus saying there's a credible chance (>1%) that it will happen. They're very different claims!

Lovelace and Menabrea probably should have regarded their time as disproportionately likely (compared to arbitrary decades) to see continued rapid progress. That's compatible with thinking it overwhelmingly likely (~99%) that they'd soon hit a hurdle.

As a heuristic, ask: if one were, at the end of history, to plot the 100 (or even just the 50) greatest breakthrough periods in computer science prior to ASI (of course there could always be more breakthroughs after that), should we expect our current period to make the cut? I think it would be incredible to deny it.

Thanks for explaining your view! 

On the first point: I think we should view ASI as disproportionately likely in decades that already feature (i) recent extraordinary progress in AI capabilities that surprises almost everyone, and (ii) a fair number of experts in the field who appear to take seriously the possibility that continued progress in this vein could soon result in ASI.

I'd then think we should view it as disproportionately unlikely that ASI will either be (a) achieved before any such initial signs of impressive progress, OR (b) achieved centuries after such initial progress. (If not achieved within a century, I'd think it more likely to be outright unachievable.)

I don't really know enough about AI safety research to comment on the latter disagreement. I'm curious to hear others' views.

I'm a bit confused by this response. Are you just saying that high expected value is not sufficient for actual value because we might get unlucky?

Some possible extinction-averting events merely extend life for 1 second, and so provide little value. Other possibilities offer far greater extensions. Obviously the latter possibilities are the ones that ground high expected value estimates.

I think Shulman's points give us reason to think there's a non-negligible chance of averting extinction (extending civilization) for a long time. Pointing out that other possibilities are also possible doesn't undermine this claim.

Distinguish pro tanto vs all-things-considered (or "net") high stakes. The statement is literally true of pro tanto high stakes: the 1% chance of extremely high stakes is by itself, as far as it goes, an expected high stake. But it's possible that this high stake might be outweighed or cancelled out by other sufficiently high stakes among the remaining probability space (hence the subsequent parenthetical about "unless one inverts the high stakes in a way that cancels out...").

The general lesson of my post is that saying "there's a 99% chance there's nothing to see here" has surprisingly little influence on the overall expected value. You can't show the expected stakes are low by showing that it's extremely likely that the actual stakes are low. You have to focus on the higher-stakes portions of probability space, even if small (but non-negligible).