I mean that we have what I call "coarse awareness" here: we conceive of crude groups of possible worlds, rather than possible worlds specified in fine-grained enough detail to assign them precise values (wrt impartial altruist axiologies). See also here for some examples. Happy to unpack more if those sections don't answer things!

Good question! I think "other theoretically possible aggregations of all or most of the possible consequences of A and B" would also suffice, yeah. (Of course, if we ourselves can't specify what this alternative is, we have our work cut out for us if we're gonna argue that we should expect our idealized self to prefer A over B on this basis.)

Interesting, that's helpful to know.

Not a comprehensive reply, but: I think many of the examples you're talking about are arguably cases of coarse awareness. People were coarsely aware of the potential backfire risks earlier on, but (arguably) the reason they didn't give these risks enough weight was that they didn't have a more fine-grained awareness of the specific causal pathways. I think such cases count as evidence for the pessimistic induction.

Thanks Vasco. I've summarized my reply on LessWrong here (figured that this might be of (more?) interest to LW readers).

I'd guess we're getting slightly better, yep. I might put less weight on the evidence you mention, than on: "We're living in a period of really unprecedented AI progress, seems like that puts in a better position to reason about the mechanisms governing the far future than ever before."

I think it's mostly (1), but I'm open to something like (2) or (3) as well.

(Following (1):) There is in principle some (a) amount of information that non-ideal agents could attain about the cosmos with non-Pascalian probability,^[1] + (b) a priori modeling and induction we could apply to that information, such that we wouldn't be clueless. So I don't think we need to observe the target variable, or empirically "validate" the theory, to be non-clueless.

But the bar to achieve such an (a)+(b) seems very high, because:

• If we do try to empirically validate the theory by appealing to calibration on near-term proxies:
  • I indeed don't see why we should expect such calibration to transfer, up to the degree of precision we need to escape cluelessness (sec. 2.3.1.1). This bites even if, say, we use AI to get much more calibrated on ~years-long time horizons.
• If we don't, and instead try to argue conceptually that the theory captures enough of the relevant considerations in fine-grained enough detail:
  • The web of factors this theory would have to capture seems ludicrously complex (the rest of sec. 2.3). Of course, good theories can compress complexity, but getting that amount of compression while keeping things computationally tractable^[2] sounds rough.

So my suspicion is that yeah, we'd still be clueless given the kind of theory you mention. But I find it hard to say, because I can't imagine exactly what "comparably good" looks like, concretely. I appreciate that that's hard to spell out on your end.

Maybe sufficiently advanced AI could get around this. Maybe not, e.g. if "the universal prior" is irreducibly imprecise, or if (following (3)) information about simulators or causally disconnected worlds is fundamentally inaccessible.

(I'm happy to unpack any of this more if useful, not sure if I answered your question properly!)

1. ^{^}

   As in, if I were to represent this probability numerically, the interval wouldn't all be less than the Pascalian threshold.

2. ^{^}

   Like, something analogous to the Schrödinger equation doesn't count. :)

Hi Ben, I like the spirit of this question, though I'm not sure it's the most relevant formulation. Thoughts on that, before I answer your literal question:

• To get clear on terms:
  • I'm assuming by "CI" here, you mean something like your precise 90% (or whatever) confidence interval for your idealized self's EV of the intervention (as per Premise 1).
  • By "robust", I don't mean a narrow / strictly positive CI. I mean that the verdict "this intervention is positive 'in expectation'" isn't sensitive to arbitrary choices about how to factor in the considerations we're unaware of.
    • I think it's fair to say most people involved in early AI risk advocacy considered their work robustly positive in that sense.
• This definition of "robust" is what matters for Premise 3. Because P3 says, our verdicts about interventions having positive "EV" are sensitive to such arbitrary choices — even if we admit we're very uncertain (i.e. we have wide CIs).
• So, if we're asking whether a given "sign flip" counts as evidence for P3, I don't see why the bar should be "narrow CIs with opposite-sign center points before and after the consideration".
  • If we've discovered a consideration that flipped us from "wide CI centered at a positive EV" to "wide CI centered at a negative EV", isn't that some evidence that our initial "positive in expectation" verdict wasn't robust, in the sense above? (And hence inductive evidence that our current "positive in expectation" verdicts aren't robust (i.e., evidence for P3), as argued here.)

Anyway, I'd agree that the clearest evidence for P3 would come from sign flips that meet your bar. Maybe the small animal replacement problem? I'd guess lots of people who care about animal welfare thought that getting people to eat less beef was clearly good before being aware of SARP, and think it's clearly net-bad after being aware of SARP. (It's harder to come by examples of sign flips by your def for longtermist causes, because non-clueless longtermists typically agree that we should be very uncertain about the far future. But per the above, this is to be expected if we're clueless.)

Hi Dan/Fable, thanks for the critique! The three most important problems I see:

1. Claiming that the unawareness argument only has practical implications if there's a privileged “default”

The sequence argues carefully for incomparability. It cannot argue that incomparability favors inaction, because if A and the status quo are incomparable, the status quo is not better. Yet the practical gloss everyone puts on the conclusion resolves every incomparability toward the default

As discussed here and in the introduction of the sequence, my claim was never “we should default to inaction”. It’s that we have no impartial altruistic reason to favor any intervention over any other option, including “inaction”.

Your response to this is that if we don't favor a default under incomparability, “the argument has almost no practical bite”. But we can have reasons for choices other than impartial altruistic ones — see here.

(This point is upstream of one of your replies to objection #2: “so everything becomes incomparable with everything, the argument again supplies no reason to resolve toward a default”.)

2. Conflating two kinds of awareness growth

In your response to the objection “The considerations that matter most may be ones no engagement reveals”, you say that the evidence for P3 is “We’ve become aware of new considerations through active engagement.” But “becoming aware of new considerations” is a much lower bar than “becoming aware of all the considerations that the sign of an action’s ‘EV’ is sensitive to”. You need the latter for your critique via Model 3 to work. This is important for the following:

3. Neglecting unawareness about/imprecision in the time horizon 

Your argument in Model 3 seems to be: Suppose you have T time steps to (1) actively grow your awareness by repeated exploration and then (2) exploit the strategy that does best w.r.t. your credences over this fleshed-out awareness set — before you die. Then (1)+(2) would beat “stick to the familiar domain”. I’m happy to grant here^[1] that this is true for some T.

But we don’t know T.^[2] If our beliefs about T are imprecise enough, we can’t say whether (a) the benefits of eventually cashing in on our grown awareness outweigh (b) the potential backfire effects of actions we take to grow our awareness. To meet the bar of awareness growth noted in (2) above, T might need to be very large indeed.

1. ^{^}

   This is just for the sake of argument. I think the model is importantly unrealistic in some ways I don't cover here for lack of time.

2. ^{^}

   Suppose we instead say “Conditional on living forever, the upsides are unbounded, so the utility from this case swamps all the finite-T cases.” One problem is that if we’re going to allow for unbounded upsides from an infinite T, we should also consider unbounded downsides. (You acknowledge this possibility in objection #2, but what matters is whether your critique via Model 3 actually works, not what the sequence as written says.)

Hmm yeah maybe I shouldn't have fully endorsed Ben's summary. I think many forms of bracketing are impartial in the sense that they don't arbitrarily favor some moral patients over others. But the forms of bracketing I'm aware of are either (1) not "impartial" in the sense that some moral patients/consequences are bracketed out for not very well-motivated reasons, or (2) not action-guiding.

That latter definition of impartial might be confusing though, so in general I'd just list my specific dissatisfactions with different forms of bracketing, "impartiality" aside.

Hey Ben — your understanding is correct. Option 2 is meant to allow for the possibility that the inference from P1-P3 to Conclusion, as stated in my summary post, is logically invalid. (Of course I think that's unlikely, but philosophy can be subtle.) Does that clear things up?