This competition entry has been selected for publication by the Forum team.

Credence: Likely.^[1]

Anthony DiGiovanni’s sequence, The challenge of unawareness for impartial altruist action guidance (henceforth “the sequence”), argues that impartial altruists have no non-arbitrary reason to prefer any decision over any other. The central problem is unawareness: “many possible consequences of our actions haven’t even occurred to us in much detail, if at all.”

I disagree. Impartial altruists should act as if we have precise probabilities for beliefs and precise expected utilities of decisions, even if we (usually) do not explicitly state them. It’s okay to act on heuristics, but those heuristics are in service of maximizing the expected value of a (non-literal) utility function, rather than being “terminal” heuristics.

“Act as if” means: either I give an explicit precise credence; or I don’t, but I operate under the assumption that I could give a precise credence if I spent enough time reasoning through my beliefs.

This post starts by explaining the sequence’s central arguments in my own words. Then it offers five defenses of acting on your best guess.

(My interpretation of) the central arguments for suspending judgment

“We should suspend judgment” doesn’t mean “we should be indifferent between all possible actions.” Rather, it means “we have no compelling reason to prefer any action over any other, nor do we have any compelling reason to be indifferent between them.”

This section describes the key premises in support of suspending judgment as I understand them, and as they bear on my response. I label them P.M (M for “Michael’s interpretation”) to distinguish them from the numbered premises DiGiovanni gives.

• P.M1. The formal framework of expected utility maximization assumes the agent has knowledge of the full possibility set and assigns a subjective probability to each possibility. In real life, we are not aware of all possibilities. Given unawareness, there is no known canonical method for assigning probabilities to beliefs, or assigning expected utilities to actions.
• P.M2. “The worry about assigning precise EVs (with respect to impartial values) isn’t that it’s difficult, but that it’s arbitrary: we have no reason to pick one precise EV over many others.”^[2]
• P.M3. This is specifically a worry for impartial^[3] altruists: unawareness about the far future is sufficiently great that different arbitrary choices (of precise probabilities) render different verdicts. Whereas for local decision-making, unawareness is small enough that any choice within the “reasonable range” renders the same verdict.

I disagree with P.M2 and P.M3 and will argue against both.

In terms of the sequence’s logical argument^[4], I dispute premises P2b^[5] and P3^[6]:

P2. Conceptual premise: If our understanding of A’s and B’s possible consequences is sufficiently coarse-grained, then we don’t have an argument for “expecting” our idealized self’s EV for A to be higher, lower, or equal to B’s. So A’s and B’s “EVs” are incomparable. In particular:

1. Against precise EVs: We shouldn’t represent actions’ degrees of c-preferability with literal precise expected values.
2. Against “best guesses”: Even if we don’t use literal precise EVs, we shouldn’t always force ourselves to compare A’s and B’s “EVs”.

P3. Empirical premise: Due to unawareness (at least), our understanding of any pair of actions’ possible consequences is indeed very coarse-grained — enough that the conclusion of (P2) follows (i.e., these actions’ “EVs” are incomparable). In particular, the actions’ “EVs” are too severely imprecise to compare them, regardless of whether we (a) formally model these “EVs” or (b) appeal to informal/heuristic arguments.

However, this post will not give much direct attention to P2b or P3. When I first read the premises at the start of the sequence, they gave me little clue what arguments the sequence would make; then, after reading the sequence, I had a hard time matching up my mental model of its arguments with the listed premises. Therefore, I will argue against my own interpretation of DiGiovanni’s arguments. This approach should better illuminate any misunderstandings.

My response will not address imprecise probabilities. The unawareness sequence introduces a dilemma: either you choose precise probabilities (which the sequence says are arbitrary^[7]), or you use imprecise probabilities, which the sequence models as probability ranges^[8]. The sequence argues that imprecise probabilities make expected utilities indeterminate, and that various proposed solutions fail. I set those aside, because I prefer to use precise probabilities (explicitly or implicitly).^[9]

Five defenses of acting on your best guess

1. Unawareness is a problem for everyone—there is no “general theory of decision-making”. If we can make ordinary decisions under unawareness, then we can also make impartial-altruistic decisions about the far future. (This replies to P.M3 and P2b.)
2. You can have good reasons for a belief or decision, even when you can’t figure out what those reasons are. People’s intuitions often outpace our understanding of how to formally justify a belief. We see concrete examples of this phenomenon in decision theory and mathematics. (This replies to P.M2 and P2b.)
3. The problem of reasoning-under-unawareness may be formally solvable, a la logical induction^[10]. (This replies to P.M1, P.M2, and P2b.)
4. Forecasting difficulty is a continuum. The far future is not a special un-forecastable category; it faces more extreme versions of the challenges ordinary geopolitical forecasting already encounters. (This replies to P.M2, P2b, and P3.)
5. Rather than treating unknown unknowns as indeterminate, it is reasonable to assume that their EVs cancel out in expectation. (This replies to P.M2 and P2b.)

Unawareness is a problem for everyone

The unawareness sequence proves too much. The problems it raises do not only apply to impartial altruists; they apply equally well to all rational agents.^[11]

If we are comfortable (enough) to make decisions in ordinary situations in spite of these problems, then we can also be comfortable (enough) when considering our impacts on the far future.

When making local decisions, or when pursuing short-term goals, the problems of unawareness still come into play. Standard expected utility theory assumes complete knowledge of possibility-space; we cannot literally apply it if we have any unawareness whatsoever, which we always do.

More importantly, we cannot put strict bounds on the scope of unawareness—even for situations we can model well—because unawareness lives outside our models. We can argue that the far future involves greater unawareness than short-term decisions, but making that sort of argument requires the same moves that we use to justify estimating EVs into the far future. If the latter is untenable, then so is the former.

In section 2.2, DiGiovanni argues that unawareness is not a problem for local decision-making:

We don’t worry that eating a sandwich could catastrophically backfire on us. So how do we non-arbitrarily bound the probability of something like, “I’m missing a hypothesis that implies this sandwich will kill me”?

He gives two reasons for bounding the probability. I will argue against both.

Knowledge of mechanisms. ... When you bite into a sandwich, you understand the relevant physical systems well enough to precisely predict that it almost certainly won’t kill you. It would be arbitrary to posit hidden mechanisms that imply drastic deviations from this prediction.

I know nothing about the mechanisms I’m unaware of—the unawareness lives outside my model of mechanisms.^[12] To justify my belief that the sandwich won’t kill me, I have to fall back to inductive evidence.

Speaking of which, here’s DiGiovanni’s second reason:

Inductive evidence of success. Second, even when we don’t understand a mundane problem’s mechanisms, we can non-arbitrarily^[13] extrapolate from historical successes if there’s no particular reason to think the given problem is disanalogous.

I’ve never experienced the far future, which makes it hard to apply inductive evidence. But I’ve never eaten this particular sandwich before, either. Applying inductive evidence requires making a leap of faith that the historical evidence is relevant to the decision at hand.

No future scenario is exactly analogous to historical evidence, nor is it perfectly disanalogous. For instance, there is much debate over which past events are most relevant to evaluating AI risk: pessimists point to the fates of less-intelligent beings in the presence of intelligent adversaries, while optimists point to the near-monotonic welfare improvements from technological progress.^[14] Neither comparison is perfect, but both provide nonzero inductive evidence.

I can apply inductive evidence to my history of sandwich-eating, and I can apply it to attempts at doing impartial altruism. Inductive evidence for the latter is weaker, but it’s still present.

Is there a principled difference between sandwiches and the far future?

From Cluelessness: Summary of the argument, why it matters, and counterarguments:

Premise P2 claims that actions are incomparable when our understanding of a decision problem is coarse-grained enough. There’s a principled threshold for “enough”, namely, when the comparison seems sensitive to arbitrary choices about how to fill in the details of our coarse-grained model.

In other words: Sandwich-eating problems are insensitive to arbitrary parameter choices. We are uncertain about how to resolve ambiguity, but that doesn’t matter because, with respect to immediate consequences, any reasonable procedure will agree that eating the sandwich has higher EV than not eating it.

(Is that actually true? Nobody has modeled out every possible set of sandwich-related parameter choices within the “reasonable range”, so we’re making this claim without having checked.)

Recall that unawareness means “many possible consequences of our actions haven’t even occurred to us in much detail, if at all.”

My reply is: How do we know that sandwich-related decisions aren’t “sensitive to arbitrary choices”?

When comparing my model of sandwich-eating to my understanding of the far future, I find that the latter has much greater model uncertainty. I have wider priors and can think of more sign-flipping considerations. But that’s not unawareness—unawareness is about what’s outside my model. How can I say that impartial-altruistic decisions are subject to more unawareness than sandwich-eating decisions? I don’t know what’s not in my model. Nor do I have any clear justification for bounding my unawareness—e.g., I don’t know how to say “there cannot be more than a 5% chance that eating a sandwich will cause an outcome that I had never considered.”

The following two claims are in the same class:

1. My sandwich-related unawareness is sufficiently bounded that it doesn’t influence my decisions.
2. I can reasonably go with my best guess when making decisions from an impartial-altruistic point of view.

In both cases, I make a leap without having a visible justification for doing so, or even a theory of how to produce such a justification.

Epistemology has problems for everyone

We have no “non-arbitrary” justification for making any decisions under any circumstances, due to various epistemological problems:

• Unknown unknowns are just that. I cannot know that I understand sandwich-eating better than I understand longtermist decision-making, because there could be some enormous sandwich-related leviathan lurking out of sight.
• Ideal Bayesian reasoning requires omniscience: assigning consistent probabilities to every proposition is computationally intractable (Good, 1950^[15]). This is true even with zero unawareness.
• For example, an ideal reasoner must assign probability 1 to Goldbach’s conjecture if true and 0 if false. Otherwise, its beliefs are logically inconsistent.
• Making predictions in general requires solving the halting problem.
• The Münchhausen trilemma: All proofs of a statement are either circular (proved by asserting that the statement itself is true), regressive (requiring an infinite chain of proofs), or dogmatic (relying on premises that cannot be defended).^[16] This remains an open problem.

Not that these problems aren’t troubling, but we don’t let them stop us from making decisions.

You can have good reasons, even when you can’t figure out what those reasons are

“I haven’t found any formal or rigorous justification for my decisions” doesn’t mean “my decisions are arbitrary”. We can act as if we have good reasons even when we can’t identify those reasons.^[17]

I do not have a formal or rigorous justification for this claim; nonetheless, I have reason to act as if it’s true.

Two pieces of supporting evidence:

1. Common-sense decision theory can solve problems that we have no rigorous account of. Formal decision theories have yet to catch up to common sense.
2. Mathematicians can intuit the truth of a conjecture before they know how to prove it.

Decision theory

Parfit’s hitchhiker^[18]:

Suppose you’re out in the desert, running out of water, and soon to die - when someone in a motor vehicle drives up next to you. Furthermore, the driver of the motor vehicle is a perfectly selfish ideal game-theoretic agent, and even further, so are you; and what’s more, the driver is Paul Ekman, who’s really, really good at reading facial microexpressions. The driver says, “Well, I’ll convey you to town if it’s in my interest to do so - so will you give me $100 from an ATM when we reach town?”

Now of course you wish you could answer “Yes”, but as an ideal game theorist yourself, you realize that, once you actually reach town, you’ll have no further motive to pay off the driver. “Yes,” you say. “You’re lying,” says the driver, and drives off leaving you to die.

Most formal decision theories^[19] have no answer to this problem—they cannot give any account of why you should pay the driver, or any account of why the driver should pick you up. But regular people have no difficulty seeing that they ought to offer to pay the driver and then follow through on their commitment. Formal decision theories get left in the desert; regular people make it out okay.

Vibe Decision Theory^[20] is a longer (and more humorous) illustration of this point. The author takes a list of dilemmas and tests three formal decision theories against them, along with “vibe decision theory” (VDT) — in which the author asks an LLM what to do. The LLM sees something that none of the formal decision theories manage to fully capture.

Mathematicians have good intuitions about conjectures

Of all possible mathematical conjectures, most are unprovable. Among the provable ones, most have proofs so complicated that no human could ever conceive of them. And yet, a majority of “interesting” conjectures are within reach. We have no proper accounting of why so many conjectures turn out to be provable, but it seems that math-in-practice is easier than math-in-theory.

Relatedly, mathematicians can often see whether a conjecture is true, even when they have no clue how to prove it. We have no explanation for how that’s possible. The obvious sources of evidence are weak:

• You can search for counterexamples, but often the first counterexample is too large to find (see: Skewes’ number).
• You can look at failed proof attempts, but you’d expect to learn little from them, because almost all true statements have proofs too long for anyone to find. (This seems related to the fact that “interesting” conjectures are surprisingly amenable to proof.)

And yet, mathematicians can perceive the truth of a statement before they can see the proof. Their brains have a power of reasoning that we do not yet understand.

For more on this puzzling state of affairs, see Gowers (2023)^[21] and DeDeo (2024)^[22].

Tying back to the challenge of unawareness

Unawareness is analogous to decision theory, or to mathematicians’ understanding of unproven conjectures. We have no formal accounting of how it’s possible to have all-things-considered beliefs about how our actions influence the future. Nonetheless, we do have such beliefs, and we can reasonably take them to be non-arbitrary—in the same way that a mathematician can make a non-arbitrary prediction about the truth of a thorny conjecture.

Forming beliefs under unawareness has a close analogy to math:

• Finding “interesting” conjectures is intractable in theory, but in practice, mathematicians do it.
• Assigning probabilities to beliefs is intractable in theory, but in practice, people do it.

The analogy to decision theory is looser, but the two have a similar smell.

Reasoning-under-unawareness may be formally solvable

Even if we don’t currently know how to express reasons for judgments, I’m optimistic that we can make progress on the problem. If the problem is solvable, then that’s reason to believe that evolution has endowed us with the capacity to implement that solution (or at least an approximation of it).

Before Bayes’ theorem was published in 1763, were people unable to update beliefs based on evidence? The lack of formal understanding of Bayesian reasoning didn’t stop people from forming correct beliefs anyway.

Logical induction^[23] attempts to describe how agents can assign probabilities to beliefs under realistic constraints (limited computation time, etc.). It remains an open problem, but researchers have made some progress. The human brain appears to implement some flavor of logical induction—probably a better flavor than any that’s been formally described to date.

Similarly, I expect that it is possible to make progress on the theoretical problem of decision-making under unawareness.

Predicting the expected impact of our actions on the far future is largely a computability problem. Can we make useful analogies to a formal computing system, and then prove theorems about it?

A basic illustration of how we might do this: Represent the universe as an N×N grid in Conway’s Game of Life. We know the contents of a smaller M×M square, and we know nothing about the contents of the cells beyond that square. Then define the “positive utility” of the universe to be the number of horizontally-oriented heavyweight spaceships, and “negative utility” as the number of vertically-oriented heavyweight spaceships. Can we make any provable claims about the expected utility in the full universe, given what we can see?

I expect it is possible to formally specify a conjecture along the lines of:

If we can only see the contents of the smaller M×M square, and all outer squares are randomly distributed, then we can predict better than chance whether the full universe will contain positive or negative expected utility.

Furthermore, I expect that the conjecture is true.

The Game-of-Life metaphor is illustrative: you could build a formal system analogous to an impartial altruist predicting the all-things-considered outcomes of their actions, and then prove theorems about that system.

Forecasting is possible

I will take as a premise that non-frequentist probabilities are reasonable^[24] in general, because that does not appear to be a crux. The relevant question is whether it’s reasonable to have Bayesian credences about how our actions affect the full scope of the cosmos.

From 2.3.1.1:

The mechanisms we’re unaware of might be qualitatively distinct from those we’re aware of. They’re not merely the minor variations we know superforecasters can handle.

In what sense can superforecasters only handle “minor” variations? Geopolitical forecasting^[25] requires making predictions about complex questions with incomplete knowledge and even incomplete awareness of the hypothesis space—qualitatively, the same problem impartial altruists face.

We can assign subjective probabilities to short-term outcomes and coherently reason about them, and some subjective probabilities are more accurate than others.

The spectrum runs from dice rolls to geopolitical forecasts to decisions about the far future. We can make coherent predictions at the right end for the same reason we can make them in the middle. We have no general theory of how to forecast geopolitical events, and no formal account of why they’re predictable at all, yet we forecast them anyway.

Geopolitical forecasts run into some of the same problems as far-future utilities. In both cases, we do not have a complete hypothesis space (something could happen that we didn’t think of), and we cannot enumerate every consideration that influences the probability of a proposition being true.

The sequence argues that EV comparisons become infeasible “when the comparison seems sensitive to arbitrary choices about how to fill in the details of our coarse-grained model”^[26]. With geopolitics, “arbitrary” changes to a forecaster’s modeling choices can alter credences in a decision-relevant way. If the arbitrariness critique defeats far-future forecasts, then it defeats geopolitical forecasts as well. Furthermore, the quoted passage permits comparisons with imprecise probabilities when imprecision is small but not when it’s large. Forecasters, by contrast, use precise credences. If probability ranges must be used, then the standard approach to forecasting becomes impossible.

The unawareness sequence denies that forecasting success should be taken as a reason not to be clueless about the far future (from Should you go with your best guess?):

First, given our calibration (if any!) in domains like geopolitical forecasting, what can we conclude about the kinds of beliefs we should have about domains we care about as effective altruists? To answer this, we need to form beliefs about how much the former transfers to the latter. And arguably these beliefs should themselves be highly indeterminate, given that we have no direct evidence that superforecasters are calibrated about the far future.

Compare:

Given superforecasters’ success at predicting events through 2025, what can we conclude about their ability to predict events in 2027? To answer this, we need to form beliefs about how much the former transfers to the latter. And arguably these beliefs should themselves be highly indeterminate, given that we have no direct evidence that superforecasters are calibrated about 2027.

This relates to the earlier argument about “inductive evidence of success”. If we accept that superforecasters’ past success in geopolitics is evidence of future success—even though next year’s geopolitical situation will be unprecedented in many ways—then we have admitted that we’re willing to use inductive evidence for situations that aren’t perfectly analogous, or that involve some unawareness. At that point, there is no barrier to having credences about the impartial-altruistic effects of decisions, even though our ability to induct is much weaker.

(To gesture at a more formal assertion: A decision has a long list of associated properties. As long as we know more than zero of those properties, we can apply inductive reasoning.^[27])

Quoting the same article again:

[T]here are some prima facie significant differences between geopolitical forecasting and the kinds of forecasts EA decision-making depends on. For example, for the latter, we need to model extremely complex and unprecedented causal pathways, like the effects of our actions on a post-ASI civilization.

Is there a qualitative difference between predicting actions’ effects on the far future versus predicting effects on complicated geopolitical outcomes? On the “predictability spectrum” from dice rolls to far-future outcomes, DiGiovanni wants to draw a line somewhere to the right of geopolitical forecasts and say we can make predictions about anything on one side of the line, but not the other. I don’t see the justification for drawing that line.

Unknown unknowns cancel out in expectation

The sequence preempts this argument in 4.1.1, “Symmetry”:

A common intuition is, “If it really is so unclear how to weigh up these updates about the catch-all, don’t the optimistic and pessimistic updates cancel out in expectation?” But our actual epistemic state is that we have reasons pointing in both directions, which don’t seem precisely balanced.

The sequence offers a thought experiment in support of this claim. From Should you go with your best guess?:

Back to the example of pausing AI. Suppose that, as one notable input into your overall beliefs, you defer a bit to the nearly 34,000 signatories of the FLI open letter supporting the pause. Nonetheless, you think the sign of pausing AI is indeterminate. But let’s say you wait a while, and the letter now has 40,000 signatures. All else equal, this means you have stronger evidence in favor of pausing AI based on deference — the “pausing AI decreases x-risk on net” view has been “mildly sweetened”. Should you now favor that view, as if you’d previously believed pausing AI is equally likely to decrease x-risk as to increase it? Arguably not. Given your previous evidence, we suppose, you ought to have no clue which effect is more likely, and a bit of evidence this small plus no clue ought to equal no clue.

I do not share this intuition. If the number of signatures grows faster than expected,^[28] then I feel like I’ve learned something.

Realistically, if my belief vacillates between the two sides when the letter has 34,000 signatures, then it will also vacillate at 40,000. But I wouldn’t model that as having an indeterminate credence. Rather, I would describe the vacillation as: when I think about the dilemma, each unit of thought updates my credence by more than the extra 6,000 signatures do.

Unlike ideal Bayesian reasoners, I can change my beliefs purely by sorting through the implications of what I already know, without learning any new facts. My credence is indeterminate in the specific sense that it may change upon reflection (even with no new evidence), but I’m reasonably satisfied to go with my credence at the moment when I decide to stop reflecting—I would not say that there is “no reason” to act on that particular credence.

This gets back to issues discussed in previous defenses, including that unawareness is a problem for everyone and reasoning-under-unawareness may be formally solvable.

Allow me to propose my own thought experiment:

Alice has two piles of heavy stuff. She’s keeping the piles somewhere I can’t see them. She asks me: “If I put my piles of weights on a balance scale, which side will be heavier?” She hasn’t told me anything about what’s in the piles, or even how big they are.

I could guess that one pile or the other might be larger. Alice is right-handed, so maybe she would naturally place more items in the right pile. But English speakers tend to think in terms of left-to-right, so maybe she would favor the left side. Neither reason seems particularly stronger than the other.

Recall DiGiovanni’s objection: our reasons point in both directions and don’t seem precisely balanced, so we should suspend judgment rather than assuming the considerations exactly cancel out.

I would be surprised if the two piles were precisely balanced. But it’s reasonable to believe that the piles are balanced in expectation. And it seems wrong to say that I cannot have a view about the expected tipping of the scale. In expectation, the scale does not tip.

Suppose Alice removes an object from the left pile and shows it to me. It’s a pebble. Now I know that the left pile is one pebble lighter. Does that change my expectation for how the scale tips?

Even though I know virtually nothing, I’m inclined to say it’s evidence that the (new) left pile is heavier.^[29] I’m not confident that that’s the correct inference, but at least I can have a view.s

Conclusion

I have reason to take whatever action maximizes EV according to my best guess; but the challenge of unawareness has not been solved. Where I diverge from Anthony DiGiovanni is that I do not consider the challenge uniquely troubling. I view it in the same category as the Münchhausen trilemma, or the impossibility of ideal Bayesian reasoning without omniscience.^[30] Nor do I consider the problem to only apply to impartial altruists—unawareness is a problem for everyone.

We can have good reasons for beliefs, even if we cannot articulate them. Forecasting is possible—forecasters express precise credences, even when those credences appear “arbitrary”. Furthermore, I expect that we can make progress on the theoretical problem of how an agent should behave under unawareness.

Ultimately, we do not have a refined theory of how human-like reasoners (as opposed to omniscient Bayesians) should make decisions. In light of this, DiGiovanni says impartial altruists must suspend judgment; I say we should make a best guess as to what maximizes expected utility, and then do that. I would like to see more progress on the abstract question of how agents should make decisions under unawareness; but until then, I’m willing to continue to operate as if—as mathematicians like to say—a solution exists.^[31]

Appendix A: The sequence’s thought experiments do not pump my intuitions

Many of the sequence’s arguments lean on thought experiments or vignettes, which depend on the reader sharing the author’s intuition. I did not find the thought experiments intuitively compelling.

Example 1: Rejecting the completeness axiom

From Should you go with your best guess? (quoting Nicolas Macé):

Say you want to tell whether working on pausing AI will decrease x-risk, or backfire.... After lots of thought, you don’t feel that the evidence and your priors (whatever those are) determinately support one side of the debate more than the other. But the cases for and against the pause also don’t seem equally plausible to you. You simply aren’t able to say whether pausing AI is more likely to decrease or increase x-risk.

I cannot imagine ever finding myself in this position. If the cases for and against don’t seem equally plausible, then it’s because one side’s case appears stronger. If the evidence doesn’t support one side or the other, then I must find them equally plausible.

The “you” in this story rejects the VNM completeness axiom. I don’t know how to make arguments for or against the completeness axiom that don’t reduce to appeals to intuition, but the anti-completeness position has no intuitive force for me.

Example 2: Sign-flipping considerations

From 1.2.1:

I want to increase total welfare across the cosmos. Seems pretty daunting! Nonetheless, per the standard longtermist analysis, I reason, “The value of the future hinges on a few simple levers that could get locked in within our lifetimes, like ‘Is ASI aligned with human values?’ And it doesn’t seem that hard to nudge near-term levers in the right direction. So it seems like x-risk reduction interventions will be robust to missing details in our world-models.”

Inspired by this logic, I set out to choose a cause area.

[long list of considerations that could flip the sign of your actions’ EV]

I concur that there are many sign-flipping considerations. This leads me to conclude that I should have wide error bars, or at worst that I’m indifferent between action and inaction. It does not give me the feeling that I’m incapable of making a judgment (not preferring one side, nor preferring the other, nor being indifferent).

The author acknowledges that one might reach this conclusion:

In the end, we could say, “Let’s shrink the EV towards zero, slap on some huge error bars, and carry on with our ‘best guess’ that it’s positive.” ... This vignette alone doesn’t show we have no reason to work on AI risk reduction.

That quote suggests that he doesn’t consider this vignette to have persuasive force. But the sequence dedicates significant text to the point. Again in 2.3.1 it raises a list of highly uncertain, potentially sign-flipping considerations (this time in the context of arguing that mechanisms are poorly understood). How much am I supposed to be moved by the existence of all these considerations? If I’m supposed to be moved, then I can report that I don’t find them to have intuitive force. If I’m not supposed to be moved, then there’s not much point in my discussing it, which is why this section is in an appendix.

1. ^{^}

   Due to the subject matter at hand, this credence is recursive: it’s likely that I can meaningfully attach a credence of “likely” to this post.

2. ^{^}

   DiGiovanni, A. (2026). Cluelessness: Summary of the argument, why it matters, and counterarguments.

3. ^{^}

   Importantly, “impartial” includes impartiality with respect to time, so that we care about actions’ impacts on the long-term future.

^{^}

DiGiovanni, A. (2026). Cluelessness: Summary of the argument, why it matters, and counterarguments.

^{^}

It would be more accurate to say that I agree with the literal statement of P2b, but I dispute the way it’s used in the argument.

P2b states, “Even if we don’t use literal precise EVs, we shouldn’t always force ourselves to compare A’s and B’s ‘EVs’.” Sure, we should not always force ourselves to make a best guess between any two actions. But we should force ourselves to reach a best guess between two actions that are decision-relevant.

For example, I have not forced myself to reach a best guess about whether it would be better for me to buy a plane ticket to Micronesia or a plane ticket to Italy, because I don’t plan on doing either of those things. I will only force myself to reach a best guess between actions I’m seriously considering. This is an implication of the fact that ideal Bayesian reasoning requires infinite computing power, so I have to make compromises for the purposes of computational tractability.

^{^}

I’m not entirely sure that I’m disputing P3 because I’m not clear on the distinction between P2 and P3. If I believe it’s possible to make nonzero-information forecasts about the far future, is that a denial of P2 or P3 or both?

^{^}

I don’t actually think it’s arbitrary. I think that if DiGiovanni’s argument goes through, then it’s arbitrary.

^{^}

DiGiovanni, A. (2025). Should you go with your best guess?: Against precise Bayesianism and related views.

^{^}

I’m okay with using probability ranges to represent something like “here is how I expect my credence might change in light of new evidence”. But I don’t think it makes sense to model agents as having fundamentally imprecise probabilities.

^{^}

Garrabrant, S., Benson-Tilsen, T., Critch, A., Soares, N., & Taylor, J. (2016). Logical induction. doi: 10.48550/arXiv.1609.03543

^{^}

Or maybe it doesn’t prove too much, and in fact we’re all making a mistake by believing that we ever have reason to choose any action over any other. But if that’s our conclusion, then it feels like something has gone wrong somewhere.

^{^}

For that matter, I don’t understand the physical mechanisms either! Human metabolic pathways are notoriously complicated.

^{^}

Supposedly, it is “arbitrary” to posit unknown mechanisms when eating a sandwich, and it is “non-arbitrary” to use induction. The sequence also uses the word “arbitrary” to describe an impartial altruist’s decision to go with a best guess in light of unawareness. I don’t understand the distinction. Why is it arbitrary to make impartial-altruistic decisions, but also arbitrary to refuse to make decisions in a sandwich-related context?

As per section 2.4 under Q4, DiGiovanni appears to have a deeper mental model of what arbitrariness means, but I failed to pick up that mental model from reading the sequence. In this LessWrong comment, he talks about “epistemic arbitrariness” vs. “formalization arbitrariness”, but again I don’t understand the difference.

^{^}

For humans, anyway; technological progress to date has plausibly been net harmful due to its effects on non-human animals.

^{^}

Good, I. J. (1950). Probability and the Weighing of Evidence.

^{^}

H/T Arepo for raising this point.

^{^}

I should note that, while there is a rich philosophical literature on what it means to have a “reason” to do something, I have read none of that literature. If I’d tried to read it, I wouldn’t have been able to finish this post on time.

^{^}

LessWrong Wiki (2022). Parfit’s Hitchhiker.

^{^}

The notable exceptions being Functional Decision Theory and its cousins.

^{^}

“L Rudolf L” (2025). VDT: a solution to decision theory.

^{^}

Gowers, T. (2023). What makes mathematicians believe unproved mathematical statements? doi: 10.67678/92a5q2vm

^{^}

DeDeo, S. (2024). Hard Proofs and Good Reasons. doi: 10.48550/arXiv.2410.18994

^{^}

Garrabrant, S., Benson-Tilsen, T., Critch, A., Soares, N., & Taylor, J. (2016). Logical induction. doi: 10.48550/arXiv.1609.03543

^{^}

Alexander, S. (2024). In Continued Defense Of Non-Frequentist Probabilities.

^{^}

There are various domains where superforecasters have demonstrated success, but for simplicity, I will take geopolitics as a representative example.

^{^}

DiGiovanni, A. (2026). Cluelessness: Summary of the argument, why it matters, and counterarguments.

^{^}

One could counter that a forecaster has to make a decision about how to weight those properties, but that’s true when making any type of forecast.

^{^}

Strictly speaking, what matters is that the growth rate exceeds expectations, not merely that the number has gone up. Seeing 40,000 signatures would be a negative update if I expected to see more than that by now. But for the sake of the illustration, let’s say 40,000 exceeds my expectations.

^{^}

My reasoning: If Alice draws an object at random from the set of objects (even if the distribution is non-uniform), then she’s more likely to draw from the left pile when the left pile is heavier. And if the left pile is heavier, then it’s probably not heavier by exactly one pebble, and therefore it’s still heavier after a pebble is removed.

Alternatively, Alice could draw an object from a random pile, in which case my reasoning would change. But if we’re making a comparison to impartial-altruistic decision-making, it makes more sense to say that Alice is drawing from the set of objects rather than the set of piles.

^{^}

Other subjects in this category include infinite ethics and the problem of interpersonal utility comparisons.

The problem of induction is an ambiguous case: modern math has made progress on formalizing inductive reasoning (Solomonoff induction, etc.), but the problem isn’t definitively solved.

^{^}

A physicist, a mathematician and an engineer stay in a hotel.

The engineer is awakened by a smell and gets up to check it. He finds a fire in the hallway, sees a nearby fire extinguisher and after extinguishing it, goes back to bed.

Later that night, the physicist gets up, again because of the smell of fire. He quickly gets up and sees the fire in the hallway. After calculating air pressure, flame temperature and humidity as well as distance to the fire and projected trajectory, he extinguishes the fire with the least amount of fluid.

Finally, the mathematician awakes, only again to find a fire in the hallway. He instantly sees the extinguisher and thinks, “A solution exists!”, and heads back into his room.

It’s an old joke, but this specific wording was taken from ProofWiki.

Show all footnotes