A simple impossibility result about rational decisionmaking under unawareness

This competition entry has been selected for publication by the Forum team.

The biggest mistake made in the sequence on unawareness is to insist on using utilities. What makes an utility an utility is how it interacts with probabilities to result in actions, namely, via expected utility maximization (EUM). There are several sets of principles from which EUM can be derived, and they come in two kinds: they either derive probabilities or take them for granted. The former has the issue that, as one of the background posts from the sequence says:

“The ‘degrees of belief’ studied in this post are not, e.g., our acceptable betting odds, or a probability distribution that (along with a utility function) rationalizes our preferences. Rather, they are our basic judgments of the plausibility of different possible outcomes.”

If we assume our actual epistemic state is more adequately represented as a set of probability distributions rather than a single (action-conditioned) probability distribution, then something went wrong with the arguments and I don’t see why their conclusions should be taken as action-guiding. Similarly, the later kind directly depends on assumptions that don’t hold, and therefore we also have no reason to accept their conclusions.

It’s not obviously correct that the solution is to stick to utilities anyway and then come up with some ad-hoc way of making them work with sets of probability distributions. Instead, a more principled approach is to take the theorems we used to justify EUM, replace the probabilities in their assumptions with sets of distributions, and then do the maths to find the proper generalization. In the special case where the sets contain a single distribution, this will hopefully still give us utilities and EUM, which the new theorem would link somehow to the behaviour in the general case. This would be analogous to how, without probabilities, EUM is equivalent to picking the/a highest precedence option according to some weak ordering, and this ordering imposes constraints on the set of utility functions that generalize it to the probabilistic case.

However, doing this with an arbitrary such theorem isn’t always clearly correct. E.g., the first VNM axiom is completeness: that given two lotteries, we either prefer one, the other, or we are indifferent between them; but if we are given two sets of lotteries instead, as the sequence insists, maybe they are just incomparable. We need some more basic set of assumptions that can work well with sets of distributions. Thankfully, those exist. You don’t need to read that post; the results presented here don’t depend on it. I was trying to adapt the theorem presented there, and in doing so and considering what happens in some simple cases, it immediately followed that there is a small set of (arguably) commonsensical properties that no decisionmaking procedure under unawareness (as formulated in the sequence) can simultaneously have.

Let’s assume that:

• We can have a finite list of actions to choose from, each identified only by its index, though I’ll name them A, B, C, etc. (and then more descriptively) for clarity.
• We can have a finite set of possible outcomes (exactly one of which will actually happen).
• As in appendix B of post 3, an agent’s epistemic state (about the outcomes) is represented as a set of action-conditioned probability distributions. That is, in the notation of the sequence, that each possibility p ∈ P is a conditional probability distribution over outcomes, that depends on the action (a strategy s in the sequence) we choose. Crucially, this is not the same as having, for each action, a set of probability distributions over outcomes. I think the definition from the appendix makes sense: if we assume we can be unaware of a bit so that it could be true or false but it makes no sense to assign a probability distribution over the two possibilities, and that if we knew the true value we could use it (and other data) to perform Bayesian inference as usual, it is natural to represent our epistemic state as a set of two possible probability distributions over the state of the world, each of which leads to a probability distribution over outcomes given each possible action.
• Agents select a probability distribution over actions. Deterministic choices are allowed (they put 100% chance on a single action, and 0% on the others). Just in case, I want to be clear that here I’m making no claims about action comparability.
• Given a set of distributions over worlds, represented in whatever way they might be, the distributions over outcomes are computed as usual for each possibility in the set (following the rules of probability), and then put together into a new set.

Then, the first two of the rationality principles that lead to the issue are as follows:

1. Consequentialism: The action distribution selected is a function of only the set of action-conditioned probability distributions over outcomes.
   • Since the epistemic state is a set, the action distribution does not depend on the order of its elements (they don‘t have any).
   • This seems sensible since we are searching for some broadly consequentialist decision procedure, and even if the “correct” method allows taking other factors into consideration, I don’t think it should be forcing it, so we can assume without loss of generality that the decisions depend completely on the set of distributions. I also won’t be considering any contrived scenarios involving, e.g. perfect predictors, multiple perfect copies of an agent existing simultaneously, etc.
2. Permuting the outcomes of the actions equivalently permutes the probability the agent assigns to those actions.
   • E.g. if we swap the consequences of actions A and B so that now, for all possibilities in the set, A behaves exactly like B did (and vice versa), then whatever probability the agent assigned to B must now be assigned to A (and vice versa).
   • This seems sensible too, because the order in which the actions are written is arbitrary (what differs between them is only the consequences), and therefore should not matter.

This already tells us how an agent should behave in the following case:

• Assume the outcome is binary (it’s something that in reality will either happen or not),
• The agent has exactly two actions available (A and B),
• And their epistemic state consists of a set of two possible conditional probability distributions (each its own bullet point):
  • Selecting A leads to the event happening with probability 100%, and selecting B leads to it happening with probability 0% (it won’t happen).
  • Selecting A leads to the event happening with probability 0% (it won’t happen), and selecting B leads to it happening with probability 100%.

Let’s call P(A) the probability the agent gives to action A (so P(B) must be 1-P(A)).

Now, for each of the possibilities in the set, let’s swap the consequences of A and B. The epistemic state then is the following:

• Selecting A leads to the event happening with probability 0% (it won’t happen), and selecting B leads to it happening with probability 100%.
• Selecting A leads to the event happening with probability 100%, and selecting B leads to it happening with probability 0% (it won’t happen).

Since we swapped (the consequences of) A and B, the new P(A), let’s call it P’(A), must be equal to the old P(B), which was 1 minus the original P(A).

The epistemic state is a set, so we can swap the order in which we write the possibilities without affecting the set they represent, and therefore without changing the probabilities the agent gives to the actions, so that now we have:

• Selecting A leads to the event happening with probability 100%, and selecting B leads to it happening with probability 0% (it won’t happen).
• Selecting A leads to the event happening with probability 0% (it won’t happen), and selecting B leads to it happening with probability 100%.

Then, notice that this is exactly identical to the original scenario, so by consequentialism the action probabilities must also be identical. This means that P(A) = P’(A) = P(B) = 1 - P(A), and therefore P(A) = 0.5.

With a drawing, where:

• The vertical lines are the scales of the probabilities of the event happening, going from 0% at the bottom to 100% at the top,
• The left vertical bar is for the event probabilities if action A is selected, and the right one for action B,
• The circles/dots in the vertical lines are associated with probabilities of the event happening, in some particular possibility from the set,
• From left to right the probability of selecting A goes from 100% to 0%,
  • Then, the probability of the event happening is interpolated separately for each possibility, represented by the two dashed lines,
  • The vertical line in the middle corresponds to the action distribution chosen by a rational agent, so that the probability of the event happening is 50% in both possibilities,

we can see that this happens to be equivalent (in this case, and by coincidence) to maximizing the minimum probability of the event happening among the possibilities:

The third and last rationality principle, which I’ll call reflective stability, is that a rational agent, given the choice of behaving like they do behave, or some other way, chooses to behave like they do behave. That something roughly like this must be true of rational behaviour should hopefully be self-evident. I won’t make it precise in full generality, instead, I will just say what this looks like in the simplest possible case, which turns out to be enough for the proof:

Given any decision scenario (with a set of actions to choose from, which will have consequences), or uncertain decision scenario (a scenario will be chosen among some set according to some unconditional probability distribution), in which, before any of the randomness/unawareness is resolved, we choose (possibly randomly) between behaving like we would usually behave (when we get to the scenario), or producing some other action distributions (when we get to a scenario, conditioned on the scenario, and with probabilities given by the setup), so that from our point of view there is no other probability distribution over the two possible behaviours other than 100% our usual behaviour that results in uncertain/unaware consequences identical to those that would result from selecting our usual behaviour (again, with probability 100%), we should choose to behave that way (with probability 100%).

For example, say we have the same setup as before, but instead of deciding between A and B directly, the agent’s choice is between selecting either some other agent or themselves to make the decision. I will call (arbitrarily since it is a set) 1 and 2 the two possible conditional distributions under consideration (same as in the previous setup). Then, assume the other agent chooses action A with probability 75%. This is depicted in the drawing below:

If the other agent is chosen, their epistemic state doesn’t matter, only that they choose A with probability 75%. This results, under each of the possible conditional distributions in:

1. The event happening with probability 75%.
2. The event happening with probability 25%.

If our (rational) agent is chosen, their epistemic state is restricted to that of the initial setup and, as concluded before, they will select A with probability 50%, which results, under each of the possibilities in:

1. The event happening with probability 50%.
2. The event happening with probability 50%.

By consequentialism, the epistemic state (or at least the part they can act on) of the agent when choosing between the other agent and themselves is, for each possibility in the set (keeping in mind the actions are not actually named “other agent” and “same agent” or anything like that as part of the state, they are just the first and second actions):

1. Selecting the other agent leads to the event happening with probability 75%, and selecting themselves to it happening with probability 50%.
2. Selecting the other agent leads to the event happening with probability 25%, and selecting themselves to it happening with probability 50%.

Given this information, the reflective stability principle says they must choose themselves (second action) with probability 100%.

But then consider the case where, after the agent is chosen, a coin is flipped. If it lands heads (50% chance) the agent will be put in the original scenario, but if it lands tails (50% chance), they will be put in a scenario where, regardless of what they choose, in the two possibilities from the set, respectively (given arbitrary numbers as before):

1. The event will happen with probability 100%.
2. The event will happen with probability 0% (it won’t happen).

The choice of agent is now between our agent and a new agent that will choose A with probability 0% (will choose B). This looks as follows:

Again, given this information the reflective stability principle says they must choose themselves with probability 100%. However, notice the epistemic state the agent is in when choosing between the agents. The possibilities are:

1. Selecting themselves leads to the event happening with probability 75%, and selecting the other agent to it happening with probability 50%.
2. Selecting themselves leads to the event happening with probability 25%, and selecting the other agent to it happening with probability 50%.

By consequentialism as far as the agent is concerned, this is the exactly same epistemic state as that in the previous diagram! (again the “our agent” and “some other agent” labels are not part of the epistemic state) But then they should do what they did before and choose the second action, which now corresponds to the new agent, not themselves, so we have a contradiction.

Reality is not forced to give us a proper way to behave under arbitrary epistemic states, and the situation could be worse than just being unaware: maybe all possible behaviours are wrong. I’m very much not saying this is necessarily the case: maybe it turns out that the right thing to do is to use some sort of equivalence classes when defining reflective stability, or maybe, just like generalizations of probability should probably have epistemic origins instead of being derived from principles of rational action, generalizations of (or alternatives to) utility functions should have their origin in the structure of minds instead of in coherence theorems (the theorems don’t tell us what the concrete utility function should be, after all), or maybe it just doesn't make sense to be unaware of a bit. In any case, that’s all I have for now.