A Note Against the Maximality Rule

This competition entry has been selected for publication by the Forum team.

In the Challenge of Unawareness sequence and particularly its second background post, Anthony DiGiovanni argues for the use of imprecise, set-valued probabilities (or a set of probability distributions). For example, a rational agent might say that the probability that an AI pause decreases x-rusk is the representor [0.45, 0.65]^[1]. They do not need to put higher-order credences on the representor (such a hierarchical Bayes model would yield precise probabilities).

For decision making under imprecise probabilities, the sequence proposes the maximality rule, which states that

A should be strictly preferred to B if and only if A has higher EV than B under every distribution in your representor.

Say our agent wants to decide between these three options:

I. Support an AI pause,
II. Oppose an AI pause,
III. Do nothing regarding an AI pause/Do something else.

Which of I., II. or III. is permissible under the representor [0.45,0.65] from above? I. is better than II. and III. under the probability 0.65, but it is worse under the probability 0.45. Hence all actions are equally allowed.

This process of looking for counterexamples in the above “for all” clause in wide representors is akin to motivated reasoning: In order to say that two options are equally permissible under maximality, you need to find, for each of your options, a plausible scenario where it fails (and the other does not), deliberately avoiding consideration of the relative likelihood of these scenarios.

Maximality also undermines the original motivation behind imprecise probabilities: If you previously said you were equally considering many different scenarios, now you are privileging two, the most extreme ones. Basically, you have replaced the interval [0.45, 0.65] by {0.45, 0.65}. If [0.45, 0.65] represented that

it would take stronger evidence to update you to “X is less likely than not-X” than to “X is more likely than not-X”,

that information is lost at the decision making step^[2], but this expressiveness was a motivating factor behind representors. For decision making using the maximality rule, it seems that imperfect information does not beat zero information as its result is the same as a (precise) Bayesian agent with exactly even odds arrives at. More precisely, at least, this is true in interesting cases: comparisons of options where your representors assign different signs to your interventions’ outcomes.

This alone is not enough to refute premise P2a (never mind the whole sequence). Instead of aggregating the probabilities, a proponent of the premise can do either of the following:

• Give up on an accurate representation of all the options in the mental state and explicitly only focus on its boundaries. (This could be aided by reformulating the maximality rule as a principle of caution: You only choose an action if the most pessimistic, but still plausible case for it is better than the most optimistic, but still plausible case for any alternative action.)
• Come up with a different decision criterion entirely.

1. ^{^}

    Side note: The post furthermore admits that the agent might be indeterminate what their exact representor is, e.g., [0.45, 0.65] might be as acceptable as [0.42, 0.66] to them. Such unknown representors to me do not seem to be consistent with other descriptions of imprecise probabilities, and, to be honest, I think at the point where you are using unknown representors you might just give up on probabilistic modelling completely. Arguing that we have Knightian uncertainty about the far future would in my view only strengthen the sequence’s argument.

2. ^{^}

    This specific discrepancy Claude pointed out.