This competition entry has been selected for publication by the Forum team.
There is a very natural pragmatic response to the problem of unawareness: we should ignore it. If it is right, it seems that we have no good reason to choose one action over any other, so we may as well assume that it is wrong.
ADG anticipates this reaction, and makes some convincing points in response. Among these:
A: We can have reasons for acting other than impartial consequentialism (ADG proposes rules like avoiding dishonesty, and virtues like compassion). If there can be other reasons for acting, and if there can be no impartial consequentialist reasons for acting under unwareness, then the argument for unawareness would appear to be a compelling argument for acting according to these other reasons instead. The argument for unawareness becomes an argument against impartial consequentialism, rather than a broad claim that we cannot say anything about ethics, and so it should not be ignored lightly.
B: It is not clear what 'ignore it' means in practice for the impartial consequentialist. Does that mean prioritising neartermist causes like malaria bednets? Or does it mean trying to account for “as many galaxy-brain considerations as possible”?
My goal in this essay is to try to defend the natural reaction to unawareness (that we should ignore it on pragmatic grounds) against ADG's reply. In particular, I will make two arguments, in the two parts of the essay:
Part 1: It is possible to have consequentialist reasons for preferring one action over another, even in some situations where credences are imprecise and the action’s expected value is not robustly better, especially once we take into account pragmatic considerations (this rejects ADG’s “first premise”, and weakens the force of reply A).
Part 2: We can coherently ignore the problem of unawareness when making decisions, by using a 'simplicity' or ‘robustness’ heuristic to evaluate arguments. Simpler, robust, arguments should be preferred over more complex, fragile ones. In other words, we should not spend our time trying to come up with “as many galaxy-brain considerations as possible”. This makes explicit what 'ignore it' means for the impartial consequentialist. If I am right that this approach provides a uniquely self-consistent way of handling unawareness, then this is a good response to reply B. The simplicity heuristic is already discussed and rejected in the sequence. I try to respond to ADG’s arguments against it here.
The problem of unawareness will play no role in Part 1. I will defend the first claim by considering sources of imprecision other than unawareness, where I think defending the claim is more straightforward. I will then consider unawareness in Part 2, turning the ideas developed in Part 1 to that problem, and using them to defend the simplicity heuristic.
Part 1 - Dealing with underspecified decision procedures
Even without unawareness, consequentialism may be underspecified. In this part I discuss two potential reasons for this, one already discussed in the sequence, one not. I then explain how an impartial consequentialist might be able to make decisions anyway. Having established this, I will apply similar ideas to the problem of unawareness in Part 2.
Two sources of imprecision
Choice of prior
As discussed in ADG's sequence, we may have imprecise credences not because we have any doubt about how to update our beliefs based on the evidence in front of us, but because we do not know which prior to use. ADG describes two non-pragmatic intuitions that might help us construct a prior: indifference, and Occam’s razor. ADG then argues, convincingly, that neither of these is likely to pin down a prior uniquely. Even an ideal agent may have imprecise credences as a result. (This also apparently applies to Solomonoff Induction.)
How to weight different individuals
We may be clueless about how to weigh the utilities of different individuals (or even about how to weigh different moments in our own life). It is easy to forget when moving in EA circles, but putting 'goodness' on a numerical scale is a very odd thing to do. It is clear that in some cases we have the ability to introspect and determine that an outcome X would be 'better' than outcome Y, but having an ordering of outcomes is not enough structure to be able to put them on a numerical scale. It is not obvious what the sentence 'outcome X is twice as good as outcome Y' is even supposed to mean.
Fortunately, for a single individual, the VNM utility theorem gives us a nice solution. An outcome X is defined to be twice as good as outcome Y if that individual would be indifferent between a 50% chance of receiving X and a certainty of receiving Y. This is all well and good when considering the preferences of a single individual, but the problem is that the utility functions defined in this way are only determined up to linear scalings (technically up to affine transformations if there is no natural zero point). This becomes a big problem if you want to aggregate utilities, as utilitarians do, because if we scale the utility function of each individual differently then the ordering of their aggregations can change. Before we can aggregate, we need to calibrate our utilities for each individual to a common scale somehow, but it is not at all clear how to do that.
I think it is worth me spending a paragraph to emphasize just how problematic this is for utilitarianism. It is not just that we don't know the correct way to aggregate utilities. It is that it is not at all obvious that there are 'correct' calibrations to use at all, in any objective sense. It is not clear what an answer to the moral weights question could look like, even in principle. For there to be a 'correct' way of summing utilities between individuals, there would have to be a common numerical unit of utility that you could meaningfully use across different individuals, and it is not clear how that unit should be defined. And I think we first need to define it before we can begin to try to measure it.
In practice, when we try to compare the utilities of different humans, there are pragmatic heuristics we can employ to address this problem, in a fairly non-arbitrary way. For example, we can pick a particular experience that we expect to be similarly pleasurable or painful for all humans, and use that to anchor everyone's individual utility functions at a common value. With that single common reference point, aggregation becomes possible. But we are in a much tougher situation once we start trying to compare utilities between humans and non-humans. The more different they are from us, the bigger the problem becomes.
Ultimately, this all amounts to another serious source of imprecision for the impartial consequentialist. There may be multiple ways of calibrating utilities that all seem reasonable to us, and in addition, we may not feel able to assign a precise Bayesian probability to each one. From ADG’s point of view, our credences with regards to the utilities of different outcomes should then be imprecise.
To sum up, both choice of prior, and choice of utility calibration, are two potential sources of imprecision, in addition to unawareness. In one respect, they are even more problematic. It is not just a problem with our lack of knowledge. It is not clear that there is an objectively correct answer at all. If there is not, then arguably our inability to assign precise subjective probabilities to the different options is not just a problem of precision. The attempt may be fundamentally incoherent.
So what is an impartial consequentialist to do?
Suppose an impartial consequentialist is trying to decide between two actions, X and Y. There is a set of plausible prior distributions and utility calibrations under which action X is preferable to Y, but there is an alternative set under which Y is preferable to X. Our impartial consequentialist is not confident in ruling out any option from either set, and nor do they feel able to assign precise subjective probabilities to the members of each set.
I agree with ADG that it would be wrong to arbitrarily assign numerical weights to these options and aggregate them into a single probability distribution. But here is, I think, my first disagreement with ADG. My understanding of ADGs position is that in such a situation, there can be no impartial consequentialist reasons for choosing X over Y. X and Y are, under impartial consequentialism, incomparable. But this feels far too strong a claim to me.
Suppose our impartial consequentialist reasons as follows: "When I look at the set of priors and utility calibrations consistent with choosing X, they feel, on aggregate, more plausible/satisfying/acceptable to me than the set consistent with choosing Y. I therefore feel more comfortable committing myself to a prior/calibration pairing from this set vs the other. I therefore choose X over Y"
I believe this reasoning would be coherent. And if someone acts like this, I think it is still reasonable to describe their decision as motivated by impartial consequentialism.
You could imagine making this decision process more formal. For example, maybe we could have a moral parliament, not of different moral theories, but of different possible choices of prior, or utility calibrations. But I don't think this is necessary. Someone can make their decision based only on a vague intuitive comparison of the options, and I would still defend my claim.
I anticipate two possible objections here. First, I have not come close to defining a formal decision theory. The reasoning I am defending feels arbitrary. As ADG explains in the sequence, one of the attractions of impartial consequentialism is that it promises to remove arbitrariness from our decisions. If we concede objectivity, what is the point? Have I been far too quick to surrender the idea that we can have truly objective reasons for acting?
My response would be that the appeal of consequentialism lies not in the elimination of arbitrariness, but in its isolation. It is really remarkable that it constrains us as much as it does. In an ideal world, ignoring unawareness for now, two ideal utilitarians would be able to boil their disagreements down into a disagreement over priors, or over utility calibrations, and thereby understand that disagreement better.
The second objection I anticipate is the appeal to intuition. A common theme in ADG's sequence is that we should not trust our intuition unless we have good reason to believe it to be truth tracking. ADG convincingly argues that on empirical questions affected by unawareness, we don't have good reason. But choosing the correct prior (and arguably choosing the correct way of calibrating utilities) is a different kind of question, and I am not sure that the same concerns should apply in the same way. At a certain meta level, it seems that we have to trust the ability of our own minds to evaluate claims like these, or we could have no reason for believing or doing anything. More importantly, it seems very plausible that there is no truth here for our intuitions to track. The choice of prior or utility calibration may be just that: a choice. But I claim that it is still a choice we can coherently make, without giving up on our claim to be impartial consequentialists.
Decoupling decision making from epistemics
There is another way that a consequentialist could approach decision making that I would like to introduce before discussing unawareness. And this procedure even more directly clashes with ADGs normative premise 1. That is: they can decouple the subjective probabilities they use for decision making from those that they would use for their epistemics.
It is helpful to consider an example. Suppose a box contains 100 balls which can only be coloured red or blue. Someone looks at 99 balls from the box, in no particular order, and they are all red. They are now asked to bet on the colour of the final ball.
For this thought experiment, we exclude any consideration of traps laid by the bookmaker, and in fact we ignore any prior knowledge that our gambler has about how the world works. We assume they appear spontaneously in this scenario with no memories, only the ability to reason.
Here are two possible ways that they could approach the problem:
They adopt a prior based on Occam's razor. The rule 'all balls are coloured red' is simpler than the rule 'all balls but one are coloured red', so is given higher prior probability. The 'all balls but one' probability mass is also split 100 ways between identical balls (no special ordering), so the 'all balls are red' hypothesis is a priori more than 100 times as likely as the 'all balls but the 100th I look at are red' hypothesis. These are now the only two hypotheses consistent with the evidence, so after Bayesian updating they now hold all the probability mass, but in the same ratio as they had in the prior. Our gambler therefore assigns a probability of over 99% that the final ball is red.
They adopt a prior based on indifference. All possible 2^100 ball colourings are given equal prior probability. After Bayesian updating, only two of the 2^100 remain, and they are both equally consistent with the evidence. The probability that the final ball is red is therefore 50% (the same as it was before they looked at any balls). This conclusion applies even if the balls were in a bag and they were being sampled at random (red random samples are evidence against there being a blue ball in the bag, but 1 blue ball being in the bag has much higher prior probability than 0 blue balls).
I expect that a lot of people's immediate intuitive reaction to this will be that the second proposed approach is bonkers. But I would respond that this is only because we have evolved to be pattern matching machines. And using our past experience to justify a choice of prior would be circular (and by hypothesis, not an option available to our amnesiac gambler in this case anyway). From a purely theoretical point of view, I claim that both priors appear natural. If anything, the second approach seems more natural, in that it allows us to define the probabilities precisely, whereas Occam's razor is more vague.
You might still object: “Surely once our gambler has seen 99 red balls, which has probability ½^99 according to their prior, they should realise that they picked a terrible prior?” But any sequence of 99 balls had probability (½)^99 according to their prior, so they should not be surprised at all that they have witnessed an outcome with this likelihood. The sequence of 99 red balls is only evidence of something if you begin with a bias that 99 red balls is a special outcome, relative to the other possibilities. That bias must be adopted before interpreting the evidence, not after.
Suppose our gambler agrees with me. On purely epistemic grounds, they can see no reason to favour one prior over the other. In fact, maybe they lean towards using the principle of indifference. They might still reason as follows, on pragmatic grounds:
"Under indifference, I can never gain any information from my observations. My position appears bleak. But if Occam's razor is true, I can gain a great deal of information from my observations. More information would mean better decisions. I will therefore act as if Occam's razor is correct (even though I can see no particular epistemic reason for favouring it over indifference)."
I believe this reasoning is coherent. And I believe this gambler can still claim to be a consequentialist, despite acting against the expected value that would be derived from their purely epistemic beliefs.
Part 2 - How to ignore unawareness
The ideal consequentialist decision procedure consists of three steps:
- Assign a prior probability to every possible world.
- Update your prior distribution in light of the evidence available to you, to get a posterior distribution.
- Calculate the expected value of each possible action available to you, according to this posterior distribution, and take the action which has the highest expected value.
We saw in Part 1 that the first and last steps may be underspecified. ADG argues that all three steps are underspecified, due to “unawareness”. The term unawareness is used to describe two related problems:
- “We can’t conceive of possible worlds in full detail. Instead, we’re only coarsely aware of them in the form of hypotheses, which are sets of possible worlds described abstractly. In practice, these descriptions don’t come close to specifying every morally relevant aspect of the futures they cover. They’re more like “misaligned ASI takes over” or “future people care about wild animals”. So, a single hypothesis might gloss over large differences in value across the possible worlds it contains.”
- “For some possible worlds, we are unaware of any hypotheses that contain them. That is, these hypotheses haven’t occurred to us in enough detail to factor into our decision-making. For example, we’re unaware of the sets of worlds where undiscovered crucial considerations are true.”
Faced with unawareness, our only hope as consequentialists is to fall back on intuitions or heuristics that we believe will approximate the idealised consequentialist decision procedure. If we are considering the effect of our actions on our own lives, or on those of our family or neighbours, ADG admits that this can work. But if we are impartial consequentialists, committed to considering the welfare of all sentient beings wherever and whenever they exist, then ADG convincingly argues that we have no reason to trust that our standard intuitions and heuristics will be of any use in selecting the action with highest impartial value.
In this Part, I respond to this by introducing an assumption which I claim would justify the use of a “simplicity” or “robustness” heuristic for evaluating arguments, even in the face of unawareness. I then defend the adoption of this assumption on pragmatic grounds.
Picking a biased prior
We argued in Part 1 that choosing a prior is difficult, and that there may be no objectively best choice. Under unawareness, the situation seems even worse, in that we can’t even write a prior down. We cannot assign a prior probability to worlds which we are unaware of. However, there is still a probability distribution that we can speculate about:
Quantity X: For a random decision we might be faced with, and for a random pair of actions we might choose to take, what is the difference in utility between those two actions?
We can treat X as a random variable, with some probability distribution. If we knew the correct prior to use over all possible worlds, then we could derive the prior distribution for X. I am now going to start speculating about the shape of this unknown ideal prior distribution for X. Given that we can’t formulate a prior over possible worlds in practice, such speculation may seem silly. But given that we may not be able to formulate a unique prior over possible worlds even in principle, such speculation may seem less silly (since the distribution of X may ultimately be a choice that we are free to make, in the spirit of Part 1). I will discuss this in more detail later on. For now, I ask you to humour me.
First, the distribution of X must be symmetric about 0, by definition (and so its expected value, if defined, is also zero).
Second, can X be infinite? I am going to assume that it cannot. It is a real number, and its prior distribution is a probability distribution over the real numbers.
Third, does the prior distribution of X have finite variance? I am going to assume that it does.
Fourth, how large is this variance? I am going to assume that it is small (on the order of the utility range that a single individual might experience).
Under these assumptions, I claim that we can largely ignore unawareness when making decisions.
If the variance of X were large, then it would be easy for evidence to cause large updates to E(X). For example, if the expected value of X given that it is positive is extremely large, then if we see evidence which has a higher likelihood under positive X than negative X, the expected value of X will immediately become extremely large. But if the variance of X is small, then this is not the case. It becomes harder to move X with weak evidence.
ADG gives some worked examples of the kinds of arguments through which unawareness might infect our expected value calculations. These arguments have two distinctive features that make their serious incorporation into decision making so problematic:
- They invoke a (small) chance of a very large impact (for example, the chance that your individual actions shift the future trajectory of humanity from one possible future to another dramatically different one, coarsely defined).
- They are highly sensitive to highly uncertain parameters, to the extent that the arguments could also easily be consistent with zero or opposite sign impact as well.
If we have to take such arguments seriously, I agree with ADG that we are in trouble. But if we adopt a heuristic that we should largely ignore arguments with these features, then we might be ok. And I think the biased prior on X defined above should justify such a heuristic. If the likelihood of an argument existing is not significantly higher for large X than for zero X, then a prior with the above properties will not move much under the argument.
We can contrast this with evaluating the impact of a decision for which robust evidence is available: donating to the Against Malaria Foundation. To do this, I make a further assumption: when we split the effects of an action into multiple parts (arising from different mechanisms) the prior distribution of each part also takes the same low variance form. When we consider the impact of donating to the AMF, we can then isolate the utility associated with the robustly measurable mechanisms (e.g. the lives saved from malaria in treated group, days of sickness prevented in treated group) from all other impacts (uncertain knock-on effects on the far future). We update the distribution of each part separately, based on the evidence available. The robustly evidenced part moves a lot, the speculative part moves not very much, and can be ignored. By linearity, the expected value of the decision is the sum of the expected values of the two parts, and only the robustly measured part should concern us.
From what I understand of “bracketing”, the approach I am describing here is similar, except that it applies to arguments rather than to beneficiaries (and it is more vague - it would be interesting to explore this decomposition idea further).
It is important to stress what I am not saying. I am not saying that we can always ignore the possibility that our actions will have a large impact. A counter-example would be voting, where the chance of your vote changing an election is tiny, but if it does, the impact could be huge. But the difference with voting is that our estimate of both the small probability and the size of the impact can be robustly evidenced (or some part of the impact can), which can move us from our prior (although note that the size of the potential large impact is not necessarily any more likely than it was in the prior). This is not true of the kinds of arguments affected by unawareness.
ADG considers “Simple Heuristics” of this kind in Part 4 of the sequence, and rejects them, but I’m not sure if the rejection (focusing on the number of steps in an argument, rather than its sensitivity to uncertain parameters) directly bears on the presentation I have given here. To the extent that it does, the argument against appears to just be an appeal to the implausibility of the shape of prior I have put forward. In the remainder of this essay I respond to this concern.
Curve fitting analogy
The idea I have presented here is nothing new. As referenced in the sequence, it will be familiar to anyone who has studied Machine Learning or Statistics as the standard approach to dealing with “overfitting”. Overfitting occurs when a model responds too strongly to each new data point, being influenced by random noise, instead of by the patterns that are actually relevant for predicting unseen data. The way to deal with overfitting is to increase the bias in your model. From a Bayesian perspective, this means picking a more constrained prior. When we encounter a consequentialist argument with conclusions that seem highly sensitive to the values of extremely uncertain parameters, then this smells a lot like an overfitting problem. It is natural to want to resolve it in the same way.
But a key difference between our situation and a typical statistical modelling problem, is that statisticians can typically use a holdout set, drawn from the same distribution as the training and validation datasets, to empirically demonstrate the soundness of their approach. Impartial consequentialists do not have this luxury. We can never observe the complete impact of any of our actions on the far future. A more appropriate analogy can be found in a special kind of curve fitting problem, involving extrapolation.
Consider the following problem:
There are two unknown curves, described by functions from x to y, with x > 0. We are given a sample of data points from each of these curves, with some random noise added. In a certain range (say x < 10) these points have been densely sampled, and we have a good sense of the shape of the underlying curves. But as x increases, the sampling density becomes more and more sparse. Eventually, we have no samples at all, although the unknown curve continues. Our job is to decide which of the two curves encloses the largest area.
This is a curve fitting problem. We could approach it with an explicitly Bayesian method, or with an alternative method that we hope approximates the idealised Bayesian procedure. Either way, when viewed with a Bayesian lens, an answer to this problem ultimately checks out as a commitment about the correct prior distribution to use over the space of all possible curves (and over the shape of the random noise). But since this is a problem involving extrapolation to regimes in which we have no data, we cannot give post-hoc justification for our choice of prior by measuring the performance of our approach on a holdout set. We seem to be in a tricky position.
But I claim that although we cannot show our choice of prior is good, we can still say that certain choices of prior are, in some sense, bad. We can do this before we have looked at any data at all, as follows:
- Repeatedly sample pairs of curves from the prior, and sets of sample points from each curve with the expected sparsity.
- For each pair of curves, repeatedly re-sample from the prior over random noise, apply the curve fitting procedure to the noisy points, and measure the area under the fitted curve.
- If the estimate of which curve has the largest area changes dramatically with each re-sampling of the noise, then this indicates that our prior has an overfitting problem. We can already say this before we have seen any real data.
Note, although I have called the prior “bad”, I am not claiming to have shown that it is not the best way to represent our actual epistemic situation. Maybe the curves we encounter in this problem really have been drawn from a bad prior. But: if we are anyway confused about the correct way to represent our actual epistemic situation, then I claim that it makes sense to rule out “bad” priors on pragmatic grounds. I claim that if we are faced with this problem, it is defensible to pick a prior under which solving the problem is possible, and to justify the choice of prior on that basis alone. In this context, this will look like choosing a prior under which the shape of the curve for x < 10 is a good guide to its behaviour for x > 10.
Conclusion
With all this groundwork laid, I will now present what I think could be a coherent argument for ignoring unawareness on pragmatic grounds. Just like the gambler from Part 1 betting on the colour of a ball, or the statistician from the previous section trying to estimate the area under some curves, we may reason as follows:
- We like the Bayesian approach to epistemics and decision making, but we are confused about the correct way to choose a prior, and about what it even means for one prior to be ‘correct’.
- Under some choices of prior, good decision making seems impossible (see ADG’s sequence).
- But there is a natural choice of prior under which good decision making is possible (see my ‘constrained prior’ section above). In particular, we can adopt a prior under which the difference in utility between a random pair of actions available to us has a variance on the order of the utility scale of a single individual. Such a prior should not move much under the kind of evidence that is sensitive to unawareness, allowing us to largely ignore this evidence.
- On pragmatic grounds, we should therefore adopt a prior with this shape.
