"We don’t usually think of achievements in terms of what would have happened otherwise, but we should. What matters is not who does good but whether good is done; and the measure of how much good you achieve is the difference between what happens as a result of your actions and what would have happened anyway." - William MacAskill, Doing Good Better
Counterfactual thinking is fundamental to economic thinking, and this approach has been incorporated into Effective Altruism. The actual impact of your choices is based on what changed, not what happened. Per the forum wiki summary, "Counterfactual reasoning involves scenarios that will occur if an agent chooses a certain action, or that would have occurred if an agent had chosen an action they did not."
In this post, I'll argue that when counterfactual reasoning is applied the way Effective Altruist decisions and funding occurs in practice, there is a preventable anti-cooperative bias that is being created, and that this is making us as a movement less impactful than we could be.
Myopia and Hyperopia
To consider this, we need to revisit a fundamental assumption about how to think about impact, in light of the fact that individuals and groups can, should, and often do cooperate - both explicitly and implicitly. And to revisit the assumption, I want to note two failure modes for counterfactual reasoning.
First, myopic reasoning. Being a doctor saves lives, but arguably has little counterfactual value. And myopic reasoning can be far worse than this. As Benjamin Todd suggests imagining, "you’re at the scene of an accident and you see an injured person. In your enthusiasm to help, you push the paramedics out of the way and perform CPR on the injured person yourself. You’re successful and bring them back to consciousness, but because you’re less well-trained than the paramedics, you cause permanent damage to the person’s spine." Obviously, counterfactual thinking, appreciating that without your "help" the outcome will be better, can prevent this type of myopia.
But there is a second failure mode, which I'll call hyperopic reasoning. Myopia is nearsightedness, the inability to see things far away, and hyperopia is farsightedness, the inability to see things nearby. What does this look like in reasoning about outcomes, and how could it fail?
In December 2020, the United Kingdom and Sweden granted emergency approval for the Oxford–AstraZeneca vaccine, while Germany and the United States authorized the Pfizer-BioNTech vaccine, and the United States authorized the Moderna vaccine as well. While all of these were significantly better than the earliest vaccine, from Sinopharm in China, or the slightly later CoronaVac or Sputnik V, the counterfactual value of each vaccine seems comparatively miniscule because there were two other western vaccines that would have been approved without the development of this third option.
The Incidental Economist puts it clearly:
When you want to know the causal effect of an intervention (policy change, medical treatment, whatever) on something, you need to compare two states of the world: the world in which the intervention occurred and the world in which it did not. The latter is the counterfactual world... What we really want to know is how the world is different due to the intervention and only the intervention.
This is viewing the counterfactual from a single viewpoint. There is no shared credit for AstraZeneca, Pfizer, and Moderna, because each counterfactual is ignoring the nearby efforts, focused only on the global ones. But we certainly don't think that the value was zero - so what are we doing wrong?
We can try to resolve this without accounting for cooperation. Prior to the creation of the vaccines, it was unclear which would succeed. Each team has a prior potential to create a vaccine, and some chance that the others would fail. If each team has an uncorrelated 50% chance of succeeding, and with three teams, the probability of being the sole group with a vaccine is therefore 12.5% - justifying the investment. But that means the combined value of all three was only 37.5% of the value of success, and if we invested on that basis, we would underestimate the value for each. In this case, the value of a vaccine was plausibly in the hundreds of billions of dollars, so it's enough - but it's still wrong as a general rule.
Cooperation Within Effective Altruism, and Without
I think this problem is dealt with within the Effective Altruism movement partially by viewing it as a single actor. Within Effective Altruism, people are remarkably cooperative, willing to forgo credit and avoid duplicating effort. If we're collectively doing two projects with similar goals, each of which may fail, we can view both as part of a strategy for ensuring the goals are achieved.
And there's a value in competition and multiple approaches to the same question, and this means some degree of internal competition is good. It's certainly useful that there is more than one funder, so that decisions are made independently, despite the costs imposed. It's probably good that there are two organizations doing career advising, instead of just having 80,000 Hours.
In business, this approach, with multiple firms engaged in related markets, Could be be called coopetition - cooperation on some things and competition on others. But the coopetition approach is imperfect. There is a natural tendency for people to seek credit for their ideas, and that often leads to suboptimal outcomes. (Which we can compensate for, at least partially.) Similarly, there is a cost to coordination, so even when people would like to cooperate, they may not find ways to do so. (But providing a list of ideas for others to pursue and helping launch projects designed to fill needs is useful.)
Where Effective Altruism more clearly falls down is external cooperation. Much of what Effective Altruism does is external-focused. (Biosecurity, Alternative protein, Global poverty reduction, and similar. Everything but community building, essentially.) Unfortunately, most of the world isn't allied with Effective Altruism, or is only implicitly cooperating. But most of the world at least generally shares its goals, if not its priorities. And that is unlikely to change in the near-term.
So who should get credit for what gets done? This matters - if we're not careful with our counterfactual reasoning, we could overcount or undercount our contribution, and make sub-optimal decisions.
Shapley Value
Lloyd S. Shapley gave us a fundamental tool for cooperation, the Shapley value, which is the (indisputably correct) way to think about counterfactual value in scenarios with cooperation. If you don't know what it is, read or skim Nuño Sempere's post, and come back afterwards. But in short, in the same way that considering counterfactuals fixes myopia by considering what happens if you do nothing, Shapley values fix hyperopia by giving credit to everyone involved in cooperatively providing value.
And a large part of the value provided by Effective Altruism is only cooperative. We donate bed-nets, which get distributed by local partners, on the basis of data about malaria rates collected by health officials, when someone else is paying for the costs of the distribution, to individuals who need to do all the work of using the bed-nets. Then we walk away saying (hyperopically,) we saved a life for $5,000, ignoring every other part of the complex system enabling our donation to be effective. And that is not to say it's not an effective use of money! In fact, it's incredibly effective, even in Shapley-value terms. But we're over-allocating credit to ourselves.
The implicit question is whether we view the rest of the world as moral actors, or moral patients - and the latter is the direct cause of (the unfortunately named) White Savior Complex. This is the view that altruists decisions matter because they are the people responsible, and those they save are simply passive recipients. And (outside of certain types of animal welfare,) this view is clearly both wrong, and harmful.
"Non-EA Money"
I've used this phrase before, and heard others use it as well. The idea is that we'd generally prefer someone else spends money on something, because that leaves more money for us, counterfactually, to do good. But this is a anti-cooperative attitude, and when we have goals like pandemic prevention which are widely shared, the goal should be to maximize the good done, not the good that Effective Altruism does, or claims credit for. Sometimes, that means cooperating on use of funds, rather than maximizing counterfactual impact of our donation.
Does this matter? I think so, and not just in terms of potentially misallocating our funds across interventions. (Which by itself should worry us!) It also matters because it makes our decisions anti-cooperative. We're trying to leverage other spenders into spending money, rather than cooperating to maximize total good done.
For example, Givewell recently looked at likelihood of crowding out funding, with an explicitly adversarial model; if the Global Fund and/or PMI are providing less funding because of their donations, they don't view their own funding as counterfactual. And when we only take credit for counterfactual impact instead of Shapley impact, we're being hyperopic.
I'm unsure if there is a simple solution to this, since Shapley values require understanding not just your own strategy, but the strategy of others, which is information we don't have. I do think that it needs more explicit consideration - because as EA becomes more globally important, and coordination with other groups becomes increasingly valuable, playing the current implicitly anti-competitive strategy is going to be a very bad way to improve the world.
Edited after more careful reading of the post
As you say in the post, I think all these things can be true:
1) The expected counterfactual value is all that matters (i.e. we can ignore Shapley values).
2) The 3 vaccine programs had zero counterfactual value in hindsight.
3) It was still the correct decision to work on each of them at the time, with the information that was available then.
At the time, none of the 3 programs knew that any of the others would succeed, so the expected value of each programme was very high. It's not clear to me why the '12.5%' figure in your probabilistic analysis is getting anything wrong.
If one vaccine program actually had known with certainty that the other two would already succeed, and if that really rendered their own counterfactual value 0 (probably not literally true in practice but granted for the sake of argument) then it seems very plausible to me that they probably should have focused on other things (despite what a Shapley value analysis might have told them).
It gets more complicated if you imagine that each of the three knew with certainty that the other two could succeed if they tried, but might not necessarily actually do it, because they will be reasoning similarly. Then it becomes a sort of game of chicken between the three of them as to which will actually do the work, and I think this is the kind of anti-cooperative nature of counterfactual value that you're alluding to. This is a potential problem with focusing only on counterfactuals, but focusing only on Shapley values has problems too, because it gives the wrong answer in cases where the decisions of others are already set in stone.
Toby Ord left a really good comment on the linked post on Shapley values that I think it's worth people reading, and I would echo his recommendation to read Parfit's Five Mistakes in Moral Mathematics for a really good discussion of these problems.
The most important thing about your decision theory is that it shouldn't predictably and in expectation leave you worse off than if you had used a different approach. My claim in the post is that we're using such an approach, and it leaves us predictably worse off in certain specific cases.
For example, I strongly disagree with the idea that it's coherent to say that all three programs would have zero value in hindsight, and that the true value is 12.5% each, because it means that in many plausible cases, where return on investment from a single working solution is, say, only 3x the bar for funding, we should fund none of them.
And regarding Toby's comment, I agree with him - the problem I pointed to in the last section is specifically and exactly relevant - we're committing to things on an ongoing and variable basis, along with others. It's a game-theoretic setup, and as he suggests, "Shapley [should be] applied when the other agents' decisions are still 'live'" - which is the case here. When EA was small, this was less problematic. We're big enough to factor into other large player's calculus now, so we can't pretend we move last in the game. (And even when we think we know we are, in fact, moving last in committing funds after everyone else, it is an iterated game, so we're not actually doing so.)
To arrive at the 12.5% value, you were assuming that you knew with certainty that the other two teams will try to create the vaccine without you (and that they each have a 50% chance of succeeding). And I still think that under that assumption, 12.5% is the correct figure.
If I understand your reasoning correctly for why you think this is incoherent, it's because:
If the 3 teams independently arrive at the 12.5% figure, and each use that to decide whether to proceed, then you might end up in a situation where none of them fund it, despite it being clearly worth it overall.
But in making this argument, you've changed the problem. The other 2 teams are now no longer funding the vaccine with certainty, they are also making decisions based on counterfactual cost-benefit. So 12.5% is no longer the right number.
To work out what the new right number is, you have to decide how likely you think it is that the other 2 teams will try to make a vaccine, and that might be tricky. Whatever arguments you think of, you might have to factor in whether the other 2 teams will be thinking similarly. But if you really do all have the same goals, and there's only 3 of you, there's a fairly easy solution here, which is to just talk to each other! As a group you can collectively figure out what set of actions distributed among the 3 of you will maximize the global good, and then just do those. Shapley values don't have to come into it.
It gets more complicated if there's too many actors involved to all get together and figure things out like this, or if you don't all have exactly the same goals, and maybe there is scope for concepts like Shapley values be useful in those cases. And you might well be right that EA is now often in situations like these.
Maybe we don't disagree much in that case. I just wanted to push back a bit against the way you presented Shapley values here (e.g. as the "indisputably correct way to think about counterfactual value in scenarios with cooperation"). Shapley values are not always the right way to approach these problems. For example, the two thought experiments at the beginning of Parfit's paper I linked above are specific cases where Shapley values would leave you predictably worse off (and all decision theories will have some cases where they leave you predictably worse off).
Let's make the problem as simple as possible; you have a simple intervention with 3 groups pursuing it. Each has an independent 50% chance of success, per superforecasters with an excellent track record, and talking about it and coordinating doesn't help, because each one is a different approach that can't benefit from coordination.
And I agree with you that there are cases where it's the wrong tool - but as you said, I think "EA is now often in situations like these," and we're not getting the answer right!
Edit: Vasco Grilo has pointed out a mistake in the final paragraph of this comment (see thread below), as I had misunderstood how to apply Shapley values, although I think the conclusion is not affected.
If the value of success is X, and the cost of each group pursuing the intervention is Y, then ideally we would want to pick N (the number of groups that will pursue the intervention) from the possible values 0,1,2 or 3, so as to maximize:
(1-(1/2)^N) X - N Y
i.e., to maximize expected value.
If all 3 groups have the same goals, they'll all agree what N is. If N is not 0 or 3, then the best thing for them to do is to get together and decide which of them will pursue the intervention, and which of them won't, in order to get the optimum N. They can base their decision of how to allocate the groups on secondary factors (or by chance if everything else really is equal). If they all have the same goals then there's no game theory here. They'll all be happy with this, and they'll all be maximizing their own individual counterfactual expected value by taking part in this coordination.
This is what I mean by coordination. The fact that their individual approaches are different is irrelevant to them benefiting from this form of coordination.
'Maximize Shapley value' will perform worse than this strategy. For example, suppose X is 8, Y is 2. The optimum value of N for expected value is then 2 (2 groups pursue intervention, 1 doesn't). But using Shapley values, I think you find that whatever N is, the Shapley value of your contribution is always >2. So whatever every other group is doing, each group should decide to take part, and we then end up at N=3, which is sub-optimal.
Hi Toby,
I do not think this is correct. If we consider a game with N players where each has to pay c to have a probability p of achieving a value of V:
In David's example, N = 3, and p = 0.5, so S = (7/24) V - c. For your values of V = 8, and c = 2, S = 1/3. This is not higher than 2.
From the formula for the Shapley value, maximising it is equivalent to maximising:
I have concluded in this Sheet the above is a strictly decreasing function of N (which tends to 0), so Shapley value is maximised for the smallest possible number of players. This makes intuitive sense, as there is less credit to be shared when N is smaller.
The smallest possible number of players is 1, in which case the Shapley value equals the counterfactual value. In reality, N is higher than 1 in expectation, because it can only be 1 or higher. So, since the Shapley value decreases with N, assuming a single player game will tend to overestimate the contribution of that single player. I think David was arguing for this.
In any case, I do not think it makes sense to frame the problem as deciding what is the best value for N. This is supposed to be the number of ("live") agents in the problem we are trying to solve, not something we can select.
Thank you for this correction, I think you're right! I had misunderstood how to apply Shapley values here, and I appreciate you taking the time to work through this in detail.
If I understand correctly now, the right way to apply Shapley values to this problem (with X=8, Y=2) is not to work with N (the number of players who end up contributing, which is unknown), but instead to work with N', the number of 'live' players who could contribute (known with certainty here, not something you can select), and then:
Have I understood the Shapley value approach correctly? If so, I think my final conclusion still stands (even if for the wrong reasons) that a Shapley value analysis will lead to sub-optimal N (number of players deciding to participate). Since the optimal N here is 2 (or 1, which has same value).
As for whether the framing of the problem makes sense, with N as something we can select, the point I was making was that in a lot of real-world situations, N might well be something we can select. If a group of people have the same goals, they can coordinate to choose N, and then you're not really in a game-theory situation at all. (This wasn't a central point to my original comment but was the point I was defending in the comment you're responding to)
Even if you don't all have exactly the same goals, or if there's a lot of actors, it seems like you'll often be able to benefit by communicating and coordinating, and then you'll be able to improve over the approach of everyone deciding independently according to a Shapley value estimate: e.g. Givewell recommending a funding allocation split between their top charities.
Since I was calculating the Shapley value relative to doing nothing, it being positive only means taking the action is better than doing nothing. In reality, there will be other options available, so I think agents will want to maximise their Shapley cost-effectiveness. For the previous situation, it would be:
For the previous values, this would be 7/6. Apparently not very high, considering donating 1 $ to GWWC leads to 6 $ of counterfactual effective donations as a lower bound (see here). However, the Shapley cost-effectiveness of GWWC would be lower than their counterfactual cost-effectiveness... In general, since there are barely any impact assessments using Shapley values, it is a little hard to tell whether a given value is good or bad.
In a single person game, or one where we're fully aligned and cooperating, we get to choose N. We should get to the point where we're actively cooperating, but it's not always that easy. And in a game-theoretic situation, where we're only in control of one party, we need a different approach than either saying we can choose where to invest last, when we can't, and I agree that it's more complex than Shapley values.
Hi Toby,
Regarding:
You are most likely aware of this, but I just wanted to clarify Shapley value is equal to counterfactual value when there is only 1 ("live") agent. So Shapley value would not leave us predictably worse off if we get the number of agents right.
Point taken, although I think this is analogous to saying: Counterfactual analysis will not leave us predictably worse off if we get the probabilities of others deciding to contribute right.
Agreed. Great point, Toby!
This isn't a problem with expected utility maximization (with counterfactuals), though, right? I think the use of counterfactuals is theoretically sound, but we may be incorrectly modelling counterfactuals.
It's a problem with using expected utility maximization in a game theoretic setup without paying attention to other players' decisions and responses - that is, using counterfactuals which don't account for other player actions, instead of Shapley values, which are a game theoretic solution to the multi-agent dilemma.