Summary: Consider some project worked on by multiple organisations A, B, C and D. The benefit of the project is x. Each of the organisations is a necessary condition of the benefit x. The counterfactual impact of A is x; the counterfactual impact of B is x; etc. Despite this, the counterfactual impact of A, B, C, and D acting together is not 4*x, rather it is x. This seems paradoxical but isn’t. This is relevant in numerous ways to EAs.
Cross-posted from my blog.
Much of the time when organisations are working to produce some positive outcome, no single organisations would have produced the benefit acting on their own. Usually, organisations work in concert to achieve valuable outcomes and in many cases the outcome would not have been produced if some of the organisations taken individually had not acted as they did. This is a particularly pervasive feature of policy advocacy and EA advocacy. This gives rise to apparent paradoxes in the assessment of counterfactual impact.
For example, suppose Victoria hears about EA through GWWC and wouldn’t have heard about it otherwise. She makes the pledge and gives $1m to ACE charities, which she wouldn’t have found otherwise (and otherwise would have donated to a non-effective animal charity let’s suppose). Who counterfactually produced the $1m donation benefit: Victoria, GWWC or ACE? Each of them is a necessary condition for the benefit: if Victoria hadn’t acted, then the $1m wouldn’t have been donated; if GWWC hadn’t existed, then the $1m wouldn’t have been donated; and if ACE hadn’t existed then the $1m wouldn’t have been donated effectively. Therefore, Victoria’s counterfactual impact is $1m to effective charities, GWWC’s counterfactual impact is $1m to effective charities, and ACE’s impact is $1m to effective charities.
Apparent paradox: doesn’t this entail that the aggregate counterfactual impact of Victoria, GWWC and ACE is $3m to effective charities? No. When we are assessing the counterfactual impact of Victoria, GWWC and ACE acting together, we now ask a different question to the one we asked above viz. “if Victoria, GWWC and ACE had not acted, what benefit would there have been?”. This is a different question and so gets a different answer: $1m. The difference is that when we are assessing the counterfactual impact of Victoria acting, the counterfactual worlds we compare are
Actual World: Victoria, GWWC, and ACE all act.
Counterfactual world (Victoria): Victoria doesn’t act, GWWC and ACE act as they would have done if Victoria had not acted.
The benefits in the Actual World are +$1bn compared to the Counterfactual World (Victoria). We take the same approach for each actor, changing what needs to be changed. In contrast, when assessing the collective counterfactual impact of the actors, we ask:
Actual World: Victoria, GWWC, and ACE all act.
Counterfactual world (Victoria+GWWC+ACE): Victoria, GWWC and ACE don’t act.
When multiple actors are each a necessary condition for some outcome, it is inappropriate to sum the counterfactual impact of actors taken individually to produce an estimate of the collective counterfactual impact of the actors.
[Edit: adding this to further clarify the point, as maybe the point wasn't clear enough.]
To assess Vic's counterfactual impact, we compare to the world in which Vic doesn't act, and GWWC and ACE act as they would have done had Vic not acted.
To assess GWWC's counterfactual impact we compare to the world in which GWWC doesn't exist, and ACE and Vic act as they would have done had GWWC not existed.
To assess ACE's counterfactual impact, we compare to the world in which ACE doesn't exist, and GWWC and Vic act as they would have done had ACE not existed.
This is the correct way to assess these agents' counterfactual impact. The principle that we can aggregate them is wrong and so the absurd conclusion about their collective impact that follows is not a reductio of that method of measuring counterfactual impact. The absurd conclusion is a reductio of the aggregation principle. We cannot aggregate the counterfactual impact taken individually because then we would be comparing the actual world to the world in which "Vic doesn't act, and GWWC and ACE act as they would have done had Vic not acted, AND GWWC doesn't exist, and ACE and Vic act as they would have done had GWWC not existed, AND ACE doesn't eixst GWWC and Vic act as they would have done had ACE not existed". This condition does not describe the relevant counterfactual world - if Vic had not acted, then GWWC and ACE would still have existed, but according to this proposition GWWC and ACE do not exist. The problem is the aggregation principle, not the method of estimating counterfactual impact.
This has obvious applications when assessing the social benefits of voting. Suppose that there is an election and Politician A would produce $10bn in benefit compared to Politician B. The election is decided by one vote. (For simplicity suppose that B wins if there is a tie.) Emma cast a decisive vote for option A and therefore her counterfactual impact is $10bn. It is correct to say that the counterfactual impact of each other A voter is also $10bn.
The alternative approach (which I argue is wrong) is to say that each of the n A voters is counterfactually responsible for 1/n of the $10bn benefit. Suppose there are 10m A voters. Then each A voter’s counterfactual social impact is 1/10m*$10bn = $1000. But on this approach the common EA view that it is rational for individuals to vote as long as the probability of being decisive is not too small, is wrong. Suppose the ex ante chance of being decisive is 1/1m. Then the expected value of Emma voting is a mere 1/1m*$1000 = $0.001. On the correct approach, the expected value of Emma voting is 1/10m*$10bn = $1000. If voting takes 5 minutes, this is obviously a worthwhile investment for the benevolent voter, as per common EA wisdom.
Assessing the impact of EA organisations
When evaluating their own impact, EA organisations will often face this issue. Community orgs like CEA, REG, and Founders Pledge will recruit new donors who donate to charities recommended by GiveWell, ACE and others. How do we calculate the counterfactual impact here? If I am right, in the manner above. This does mean we should be careful when making claims about the collective impact of EA as a whole movement. It would be a mistake to aggregate the counterfactual impact of all the EA orgs taken one by one.
Assessing the impact of policy organisations
For most policy advocacy campaigns, numerous actors are involved and in some cases numerous organisations will be a necessary condition for some particular policy change. Notwithstanding difficulties in finding out which actors actually were necessary conditions, their counterfactual impact should be calculated as per the above methodology.
A note of caution: leveraging and funging
The approach I have outlined needs to be applied with care. Most importantly, we need to be careful not to confuse the following two counterfactual comparisons:
Comparison 1 (correct)
Actual World: A, B, and C all act.
Counterfactual world (A): A doesn’t act, B and C act as they would have done if A had not acted.
Comparison 2 (incorrect)
Actual World: A, B, and C all act.
Counterfactual world (A): A doesn’t act, B and C act as they did in the actual world.
Confusing these two comparisons can lead one to neglect leveraging and funging effects. Organisations can leverage funds from other actors into a particular project. Suppose that AMF will spend $1m on a net distribution. As a result of AMF’s commitment, the Gates Foundation contributes $400,000. If AMF had not acted, Gates would have spent the $400,000 on something else. Therefore, the counterfactual impact of AMF’s work is:
AMF’s own $1m on bednets plus Gates’ $400,000 on bednets minus the benefits of what Gates would otherwise have spent their $400,000 on.
If Gates would otherwise have spent the money on something worse than bednets, then the leveraging is beneficial; if they would otherwise have spent it on something better than bednets, the leveraging reduces the benefit produced by AMF.
Confusing the two comparisons can also lead us to neglect funging effects. Suppose again that AMF commits $1m to a net distribution. But if AMF had put nothing in, DFID would instead have committed $500,000 to the net distribution. In this case, AMF funges with DFID. AMF’s counterfactual impact is therefore:
AMF’s own $1m on bednets minus the $500,000 that DFID would have put in plus the benefits of what DFID in fact spent their $500,000 on.
The effect of funging is the mirror image of the effect of leveraging. If DFID in fact spent their $500,000 on something worse than bednets, then the funging reduces AMF’s benefit; if DFID spent the $500,000 on something better than bednets, then the funging increases AMF’s benefits.
Thanks to James Snowden and Marinella Capriati for discussion of some of the ideas developed here.
 The ideas here are James Snowden’s