Comments
The Attribution Moloch
Summary: I argue that sufficient resource scarcity can exacerbate the effects of tiny differences in value alignment to the point where charities with almost identical goals will compete rather than cooperate. Further, a skewed perception of how impact is created as well as mere ignorance can cause prioritization to aggravate failures of coordination. (Reposted from my blog.)
There’s something that I’m worried about that is fairly close to the core of EA. I don’t know to which degree I should be worried, and I don’t know how pervasive it is, but nonprofit insiders have written about it and other insiders have told me about it with regard to specific cause areas, and I’m afraid prioritization that ignores it may exacerbate it.
Many thanks to Lukas Gloor, Melanie Joy, Sara Nowak, Jacy Reese, and Anne Wissemann for proofreading the draft of this article and adding many valuable thoughts!
I don’t want to imply that any specific space suffers from this problem, so here’s one that I made up.
Everyone agrees that paperclips are not treated with the dignity that their charming smiles deserve. Few people, unfortunately, do something about it. Not so with us. Five years ago I founded Clippy Guards International (CGI). I found a small number of donors that are passionate about paperclips and donate to CGI so that I could employ two of CGI’s most diligent volunteers full time, Erin and Pat.
Recently acquaintances of ours founded a second organization, the No Staples Association (NSA), with the express goal of instituting a staple tax and banning staple ads – a promising but risky idea we had considered as well that is sure to increase demand for paperclips if it succeeds. If we had more resources, we could afford to take more risks, but as it is, we rather decided to rely on more incremental interventions so we don’t lose donors if a risky campaign fails.
The new organization is prominent in the social media circles that our donors read, and I’ve heard talk that some of our donors are interested in it. The week after NSA published its strategic plan for 2016, two of our donors cancelled their recurring donation orders to us.
Today Alice, the executive director of the No Staples Association, emailed us. NSA’s campaign is at an impasse, but Alice heard through the grapevine that we’re in touch with several key politicians. If I were to put Alice in touch with them, she’d have a shot at convincing them. I know Alice and know that she’s competent and won’t ruin our rapport with the politicians, I think that NSA’s campaign is highly promising, and I know that all of us only want the best for all the Clippies of the world.
But if NSA succeeds, more of our donors will reconsider their donations. On our tight budget that means that one of us will have to agree to leave our organization or work part-time. Erin is the one many of the politicians know and trust. Pat is the public face of the organization for our donors; if he left, we could lose even more donors. If I, the founder, were to leave, surely the organization would founder and donors would consider that I may not believe in it anymore.
I’m really sorry for NSA, and I wish there were another way, but it’s either us or them, and our campaigns are the less risky ones, so I vote for us. I deleted Alice’s email from our shared account before Erin and Pat could see it. I’m so sorry, Alice!
This is what I call the Attribution Moloch – in reference to Scott Alexander’s Moloch, a personification of defection in the prisoner’s dilemma–like situations that makes everyone worse off.
The basic idea of the prisoner’s dilemma is the following (adapted from Douglas Hofstadter):
Two people meet [without being able to identify each other] and exchange closed bags, with the understanding that one of them contains money, and the other contains a purchase. [The seller desires the money more than the product, and the buyer desires the product more than the money.] Either player can choose to honor the deal by putting into his or her bag what he or she agreed [can cooperate], or he or she can defect by handing over an empty bag. [Emphasis added.]
The best situation for both, together, is to realize their mutual gains from trade by cooperating, but it is rational for either to defect, since (1) they may get the money and the purchase, and (2) it is rational for the other to defect, so that each expects to lose money and purchase if they cooperate.
I think resource scarcity is not the only problem facing our two paperclip charities. Another problem is lacking coordination1 and a third is attribution.
To explain the problem with attribution – crediting agents for impact – I need to reference the concept of the INUS condition that John Mackie introduced in Causes and Conditions.
According to Mackie, what we typically think of as causes of an effect are really clusters of factors, each of which is necessary but insufficient to bring about the effect. Together this cluster of factors is sufficient but not necessary for the effect since other such clusters may have had the same effect.2 One typical example:
So to say that short circuits cause house fires is to say that the short circuit is an INUS condition for house fires. It is an insufficient part because it cannot cause the fire on its own (other conditions such as oxygen, inflammable material, etc. should be present). It is, nonetheless, a [necessary] part because, without it, the rest of the conditions are not sufficient for the fire. It is just a part, and not the whole, of a sufficient condition (which includes oxygen, the presence of inflammable material, etc.), but this whole sufficient condition is not necessary, since some other cluster of conditions, for example, an arsonist with gasoline, can produce the fire.
What donors do all too often when extrapolating from average cost-effectiveness to marginal cost-effectiveness is that they partition impact: They see some of the factors that were necessary for the impact and credit these factors with some fraction of the impact so that all fractions add up to one. But none of the impact would have manifested if even a single one of the factors had been missing, so there is no valid partition of the impact.3
School-based mass deworming programs, for example, are only as effective as they are because there are already schools and teachers in place. If there were enough teachers, no worms, and no schools, building schools would be more effective (or advocating for it on government level).4
The Ultra Poor Graduation Program is an explicit attempt at providing several factors following the realization of their necessity but individual insufficiency:
The Ultra Poor Graduation program is designed to graduate ultra poor households out of extreme poverty to a more stable state. This 24-month program provides beneficiaries with a holistic set of services including: livelihood trainings, productive asset transfers, consumption support, savings plans, and healthcare. By investing in this multifaceted approach, the program strives to eliminate the need for long-term safety net services.
But when we partition impact as if one of the factors were necessary and sufficient, then we greatly overestimate the difference in overall cost-effectiveness between the final factor and the factors that paved the way for it for three reasons:
When we’re looking for cost-effective interventions, we’re looking for the currently worst bottlenecks to creating impact and the ones that can be reduced most efficiently. We’re not looking for an intervention that can – by some artificial algorithm or other – be attributed with the greatest impact.
When we, as funders, make that mistake, we encourage charities to game the algorithm and hog credit rather than to bring about the best version of the future, and I know of no algorithm that could be used in practice to attribute impact in a way that would not create perverse incentives, that is, an algorithm that grants greater credit to an agent when the agent chooses actions that increase the overall impact compared to a counterfactual case where the agent chose a worse action. Hence (absent such an algorithm) we should abstain from attributing impact and think of top charities not as the charities that should be credited most highly (as the word “best” may imply) but as the charities that address the currently most compelling (i.e., important, tractable, and neglected) bottlenecks to achieving impact.6
As an example, consider an algorithm that attributes impact according to the staff time and budget invested by the agents: Two charities agree to fund half of the nets of a universal coverage campaign of long-lasting insecticide-treated mosquito nets. Universal coverage means that every bed is protected, which comes with additional valuable community-wide effects. The first charity buys the nets and ships them to the country. The other charity goes to some length to find the closest supplier to save shipping costs and makes a contract with the supplier that also covers a future distribution to get a greater bulk discount, resulting in 50% savings. The number and quality of the nets that the charities provide is identical, yet the second charity gets awarded only 33% of the impact. That’s not a good incentive structure.
Or consider an algorithm that attributes impact according to the capacities of the agents: In the case of school-based deworming mentioned earlier, schools, teachers, and students are in place, so a charity only needs to provide the pills. Great impact will be generated and the charity will get attributed a share of it. But if the charity heavily invests into building new schools right next to the old ones, hiring teachers for those schools, and luring over all the students from the old schools, it implements many of the factors that cause the impact itself. Hence it will get attributed more of the impact even though, compared to the counterfactual, it merely achieved the same impact at a greater cost.
The only attribution algorithm that I can think of that could avoid these problems is one that compares the actual outcome to the optimal outcome, determines the agent whose decision led to the degradation of the actual outcome, if any, and penalizes it proportionately. This approach is rendered more complex as soon as several agents make less than optimal decisions, but even apart from that, the practical impossibility of knowing the optimal outcome and determining the decision that prevented it renders it purely hypothetical.
Who, for example, is ever going to find out about the email that the founder of CGI deleted in the introductory tale? CGI has impact-minded donors who donate to the best value-aligned charity they can find, but it’s opaque to them whether the “best charity” holds that position in the ranking fairly because of the impact it achieves or whether it hold the position unfairly because of the impact it prevents its “competitors” from achieving, thereby decreasing the overall impact. Such is the nature of rankings, which I am subsequently to critique.
Prioritization that does not heed the above problem and does not go to some lengths to avoid it, will exacerbate it.
GiveWell need not worry though. Its top charities live in a postscarcity world thanks to Good Ventures and they have no reason to compete for funding. They also have fairly different programs and thus inhabit fairly different cause areas, reducing any potential for competition (and maybe also the potential for cooperation). But as we create more impact-minded donors and thus incentivize more charities to prioritize impact, comparative research into charities and the number of funding gaps that are compared will increase. And that’s awesome!
But resource scarcity will tend to increase among the majority of charities that are not considered top charities and thus lose some donors to the top charity group. Hence, naive prioritization that does not incentivize cooperation sufficiently can instead incentivize many kinds of defections among potential top charities, which can be very inconspicuous especially when they consist of omissions rather than actions. Such inconspicuous defection may include the following:
The total impact will be much curtailed, but no one will know the counterfactual.
What’s worse, the organizations that defect so clandestinely will emerge as seemingly more effective than those they defect against, which will be rewarded through increased donation flows. Those that are defected against will see diminishing donation flows and, when taken to the extreme, will be discontinued until the whole space is one of organizations that defect against each other at every opportunity – the worst case for everyone involved since really everyone cares about the agent-neutral impact.
These are open questions with preliminary ideas for answers. I would be happy to start a discussion on these questions.
Imagine, for example, that there were a sheriff who punishes anyone who hands over an empty bag. ↩
Fun fact: Peter Singer had taken over most of Mackie’s lectures so Mackie had time to write the book. ↩
And that is not even fully true because the impact may have manifested later due to another cluster of factors, but I think this type of replaceability is well understood in the movement. ↩
One might also credit the parents for sending their children to school, but the number of school children with intestinal worms is probably much greater than the number of children reached through school-based deworming, so that the parents’ contribution is mostly replaceable. If one parent hadn’t sent to school a child who benefitted from deworming, another child likely would have received the pill – though maybe with less elasticity than this example implies. ↩
Only if these experiments were conducted by the same organization that finally succeeded, the status quo is probably already to consider them. ↩
As an aside, impact certificates are probably also just as good incentives as the buyer makes them so that naive buyers can create perverse incentives just like normal funders. They may also send the often incorrect message that one necessary but insufficient factor owns part of the impact to sell it. That problem may be solved with different phrasing or disclaimers, but actual pricing according to an organization’s or person’s achievement would require perfect knowledge of the hypothetical optimal outcome and its mechanics, whose actions or omissions have led to degradations compared to that outcome, and the degree of degradation that all combinations of actions and omissions incur – which seems infeasible. ↩
One danger of rewarding cooperation is that sometimes it’s clear from the start that some cooperative effort would only come at a cost for one organization without benefit for anyone. In such a case an organization should not be incentivized to sacrifice its time for improved reputation. Some sort of Whuffy- or PageRank-based scoring can alleviate this problem by giving organizations with little reputation less power to award reputation. Such a system may suffer from some of the problems of startup valuation, since the reputation of new organizations will be based on hopes rather than track records and will thus be much less reliable. Since social status suffers from the same and worse limitations, the system may still be an improvement over the status quo. ↩
Great post, and indeed something to think about.
A specific solution that might be easier and more practical than Whuffies would be for organizations to write endorsements of each other that describe how they cooperated, which can then be shown to donors as part of annual reports, etc. Then, donors would have a chance to reward cooperative organizations directly.