Earlier this year, GiveWell Experimental predicted that we, the team behind Charity Science Health (CSH), have a 15% chance of becoming a GiveWell top charity by giving season 2019. That sounds pretty cool. But what does it mean for our plans? How good is it to be a top charity? And how does having a 15% chance of achieving top charity status compare to other things we could be doing with our time?
While it would take a lot of time to treat this question rigorously, a rough initial sketch seems very prudent for guiding our staff plans as we enter 2017 and decide whether to continue as-is, scale up, or scale down, with a keen mind to what we could be doing instead. An initial analysis and back of the envelope calculation seemed like the best way to inform our intuitions on this topic, so we set out to make one.
Our estimate is based on assuming that GiveWell top charities have impact according to four factors: (1) funding from GiveWell (both from Good Ventures and directed by the interest of GiveWell donors, which will likely add up to $108M a year or more), (2) funding from non-GiveWell sources (e.g., other foundations), (3) how cost-effective they are, and (4) the total room for more funding of the organization.
Funding from GiveWell
Between 2011 and 2015, the four top GiveWell charites collectively received $166M, with $108M coming from just 2015. If we divide the $166M over five years evenly among all four charities, that’s $8.3M per year per charity. A 15% chance of us achieving this is thus worth an expected value of $1.3M a year, which divided among three full-time employee equivalents (FTE) is $433K in expected value per person.
However, GiveWell is moving a lot more money now than it used to, especially with Good Ventures (though due to an unusually large 2015 grant from GiveWell to GiveDirectly, the trend may be exaggerated). Additionally, there’s somewhat of a power law even within GiveWell top charities. In 2015, GiveDirectly got ~50% of all the funding, the Against Malaria Foundation (AMF) got 35% of the funding, and SCI only got 3% of the funding. While we have a 15% chance of being a top charity, there’s still a good chance we’d end up on the lower half and get only 3%.
So let’s assume now that instead of it being the 2011-2015 track record of $166M over five years (averaging to $33.2M per year), giving instead continues linearly at roughly $80M a year. We’ll also assume that we’ll break up our 15% chance into a 5% chance of getting into the top three (along with AMF and GiveDirectly) and receiving roughly one third of the funding and a 10% chance of getting ~5% of the funding like Deworm the World and SCI. 5% chance of 33% of $80M/yr plus a 10% chance of 5% of $80M/yr is $1.32M/yr + 400K/yr = $1.72M/yr or $344K per person per year.
Lastly, it seems quite plausible that the addition of a new GiveWell top charity would bring in more overall money to GiveWell top charities. At minimum, an eighth top charity would likely lead Good Ventures to make an additional incentive grant of $2.5M that they would not have otherwise made. However, it also seems plausible to me that a new charity, especially one focused on something other than malaria or deworming, would bring additional interest to GiveWell from new donors or old donors interested in new areas. If we thought this effect would, together with the additional Good Ventures grant, boost the total money moved by an additional $10M, this would boost our expected of $344K per person per year to $364K per person per year.
Comparative cost-effectiveness
Another major factor is how cost effective we will be, relative to other GiveWell top charities. For example, if we take GiveWell’s 2016-2017 estimates literally, AMF is ~4x as cost-effective as GiveDirectly (though I expect GiveWell would kindly ask us not to take these estimates literally).
While our current estimates don’t place us to be as cost-effective as AMF, we think we have a good shot of being more cost-effective than GiveDirectly (GD). If we could even be 2x GD (or 0.5x AMF) and shared 50/50 with GD for money from GiveWell’s sources, we’d be effectively doubling the impact value of that share.
To make this more concrete, GD got $19M in 2015 excluding Good Ventures and AMF got $15M. If we adjusted those numbers to the literal cost effective estimates and did not adjust for diminishing marginal returns, we might be able to say something roughly like the $19M to GD being worth the same as $4.75M to AMF, or 1.9M in cost-effectiveness adjusted dollars. The total among AMF and GD would then be 19.75M cost-effectiveness adjusted dollars.
If Charity Science Health was a 2x GD charity, and we still assumed that AMF got $15M but split the remaining money 50-50 between GD and CSH (So that GD got $9.5M and CSH got $9.5M), the new cost-effectiveness adjusted total (relative to AMF) would be 15M cost-effectiveness adjusted dollars derived from AMF’s impact, 2.375M cost-effectiveness adjusted dollars from GD and 4.75M cost-effectiveness adjusted dollars from CSH, for a total of 22.125 cost-effectiveness adjusted dollars. Thus the addition of CSH could be modeled as a net gain of 2.375M cost-effectiveness adjusted dollars, even though the total amount of money donated has not changed.
The more cost-effective the new top charity, the better. If CSH were instead the same as AMF and still split with GD 50/50, the net gain would increase from 2.375M cost-effectiveness adjusted dollars to 7.125M cost-effectiveness adjusted dollars.
Overall, while I think the concept of “cost-effectiveness adjusted dollars” is useful for sketching out one potential area of impact for CSH, I hestitate to take the notion literally and think that a lot more work would be needed to sketch out the concept more rigorously. In practice, however, comparing our internal rough guess cost-effectiveness estimate of Charity Science Health with a rough guess of the average cost-effectiveness of GiveWell top charities from 2016-2017, the two numbers seem to be roughly even, so I don’t expect any practical benefit from this factor.
Funding from sources outside GiveWell
While people in the EA movement may not think about it much, GiveWell and Good Ventures are not the biggest funders of non-profits out there. A potential GiveWell top charity could also hope to vie for the attention of other funders, like the Gates Foundation, the Lampert Family Foundation (funders of New Incentives), the Global Innovation Fund, YCombinator, and others. While the GiveWell stamp of approval could certainly help lead these funders to a charity, the best charities could certainly find funding independently from GiveWell’s endorsement.
A new GiveWell top charity could unlock hundreds of thousands, if not millions, of dollars from large funders, and it’s quite likely the money counterfactually would not have gone to as good of a charity (by GiveWell’s standards).
Room for more funding
It doesn’t matter if you can attract tens of millions of dollars if you cannot effectively spend tens of millions of dollars. For example, GiveWell determined that Deworm the World could only make use of ~$15.5M in “execution level 1 and 2” funding over 2017. We’re highly uncertain about our own potential for room for more funding in future years, so this could really cap our potential for impact even if we otherwise succeed. If we were only able to take in $5M, our ability to move $20M would not matter.
Staff allocation
Right now, we have all our high-talent “senior” staff allocated to running CSH. However, there could come a time when we could run CSH in a more “passive” mode, continuing to have nearly as high impact, though without seeking as strongly to scale. This is similar to what CSH staff have already done with our prior project of Charity Science Outreach, dropping the team from ~5.5 FTE to ~1 FTE while still continuing to move a good amount of money.
Thus, in a hypothetical future, perhaps CSH could spend $10M per year almost indefinitely while the CSH “senior” staff move on to start a different charity. If we assumed that “passive” mode lasted for a decade, the present value of that $100M over ten years would be worth $55.8M today (assuming a 6% interest rate), or $18.6M per current “senior” staff member (3 FTE), though we still would have to adjust for the counterfactual value of the non-senior staff members.
Conclusion
A new GiveWell top charity will have impact by (a) attracting some new funding through GiveWell that would not have otherwise been attracted, (b) being more cost-effective than current GiveWell top charities, and (c) by attracting new funding from non-GiveWell foundations. However, it is capped by a large chance of failure, room for more funding, staff time from people who could otherwise be doing other great things.
When I try to model all these factors out in the following Guesstimate model, I get an estimate that the total efforts of our team (including senior staff, non-senior staff, and volunteers) are roughly equal to, in expectation, the work of a team of full-time equivalent people each earning to give $400K a year to GiveWell top charities, with our 95% confidence interval ranging from $220K/yr to $720K/yr. This compares favorably, as $400K donated per year is, I think, higher than what a typically ambitious and skilled EA could be expected to earn to give and is much higher than the earning to give levels found in 80,000 Hours 2014 report.
However, more work has to be done to refine this estimate. For example, this does not take into account the tricky aspect of double counting, since our impact would be partially in moving the donations of others. Also, this does not separate out the fact that the counterfactuals and individual impact of all our staff and volunteers are definitely not the same. Using the same model, attributing 100% of the impact to senior staff and 0% of the impact to anyone else, shifts the estimate to $4.1M per senior staff year ($550K/yr to $25M/yr).
But overall, this makes founding a GiveWell top charity could be a very high earning (though also somewhat high variance) career choice. Hopefully this estimate will continue to be further refined as we learn more about our chances of success and failure!
-
Thanks to Joey Savoie, Justis Mills, and Marcus Davis for reviewing a draft of this essay.
Update 19 Dec: Thanks to Vipul Naik and Owen Cotton-Barratt for pointing out problems with the model. After taking their feedback into account, my estimate for total staff time (including volunteers) was revised from a mean of $470K/yr (95% interval: $150K/yr to $1.4M/yr) to a mean of $400K/yr (95% interval: $220K/yr to $720K/yr). The estimate for senior staff time was revised from a mean of $4.1M/yr (95% interval: $550K/yr to $25M/yr) to a mean of $3.7M/yr (95% interval: $700K/yr to $12M/yr).
Thanks also to Vipul Naik for spotting several typos.
Can you explain why attributing all impact to senior staff increases the width of the confidence interval (in log space)? I'd naively expect this to remove a source of uncertainty.
I had a quick look at the Guesstimate model, and what I think is going on is that you just have much wider error bars over how much senior staff time will be taken; but you include scenarios with negative senior staff time(!), which may contribute significantly to expectation of the value-per-year figure, but isn't very meaningful. Am I just confused?
That part is now fixed, but it doesn't look like it contributed meaningfully to the end calculation.
This doesn't look fixed to me (possible I'm seeing an older cached version?). I no longer see negative numbers in the summary statistics, but you're still dividing by things involving normal distributions -- these have a small chance of being extremely small or even negative. That in turn means that the expectation of the eventual distribution is undefined.
Empirically I think this is happening, because: (i) the sampling seems unstable -- refreshing the page a few times gives me quite different answers each time; (ii) the "sensitivity" tool in Guesstimate suggests something funny is going on there (but I'm not sure exactly how this diagnostic tool works, so take with some salt).
To avoid this, I'd change all of the normal distributions that you may end up dividing by to log-normals.
Okay, I've now done this.
Let me know if you think the model is better and I can update the post.
re (1), that is true because Guesstimate uses a Monte Carlo method with 5K samples I think.
re (2), I don't know how to read the sensitivity outputs well, but nothing looks weird to me. Could you explain?
I think this has removed the pathology. There's still more variation in this number, but that comes from more uncertainty about amount of senior staff time needed. If the decision-relevant question under consideration is "how many of these could we do sequentially?" then this uncertainty is appropriate to weight like this.
Thanks. I updated the post accordingly.
No, that's a good and valid question and I'm unsure of the answer. At first I thought it was because I had not properly separated out some of the calculations (e.g., I had a single cell that accounted for both number of years to take to become a top charity and number of staff years per year). But once I separated them out (see "Alternate Calculation" in the model), the uncertainty range is still larger for senior staff than all staff, even though the all staff calculation unambiguously includes all the values of the senior staff calculation and more.
So now I think it's either possible there's an error somewhere in my model or maybe some of the uncertainty are interfering with each other and cancelling out. I think this second one can happen when you divide two intervals.
Also, the fact that the senior staff number range includes a number close to 0 would create the potential for a very large tail of high impact, which would not be present on the total staff year model which does not include numbers close to 0.