Comment Permalink

When you say your aim is

I can very honestly, and with a lot of authority, tell a potential donor that if they donate to a particular charity, they will avert at least one DALY or save at least 1 child's life for the given cost

I think you're looking for a level of certainty that basically doesn't exist in any of the top recommendations, whether by GiveWell, GWWC, etc let alone the higher-EV lower-certainty giving opportunities (policy reform, field-building, etc), and that the kinds of adjustments you make won't get you where you want. You can and should argue that a particular giving opportunity is great based on multiple lines of reasoning, but you can't make your exact claim as phrased (apologies if the following may seem nitpicky — I used to require the same sort of certainty too, so consider it an attempt to legibilize why I changed my mind).

The reasons for this are hard to summarise in a persuasive way if you haven't built intuition by scrutinising a (good) CEA. You can get a sense of them by picking a charity everyone agrees is a long-running top recommendation for cost-effectiveness + evidence robustness + track record (proxying your search for authoritative certainty), e.g. the Against Malaria Foundation, and seeing what GiveWell says about them. So let's do that:

it turns out that the final cost-eff estimate depends a lot on which region you're looking at (from 4.8x cash in Chad to 22.6x in Guinea), so let's pick the DRC (14.5x) to illustrate
GW begin with a $1M grant size as placeholder value, which in the DRC covers 94.49% of total costs with the small remainder covered by the government, yielding $1,058,303 total spending by all contributors (it's more complicated than this, see this cell's note, but the complication doesn't affect the bottomline)
this spending leads to the distribution of 180,896 insecticide-treated nets at $5.85 per net, but GW calculate here using data from the Pryce et al 2018 meta-analysis that only about 70% of them were used in trials, and GW discounts this further by -10% in program contexts out of skepticism (trial-to-program adjustment is quite complicated, see GW's analysis here), so their best guess is only 113,965 (63%) of those 180,896 nets actually end up being used. This varies a lot by region already, e.g. in Chad (whose cost-eff being ~1/3rd of the DRC's above might make you think is due to fewer nets being used) GW actually estimates 167,299 nets used (+47% more than in the DRC)
GW's analysis indicates that ~2.15 people sleep under each net in the DRC (this varies by region too, going down to 1.72 in Togo for instance), so those actually-used nets above cover 245,154 people. Another very complicated calculation with a plethora of best-guesses adjustments yields an estimate of ~1.2 effective coverage years provided per net (lower than in many other regions which get ~2.0 ec-years), hence 292,350 person-years of coverage
~16% of DRC's national population is under-5 children, so GW assumes ~16% or 45,449 of those person-years are for under-5s, for whom benefit estimation is calculated separately because child mortality rate is much higher than in older groups — so much so that GW estimates 81% of the lives saved (and 87% of the mortality reduction benefit) are from these 16% of beneficiaries. As an aside, given this context you'll be glad to hear that GW estimates 59.1% of under-5 children sleep under nets in the DRC (varying from 31-80% in the regions AMF operates that GW analysed) using data from the Malaria Atlas Project combined with GW's ITN coverage estimates
GW estimates a 56% expected reduction in malaria incidence among children who sleep under nets in the DRC. This comes from Pryce et al 2018's 64% implied reduction with 3 further discounts: -5% for internal validity ("to account for the possibility that the summary effect reported in Lengeler's 2004 meta-analysis might not have been the true effect ITNs had on the populations being studied"), -5% for external validity (to account for differences between trials and program implementation — see this Asterisk essay for a particularly stark example of how different scale-up results can be, and note that GW still assign a discount out of general skepticism even after separately discounting for their 3 major external validity concerns of net usage, country-specific mortality rates, and insecticide resistance), and -4% for insecticide resistance (in yet another complicated calculation that shows how region-dependent net effectiveness reduction due to resistance is, going as high as -59% in Chad which partly explains its low cost-eff)
GW estimates that in the absence of nets, the total under-5 mortality rate attributable to malaria alone would be 0.80%. This is full of adjustments too: GW start with the fact that in the DRC 0.31% of under-5 deaths in 2019 were directly attributable to malaria (40,985 deaths among 13.4M children), adjust this up +75% to 0.53% to account for indirect malaria deaths (GW's complicated reasoning concludes "we're highly uncertain about the exact value of this input", but it's more skeptical than the best literature implies and is roughly the midpoint of expert opinion), adjusts for deaths averted due to seasonal malaria chemoprevention or SMC programs (delivered by another top GW charity) to avoid double-counting, and finally adjusts for the 59.1% net coverage in the DRC mentioned earlier
combining everything together: $1M GW grant → $1,058,303 total spending including DRC government funds mobilised → 180,896 nets distributed → 113,965 nets used → 245,154 people covered → 292,350 person-years of effective coverage → 45,449 person-years of under-5 coverage → 203 under-5 deaths averted. A similarly adjustment-filled calculation yields 48 deaths averted among older people, so 251 total deaths averted (this is only halfway through GW's main CEA calc but the rest won't be germane to the point)
Naively we might say based on this that a $3,989 donation saves one life in the DRC, and that this varies from $1,700 in Guinea to $14,493 in Chad, but after seeing the kinds of adjustments that GW had to make above (do dig into those links to get a better sense of what I mean), I'm not sure you would be comfortable telling prospective donors that (say) giving $15,000 to AMF would save "at least one child's life". All you can say is that it would do so in expectation, and that the perspective shift is worth making as there are good reasons for giving (and doing good) guided by an expected value lens, while also being aware of this lens' shortcomings

Notice how difficult it is to get the level of certainty you seek in every adjustment step of the calculation above, despite GW's almost pathologically-skeptical thoroughness. This doesn't mean AMF is "a bad recommendation" — it means virtually every other top giving opportunity is "worse" (by the standard you seek) because giving well is really hard, yet we still have very good reasons to think that AMF and all those other top recommendations are excellent giving opportunities, so you should recalibrate your expectations and communicate this to potential donors as well who seek the same.

re: the "multiple lines of reasoning" alternative for recommending great giving opportunities, GiveWell's approach is solid I think. For AMF that looks like

We recommend AMF because of its:
Implementation of a program (mass distribution of ITNs) that we think is very cost-effective (more in our separate report on ITNs).
Long track record of supporting campaigns in a number of countries. (More)
Processes for tracking other actors’ malaria spending in the countries where AMF works, giving us more confidence that it is filling genuine funding gaps. (More)
Transparency – AMF shares significant information about its work with us. (More)

and GW detail their reservations as well (same link).

Showing 3 of 5 replies (Click to show all)

NickLaing

Mar 20

Nice one David! Mo has done as great job explaining here do I won't add too much. To reiterate the costs of overheads are built into all that estimations, so I don't think you need to worry about that as a separate issue. I love your intuition on family planning, and there are some great Organizations like FEM. And Lafiya Nigeria which have solid first effectiveness analysis showing they in probably have similar cost t effectiveness to other to charities, spend very little on overhead and help women in Northern Nigeria to take control of their family's future and improve women's health. I love the way you are thinking and all the great on your donation journey.

David Hammerle

Mar 19

I've only spoken to a few donors, mostly friends and family. They seemed to especially like the idea of donating to charities that are local to their communities, which most highly cost-effective charities don't, because they are mostly in low income places, not the USA. They were also concerned with what portion of the donated money would be spent on the charity's own overhead or administration. And they especially didn't want to receive spam mail or email from charities as a result of donating. I suppose my assumption is that I should start with an argument I think would work on myself, if I hadn't studied global health as much as I have, and then try to figure out what part of that people are resistant to. If I hadn't studied global health, and wanted to be able to donate effectively without having to study it, I would have been looking for that upper limit with a likelihood of somewhere around 95% that the cost would be below it, even if the upper limit needed to be pretty high in order to reach that level of certainty. So now I'm trying to figure out such an upper limit myself.

Mo PuteraMar 212

I again want to say that I resonate a lot with what you're trying to do, since I've tried (and mostly failed) to do something similar myself before.

I worry a bit that you're prematurely optimising approach-wise when you conclude the thing to focus on is figuring out the cost-effectiveness upper limit at which you can tell people honestly and confidently that their donation is doing that much good, instead of asking donors how they think about giving (which you did). For instance, Sawyer's comment reiterates the sentiment I mentioned earlier that ... (read more)

See in context

Estimating Global Health Overhead Costs

by David Hammerle

Mar 153 min read 6

5

Cause prioritizationEffective givingGlobal health & developmentAdjusted life yearCharity evaluationCost-effectiveness analysis

Frontpage

I often focus on trying to improve my ability to estimate how much it would cost to avert a DALY or to save the life of a kid under 5 years old by donating to a particular charity. I am especially trying to find an upper limit to these amounts, so that I can very honestly, and with a lot of authority, tell a potential donor that if they donate to a particular charity, they will avert at least one DALY or save at least 1 child's life for the given cost. I want to find such an upper limit so that there can be somewhere around at least a 95% chance that it is correct.

I think a lot of people that might be interested in donating to highly cost-effective charities don't end up doing that, because they just don't trust those charities and the estimates for how cost-effective they are. And, I think trying to find better upper limit values for this kind of thing could help improve the argument for giving money to these kinds of highly cost-effective charities.

As part of this, I recently started trying to figure out how to factor in the costs from various effective altruism or global health organizations that aren't directly doing charity work or even directly giving money to the charities that are directly doing the work, but that do work that facilitates the work that those charities do. For example, the WHO seems to do a lot of work that makes it possible for lots of charities to do their work, so it seems likely that I should try to account for those costs for the WHO.

So far, the best method I have come up with for estimating these kinds of costs is to:

Add up the money spent on overhead activities like this.
Compare it to the total impacted charity money that is flowing through the charities and organizations that are being helped by those overhead activities.
Figure out a good estimate of an upper limit on how much good all the impacted charity money together does in dollars per DALY.
Multiply the dollars per DALY by the amount spent by the organization and divide by the amount flowing through the applicable charities being helped. This should give you a final number of dollars per DALY of overhead cost from these types of organizations (like the WHO).

For example, doing this calculation for the WHO:

Start with the budget of the WHO for both sub-Saharan Africa ($496 million) and the HQ ($597 million) added together ($1,093 million).
Find the total impacted money flowing through the impacted organizations. I'm estimating this to be $114 billion because that is about how much the world spends on health interventions in sub-Saharan Africa.
Figure out an upper limit of cost per DALY for health interventions in sub-Saharan Africa. It seems like $150/DALY is a good estimate for this.
Multiply dollars per DALY ($150/DALY) by amount spent by WHO ($1,093 million) and divide by amount flowing through impacted organizations ($114 billion). That comes out to $1.44 per DALY that is overhead costs from the WHO.

Thus far I'm thinking it might be good to account for these kinds of costs from the following organizations: WHO, GiveWell, Rethink Priorities, Giving What We Can, The Life You Can Save, The Centre for Effective Altruism, 80,000 Hours

So I'm wondering:

Does anyone have any ideas on how I could improve this calculation?
Or which organizations I should include?
Or other kinds of costs that I should try to account for?

And also:

Does anyone have any arguments for why this calculation is unnecessary?
And why it is only necessary to include the costs of the charity itself?
Or how estimates from organizations like GiveWell are already accounting for these types of costs somehow?
Or any other feedback?

5 Reactions

More posts like this

Comments6

Sorted by

New & upvoted

Click to highlight new comments since: Today at 6:20 AM

Mo PuteraMar 1614

When you say your aim is

I can very honestly, and with a lot of authority, tell a potential donor that if they donate to a particular charity, they will avert at least one DALY or save at least 1 child's life for the given cost

it turns out that the final cost-eff estimate depends a lot on which region you're looking at (from 4.8x cash in Chad to 22.6x in Guinea), so let's pick the DRC (14.5x) to illustrate
GW begin with a $1M grant size as placeholder value, which in the DRC covers 94.49% of total costs with the small remainder covered by the government, yielding $1,058,303 total spending by all contributors (it's more complicated than this, see this cell's note, but the complication doesn't affect the bottomline)
this spending leads to the distribution of 180,896 insecticide-treated nets at $5.85 per net, but GW calculate here using data from the Pryce et al 2018 meta-analysis that only about 70% of them were used in trials, and GW discounts this further by -10% in program contexts out of skepticism (trial-to-program adjustment is quite complicated, see GW's analysis here), so their best guess is only 113,965 (63%) of those 180,896 nets actually end up being used. This varies a lot by region already, e.g. in Chad (whose cost-eff being ~1/3rd of the DRC's above might make you think is due to fewer nets being used) GW actually estimates 167,299 nets used (+47% more than in the DRC)
GW's analysis indicates that ~2.15 people sleep under each net in the DRC (this varies by region too, going down to 1.72 in Togo for instance), so those actually-used nets above cover 245,154 people. Another very complicated calculation with a plethora of best-guesses adjustments yields an estimate of ~1.2 effective coverage years provided per net (lower than in many other regions which get ~2.0 ec-years), hence 292,350 person-years of coverage
~16% of DRC's national population is under-5 children, so GW assumes ~16% or 45,449 of those person-years are for under-5s, for whom benefit estimation is calculated separately because child mortality rate is much higher than in older groups — so much so that GW estimates 81% of the lives saved (and 87% of the mortality reduction benefit) are from these 16% of beneficiaries. As an aside, given this context you'll be glad to hear that GW estimates 59.1% of under-5 children sleep under nets in the DRC (varying from 31-80% in the regions AMF operates that GW analysed) using data from the Malaria Atlas Project combined with GW's ITN coverage estimates
GW estimates a 56% expected reduction in malaria incidence among children who sleep under nets in the DRC. This comes from Pryce et al 2018's 64% implied reduction with 3 further discounts: -5% for internal validity ("to account for the possibility that the summary effect reported in Lengeler's 2004 meta-analysis might not have been the true effect ITNs had on the populations being studied"), -5% for external validity (to account for differences between trials and program implementation — see this Asterisk essay for a particularly stark example of how different scale-up results can be, and note that GW still assign a discount out of general skepticism even after separately discounting for their 3 major external validity concerns of net usage, country-specific mortality rates, and insecticide resistance), and -4% for insecticide resistance (in yet another complicated calculation that shows how region-dependent net effectiveness reduction due to resistance is, going as high as -59% in Chad which partly explains its low cost-eff)
GW estimates that in the absence of nets, the total under-5 mortality rate attributable to malaria alone would be 0.80%. This is full of adjustments too: GW start with the fact that in the DRC 0.31% of under-5 deaths in 2019 were directly attributable to malaria (40,985 deaths among 13.4M children), adjust this up +75% to 0.53% to account for indirect malaria deaths (GW's complicated reasoning concludes "we're highly uncertain about the exact value of this input", but it's more skeptical than the best literature implies and is roughly the midpoint of expert opinion), adjusts for deaths averted due to seasonal malaria chemoprevention or SMC programs (delivered by another top GW charity) to avoid double-counting, and finally adjusts for the 59.1% net coverage in the DRC mentioned earlier
combining everything together: $1M GW grant → $1,058,303 total spending including DRC government funds mobilised → 180,896 nets distributed → 113,965 nets used → 245,154 people covered → 292,350 person-years of effective coverage → 45,449 person-years of under-5 coverage → 203 under-5 deaths averted. A similarly adjustment-filled calculation yields 48 deaths averted among older people, so 251 total deaths averted (this is only halfway through GW's main CEA calc but the rest won't be germane to the point)
Naively we might say based on this that a $3,989 donation saves one life in the DRC, and that this varies from $1,700 in Guinea to $14,493 in Chad, but after seeing the kinds of adjustments that GW had to make above (do dig into those links to get a better sense of what I mean), I'm not sure you would be comfortable telling prospective donors that (say) giving $15,000 to AMF would save "at least one child's life". All you can say is that it would do so in expectation, and that the perspective shift is worth making as there are good reasons for giving (and doing good) guided by an expected value lens, while also being aware of this lens' shortcomings

re: the "multiple lines of reasoning" alternative for recommending great giving opportunities, GiveWell's approach is solid I think. For AMF that looks like

We recommend AMF because of its:
Implementation of a program (mass distribution of ITNs) that we think is very cost-effective (more in our separate report on ITNs).
Long track record of supporting campaigns in a number of countries. (More)
Processes for tracking other actors’ malaria spending in the countries where AMF works, giving us more confidence that it is filling genuine funding gaps. (More)
Transparency – AMF shares significant information about its work with us. (More)

and GW detail their reservations as well (same link).

David HammerleMar 175

Thanks for the response, Mo.

I have also spent a significant amount of time looking at cost-effectiveness analyses from GiveWell and I think they're great. I even wrote a summary of GiveWell's cost-effectiveness analysis for Against Malaria Foundation similar to yours, although I think yours might be better. Mine is on this webpage (https://davehammerlecoder.com/against-malaria-foundation) on my personal website under the section 'Appendix A: GiveWell's Cost-Effectiveness Analysis for the Against Malaria Foundation'.

I'm not so much concerned with finding a really accurate measure of the cost per DALY averted or life saved as I am with finding an upper limit, even if the upper limit is pretty far above the estimated average. For example, looking at the Against Malaria Foundation, according to GiveWell, it looks like the upper end cost to save a life would be $14,493, specifically in Chad. If that's the upper end of GiveWell's estimate, then it seems like $50,000 cost per life saved might be a pretty reasonable upper limit. Meaning it seems plausible to me that someone could have a 95% confidence that AMF will save a life for less than $50,000 on average.

And if I were a normal person living in a rich country like the US, considering donating some money, saving a life for $50,000 would seem like a really good deal to me. Even if saving that life only averts 35 DALYs, that would translate into ~$1,400 per DALY or ~$4 per day of life added. So if I work an extra 1 hour for about $30/hour, I can extend someone's life by 7 days, or 168 hours. Which means that for whatever span of time extra I spend working, somebody else gets to live for ~168 times that span time. From an altruistic perspective, that's an amazingly good deal.

I can also definitely understand that being able to donate money and have it be as cost-effective as it probably is, is very impressive. I'm sure it takes a lot of work from a lot of different people in order to make that possible, including the people at GiveWell and many government entities, both African and not.

But it took me a lot of time and effort, including a bachelor's degree and a trip to EA Global in Boston to attain the degree of confidence that I currently have in my donations. To me it seems like maybe we could get more normal people interested in donating if it didn't require as much free time and effort.

I think maybe the reason most people don't want to donate to these kinds of charities is that they don't consider charities trustworthy enough. And that seems like a reasonable response to me, especially considering the kinds of ads from African charities pop up in my web browser. But I think if we can get a more consistent message out to more normal people who don't want to spend as much time on this, donating money to global health charities could become more normal in rich countries like the USA.

I think in order to accomplish that, we need a message that is consistent, and that can be verified by people who decide to really look into it. This way, normal people would have more of a reason to trust highly cost-effective global health charities. And I think a fairly consistent upper limit on the cost to avert a DALY or save the life of a kid under 5 via any given charity could help with this.

I want to account for the money spent by organizations like GiveWell, the WHO, the Centre for Effective Altruism, and Open Philanthropy in an attempt to improve these upper limits. I'm especially concerned about these costs because I recently started donating to family planning global health charities. These charities can be viewed as highly cost-effective just by reducing maternal mortality and from various other positive impacts.

But the thing I really like about family planning is that it can avert births.
Each averted birth avoids the 7.4% chance kids in sub-Saharan Africa have of dying before the age of 5 years old.
Looking at it this way, these charities can avert the death of a kid under 5 years old for under $1000, and probably substantially less.
The amount depends on odd things like which utilitarian theory one subscribes to. I go by an average-based utilitarian theory.
If you want more info about that, my post on the EA forum about it is here: https://forum.effectivealtruism.org/posts/stWZ26t9W7qi3ieJS/why-aren-t-relocated-births-accounted-for-in-cost

My problem now is that, at that degree of cost-effectiveness, costs from organizations like the WHO and GiveWell become much more significant relatively. And that's why I started trying to figure out how to estimate applicable costs from those kinds of organizations.

NickLaingMar 204

Nice one David! Mo has done as great job explaining here do I won't add too much.

To reiterate the costs of overheads are built into all that estimations, so I don't think you need to worry about that as a separate issue.

I love your intuition on family planning, and there are some great Organizations like FEM. And Lafiya Nigeria which have solid first effectiveness analysis showing they in probably have similar cost t effectiveness to other to charities, spend very little on overhead and help women in Northern Nigeria to take control of their family's future and improve women's health.

I love the way you are thinking and all the great on your donation journey.

Mo PuteraMar 194

Thanks for the considered response David, upvoted. I'll preface by saying that I appreciate the effort you're putting into this.

When I first read your post, I anchored hard to "... so that I can very honestly, and with a lot of authority, tell a potential donor..." which made me assume you've spoken to potential donors, which I resonated with. I've only spoken to potential small local donors in a very limited personal capacity, and a recurring concern they raise is that they simply don't want to be misled by claims of effectiveness (forget cost-effectiveness). I recall speaking to someone who felt disquieted by not just the number but also the kind of uncertainty-driven adjustments I listed above for instance, and also appreciative of me pointing them out. I recall speaking to another person who was disquieted by the nature of helping in statistical expectation, when what they wanted was certainty; they appreciated me pointing this out as well (and were unpersuaded by the arguments for expectation-oriented giving).

This disquiet is of course a personal matter, very case-by-case; instead of guessing you should just ask potential donors. In this context your comment makes me wonder if you've spoken to any, and if so what did they think of your approach? To be more direct: when you say things like "it seems plausible to me that someone could have a 95% confidence", is that from talking to them or just your own guesses for now?

David HammerleMar 193

I suppose my assumption is that I should start with an argument I think would work on myself, if I hadn't studied global health as much as I have, and then try to figure out what part of that people are resistant to. If I hadn't studied global health, and wanted to be able to donate effectively without having to study it, I would have been looking for that upper limit with a likelihood of somewhere around 95% that the cost would be below it, even if the upper limit needed to be pretty high in order to reach that level of certainty. So now I'm trying to figure out such an upper limit myself.

Mo PuteraMar 212

I again want to say that I resonate a lot with what you're trying to do, since I've tried (and mostly failed) to do something similar myself before.

Most potential donors are not really risk neutral, and would rather spend $5,001 to definitely save one life than $5,000 to have a 10% chance of saving 10 lives. Risk neutrality is a totally defensible position, but so is non-neutrality. It's good to have the option of paying a "premium" for a higher confidence (but lower risk-neutral EV).

and Jason's comment seems relevant as well:

Orthogonally, I think most people are willing to pay more for a more legible/direct theory of impact.
"I give $2800, this kid has lifesaving heart surgery" is certainly more legible and direct than a GiveWell-type charity. In the former case, the donor doesn't have to trust GiveWell's methodologies, data gathering abilities, and freedom from bias.

and these aren't necessarily obvious in advance, so if you start from a (simplistic) model of giving advisory effectiveness as being [number of prospective donors in your circle] x [fraction of donors who find argument X persuasive] x [$ per donor] x [expected good per $], then starting with an argument that works on yourself (and on me too — I don't think we're that representative of the donor pool) doesn't let you get empirical input on the 2nd term, while asking donors does.