## Effective Altruism ForumEA Forum

Global health and development

# Summary

In April 2022, GiveWell made major updates to their estimates of how cost-effective water chlorination is at improving well-being, with their most recent estimates indicating that such interventions are 4-9x as cost-effective as GiveDirectly. We examined GiveWell’s water quality intervention cost-effectiveness model for errors or methodological issues. We discovered five potential findings and developed approaches to correct for these findings as best as possible given limited available public data on some of these topics. Our major changes to the cost effectiveness analysis were:

• Redoing GiveWell's meta-analysis on the effect of water chlorination on all-cause mortality to include Haushofer et al, 2021. This increases the estimated mortality effect size by ~40%, but due to structure of GiveWell's analysis, has no independent effect on overall cost effectiveness.
• Using the studies on water chlorination and illness to calculate a full probability distribution for the effect of chlorination on all-cause mortality, not just a biologically plausible maximum. Together with the updated meta-analysis, this reduces the cost effectiveness of the interventions by ~7% for in-line chlorination and ~17% for Dispensers for Safe Water.
• Adjusting the moral weight calculations to account for deaths averted from non-waterborne illnesses. This reduces the cost effectiveness of the interventions by ~2%.
• Revising the cost estimate for in-line chlorination to use intervention-specific estimates rather than reuse cost estimates from Dispensers for Safe Water. This increases the cost effectiveness of in-line chlorination by ~13%.
• Discounting future spending and benefits to account for time delays between funding provision and health impacts. This reduces the cost effectiveness of the interventions by ~12% in the long term and ~5% in the short term.

Using our updated approach, the cost-effectiveness of in-line chlorination falls by 0.21% and the cost-effectiveness estimate of Dispensers for Safe Water falls by 23%. Our results are incorporated into an updated version of GiveWell’s water quality cost-effectiveness model spreadsheet, where cells with updated input data are highlighted in red.

# Updated Results

Our results are incorporated into an updated version of GiveWell’s water quality cost-effectiveness model spreadsheet. Cells with updated input data are highlighted in red, and a summary of the changes is included in the “Results Summary” tab. Tables 2-4 summarize our results, with cost-effectiveness given in multiples of the cost-effectiveness of cash transfers. All together, our changes lead to a negligible change in the cost-effectiveness of in-line chlorination and a decrease of 17-23% in the overall cost effectiveness of Dispensers for Safe Water. The decreases are larger for the long-term estimates than for the short-term estimates, and larger for Uganda and Malawi than for Kenya.

Table 2: Updated Cost-Effectiveness for In-Line Chlorination

Table 3: Updated Cost-Effectiveness for Dispensers for Safe Water (Short-Term)

Table 4: Updated Cost-Effectiveness for Dispensers for Safe Water (Long-Term)

Different mechanisms of benefit are affected in different ways by our revisions. Figure 2 shows how each of the five benefit areas considered by GiveWell change for each of the intervention areas. Benefits via under-5 deaths averted, over-5 deaths averted, and developmental effects decrease in all intervention areas. Benefits via medical costs averted increase in some areas and decrease in others. Morbidity benefits increase in all areas, but the changes are negligible compared to the benefits via other mechanisms.

Figure 2: Units of Value from Each Benefit Mechanism in the Original and Updated CEA (Short-Term).

# Discussion

Our evaluation of GiveWell’s water quality CEA identified five areas for improvement, which combined to reduce our estimate of the cost effectiveness of DSW by approximately 20% and had no overall effect on the cost effectiveness of ILC. This result is the opposite of what we expected when we started the project, as our initial exploration of the topic area made us think that GiveWell was likely underestimating the cost effectiveness of water chlorination. We were surprised as we investigated more thoroughly that our analysis showed the opposite.

While we found several areas of disagreement with GiveWell’s analysis, our examination of the CEA made us more convinced that the overall approach is sound and that water chlorination has a noticeable impact on all-cause mortality. Our five areas of improvement are all focused on improving how specific values within the CEA are calculated, rather than reconsidering the fundamental assumptions behind the CEA.

As two outsiders, we found GiveWell’s water quality CEA to be clearly explained and deeply researched. But at the same time, it is clear that the water quality CEA is newer and therefore less well developed than the CEAs for GiveWell’s top charities. The benefits from medical costs averted and development effects appear to have been investigated in much less detail than the mortality benefits, even though these two mechanisms account for an equal amount of value as the deaths averted.[11] Similarly, the downside adjustments and cost estimates appear more superficial than in GiveWell’s other CEAs. Many of our edits to the CEA involve replacing an ad-hoc adjustment with a more developed technique from other CEAs. This includes our discounting of future lives saved, which follows the technique from the New Incentives CEA, and our use of Bayesian analysis to combine evidence from multiple lines of reasoning, which is similar to the approach used in the Deworm the World analysis.

A 20% change in cost effectiveness is relatively modest compared to the range of cost effectivenesses considered by GiveWell. For comparison, the country-specific numbers in GiveWell’s other cost effectiveness estimates range from -0.5x as effective as cash to 51x as effective. A 20% change in cost effectiveness in either direction isn’t enough to make water chlorination GiveWell’s most effective charity or its least effective, nor would a 20% increase be enough to push either DSW or ILC over GiveWell’s current estimated 10x cash effectiveness bar for funding

At the same time, a 20% decrease in cost effectiveness is large enough that it could potentially affect funding decisions. In early 2022, GiveWell used a 6x cash effectiveness bar for funding, which DSW meets according to GiveWell’s CEA but no longer does in our updated version. Our revised analysis also suggests that ILC is a more promising intervention than DSW. In GiveWell’s original analysis, ILC and DSW were approximately equally cost effective. In our updated analysis, ILC is 33% more cost effective than DSW, although this result is highly sensitive to the estimated cost of ILC systems. Efforts to scale up and widely deploy ILC could therefore be more cost effective than expanding the DSW program. ILC is also a relatively new intervention, so there may be more potential for further development to reduce its cost or improve its adherence rates than there is for DSW.

# Appendix 1: Parameters used in Bayesian Analysis

The R and Stan code for our model is available on GitHub. Below is an explanation of how we chose the point estimates and probability distributions for each of the parameters.

### Direct Approach

1. Mortality effect size: Lognormal probability distribution, with mean -0.214 and sd 0.120, from https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=123265793&range=B12

### Indirect approach

1. Morbidity effect size: normal distribution with mean 0.77 and standard deviation 0.13, from the Clasen et al., 2015 meta-analysis
4. Morbidity internal validity: normal distribution with mean 0.9 and standard deviation 0.05, based on https://docs.google.com/spreadsheets/d/1BnuHlq4b0NSDjDmcEZ-79gjqrsQdi89zVn4ifSkPQ5k/edit#gid=1674952052&range=E3 and accompanying note saying "We have not put much thought into this adjustment".
5. Relationship between diarrhea morbidity and diarrhea mortality: lognormal distribution with mean 0 and standard deviation 0.171, based on the calculations described in this tab.
6. Mills-Reincke effect: Modeled as the fraction of all deaths that the X% reduction in mortality applies to. Normal distribution with mean equal to the sum of deaths due to enteric infections, respiratory infections, and a quarter of other infections and nutritional deficiencies, and standard deviation such that "Assumption 4" in GiveWell's mortality plausibility modeling is 2 standard deviations above the mean.
• We are interpreting GiveWell's "maximum plausible reduction" to be equivalent to a 95% UCL.
• Our assumption sets the mean roughly halfway between Assumption 2 and Assumption 3 in GiveWell's mortality plausibility modeling.
7. Correction for deaths due to enteric infection being more common in areas without water treatment (new adjustment): GiveWell's plausibility mortality modeling uses the fraction of deaths from different causes on a nationwide basis. However, intervention areas have lower levels of water treatment than the country as a whole and therefore will have a higher percentage of deaths due to enteric infections and other affected causes. We correct for this algebraically using the data reported here.

# Appendix 2: Addendum Regarding Funging

In addition to the five issues discussed in the main report, we identified a possible additional issue with GiveWell's downside adjustment for funging. While we did not have sufficient time to fully investigate this possible issue and to incorporate it into our updated version of the model, we think it is worth flagging for further review. In line 138 of GiveWell's DSW CEA spreadsheet, a value is computed for the units of value created if another philanthropic actor were to fund DSW instead of GiveWell. This value is calculated by multiplying the units of value per $10,000 created by DSW by the percent of the DSW program funded by another donor (100%), then subtracting an estimate of the units of value that the other funder would have created had they not funded DSW. However, the subtracted units of value estimate is computed by multiplying the benefit per dollar of other funders' non-DSW spending, the percent of the program funded by the other donor (100%), the cost per person covered, and the number of dollars spent ($10,000). We think including the cost per person in the  multiplication is most likely incorrect, as it leads to the result being the counterfactual value per 10,000 people covered, not per $10,000 spent. Since the downside adjustment section operates entirely in units of value per$10,000 spent, subtracting the counterfactual units of value per 10,000 people covered does not make sense to us. Because the cost per person covered is relatively close to \$1/year, this potential error leads to a modest change in the overall cost effectiveness for DSW, which we estimate is approximately 2%. However, we have not done as deep an investigation into this item as we have for the others, and we may be misunderstanding the intent of this calculation.

1. ^

The authors shared an early draft of the meta-analysis with GiveWell in 2020

2. ^

See lines 147-152 of GiveWell’s DSW CEA spreadsheet

3. ^

See comments in cell A106 of GiveWell’s DSW CEA Spreadsheet and cell E7 of GiveWell’s Funging Spreadsheet

4. ^

See lines 87-141 of GiveWell’s DSW CEA spreadsheet

5. ^

"First, it is a follow-up of the Null 2018 RCT, so it does not meet the independence assumption underlying this meta-analysis method."

"Second we find the effect size implausibly large (including the bottom of the 95% confidence interval)  given the modest intervention effect on chlorination rates"

Source

6. ^

See the “Methods” section of Null et al. (2018)

7. ^

See line 124 of GiveWell’s cost-effectiveness analysis for New Incentives

8. ^

See line 37 of GiveWell’s moral weights and discount rate sheet

9. ^

Because decreases in deaths due to diarrheal disease may be caused in part due to increased health spending, it is not clear that health spending on diarrheal disease would be expected to decrease at the same rate as deaths due to diarrheal disease. As a result, we chose to apply only the 4%/year discount rate associated with economic benefits to benefits from decreased health spending, and did not apply any discounting for decreasing diarrheal deaths.

10. ^

Cell B72 in the GiveWell’s ILC Kenya CEA spreadsheet contains the formula “=DSW!B74”

11. ^

As an extremely crude measure, GiveWell's Water Quality Interventions report spends 4100 words discussing mortality effects and only 680 words discussing the developmental and medical cost effects.

New Comment