Economic losers: SoGive's review of deworming, and why we're less positive than GiveWell

Primary author: Lucas Lewit-Mendes

Secondary authors: Sanjay Joshi, Isobel Phillips, Matt Sharp

tl;dr: We (SoGive) estimate that mass deworming is 6-10x more cost-effective than GiveDirectly's cash transfers. This is less cost-effective than GiveWell's estimate of 12-19x, but remains close to, albeit slightly below, their current 10x funding bar. The discrepancy is primarily driven by our conclusion that gains in adult earnings for some dewormed children may be at the expense of other workers (i.e. "economic losers"). 

Red-Teaming GiveWell's Recommendation of Mass Deworming

We (SoGive) have been conducting red-teaming of some of GiveWell's recommended top charities, starting with mass deworming programmes. Much has been written on deworming, from a landmark trial in Kenya, to the infamous "worm wars", to an apparently encouraging 20-year follow-up study. GiveWell moved around $40 million to support mass deworming programmes in the year to June 2022. After ~200 hours of research, we hope to shed some light on some relatively unexplored aspects of the mass deworming conversation. 

Our major update is that we now expect deworming to have less chance of huge economic impact, because competition between workers in the labour market may result in "economic losers". Alongside other small updates, this reduces cost-effectiveness to 6-10x GiveDirectly's cash transfers (varying by charity), down from GiveWell's estimate of 12-19x. 

We would like to express our gratitude to GiveWell for their correspondence throughout the research process. 

This post will highlight our key findings from our full intervention report, covering the following topics: 

Table 1

Despite the significant amount of effort already expended, our work on deworming is not complete. We remain excited to continue exploring some of the other areas that we outline towards the end of this post, including unprogrammed deworming (i.e. how much deworming would happen anyway without mass deworming programmes), the implications of climate change for deworming, and more on the health aspects of deworming.

Background on Mass Deworming 

GiveWell-recommended mass deworming programmes currently operate in sub-Saharan Africa and South Asia, treating children with two types of parasitic worm infections - schistosomiasis and soil-transmitted helminthiasis (STH). The scale of infections is enormous. Around 200 million people have schistosomiasis, while around 1.5 billion have STH. The health effects of worm infections can be severe in rare cases, including organ damage, intestinal inflammation, intestinal obstruction, and impairment of nutrient intake. But since worm infections often coexist with other morbidities, precise data on how often worm infections cause particular severe symptoms is hard to come by.^[1] 

Worm infection is not binary - worm burdens range from light, to moderate, to heavy. Around 1.3%^[2] of children in populations treated by GiveWell-recommended programmes are considered to have "moderate" or "heavy" worm burdens, which means they are infected with a large number of worms and are therefore more likely to have the morbidities described above. The remaining children have either no infection or "light" worm burdens, which are often asymptomatic.^[3]

Mass deworming involves treating everyone in a school or community with parasite-killing drugs. Individuals are not tested for the presence of infections, as treatment only costs around $1 per person, which is cheaper than infection diagnosis. GiveWell assigns "top charity" status to four charities that undertake mass deworming. This is primarily based on the results from an experiment in Busia, Kenya (the Busia experiment), which used a novel study design (randomisation clustered by school) to account for health externalities.^[4] Children who were dewormed for longer earned remarkably more and had higher consumption (spending) as adults 10, 15, and 20 years later. 

SoGive's mass deworming intervention report builds on GiveWell's comprehensive analysis by replicating their analysis (including calculating our own replicability adjustment), highlighting key uncertainties, exploring potential points of contention (summarised in Table 2), and providing an updated cost-effectiveness estimate.

Table 2 

Our key findings are described below: 

1.  The notably large effects on earnings and consumption from the Busia experiment are not statistically significant, but GiveWell sufficiently accounts for this uncertainty. 

GiveWell's cost-effectiveness analysis relies heavily on a single piece of evidence - a ~10.9% increase in long-run earnings/consumption 10-20 years after the Busia experiment. One concern is that the treatment effect on earnings is non-significant (p-value 0.297)^[5], though consumption is almost significant at conventional levels (p-value 0.058)^[6]. In isolation, we believe this would raise serious doubts about whether the positive results are driven by deworming or random chance. 

However, GiveWell addresses the lack of statistical significance by applying a large replicability adjustment to the published effect size from the Busia experiment. A replicability adjustment answers the question: if we were to run 100 perfect randomised controlled trials (RCTs) (placebo-controlled and no attrition) in exactly the same context as the Busia experiment, what would a hypothetical meta-analysis of these trials conclude? In other words, how much lower is the real effect of deworming in an identical context? GiveWell uses formal and informal Bayesian approaches to estimate that the real effect would be 13% as high as the headline results. 

How reasonable is an 87% cut to such imprecise results from a single experiment? Two additional studies^[7] provide additional evidence in favour of the headline results. A study of the siblings and neighbours of treated children in the Busia experiment, and a second, smaller experiment in Busia, also point to large cognitive or economic gains for children less exposed to worm infections. By Occam's razor, the most plausible explanation to reconcile these findings is that deworming did in fact increase earnings for treated schools and communities in Busia. 

Overall, we believe GiveWell has sufficiently accounted for the lack of statistical significance on long-run earnings and consumption. 

2. The causal mechanism between deworming and long-run earnings gains is a "black box". 

GiveWell’s model of the impact of deworming is based on the expected increase in income that dewormed children will enjoy in later life.

Surprisingly, two meta-analyses (a Cochrane review and Campbell review) of around 50 RCTs find that mass deworming provides minimal short-run health, nutritional, or educational benefits. This leaves us with no clear mechanism driving the long-run earning and consumption results from the Busia experiment. Hence, the causal pathway is a "black box".

On average, mass deworming probably does improve child weight, an indicator of nutrition. GiveWell senior adviser David Roodman's interpretation of the Cochrane review, Campbell review, and a third more optimistic meta-analysis (which includes additional data) is that there is a robust effect of mass deworming on weight gain. 

In addition, there is stronger evidence^[8] that deworming children known to be infected (i.e. those who show symptoms due to moderate or heavy worm burdens) improves child weight. This suggests that, at the very least, mass deworming should improve nutrition for children with the heaviest worm burdens. However, most children across a treated population have lighter worm burdens. We follow GiveWell’s approach of estimating the conversion from child weight gain to adult earnings by using evidence on the effects of random birthweight variation on earnings. Our calculations indicate that the average improvement in child nutrition from mass deworming is too small to explain the full long-run effects in Busia.  

More generally, we find it counterintuitive that there should be a small effect size for the most immediate effects (nutrition/education/health) and then a large effect size for the more indirect effects (earnings in later life). This is because, in our experience of reviewing charities and their theories of change, there is normally a dilution effect from one stage to the next (e.g. if, say, there were an immediate nutrition benefit, then not everyone who received the nutrition benefit would necessarily also benefit from earnings in later life). This leads us on a quest for alternative explanations of the long-run earnings/consumption results. 

We have argued that GiveWell has adequately explained these two points (the "black box problem" and the lack of statistical significance), but we mean this only in a very theoretical sense – i.e. if we imagine a hypothetical reader who carefully reviews everything GiveWell has written. In practice, most readers do not study every detail, and we know that some donors believe that the deworming recommendations are more robust than we or GiveWell believe them to be. This is despite the fact that what we call the "black box problem" is explicitly set out in GiveWell's blog posts.

3. Gains in adult earnings for some dewormed children may be at the expense of other workers (i.e. economic losers). 

A "naive" cost-effectiveness model (i.e. one which took the results from the Busia experiment at face value) would find that mass deworming is extremely cost-effective. GiveWell’s replicability adjustment of 13% accounts for a number of concerns with this approach, including the “black box problem” described above. However, we believe that this adjustment doesn't adequately address our concerns around economic losers. 

One possible explanation for the “black box problem” is that dewormed children became slightly more productive, allowing them to obtain higher-earning jobs and earn disproportionately higher wages. In particular, children in treatment schools from the Busia experiment work fewer hours in agriculture^[9], which is typically a low-wage industry. 

Our concern is that deworming, while providing gains in adult earnings for the treated population, may have also caused the loss of higher-earning employment opportunities for other workers (including but not limited to the experiment's control group). In this scenario, gains for some workers would have been at the expense of other workers (i.e. economic losers). Compared to a counterfactual world without deworming, these economic losers would be pushed into lower-earning jobs. Crucially, a simple comparison of treatment and control schools would conceal this effect. What really matters is the effect of deworming on overall productivity, which is a key driver of per capita living standards. Against the backdrop of minimal evidence for any substantial short-run impacts of mass deworming, the economic losers effect strikes us as a plausible explanation for the long-run Busia results. 

We are more pessimistic than GiveWell about the extent to which economic losers should be incorporated into the analysis. Our understanding, based on our careful review of GiveWell’s work, and based on our conversations with GiveWell, is that the economic losers consideration is:

• Not at all incorporated into GiveWell’s replicability adjustment of 13%
• Incorporated via a modest 3% reduction in GiveWell’s cost-effectiveness model

Our estimated adjustment is significantly more material than this, as outlined below. The details of how we constructed this estimate are set out in our full report. In short, the quantification of economic losers was based on (a) reviewing the details of the Busia experiment’s follow-up papers and seeking clarifications from the authors (b) a number of intuitions and suppositions, honed partly by time spent by some of the authors in the Luo part of Kenya, in areas not dissimilar to Busia. We believe that others could reasonably disagree with the details of our quantification approach. 

However, our core claim is that given the “black box problem”, we think it would be surprising if the correct adjustment were as modest as the adjustment incorporated by GiveWell. This is because, given the “black box problem”, we are significantly more inclined than GiveWell to believe that the economic losers issue is a material explanation for the data that we see.

We have calculated an alternative replicability adjustment, by following a formal Bayesian approach, where we treat the Busia results as an update to the short-run evidence. After accounting for economic losers, the result is an adjustment of 8.6% (compared to 13% in GiveWell’s model), and 7.2% after accounting for other factors (explained here). The economic losers consideration therefore reduces our cost-effectiveness estimate to 7-12x cash transfers (down from 12-19x), and is the main factor that drives down final cost-effectiveness to 6-10x.
 

4. The direct health effects of mass deworming appear to be small on average. 

GiveWell's cost-effectiveness model focuses primarily on the impact of mass deworming on long-run earnings and consumption, while direct health effects and short-term anaemia effects are relegated to supplementary adjustments (+1% and +9%, respectively). As noted above, we are confident that mass deworming increases child weight on average, but these improvements appear to be small. 

We estimate that mass deworming improves child weight by around 0.11 standard deviations in a high worm burden setting (similar to Busia). For context, a child who is one standard deviation below median weight-to-height is considered to have "mild wasting", which is commonly used as a measure of undernutrition. Since a 0.11 standard deviation weight gain is far from enough to reverse mild wasting, it seems unlikely that it represents a substantial nutritional gain. However, little is known about the subjective experience associated with weight gain (e.g. lessened feelings of fatigue, hunger, or weakness). The Cochrane and Campbell reviews reject claims that mass deworming improves other health outcomes, such as reductions in anaemia or mortality. 

Our work on modelling the direct health effects of mass deworming is ongoing, and we expect to publish further analysis in the future, such as explicitly modelling effects on disability-adjusted-life-years (DALYs). This would require a consistent approach to converting various effects of deworming into DALYs, including those based on deworming RCTs (i.e. weight gain), and those based on the rare severe effects of worm infections (such as intestinal obstruction).

5. Health effects are especially uncertain in the presence of comorbidities. 

The impact of deworming on morbidity caused by worm infection is clearly positive. However, worm parasites may also interact with comorbidities, such as malaria, HIV, and tuberculosis, leading to ambiguous net effects on health. Whether worm parasites overwhelm the host's immune system, or stimulate a stronger immune response, may be context-dependent. In fact, reported effects of deworming on the incidence and severity of comorbidities are inconsistent across studies. 

GiveWell's cost-effectiveness analysis includes a small positive adjustment (+6%) for HIV reduction. They also assess effects on malaria, concluding that deworming may increase the presence of malaria parasites in the blood, but there is little evidence that deworming increases cases of severe malaria or incidence of symptomatic malaria. We conclude that worm infections may be protective against some diseases (inflammatory diseases and possibly malaria) and exacerbate the harms of others - some studies suggest that worm infections may reduce the efficacy of vaccines, increase the risk of HIV acquisition, and increase the risk of progression from latent to active tuberculosis. The literature on HIV outcomes has some inconsistency, so we reduce the HIV adjustment to +3%. 

There is some tentative theoretical evidence to suggest that light worm burdens might reduce the health impact of comorbidities. This would make current mass deworming programmes less cost-effective, due to reductions in worm burdens over time. However, as far as we are aware, there is no strong empirical evidence to determine whether light worm burdens are beneficial. 

Overall, interaction with comorbidities increases both the upside and downside risk associated with mass deworming, but does not significantly change its expected value given existing research. Additionally, if, for example, deworming had caused a sharp increase in child malaria deaths, this would likely have become apparent in the results from past RCTs of child mass deworming, i.e. through measured effects on mortality (which was reduced by one child per 1000, but was not statistically significant). 

Overall, we believe that GiveWell gives a surprisingly small amount of attention to the health effects of deworming, given that deworming is primarily a health intervention. While we agree with GiveWell that the direct health effects of deworming are small (see section 4 above), our conversations with deworming NGOs suggested that the interaction with HIV was potentially material. Yet based on our review of the evidence, we are less convinced about the scale of this impact – perhaps even slightly less so than GiveWell. Assuming we hold GiveWell to a high standard for depth of analysis and transparency, we would expect GiveWell to have provided more clarity on this topic, and more generally on the health effects of deworming. 

6. GiveWell accounts for the possibility that drug resistance could develop, but we are somewhat more pessimistic about this risk.  

After around 15-20 years of large-scale mass deworming, there is not yet conclusive evidence^[10] of anthelmintic drug resistance in human worms. However, resistant Staphylococcus aureus emerged^[11] after 30 years of vancomycin use in hospitals in Japan and the US. Among livestock, drug resistance in soil-transmitted helminths is widespread^[12], although livestock deworming is typically more frequent (often reaching five times^[13] per year compared to once or twice for humans). 

While GiveWell includes a reference to the risk of drug resistance in their intervention report, the issue is covered only briefly. They apply a -4% adjustment in their cost-effectiveness model, but the impact of drug resistance is not modelled in depth.

Since mass deworming increases the volume of drug consumption, it increases the risk of drug resistance^[14], which would reduce the efficacy of deworming. In addition, in a world without mass deworming programmes, we tentatively estimate that, on average, around 24% of children in worm prevalent locations would still have access to “unprogrammed deworming” through sources such as chemists and healthcare centres. The 24% estimate is preliminary; as outlined in our full report, we believe this is an area for further exploration. For these children, mass deworming may be unnecessary and even harmful, by increasing the risk of drug resistance and therefore impeding their ability to obtain effective deworming drugs in the future. Nonetheless, given the majority of children would not have access to treatment without mass deworming, we still expect the benefits of mass deworming to far outweigh the expected future costs of foregone effective treatment. Our cost-effectiveness model includes a -7.7% adjustment for drug resistance (compared to GiveWell’s -4% adjustment), based on the potential risk to the effectiveness of future unprogrammed deworming. 

An alternative for mass deworming charities would be to provide targeted deworming only to children with moderate-to-heavy worm burdens, reducing the volume of drug consumption and therefore the probability of drug resistance. We have not reviewed the cost-effectiveness of targeted deworming, which may be prohibitively costly due to the need for infection diagnosis. 

Conclusion

In our adjusted cost-effectiveness model, we estimate that although some GiveWell-recommended mass deworming programmes outperform their funding bar (10x cash transfers), average^[15] cost-effectiveness is reduced to 6-10x cash transfers (varying by charity), down from GiveWell's estimate of 12-19x. This implies that a £23-£40 donation to a mass deworming charity will provide benefits equivalent to doubling someone’s consumption for one year. 

Our key findings are:

• Mass deworming robustly improves average child weight, a measure of nutrition.
• The cost of treatment is very low, and development benefits continue to accrue over a treated child’s working life.
• Extrapolating from short-run development benefits to expected long-run economic benefits, the Busia experiment's earnings/consumption results appear unlikely to be driven by productivity improvements alone.
• Competition between workers for higher-earning jobs and the associated economic losers could be a material explanation for the earnings/consumption gains in Busia.
• We are particularly sceptical of claims that deworming might have a huge economic impact. Productivity is a key driver of living standards, but the short-run effects on productivity-enhancing health, nutrition, and education are small.

Future research could focus on: 

• A deep dive on unprogrammed deworming: our tentative, not fully researched view is that the rate at which beneficiaries would have been dewormed anyway (i.e. unprogrammed deworming) is higher than GiveWell’s cost-effectiveness model suggests; we believe there is a high probability that a materially larger downward adjustment is justified.
• Exploring Happier Lives Institute's observation that small increases in annual earnings may have smaller effects on subjective well-being than large lump sums.
• Quantifying direct health effects using more formal metrics (e.g. DALYs), which could lead to stronger cost-effectiveness.
• Better understanding the likelihood that mass deworming will lead to drug resistance.
• A systematic empirical review of the contexts (e.g. lighter/heavier worm burden, younger/older children) in which deworming leads to more positive or negative effects on comorbidities.
• Forecasting scenarios in the coming years in which higher prevalence of moderate-to-heavy worm burden may arise (including those caused by climate change).
• Investigation into whether using an average earnings effect size overestimates improvements in subjective well-being, given some children may benefit from deworming significantly more than others (and experience diminishing returns). 
   

1. ^{^}

   GiveWell notes evidence of two severe symptoms of worm infections. Ascariasis causes intestinal obstruction for around one in every 3800 school-age children in Sub-Saharan Africa each year, resulting in around 3 deaths per million children per year. Intestinal obstruction may require hospitalisation and surgery and lasts around 4 weeks. Trichuriasis causes colon bleeding and inflammation, which lasts more than 12 months for around one in every 700 school-aged children in Sub-Saharan Africa each year. 

2. ^{^}

   Average of column E + F

3. ^{^}

   p.11 

^{^}

Treatment was not technically randomised, but GiveWell senior adviser David Roodman finds no substantial evidence of statistical imbalance at baseline. 

^{^}

Table 1, panel B

^{^}

Table 1, panel A 

^{^}

See "…but there are reasons to trust the Worms experiment too"

^{^}

See "Data and Analyses - Comparison 1"

^{^}

Table 2, panel B

^{^}

See "Evidence of reduced efficacy" 

^{^}

p.1152

^{^}

Footnote 85

^{^}

p.17

^{^}

p. 1156

^{^}

Weighted by the proportion of funding that GiveWell expects the charity would allocate to each country if it were to receive additional funding

Show all footnotes