GiveWell’s approach to supplementary adjustments uses questionable priors

Primary Author: Sanjay Joshi, with thanks to multiple reviewers in SoGive

This was submitted to the GiveWell Change Our Mind Contest. 
 

Exec summary

1. One section of GiveWell’s cost effectiveness model is dedicated to various “supplementary adjustments” for the different charities in the model.
2. For these adjustments, GiveWell’s priors are (implicitly) that the effect size of each adjustment is zero.
   1. While this prior is legitimate in some cases, it is not in general well founded.
      1. If the adjustment is positive, the prior that the adjustment is zero is consistent with GiveWell’s usual sceptical approach of penalizing uncertainty.
      2. This is no longer true if the adjustment is negative (i.e. if it’s a downside of the intervention). A prior of zero on a negative adjustment increases the final cost-effectiveness number, which rewards uncertainty and is not consistent with erring towards skepticism and penalizing uncertainty.
   2. We recommend investing more effort in modelling some of the adjustments.
3. Our exploration of one adjustment – unprogrammed deworming – led to an outcome similar to GiveWell’s, by chance.
   1. We explored the negative adjustment with the highest impact on the final number in more detail. The adjustment accounted for “unprogrammed deworming”; i.e. deworming which might have counterfactually occurred outside of the programme being modelled or led to repeat deworming tablet doses.
      1. After the exploration, we felt that GiveWell's adjustments were not adequately justified given their a priori potential magnitude
      2. We also believed the approach was inconsistent with a sceptical prior
      3. While our adjustment coincidentally did lead to a very similar final number to GiveWell’s, we believe that this is just a coincidence, and a deeper dive might lead to more significant changes.
4. An extremely crude estimate of the potential impact of this would be an 8% decline in the cost effectiveness of New Incentives

1. One section of GiveWell’s cost effectiveness model is dedicated to “supplementary adjustments”

Supplementary adjustments refer to considerations which GiveWell considers to be potentially relevant, but which are not rigorously modelled. 

Examples include the possibility that malaria nets might prevent stillbirths, that mass deworming could lead to drug resistance, or that seasonal malaria chemoprevention could have side effects.

The full list of “out of model” supplementary adjustments is provided in Appendix 1

2. GiveWell’s priors are (implicitly) that the effect size of each adjustment is zero

GiveWell’s methodology for unmodelled supplementary adjustments is described below. You can see how this is implemented by looking at the table in Appendix 1 or directly in GiveWell’s model.

• GiveWell comes up with a figure to capture the size of the adjustment - the rough best guess of effect size
• GiveWell assesses:
  • how well-justified the estimate is
  • How easy the estimate is to model
  • How consistent it is with the way GiveWell generally models things
• The assessments are used to calculate a factor which is applied to the rough best guess
  • If the assessments are high for each of those items, the factor could be as high as 90%, meaning that the adjustment is almost given full credit in the model
  • If the assessments are low for each of those items, the factor could be as low as 10%, meaning that the adjustment is almost given no credit in the model

At a high level, many aspects of the methodology seem reasonable at first glance

• Implicit in the methodology is a prior that the effect size is zero
• GiveWell often applies a sceptical prior of zero impact, and there are typically good reasons for this. GiveWell has justified this position at length.
  • In short, it’s because GiveWell believes (and SoGive agrees) that achieving positive impact is a very difficult endeavour, so uncertainty should be penalized and in the absence of evidence we should default to the assumption that little-to-no impact is achieved.

2a. A prior of zero is not sceptical and uncertainty-averse if applied to negative adjustments

• If the adjustment is positive, the prior that the adjustment is zero is consistent with GiveWell’s usual sceptical approach.
  • For example, one of the adjustments is for the possibility that deworming could have benefits relating to HIV
  • Applying a prior of zero is appropriate here – our default or “prior” assumption on this should not be that tackling worms reduces the risk of HIV, it’s only something we should believe with sufficient evidence.
• This is no longer true if the adjustment is negative (a downside of the intervention)
  • In this case, choosing a prior of zero is actually favourable to the charity’s intervention; this is inconsistent with GiveWell’s usual sceptical prior.
  • For example, defaulting to a prior of zero impact means that GiveWell’s default/prior position is that:
    • in the absence of deworming programmes, people would never receive deworming pills (e.g. by going and getting them themselves); or 
    • mass deworming has zero chance of leading to drug resistance; or 
    • there is zero chance that there would be negative effects from decreased immunity when people are protected from malaria.
  • We do not believe that any of the above statements are reasonable priors.

2b. We recommend investing more effort in modelling some of the adjustments

In order to improve on this approach, we believe that one of the two following approaches is legitimate:

1. Use a prior which is better justified than a prior of zero, in particular avoiding the scenario where an unjustified prior tends to reward higher uncertainty adjustments
2. Invest more effort in determining the estimate of the adjustment

Given that it’s hard to come up with a suitable prior without investing more effort in determining the size of the adjustment (i.e. it’s hard to do the first without doing the second), we recommend the second approach.

Step 1 is to narrow down the list of adjustments – the full list has more than 60 items (see Appendix 1). Doing more detailed analysis on all of these would be excessive.

Here are some criteria for narrowing down the list:

• Adjustments where the sign is positive are arguably covered adequately by the current method: the prior is more suitable as outlined above.
• There are 18 adjustments where the sign is negative.
• Of those, 11 adjustments have a magnitude of ≥ 10% (i.e. are ≤ -10%), and of those, 9 have magnitudes exactly equal to -10%, according to GiveWell’s raw “rough best guess of effect size” (listed below).

Hence it seems that, at the least, it would be suitable for GiveWell to invest more effort into the two adjustments with the largest magnitude of raw “rough best guess of effect size”, which are:

• Unprogrammed deworming: -35%
• Crowding out of New Incentives: -25%

It may further be justifiable to invest effort into all of the 11 items listed below.

Table 1: List of adjustments where the effect is negative

In order to illustrate our point, we selected the item on the list which had the largest magnitude (unprogrammed deworming) and modelled it more carefully.

(3) Our exploration of one adjustment – unprogrammed deworming – led to an outcome similar to GiveWell’s, by chance.

Unprogrammed deworming refers to deworming which occurs outside of Mass Drug Administration programmes. This could be self-administered (the beneficiary – or their family – purchases deworming tablets themselves). It could also happen because other NGOs sometimes include deworming alongside other programmes (e.g. Helen Keller is a GiveWell-recommended charity which does this).

We did an exercise in considering unprogrammed deworming in more detail.

Here are the results, which show that the results are about 2% more favourable. For example, for SCI, the cost effectiveness went from 13.5x cash to 13.8x cash – a modest 2% improvement.

Table 2: Impact of using SoGive’s choice of unprogrammed deworming adjustment on cost-effectiveness

We calculated this by taking a copy of the GiveWell cost effectiveness spreadsheet and adjusting cell M47 in the supplementary adjustments tab.

How GiveWell calculated an unprogrammed deworming adjustment of 14%

Table 3: How GiveWell modelled unprogrammed deworming 

• The calculation starts with a -35% “rough best guess” adjustment; very little indication is given of how this was determined. GiveWell provides a comment which says: “Deworming separate from national MDAs may be common, but we have little sense of how common it is. Harris et al. 2015, Hafiz et al. 2013, and Addiss 2015 all found relatively high rates of unprogrammed deworming in the studied populations.” (Source: Cell I47 of supplementary adjustments tab in GiveWell’s model)
• The -35% figure is then adjusted downwards by 40% to reach a final number of -14% as described above. This means GiveWell's final number is more optimistic because of the fact that unprogrammed deworming is hard to justify using objective measures and diffcult to model, violating the convention of having a sceptical prior which penalizes uncertainty.

As we’ve indicated, we are unconvinced that the logic behind this approach is appropriate in this case.

How we calculated an unprogrammed deworming adjustment of 11.7%

We modelled unprogrammed deworming in this spreadsheet. Our calculations are summarised below:

Table 4: Overview of SoGive’s calculation of the unprogrammed deworming adjustment

As can be seen from the above, the difference between the SoGive adjustment (11.7%) and the GiveWell adjustment (14%) is modest, and leads to a modest (2%) difference in the cost-effectiveness result.

However, this occurred by chance.

As can be seen from the above, the methods used to calculate the two figures are very different, and it would have been unsurprising if the two figures had come out to be very different. Furthermore, several elements of the model have been determined using subjective judgement, and further research could lead to the number changing materially.

Hence we believe that, at least for the other large negative adjustment (Crowding out of New Incentives) and possibly for some of the others, further modelling such as the above should be conducted.

(4) An extremely crude estimate of the potential impact of this would be an 8% decline in the cost effectiveness of New Incentives

This estimate was determined simply by skipping the adjustments and setting the used adjustment equal to the raw “rough best guess” figure. This is a substantially cruder method than the approach outlined above for unprogrammed deworming, however it gives a reasonable indication of the potential scale of the effect.

This causes the New Incentives cost effectiveness to decline by 8%, as shown in this spreadsheet.

Appendix 1: Full list of GiveWell’s “out of model” supplementary adjustments

The below table is taken from GiveWell’s Supplementary adjustments tab in their cost-effectiveness model.

It includes only the “out of model” adjustments, i.e. items which GiveWell captures more rigorously in the main model are left out of the list below.

Veteran followers of GiveWell’s models will recognise them as “unmodelled” adjustments, to use the term employed in earlier models.