Hi Martijn, thanks for such a careful read of the underlying study. I’ll respond to your points in turn - several are fair, and in one or two places our report reaches the same conclusion you do.
Regarding the HDU estimate: We agree with your take. Our report flags that the TMLE-adjusted 9.8pp reduction exceeds pre-period mortality and is unreliable - as a result, we didn’t use the HDU estimate in our model. Our paediatric estimate is based on the general-ward results. I take your broader point, though: if the only analysis restricted to monitored children shows no clear effect, that weakens the case for the ward-wide spillover our model relies on.
For patient mix and SLH: Both are valid limitations. The study adjusts for diagnosis category but not severity, and the patient-mix shifts go in different directions at the two sites, so the direction of the remaining bias is unclear. Without a breakdown between HDU and the rest of the ward at SLH, we also can’t separate direct effects from spillover. A desk review like ours can’t resolve these questions and unfortunately the reproduction checks you suggest are beyond the scope of a typical RP analysis, although we would welcome them. In the meantime, this is why we don’t think our conclusion should rest on the size of the effect…
Re. discounting and whether our estimate is conservative: You’re right that our internal validity discount is subjective. The report says so explicitly: it’s a judgement broadly in line with GiveWell’s adjustments for non-randomised designs. We also agree the resulting central estimate shouldn’t be described as conservative in itself, given its high uncertainty and reliance on spillover. Where we’d gently push back is on what our conclusion depends on. Because the system costs about $6,100 per ward per year, it only needs about a 1% relative reduction in ward mortality to clear the funding bars we used. On our admission assumptions, that’s roughly one death averted per ward per year. So the claim we’d stand behind is that IMPALA is likely cost-effective if it has any meaningful effect at all, not that the 12% figure is correct. The real crux is the probability that the true effect is close to zero. Our sensitivity ranges didn’t model that directly, but as we noted in reply to Nick above, even assigning a 30% probability of zero effect leaves the expected value well above the bar (~8,000x in Coefficient Giving terms).
We fully agree that independent, well-designed evaluation is the most important next step, and we’d encourage funders considering scale-up to build it in.
It's fair to suggest that we could have included a higher probability that the intervention has no effect, given uncertainties over the study designs and evidence available.
On that basis, we ran an expected value calculation, including a 30% probability that there's actually zero effect (but a 70% probability of our heavily-discounted effect size). Under these conditions our estimate is still way above the bar (~8,000X in CG terms).
Thanks so much for interacting with our work. For broader context around this report, it’s important to note that our approach from the outset was to build a series of models to show what you would need to believe for climate to be competitive with GHD. We therefore aren’t wedded to any particular SCC, and certainly can imagine different ways of approaching the problem.
To your specific points:
It is our view that Rennert is the best middle ground available from current peer-reviewed work. It sits between DICE-2023 (~$66) at the lower end, and estimates that incorporate growth effects likeBilal & Känzig (2024) (~$1,000+ at the same 2% discount rate) at the higher end. So, whilst we agree that some parameters in the GIVE model can be questioned in isolation (your discount rate and impact horizon critique seems reasonable), we feel it is still a sensible middle ground when taken as a whole, in comparison to other peer reviewed estimates currently available.
On adaptation, we feel less optimistic that the principle of ‘richer future = strong adaptation’ will hold for all people (e.g. those most exposed and least able to adapt in LMICs) or all sectors (e.g. agriculture), with the intersection between the two being particularly noteworthy. However, we accept this is genuinely uncertain and that reasonable people will weigh it differently.
Overall, we think our analysis at least supports the re-visiting of climate as a cost-effective cause area within the EA space and we appreciate the various thoughtful comments so far. Thank you for engaging!
Hi Martijn, thanks for such a careful read of the underlying study. I’ll respond to your points in turn - several are fair, and in one or two places our report reaches the same conclusion you do.
Regarding the HDU estimate: We agree with your take. Our report flags that the TMLE-adjusted 9.8pp reduction exceeds pre-period mortality and is unreliable - as a result, we didn’t use the HDU estimate in our model. Our paediatric estimate is based on the general-ward results. I take your broader point, though: if the only analysis restricted to monitored children shows no clear effect, that weakens the case for the ward-wide spillover our model relies on.
For patient mix and SLH: Both are valid limitations. The study adjusts for diagnosis category but not severity, and the patient-mix shifts go in different directions at the two sites, so the direction of the remaining bias is unclear. Without a breakdown between HDU and the rest of the ward at SLH, we also can’t separate direct effects from spillover. A desk review like ours can’t resolve these questions and unfortunately the reproduction checks you suggest are beyond the scope of a typical RP analysis, although we would welcome them. In the meantime, this is why we don’t think our conclusion should rest on the size of the effect…
Re. discounting and whether our estimate is conservative: You’re right that our internal validity discount is subjective. The report says so explicitly: it’s a judgement broadly in line with GiveWell’s adjustments for non-randomised designs. We also agree the resulting central estimate shouldn’t be described as conservative in itself, given its high uncertainty and reliance on spillover. Where we’d gently push back is on what our conclusion depends on. Because the system costs about $6,100 per ward per year, it only needs about a 1% relative reduction in ward mortality to clear the funding bars we used. On our admission assumptions, that’s roughly one death averted per ward per year. So the claim we’d stand behind is that IMPALA is likely cost-effective if it has any meaningful effect at all, not that the 12% figure is correct. The real crux is the probability that the true effect is close to zero. Our sensitivity ranges didn’t model that directly, but as we noted in reply to Nick above, even assigning a 30% probability of zero effect leaves the expected value well above the bar (~8,000x in Coefficient Giving terms).
We fully agree that independent, well-designed evaluation is the most important next step, and we’d encourage funders considering scale-up to build it in.
Hi Nick,
It's fair to suggest that we could have included a higher probability that the intervention has no effect, given uncertainties over the study designs and evidence available.
On that basis, we ran an expected value calculation, including a 30% probability that there's actually zero effect (but a 70% probability of our heavily-discounted effect size). Under these conditions our estimate is still way above the bar (~8,000X in CG terms).
Thanks so much for interacting with our work. For broader context around this report, it’s important to note that our approach from the outset was to build a series of models to show what you would need to believe for climate to be competitive with GHD. We therefore aren’t wedded to any particular SCC, and certainly can imagine different ways of approaching the problem.
To your specific points:
Overall, we think our analysis at least supports the re-visiting of climate as a cost-effective cause area within the EA space and we appreciate the various thoughtful comments so far. Thank you for engaging!