Thanks, Nick, these are helpful challenges, and as co-authors of the RP report, we (Abbie Clare and I) wanted to respond to two of them.
Blinding and attention effects: We agree with you that staff seeing the monitors, and any resulting change in how the ward operates, is part of the intervention rather than a bias. If anything, the evidence points that way: in the ZCH pediatric estimate, around 93% of children in the population showing the benefit weren’t directly monitored, which suggests ward-level changes may be doing a lot of the work. (For clarity, the point about blinding in the post is GOAL 3’s, made in the context of trial design; our report doesn’t argue blinding is definitely needed.)
The point we were making is a narrower one about the conditions under which the effect was measured. In the study periods, the monitors arrived alongside training, research teams and the novelty of a new system. Some of the observed effect may reflect that heightened attention, which could fade as wards get used to the system and outside attention moves on at scale. That’s why we flagged it as a reason the effect in routine rollout might be smaller. Ongoing support through GOAL 3’s subscription model may reduce this concern, but we can’t tell from the current evidence.
Our central estimate and uncertainty: We share your view that the evidence on effect size is weak. It’s entirely non-randomized and only partly controlled, and our report says so. Because of that, we didn’t take the observed effects at face value. Our internal and external validity discounts took away ~75% of the observed effect size in our central estimate and ~90% in our lower bound estimate. Reasonable people could argue for steeper discounts, and we’d welcome views on what you’d consider more appropriate. Our view was also that we shouldn't treat weak evidence as being equivalent to having zero evidence at all. And given that the counterfactual is manual checks every six hours, which often isn't met in practice, it seemed unlikely to us that improving monitoring truly has no effect, even if the real mortality reduction is much smaller than observed.
The reason we still conclude the system is very likely cost-effective is less about our central estimate and more about the break-even point. Because the system is inexpensive (~$6,100 per ward per year), it would need only about a 1–2% relative reduction in mortality to clear the funding bars we used. On our assumed admission numbers, that’s roughly one to two deaths averted per ward per year. Our uncertainty ranges also go below the cash-transfer benchmark at the lower end (around 3x), so we’re not claiming the intervention is certainly highly cost-effective. Our claim is that even if the true effect is far smaller than the observational studies suggest, it would likely still be worth funding.
We’d also gently note that the RP report does not claim this is the most cost-effective global health intervention. Our conclusion is that it’s very likely cost-effective, with substantial uncertainty about how big the effect is. We fully agree that a well-designed trial would be highly valuable, and that generating better evidence alongside any scale-up would be the ideal path.
Thanks, Nick, these are helpful challenges, and as co-authors of the RP report, we (Abbie Clare and I) wanted to respond to two of them.
Blinding and attention effects: We agree with you that staff seeing the monitors, and any resulting change in how the ward operates, is part of the intervention rather than a bias. If anything, the evidence points that way: in the ZCH pediatric estimate, around 93% of children in the population showing the benefit weren’t directly monitored, which suggests ward-level changes may be doing a lot of the work. (For clarity, the point about blinding in the post is GOAL 3’s, made in the context of trial design; our report doesn’t argue blinding is definitely needed.)
The point we were making is a narrower one about the conditions under which the effect was measured. In the study periods, the monitors arrived alongside training, research teams and the novelty of a new system. Some of the observed effect may reflect that heightened attention, which could fade as wards get used to the system and outside attention moves on at scale. That’s why we flagged it as a reason the effect in routine rollout might be smaller. Ongoing support through GOAL 3’s subscription model may reduce this concern, but we can’t tell from the current evidence.
Our central estimate and uncertainty: We share your view that the evidence on effect size is weak. It’s entirely non-randomized and only partly controlled, and our report says so. Because of that, we didn’t take the observed effects at face value. Our internal and external validity discounts took away ~75% of the observed effect size in our central estimate and ~90% in our lower bound estimate. Reasonable people could argue for steeper discounts, and we’d welcome views on what you’d consider more appropriate. Our view was also that we shouldn't treat weak evidence as being equivalent to having zero evidence at all. And given that the counterfactual is manual checks every six hours, which often isn't met in practice, it seemed unlikely to us that improving monitoring truly has no effect, even if the real mortality reduction is much smaller than observed.
The reason we still conclude the system is very likely cost-effective is less about our central estimate and more about the break-even point. Because the system is inexpensive (~$6,100 per ward per year), it would need only about a 1–2% relative reduction in mortality to clear the funding bars we used. On our assumed admission numbers, that’s roughly one to two deaths averted per ward per year. Our uncertainty ranges also go below the cash-transfer benchmark at the lower end (around 3x), so we’re not claiming the intervention is certainly highly cost-effective. Our claim is that even if the true effect is far smaller than the observational studies suggest, it would likely still be worth funding.
We’d also gently note that the RP report does not claim this is the most cost-effective global health intervention. Our conclusion is that it’s very likely cost-effective, with substantial uncertainty about how big the effect is. We fully agree that a well-designed trial would be highly valuable, and that generating better evidence alongside any scale-up would be the ideal path.