At the end of the day, the Ben et al. study is of low quality and questionable integrity. RP's subjective discounting and limited engagement with the actual validity of the study, its statistical methods and its confidence intervals obscures how unreliable this study seems. Therefore, this independent review from RP should not change much with respect to our confidence in Impala's effect size. Better, actually independent research is needed before we can estimate the effectiveness of the intervention with any certainty.
A few of the issues with the study, beyond what has been raised by RP:
From reviewhistory, it seems that conflict of interest was only declared by version 3, after peer review was done. Reviewers possibly were not aware of the massive conflict of interest from 3 of the study's co-authors, and it is unclear to what extent these co-authors were involved in the study. Peer-reviewers could only see that Niek, the listed co-author, worked at Goal3. It is very possible that they were not aware that Goal3 markets and commercializes the system being studied and therefore didn't register the enormous COI.
The HDU estimate is very fragile. The adjusted 9.8 pp reduction implies that, without IMPALA, these children would have died at well above the 8.1% actually observed before IMPALA, an extrapolation resting on only 20 pre-period deaths. The authors’ own matched analysis gives just 1.7 pp (95% CI −4.3 to 8.4). In the first manuscript version, both the methods and the table labelled this as the effect among children actually monitored. Later versions relabelled it as a whole-cohort comparison without changing the numbers. If the original description is correct, the only estimate for monitored children shows no significant mortality effect.
The patient mix changed significantly from the pre to the post period, and this was accounted for insufficiently. Only a subset of statistically significant difference between the groups are accounted for, without clearly arguing why. They leave out differences in diagnose severity alltogether. It is very hard to perform any analysis of effect size and statistical significance without better information on the causes of mortality and the severity of the diagnoses at entry. RP can't perform primary research, but they can make an effort to request and re-analyze the data for some basic reproduction checks, rather than taking the results and take TMLE-adjusted confidence intervals on face value for statistical tests. The odd TMLE-adjusted mortality effect at the ZSH HDU should flag to the authors that the TMLE method shouldn't be trusted blindly. To me, this seems to be clearly the best use of their time, and I am surprised they didn't attempt to do this more thoroughly.
In ZSH HDU, mortality increases post-impala, but at SLH, we don't have data on mortality and characteristics of the patients that actually were monitored with impala. ZSH seems to be the cleanest evidence of the effect size due to seeming similarities between pre and post groups, but we can't do the analysis without the HDU and non-HDU breakdown.
tl;dr: subjective discounting is not a replacement for critically engaging with evidence quality. The 3x discounting can give the reader the impression that the researchers are being very conservative, but a 12% effect size is still moderately large in absolute terms and large relative to other interventions. It is huge given that most of the effects are from spillover, and mostly covers a reduction in deaths from children not actually monitored by the device. So I warn any readers against accepting the framing that a 0.5-1pp mortality reduction across the entire hospital is a very conservative estimate.
Thanks Martijn. This is the kind of critical reflection we hoped for when we opened this up, and I’m grateful you took the time to go through the paper, its supplement and the review history in this much detail. These are important points, and they add nuance to how the evidence should be read.
On the conflict of interest: you are right. Our commercial interest should have been declared from the first submission, and it wasn’t until the final version. That was an oversight because we did not handle the submission process and we assumed it was done, not a deliberate choice, but it remains a mistake and I share responsibility for it as a co-author. Readers should weigh the paper with that in mind. The same applies to the RP report: GOAL 3 commissioned and paid for it.
I do want to respond to one word: “integrity.” I understand why the late COI declaration raises that question, and that one is on us. But I’d ask you to separate it from the rest of the work. The study was designed, run and analysed by researchers from six independent academic institutions in Malawi and the Netherlands, with GOAL 3 staff contributing to data collection and writing, not the analysis. The full results were published, including the HDU result that was not significant and the increase in crude HDU mortality, and the data are available for replication. Questioning the methods is fair and useful, and I will aim to get the answers as soon as possible. Questioning the integrity of the researchers, many of whom have no stake in IMPALA, is a different claim, and I don’t think that does justice to all the people who have contributed. I’d appreciate it if you’d reconsider that framing.
On the statistical points: I’m a clinician, not an epidemiologist, unfortunately the methods in this study (TMLE, the matched sensitivity analysis, how the estimates are defined) are beyond what I can judge with confidence. I don’t want to explain/defend something I can’t fully assess. I’ll ask the lead authors to respond to your points on the HDU estimate, the matched analysis, the case-mix adjustment and the data breakdown at St Luke’s, and I’ll share their answer here. The paper says anonymised data are available for replication, and I’d support an independent reanalysis. I’ll also ask Rethink Priorities to respond to your point on how they weighed this study, since that question is fairly theirs to answer.
Some broader context. I agree the Zomba HDU result can’t carry weight on its own, and that the ward-level results depend on spillover, which this study did not test directly. That’s why it may help to look at the rest of the evidence:
Neonatal units: we have five estimates from units where the monitors were actually installed, so these don’t rely on spillover. The strongest design is a controlled interrupted time series in two Malawian NICUs (Chihana et al., 2026). It found a 35% reduction at Mchinji (significant) and 27% at St Luke’s (not significant). Sengerema in Tanzania saw a 27% reduction (uncontrolled), Nyamata in Rwanda 20% (not significant), and Rwamagana no change.
Critical illness events: these fell sharply at both Malawian sites (Zomba HDU −47 pp, St Luke’s −25 pp). The study was powered for this outcome. It isn’t mortality, but it is the more precise signal and pathophysiologically very much aligned as an intermediate outcome towards mortality.
Mechanism behind spillover: health workers report monitoring time dropping from 3.3 to 1.8 hours per day. Whereas in other (unpublished) surveys in 4 health facilities with 116 participants, we have also found 92% of health workers reported reduced workloads. That makes a ward-wide effect plausible. Important to note here is that in all hospitals where these studies took place the same nurses working in the HDU are the ones taking care of the patients in the general ward.
We understand our evidence is not super-strong, eight of nine mortality estimates point in the same direction, but only some are statistically significant. None are randomised, and most were co-authored by GOAL 3 staff. A consistent direction makes chance less likely, but it doesn’t remove biases the studies share, such as before-after confounding. So my honest summary is that the evidence points towards a benefit but does not establish its size, and your critique makes clear the uncertainty is wider than a single discounted figure suggests. That’s also why we still want an independent, prospective controlled evaluation alongside scale-up, as discussed with Nick above.
Thanks again. This is exactly the kind of scrutiny that makes the evidence base better and helps to improve our understanding as well where we have blind spots and need to make additional steps, which was also one of the key reasons to ask RP to assess our evidence.
I hope the process will also help to build your confidence in us and that we are as truthful and transparent as we can be in search of improvement and sharpening our reasoning, evidence (including interpretation) and expanding our impact.
Thanks for the reply Niek. Perhaps I will have time to respond more in depth later but let me just say an important bit:
I said the study and the way it has come about is of questionable integrity. It wasn’t an assessment of the integrity of the people involved.
That being said, we shouldn’t just assume everyone involved in the study to have acted under complete integrity either, given the COI. We have no visibility on the extent to which COI’d authors were involved in editorial decisions during writing, methodological decisions during analysis and data-presentation as well as operational decisions during the study’s actual implementation. In all of these, I’m more concerned about bias (as distinct from lack of integrity), so that’s a good reason to keep matters of personal integrity out of it.
One more addition --> we have planned to use all feedback and insights from this Forum to further sharpen our own evidence overview which we have created on notion. So our own interpretation and representation of the evidence will be improved. If you are interested, please have a look. All feedback (positive and negative) is welcome and will be used to further improve the overviews. https://goal3knowledgehub.notion.site/public-knowledge-hub?pvs=143
Specific recommendations or suggestions for further research (beyond a trial as we already have planned that but will need funding first) are also much welcomed.
I think this could be a great intervention, but I disagree on the RCT front. I strongly believe that in the medium to long-term, this intervention should, and will only go to scale after high quality RCTs.
Yours and RP's reasons for not doing it aren't compelling to me. I'm almost a bit confused.. This might actually be one of the easier RCTs to do in the development world. If I was a funder, I would be asking why you aren't in the middle of a big RCT at the moment, or at least why you aren't getting started on one. I agree with the funders.
"Funders often tell us to come back with an RCT, which we estimate would need around 72 hospitals across three countries, US$5 million and three to four years." If this is what's needed, then yeah do it. You can get the 5 million - have you asked GiveWell? But I'm skeptical you really need 72 hospitals. Mortality rates are high enough in massive hospitals I would imagine if you are in national referral hospitals in capital cities, or similar, then this number would drastically drop. Maybe you are considering a bunch of smaller hospitals?
From your article "conventional RCT is hard to run for this intervention. It can't be blinded, because health workers know when monitors are on the ward. It has to randomise whole hospitals, because the central patient overview and the positive impact on workload affects care for every child on the ward, not only those attached to a monitor. Death is a relatively rare outcome, and baseline neonatal mortality across our sites ranges from 4.3% to 27.8%, which makes a trial harder to power. Ministries are also reluctant to keep hospitals in a control arm for years while neighbouring hospitals have monitoring. So is an RCT needed before scaling? If the effect is real, waiting three to four years for trial results means children in hospitals that could have had monitoring go without it. Last but not least it is important to emphasize that continuous monitoring is already considered standard of care in most parts of the world, so perhaps it is less about if it works but rather to what extent it should be prioritized amidst many other priorities and gaps.
From RP "Note that a traditional randomized controlled trial may be genuinely hard to run in this context: a monitoring system cannot be disguised as a placebo, and it may change how the whole ward operates. For example, staff know when the monitors are present, and the intervention is bundled with training and supervision. The observed effect size therefore reflects both the device plus the attention surrounding it (i.e., the Hawthorne effect[5]),
1. Your current evidence is fairly weak, and low on the evidence ladder. Before and after studies are notorious for being unreliable because there are so many other factors involved. Hospitals are always improving their systems and there are many reasons why mortality could drop. Seeing consistency between studies is encouraging but evidence needs to be better.
2. Blinding is not very important - the staff seeing the monitors is part of the intervention no? I'm not sure why RP and you think this is a problem? If it changes how the whole ward operates that is part of the benefit of the intervention. In fact that might be where you main benefit is! I'm confused about this comment by RP here and not sure what point they are making.... The training and supervision front could be a big deal though, especially when you believe your main impact to be ward spillovers. That's why I think training and supervision can't really last more than a few months, then you need to largely leave the hospital be with the monitors with minimal supervision and see if mortality reductions are sustained over 2-3 years. This isn't that long an RCT!
4. Randomising whole hospitals isn't an ethical issue. Ministries will not refuse to allow hospitals being in a control arm (they are getting free monitors) and the evidence isn't nearly good enough yet to argue an RCT is unethical. If the difference in mortality between hospitals is big after year 2 say, the study will be triggered to stop for ethical reasons anyway.
5. The range of mortality rates doesn't make powering a study harder I don't think? (might be missing something) Its low mortality rates and low sample sizes that might be the issue here. But there are lots of hospitals where multiple kids die every week (I have one 500 meters from my home) If you are in the busiest hospitals where 150+ kids die every year, then I would think you wouldn't need neadly that amount (I might Claude this later). If you need to do a multi-country RCT so be it, that seems likely needed.
6. The urgency argument seems unrealistic. Because of the low level of evidence intervention isn't "obviously" better than other things we could spend money on. I don't think your case is nearly strong enough here to make a "bypass the RCT" argument. I think actually even framing it like this could make people lose confidence in the intervention. It feels more like a sales pitch than a good argument.
7. What is considered "Standard of care" in rich countries is close irrelevant in healthcare interventions in low-income countries. I don't think it should even be a factor really. There are 100 "standard of care" things you could introduce to a pediatric ward in Malawi (Better antibiotics, Dr. nos, nursing ratios, frequency checks, infection control, blood cultures, blood gases etc.). This doesn't mean you don't study their benefit in a low-income setting.
As an unrelated point, I feel like the presentation of the intervention here might be too branded. The intervention is a monitoring device, yet you are at pain to emphasise how your intervention is different from other devices. If your plan is to scale through government, its the monitoring system that matters not the branding. It might be better even to brand machines as "Ministry of Health" machines and think of yourself as a support org as much as an innovation org? This is a tricky one though I know.
I absoutely love this intervention - I'd be keen to get on a call and chat about it even if you might be. Sadly someone ffrom your org was pegged to meet me at EAG New York, but I can't go because of travel restrictions due to ebola :(
Hi Nick, thanks for your message. Happy to jump on a call myself as well to dive deeper feel free to email ([email protected]) or text via the chat function of the forum (if it has that)
as a first quick reflection/response, hopefully I have more time later this week but here are a few reflections, and apologies for not following your exact numbering.
A) we have been trying over the past 4 years to get trials funded (EDCTP, givewell, EU funding, Gates) without success. There are some ways to make it lower cost (lowest we got was about 2,5M, but that also reduces quality and strength of evidence). In between time we have done what we could through different smaller sources of funding to get the best evidence possible. Agree by the way that blinding is not needed and not relevant and that a large effect actually comes from the behavior change due to its clear presence on the ward. Also randomization is not an issue as long as control sites also get the intervention.
B) trials can be great, but if we wait for those results to come out before scaling we would likely not survive as an organization. This is not unique for us, but for all innovations and a major bottleneck for innovative solutions in general. Moreover, 3 years of waiting to scale and finding out that the results are same, would create a huge amount of deaths not averted, happy to provide more precise estimates at a later stage. Or plea is not to skip the trial, our plea is to do it in parallel.
C) While I like trials I actually think real world evidence is better or at least as important. With monitoring and many other interventions it is not about if they work, but how do they work in a very complex environment that is under-resourced and understaffed. In other words, if I would be the minister of health, I would value evidence from 100% of 50 sites in a real world setting a lot higher than the evidence from the intervention under trial circumstances. Just to be clear --> we are doing all this (we are currently running a controlled interrupted time series in multiple countries and multiple other impact evaluations are still ongoing) and still want to do the trial if we can get the funding for it.
D) A trial as the definitive answer to answer the question if something should be scaled is not rational from a perspective of levels of uncertainty. In my view the lower bound of the margin of certainty just needs to be above (or close to) the treshhold because if it is above that level it would not be rational to say we need a trial first before scaling it up, because the statistics already say it is highly unlikely that it is. I sometimes use the picture at the bottom to describe that (NB if evidence changes the direction of the arrow may also change). Of course it is good to discuss if you agree to the estimated impact and the margins of uncertainty to that which Rethink priorities found, because I do think that is important to agree on (or disagree and find ways to close the gap)
E) too salesy --> I think you may underestimate how important the technology, service-model and business model are compared to anything else that is out there. There are dozens if not hundreds of changes we have done and are continuously doing to improve the intervention (at GOAL 3 level) and at the facility (to improve adoption and usage at facility level) all embedded in the business model which allows us to do continue to do this over time. We do not describe this here in this post to sell or brand our solution, but also to make clear that just putting a monitor down will not have the same effect.
Last but not least: it is also a product of passion and enthusiasm that comes from building 8+ years towards this solution. I just can't be fully objective in how I describe our solution. Firstly because it is the answer to the problems I experienced myself when working in the field, secondly I think there are a lot of lessons to learn from our success that can be used by others. I hope you can accept that (and maybe even appreciate it ;-) )
F) thanks again. Really appreciate the open discussion. Would be great to connect
Nice one, some quick replies and lets set up a call. So great you are so passionate about this, and I for one think that your org should be well funded and an RCT should be done.
A) Sorry you can't get the RCT funded. Have you tried the new DIV? You're right randomising at hospital level makes it really tough. I feel like you probably need national referral hospitals for the required power. From Claude if you had hospitals roughly with 1,000 admissions a month at 3% mortality, looking for a 20% reduction in mortality you'd need 4-5 massive (national referral) hospitals, or maybe 2 massive ones and 15-20 smaller? 72 feels like too much but I obviously haven't looked into the details, just asked Claude. As your current results seem to show 30ish percent mortality reduction this seems on the difficult but feesable end?
B) I think you are in a great position continue to grow steadily if not scale without better evidence and survive - your organisation is still so young there's plenty of room to grow and then scale. You are in a pretty good position for philanthropy to scale the setup costs (rather than government) if an RCT shows it is this amazing.
C) This one confuses me I don't really understand what you mean by "Real world evidence". Your RCT should ideally be basically just what you do anyway, so wouldn't that be real world evidence? The problem is that you're looking at these mortality drops as evidence enough, where I would see them more as higher level M&E of your program. Your own M&E can never touch an external study for importance. Orgs like ANSH used mortality drops as a marker for their progrram, but they already know the intervention works from RCTs. I just dont' think it can fly for before/after studies or your own M&E to be the evidence which carries you to scale.
D) I like your chart a lot and I agree with the concept completely! But I don't agree with the levels of uncertainty RP cites, although at least they are pretty big. Given the variation even in your own data, relying on before/after studies, the nature of the effect of these kind of interventions to reduce over time (I think a 7 year horizon is too long for this CEA), and the general trend of hospital mortality improving over time I would put your error bars much wider. Claude told me Kamuzu hospital mortality dropped fom 9% to 3% between 2012 and 2015 of its own accord, so these kind of drops seem not unusual for a bunch of reasons. Also Claude and RP cited a bunch of studies in Western pediatric wards which showed benefits in before/after studies, but then when high quality RCTs were pooled the benefit disappeared. I would have anchored on the best quality research we have on the topic much more heavily than RP did, even though its in a different context. I think if the evidence isn't clear in high income countries, it increases the burden for you somewhat.
A) We want it to be representative so only selecting referral hospitals does not do the job as it will not inform a ministry of health what will happen in rural areas. The challenge is that hospitals vary quite a bit, so you have to match them first for certain characteristics and then randomly assign them. The larger the variation the more hospitals you will need, hence we came at 72 through our power calculations. The challenge is most funds only go up to 2 or 3 million USD
B) I think you are right from where we are today, but there have been many moments where it was likely we would not survive. We have our own development team, registrations to maintain all requiring significant investments. If it takes too long, likelihood of dying in the valley of death will increase. There are very few innovators who are actually successful in public health systems for this reason.
Scaling slow also has high opportunity costs. We have made calculations based on our growth projections (hospitals reached) and our impact per hospital (mortality data). I can not share these now, but let's use a fictive scenario that is close to our estimates:
Let's assume Rethink Priorities assessment is right and each hospital using IMPALA will save 7 newborn lives per year and 12,2 pediatric lives per year. If we grow according to our plans we will reach~1000 hospitals by 2030 and IMPALA will help avert ~19,200 deaths every year.
If this is delayed and only 500 hospitals are reached, this means a difference of ~9,600 lives saved in a single year and cumulatively a lot more. At the levels of our own results this would be 4-5 times higher. Of course I also understand it could be significantly lower (but as rethink priorities said even at 1-2% reduction it would still be a good investment). The longer we wait the higher the opportunity costs will be.
My point is not that we should release all breaks and all funding should go to us to scale. However I think it is more than justifiable to invest significantly in a large(r) scale program with thorough evaluation, which could yield results a lot faster and generate more decision critical evidence.
As a few follow-up question to you (and the rest of the forum): What study results (number of sites, quality etc), not from a trial, would convince you and why? If only a trial can convince you why is that? What evidence does it deliver that routine evaluation at scale could not?
C) In any research setting there will be research staff and a lot more data collection, this by itself has a huge effect on how care is provided. Moreover not all sites will be suitable for it, creating a significant selection bias. Therefore the performance in (optimal) trial circumstances often has limited meaning for what it means in real life. Moreover variations across countries, will likely influence success of the intervention. So generalizability is also limited (Why would someone in West Africa, South-East asia or the middle east believe the trial from East Africa) Real world evidence solves a large part of this, whether if it is implementation research or derived from the real world evaluation. We look at each installation for usage, adoption and impact on health workers stress and burnout, where possible (if funding allows) we look at impact on mortality. While the quality of the evidence may be lower, it says a whole lot more about what is actually happening within the specific hospital/country and provides opportunities to intervene (either through us, or the hospital itself) Seeing this over multiple hospitals in one country will help a lot more with decision making then a (potentially outdated) trial from 5000km away (as a figure of speech).
Again, not opposing a trial here just wonder if it is the right instrument for the stage we are at. Just to provide a different perspective. A 5 million implementation program, would allow me to go to 200-300 hospitals in different countries and perform a thorough monitoring and evaluation program in parallel which can look at before-after differences, but also at differences between other hospitals. Those results would be available within 1 year after implementation, so likely 2 years after a grant was awarded.
D) Don't think Claude is accurate. Without a verifiable source on the mortality drop that answer is pretty useless, beyond that it is not wise to assume a conclusion that this is common based on one site. Data quality is notoriously bad in this setting and there is severe under reporting. We have solved this by diving into individual patient files hand by hand (not us, but our research partners)
The study you cite is hardly comparable in my view - general wards versus ICU/HDU - High resource settings only - high levels of staffing and no lack of monitoring equipment - Only adults - Vast majority is surgical patients - total number of patients only 1284 over all studies combined - relative rare occurance of the events making them statistically underpowered to proove significant impact
Moreover if you dive into their discussion you also find that over multiple systematic reviews all findings are consistently positive favoring monitoring, yet not significant. That is relevant to our discussion, because one of the key conclusions is that even at 1-2% mortality reduction IMPALA would still be cost-effective against any bar.
Do you need to convince Minsitrys of Health here? I would imagine they will already think the monitoring machines are great and are keen to sign up. The research is to convince funders and international bodies like the WHO. IMO any positive result from a big RCT, even just in the biggest hospitals would be fine to generalise to smaller hospitals. Like you said if there's any difference detected at all, cost-effectiveness will be very high and unlikely to degrade to meaningless levels at smaller hospitals. The sample size you need is big, so for pragmatism perhaps it would be best to go for the largest, easiest hospitals.
To your overall point, I don't agree that RP's assessment is a reasonable benchmark midpoint on mortality and I don't agree with their confidence intervals. Claiming to be something like the most cost-effective global health intervention on the planet is a bold claim based on the evidence you have so far. In fact there's an argument that confidence intervals should include zero given the lack of RCT evidence around monitoring devices.
I agree with you if we assumed RP's assessment is sound, then your case would be compelling. On a pragmatic front, lets assume that you and RP are right about this, and the current evidence is compelling enough for scale. Even if you're in the right I think you'll struggle to convince institutional funders of this, because they want to hold ambitious scale plans to the same rigorous standards as other orgs. Most other orgs have RCTs or almost-as- trong evidence showing that what they do can save lives. With the anthropalypse though perhaps you can convince some more risk-taking folks.
On the RCT, I think from an implementation perspective an RCT seems managable as you expand. If you're telling me can go to 200 hospitals in 2 years, then I think you can manage the RCT on that path. Yes it will be harder than your other suggested monitoring but not astronomically harder. Otherwise you will be bigger in 3 years time and funders will still ask the same questions. I get your timing question - it would be OK to wait a year or two but it will need doing at some stage. What if you bundled RCT funding together with the scale funding and bite the bullet that you might roll out to half the number of hospitals?
On the study I cite - is there a better one you would suggest as the closest to your work? I found 2 pediatric RCTs in low-income countries which looked at different types of monitoring and found no mortality difference. None of these studies not directly comparable, but best practise is often to look at the the best studies we have on a similar topic, and that was the best I could find (it was the one RP quoted too). On the trends in the data I agree, but without statistical significance its pretty weak.
I think it is an important note that there is likely to be skepticism also due to a mixed history of evidence on patient monitoring. Outside of a number of negative results in adults and kids, famously after big studies continuous fetal monitoring doesn't seem to reduce neonatal mortality which still boggles my mind a bit. I know all of these are very different to what you do, but we still need whatever is the closest reference point, and I think unfortunately it might make the RCT more important than for other intevention. Our priors might be more negative here than for other interventions.
Ministries of health are not passive recipients of aid in our model, their buy-in and willingness to commit resources is fundamental for success and in our perspective the pathway to both scale and sustainability. It is indeed true however, that if I would come with funding they would gladly accept it. Increasingly we see that they are requesting funders to support in purchasing IMPALA One of the key reasons we would like to the Trial is that it will make it easier for them (and funders like UNICEF, UNITAID etc) to justify this in policy and budgeting. So at this stage we see significant buy-in and commitment from governments in the scale-up, but freeing up own resources at sufficient degree to buy entire systems is not yet feasible. We think we can also get there without the trial, but the trial would help a lot. We are also confident that we can easily deliver this trial together with our partners, as we already drafted the protocols and plans to that.
As with regards to your standpoint on RP's assessment --> would you be able to explain why you are so steadfast about it not being correct? Right now you make a claim that they are overstating the effect, but there is a 72% discount already in their estimated effect, which has been largely subjective based on study design. I wonder what arguments you would have to say their assessment is not correct and that the effect should be closer to 0 beyond the weaker study designs.
In my view the fact that the patients before and after where matched for severity in the pediatric design and that we have a controlled interrupted time series for neonatal care is also a quite strong design. (https://researchonline.lshtm.ac.uk/id/eprint/4648480/) And last but not least that all evidence points in the same direction makes chance alone very unlikely.
One thing that is also worth considering You mention the confidence interval is very large which is because the occurrence of the event (death) is quite rare. However if you look at the primary outcome of Critical Illness Events which is an intermediary outcome to mortality and is a lot more frequent, the confidence interval is a lot smaller so the claims about those are quite robust. Moreover the supportive evidence relating to impact on workload, stress levels and confidence of health workers is also important. In general our intervention achieves its effects not just through better monitoring, but also by freeing up time of health workers. This has also been quantified in multiple studies.
As with regards to other studies, I am afraid I must agree that there is hardly any good studies out there within our population. There are several studies on early warning scores, but few on monitoring and especially not in children. Why I believe it works well in this population and why this is different from many other monitoring interventions
a) it is as much about unburdening health workers and freeing up their time as it is about getting better data on what is happening with the child. Our data show health workers consistently report reduced workload and stress levels after implementation. Also direct observation shows it saves a lot of time (from 3.3 hours per shift to 1.8 hours per shift)
b) these are very high risk children, the standard of care is very little if any monitoring, it is easy to improve from there and get better outcomes
c) interventions are relatively simple and readily available (IV fluids, oxygen, antibiotics and glucose, for neonates it can be as simple as rocking the body to prevent severe apnea)
d) as mentioned it is not just monitoring, it is embedded in an entire program of training and continued support with a quality improvement component. Therefore we address a much broader range of potential barriers to timely, quality care then just the lack of vital signs monitoring
In general I feel that there is a huge lack of attention on monitoring in these settings. It is often an afterthought and even in very large programs oxygen and other treatments are prioritized over other things. So for me this is one of the main reasons our effect sizes can be so great. To me monitoring is the foundation of hospitalized patients, especially the most critical/vulnerable patients. The lack of attention on this and lack of awareness about the gaps are huget bottlenecks to improving quality of care.
These are great answers. I do agree that your intervention is more likely to reduce mortality than other montoring interventions for your compelling reasons. I don't think that makes the studies RP and I cited completely irrelevent though and do need to inform our prior to some extent.
My reason for saying I don't agree with RPs confidence intervals are 1) There isn't compelling prior evidence to anchor on 2) Your study design - Interrupted time series in general is great, but confounding can still be huge especially with all the different factors that can affect hospital mortality (no. of admissions, sickness of patinets coming in, seasonality, hospital management quality, staff quality etc.)
I don't think RP's assessment is not "correct" as such. They are doing the best they can on a broad set of assumptions with the evidence you have. Their confidence intervals though in theory give your intervention an under 5% chance of the real effect of your intervention being zero - and I just don't think we can ever be 95% sure from before and after studies in complex settings like a hospital. This statement is based on a number of other occasions where before-and-after results in the development world didn't translate to RCT findings. I would suspect other development experts would agree that. I think
And if there is even an over 5% chance that your results were due to confounders and chance, then the confidence interval should still include no effect - if that makes sense.
Thanks for acknowledging the difference between us and monitoring. It is very important to me that people see it is not just about putting a monitor down.
As with regards to confounding factors,
Everything was modelled for seasonality. (that is why we always compare a full year)
As with regards to sickness of patients --> this was corrected for by comparing individual patient characteristics and matching them on propensity score in the pediatric study.
Patient numbers were looked at and there were both significant increases and decreases. Hard to say what the impact of that was on mortality.
In all sites we have done qualitative interviews to assess if there were any other programs or changes that could explain the difference. We did not identify them. Health workers themselves attributed it to IMPALA.
As with regards to should the zero still be included. Maybe at this point yes, but as we progress and we get results from more hospitals with similar results a zero results becomes increasingly.
What I would like to put in here: let us assume Rethink priorities assessment as the base case with our current results as the upside scenario, the zero case would be the worst case (think we can agree that a negative effect of monitoring would be highly unlikely?)
My point is that we should not wait until we have certainty that it is not zero, but that we should act on what is the most likely case at this stage, because that is what will determine the return of investment. In this we should not just look at what the procurement buys now, but also on what effect it will have downstream.
We often call this the impact business case: the impact per hospital, how many hospitals we can reach in a given time (through sales, donations and programs), and the total impact per year that follows. Set the growth numbers and you get a low, medium and high estimate of impact over time. Every piece of evidence reduces the uncertainty and improves the estimate. From this you can estimate the return on investment, balancing the risk of no impact against the potential for impact. Waiting is not free: it slows growth and growth potential, with knock-on effects downstream.
Let's say you are a fund manager (or several funders making the same call) with three options:
A) A USD 500k grant, in line with our base case: about 1,170 new hospitals by 2030.
B) No funding: wait for a trial or more results. We assume about 15% fewer hospitals by 2030 (about 990). At our current stage USD 500k has an outsized effect: without it, the risk of operational disruption and bankruptcy rises, and other funding would likely shrink too, because EA funding decisions are closely linked.
C) Double down with USD 2.5M for scale-up and further evaluation: about 15% more hospitals than the base case (about 1,340).
And three impact scenarios, roughly normally distributed:
Median: Rethink Priorities' estimate, 12% fewer deaths on pediatric wards and 7% in neonatal units
Low: 5% chance: zero or negative effect
High: 5% chance: our measured effects (42% for children, 26% for neonates, as weighted in RP's report) are true, or higher
The outcome, in DALYs averted by the end of 2030:
Under these assumptions, the USD 2M catalytic grant (C instead of A) has about a 93% chance of beating the same money given to Helen Keller Intl's vitamin A supplementation, a GiveWell top charity. It breaks even at about a 1% cut in in-hospital deaths. Median return: about 359,000 DALYs, 9.8x vitamin A. Expected return: about 500,000 DALYs, 14x. If the measured effects hold: about 1.3 million. - The cost of waiting (B): at the median, about 175 fewer hospitals and 359,000 fewer DALYs than A, and 718,000 fewer than C. In expectation, about 500,000 and 1.0 million fewer.
With growth moving 15% either way, waiting costs about as much as the grant adds. If IMPALA has no effect, no option averts anything, and the USD 2M would have done more at vitamin A: about 36,600 DALYs, or 570 lives. Research that shows this early stops spending on something that does not work.
Simply put: if you can choose between Vitamin A and IMPALA, based on this model the chance is 7% that you will get a better (social) return on investment and a 93% chance that you get a better return on investment with an upside that is a lot higher.
The model is purely for discussion. I do not want to claim any of these effects as definitive outcomes. - Question 1) do you agree with the framing of this choice and the impact business case it represents - Question 2) what are the assumptions that we use for impact and growth
Full analysis, with the model, slides and all assumptions (generated by Claude reviewed by me): https://docs.google.com/document/d/1Up2M0HCa3vhK2ouK9BmbGpXGz2EHKtKqynSyuupwJOo/edit
Thanks, Nick, these are helpful challenges, and as co-authors of the RP report, we (Abbie Clare and I) wanted to respond to two of them.
Blinding and attention effects: We agree with you that staff seeing the monitors, and any resulting change in how the ward operates, is part of the intervention rather than a bias. If anything, the evidence points that way: in the ZCH pediatric estimate, around 93% of children in the population showing the benefit weren’t directly monitored, which suggests ward-level changes may be doing a lot of the work. (For clarity, the point about blinding in the post is GOAL 3’s, made in the context of trial design; our report doesn’t argue blinding is definitely needed.)
The point we were making is a narrower one about the conditions under which the effect was measured. In the study periods, the monitors arrived alongside training, research teams and the novelty of a new system. Some of the observed effect may reflect that heightened attention, which could fade as wards get used to the system and outside attention moves on at scale. That’s why we flagged it as a reason the effect in routine rollout might be smaller. Ongoing support through GOAL 3’s subscription model may reduce this concern, but we can’t tell from the current evidence.
Our central estimate and uncertainty: We share your view that the evidence on effect size is weak. It’s entirely non-randomized and only partly controlled, and our report says so. Because of that, we didn’t take the observed effects at face value. Our internal and external validity discounts took away ~75% of the observed effect size in our central estimate and ~90% in our lower bound estimate. Reasonable people could argue for steeper discounts, and we’d welcome views on what you’d consider more appropriate. Our view was also that we shouldn't treat weak evidence as being equivalent to having zero evidence at all. And given that the counterfactual is manual checks every six hours, which often isn't met in practice, it seemed unlikely to us that improving monitoring truly has no effect, even if the real mortality reduction is much smaller than observed.
The reason we still conclude the system is very likely cost-effective is less about our central estimate and more about the break-even point. Because the system is inexpensive (~$6,100 per ward per year), it would need only about a 1–2% relative reduction in mortality to clear the funding bars we used. On our assumed admission numbers, that’s roughly one to two deaths averted per ward per year. Our uncertainty ranges also go below the cash-transfer benchmark at the lower end (around 3x), so we’re not claiming the intervention is certainly highly cost-effective. Our claim is that even if the true effect is far smaller than the observational studies suggest, it would likely still be worth funding.
We’d also gently note that the RP report does not claim this is the most cost-effective global health intervention. Our conclusion is that it’s very likely cost-effective, with substantial uncertainty about how big the effect is. We fully agree that a well-designed trial would be highly valuable, and that generating better evidence alongside any scale-up would be the ideal path.
My only real quibble is with this statement "It seemed unlikely to us that improving monitoring truly has no effect, even if the real mortality reduction is much smaller than observed." I agree with this statement. But how unlikely? You have made the call here that no effect is not just unlikely, but that there is a well under 5% chance that monitoring has no effect. This is where I disagree. Given other null results when RCTs were done on monitoring (in different contexts), and an inconsistent history of before and after study results, I think it would have been better to include null in your confidence interval. I didn't do the research so I have little clue but I would maybe have guestimated the probability of no effect in the 10%-30% range given these priors.
I might have discounted further on the point estimate, but this is perhaps a "how short is a piece of string" situation where you just have to make subjective calls so fair enough here - and you did discount a lot already.
"the RP report does not claim this is the most cost-effective global health intervention." Point estimates matter a lot as this is your best estimate as to the effect. This an interesting philosophical question, but with your eye-popping point estimate and a confidence interval that doesn't include zero effect, I think there is a decent argument that your math at least does seem to claim this is one of most cost-effective global health interventions on earth, even if that isn't your intent.
Fairly minor, and not at all specific to this intervention, but I'd like to see more caution in converting DALYs into multiples to cash transfers. People often seem to think this is a straightforward factual claim, when it's primarily a subjective value judgement.
Health outcomes and income increases are, of course, different things - apples and oranges. The typical EA move to compare them is to assign a conversion rate between a DALY and an income doubling, i.e. to say that N income doublings is equal in value to 1 DALY. This is usually called 'moral weights', and these moral weights are ultimately subjective value judgements. One person might say an apple is 'worth' 1 orange, another might say it's worth 10, and so on. So, what's doing the work in claims like "this health intervention might be 76x cash" is a value judgement reasonable people will disagree about. It's not like, if you disagree with it, you're wrong because you've misunderstood the facts.
In contrast, if you're comparing two interventions that improve income, and one provides 76x more income doublings per dollar invested, that is more of a straightforward apples-to-apples comparison.
I don't mean this is a criticism of the authors, as they are applying a standard methodological move. My concern is with the standard methodological move.
Wow these guys are taking some punches graciously from the rest of us and @MichaelPlant weighs in with a curveball ha.
My main comment here is to reiterate that this is a completely standard thing which most GHD people do, largely because the biggest player GiveWell uses it as their standard.
My take here is that there are different degrees of "removal" as it were, all with different levels of understanding. We can't who's subjectivity it's just how much we are willing to live with.
You could classify these differently but I'm my mind there's at least 3 degrees of this
Even within DALY measurement there is enormous subjectiity. How does hearing loss compare with itching compare with the death of a child?
Comparing across human realms like your example here of income vs health.
Comparing across cause areas line animal welfare vs human welfare, where we can't even talk with animals to get their subjective opinion.
Personally I can liver with the end of the first 2 but currently 50/50 on whether it's worth trying to quantify number 3.
There are also these people who would rather look at this thing called wellbeing which solves this comparison-between-human issue at least, but I'm not sure if they are really serious.
"There are also these people who would rather look at this thing called wellbeing which solves this comparison-between-human issue at least, but I'm not sure if they are really serious."
The only thing that I take seriously ... is happiness. :)
But yes, while it is a completely standard thing to do, we standardly conclude, on reflection, that we should change what we standardly do.
Thanks for this valuable comment. The RP report was the first time for me to learn about the attribution of a certain value to a life saved based on age. Once you are aware about it, you see it and understand it, but it is not really discussed. I found it quite difficult to work with these values as it somehow dilutes/obscures what the findings were. Additionally I find it morally challenging as the people who benefit from the intervention might value these lives completely differently.
Not sure how we can handle that well in our communication without getting people lost in technicalities, so if you would have suggestions for that I would greatly appreciate that.
This programme is focused on saving lives, right? In your shoes, I would probably lead with the cost to save a life, followed by cost per DALYs. I'd probably mention the multiples of cash transfers, but if I did I'd always try to follow it up by saying this relies on some standard methodological assumptions from XYZ, and I recognise income and health are not easily comparable.
People pretend they understand DALY, QALYs, and WELLBYs, but I don't think anyone really has much of a feel for how much good each represents...
Full disclosure: I’m also a co-founder of GOAL 3, so definitely not an unbiased observer here.
What I find an interesting dilemma is how we should balance further RCT/evidence generation vs. scaling a proven intervention (for medical professionals)? What are the ideas on this for the forum?
76x cash transfers: too good to be true or a (missed) opportunity? An invitation for dialogue on a recent Rethink Priorities Report
TL;DR
* GOAL 3 is a social enterprise that created IMPALA, a patient monitoring and digital platform system for low-resource settings. There is also a GOAL 3 foundation that raises funds to ensure underserved communities will be reached.
* A recently completed report from Rethink Priorities calculated IMPALA to have a cost per DALY averted of $7–17 in paediatr...
This month effective altruism has gotten lots of attention, including a cover story in The Economist. It feels pretty surreal to be on the receiving end of it. But I feel grateful these ideas have caught on, and even more grateful for the drive, earnestness, and compassion of so many people who took strange but well-argued ideas seriously and then actually acted on them....
Note: This post was crossposted from the Coefficient Giving Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: On vegan advocacy, effective altruism, and FIFA
I turn 40 today. Here are some hot takes.
On factory farming
1. Our biggest challenge is salience. If factory farming led the evening news, it wouldn’t last lo...
GOAL 3 is a social enterprise that created IMPALA, a patient monitoring and digital platform system for low-resource settings. There is also a GOAL 3 foundation that raises funds to ensure underserved communities will be reached.
A recently completed report from Rethink Priorities calculated IMPALA to have a cost per DALY averted of $7–17 in paediatric care and $12–30 in neonatal care. This translates to around 76× and 28× unconditional cash transfers respectively for paediatrics and neonates against GiveWell's bar, and roughly 11,700× and 4,400× on Coefficient Giving's scale against their 1,000× bar.
RP corrected for study design and robustness of findings by applying a 60% internal-validity and a 30% external-validity discount to our measured mortality effects, arriving at roughly 12% for paediatric wards and 7% for neonatal units.
RP concluded that even at 1-2% relative reduction, the intervention would still be cost-effective.
The post concludes with an invitation for discussion on what these results mean for further research and scaling of IMPALA.
Introduction
GOAL 3 commissioned the Rethink Priorities Global Health and Development team to provide an external analysis of the impact and cost-effectiveness of our product ‘IMPALA.’ IMPALA is a continuous patient monitor combined with a digital health platform designed for low-resource settings (links with more info in the appendix). The report estimates that IMPALA is highly cost-effective, far surpassing every funding threshold considered. We would like to use this post to discuss the studied intervention, the results, the questions that may arise as a result, and how we can address these now or in the future.
Understanding the problem in its context
Sub-Saharan Africa accounted for 2.8 million of the 4.9 million deaths of children under five worldwide in 2024.[1] Access to care alone does not prevent these deaths. In low- and middle-income countries, an estimated 58% of all amenable deaths from conditions treatable by health care occur among people who did use the health system but received sub standard-quality care.[2] Important root causes to this challenge are lack of adequate equipment and shortage of staff.
In many low-and-middle-income countries (LMICs) there is a huge shortage of staff. WHO uses 4.45 doctors, nurses and midwives per 1,000 people as an indicative minimum for progress on the health-related Sustainable Development Goals.[3] A survey of 47 countries found that the WHO African Region had an average of 1.5 doctors, nurses and midwives per 1,000 people, about a third of WHO's minimum (Malawi was even below 0.5, less than a ninth).[4] Moreover, staff lack the equipment they need to monitor and manage their patients with up to 40% of equipment malfunctioning. Qualitative studies in paediatric high-dependency units in Malawi describe monitoring as intermittent, with nurses forced to focus on the sickest children while staff shortages, power cuts and too few working devices get in the way. [5]
In these conditions, health workers are often unable to recognize and act upon patient deterioration, forcing them into reactive care; leading to late stage response to emergencies when treatments are more costly and more time consuming while outcomes are poor. This sets off a vicious cycle whereby scarce time and resources are used on late-stage escape treatments, further hampering the ability of the health systems to respond in time to changing patient conditions.
Theory of Change
IMPALA enables health workers to monitor patients continuous with health workers receiving alerts when a patient deteriorates. As a result, health workers can recognise patient deterioration earlier and prioritise care more effectively. Monitoring also reduces staff workload. By automating the process of checking vital signs, IMPALA helps save hours of work for health workers every shift[6], allowing them to focus their attention where it is most needed.
Additionally, when patients are treated earlier, complications can be prevented and treatment demands fewer resources. This means less medicine use, less patient costs, and reduced time spent on treating the child by the health workers. The result is reduced health expenditure. [7]
Importantly, IMPALA is not an intervention that increases the burden on already overworked health workers but is embedded into their existing workflows and makes this easier. Each implementation includes on-site training for the ward team, a group of trained IMPALA champions among the hospital's own staff who support colleagues and onboard new starters, and ongoing follow-up from our in-country team through IMPALA Care.
Figure 1: GOAL 3 Theory of Change of IMPALA
What the RP report tells us
Rethink Priorities' Global Health and Development team took on the challenge of evaluating our evidence, commissioned by GOAL 3. RP reviewed nine mortality estimates from hospitals in Malawi, Rwanda and Tanzania. Eight of these point towards lower mortality. [8]
RP's reading is that our evidence consistently points to a mortality benefit but does not establish its size, so its model discounts heavily. RP combined our evidence into weighted averages which led to the number of 42% mortality reduction for paediatric wards and 26% mortality reduction for neonatal units. It then applied a 60% internal validity discount and a 30% external validity discount, in line with how GiveWell treats non-randomised evidence.[9]That left a modelled effect of 12% mortality reduction in paediatric wards and 7% in neonatal units.
RP then estimated cost-effectiveness using its full model (see full model parameters and results here). It puts the full cost to the health provider at $6,100 per ward per year (annualized over IMPALA's lifetime), and assumes 3,000 admissions per year on a paediatric ward and 1,000 on a neonatal unit. On that basis, it estimates that IMPALA costs $7–17 per DALY averted in paediatric care and $12–30 in neonatal care. This translates to around 76× and 28× unconditional cash transfers against GiveWell's bar, and roughly 11,700× and 4,400× on Coefficient Giving's scale against their 1,000× bar for pediatric and neonatal care respectively. The full table from the report can be seen below in figure 2.
Figure 2: Cost-effectiveness of IMPALA from different funder perspectives.
Interpretation, strengths and limitations
A model that returns 76x cash transfers could make any reader sceptical and we understand that. It is important however, to root this in understanding of the intervention and its context. The intervention works (in contrast to many public health interventions like malaria nets or vitamin A) within an environment in which already many resources are deployed. IMPALA allows hospitals to optimize the use of these scarce resources for the patients' benefit by automating repetitive tasks and ensuring essential information is readily available. Because patients have less complications and go home faster it further relieves pressure on the health system which benefits quality of care but also contributes to workload reduction and cost savings. This was found in a cost-effectiveness study recently published in the BMJ Paediatrics Open.[7] We like to think of IMPALA as adding oil to a rusty machine. All components are already there, we just allow them to work more effectively together.
However, we want to be clear about what the evidence does not yet show.
There is no randomised trial as part of our evidence. Most estimates compare a ward before and after IMPALA, so anything else that changed between those periods can look like an IMPALA effect. In our only controlled study, economic and health effects (incl. cyclones, drought, and economic crises) outside of our control may have influenced the results. As the study had a single control per site, RP notes, a shock at that one facility is hard to separate from a real trend.
Several estimates also show little or no effect of IMPALA. Their confidence intervals cross zero, or, the lower bound sits at a 1-2% reduction. Furthermore, large effects measured in small samples tend to shrink as more data comes in. GOAL 3's service model includes ongoing support, which may preserve some of that, but we cannot yet say how much. GOAL 3 has also published its non-significant results and has been transparent with all the data, including one hospital where mortality did not change, which RP took as a sign that they were assessing the full body of evidence rather than a favourable selection.
Reasons the estimate may be too low or too high
May be too low
May be too high
Mortality is the only benefit counted. Averted morbidity is excluded, though on our figures it is 3–4% of DALYs.
The model contains no counterfactual adjustment: no allowance for wards that would have obtained monitoring anyway.
Hospital and family cost savings are excluded. Avoided bed-days are directionally favourable at 8,400-10,920 per hospital per year against an annual system cost of $6,100, though not statistically significant.
No effect decay is modelled across the seven-year equipment life. Our longest follow-up is 24 months. So far we see increasing adoption and usage, but this has not been systematically assessed.
Costs are taken at maximum. No at-scale pricing is assumed, though we expect 20–30% reductions when implementing at volume.
Post-discharge mortality is unmeasured in every study. Some deaths may be deferred rather than averted.
The validity discounts may be too steep, particularly the 30% external validity discount. RP notes that quasi-experimental data is a reason not to discount external validity too heavily, because the intervention was observed in ordinary wards rather than trial conditions.[10]
The largest paediatric result depends on spillover effects. At Zomba, roughly 93% of the children in the population showing the benefit were not on a monitor.
Impact on health worker stress levels and potential health worker retention is not valued. Attrition is a first-order constraint in these systems and we have satisfaction data but no retention data.
Several individual estimates are statistically compatible with no effect.
The results that were currently found were mostly from 2023 and 2024, since then we have improved both the product and the service model to further enhance adoption
Early hospitals may have had an above average level of innovation readiness compared to scaling hospitals (likely already accounted for in applied discounts).
Please note we are not claiming the pros and cons balance. We are setting out both so that readers can apply their own weights.
Simultaneously, we still think the case holds up. As RP notes: “A consistent direction across independent implementations is harder to explain by chance than any one result considered alone.” The body of evidence is wide enough to show that IMPALA consistently points in the direction of mortality reduction across settings.
The starting point also matters. Standard care in the Malawian HDU in these studies is manual observation every six hours. A study in Kenya found hospitals completing only about half of the intended manual checks.[11] Against that baseline, a modest improvement is far more plausible than it would be in a well-resourced hospital.
Finally, the cost-effectiveness calculation from RP does not depend on a large effect. Implausible cost-effectiveness figures usually come from implausible effect sizes. Here the modelled effects of 12% and 7% sit well below every favourable site estimate. As RP also noted, “We believe this is a robust conclusion because it is driven by low cost rather than a large effect: the break-even mortality reduction is only ~1% (pediatric) and ~2% (neonatal), so the conclusion holds even if there is only a very minor mortality benefit.”
For IMPALA to fall below GiveWell's bar, it would have to save almost no lives, and the evidence (not just our own, but also evidence from similar interventions) points the other way.
Discussion: What would it take to fund scale-up now? How much more evidence, and of what kind?
We are publishing this because we want to hear what you as the EA community have to think about this result.
The dilemma. We are in more than 60 hospitals, with about 1,000 monitors. The main binding constraint on starting in new hospitals is capital: governments in Malawi, Rwanda, Tanzania and Kenya are asking for this and will fund recurring service costs (Malawi's Ministry of Health has committed to this in writing), but cannot fund the upfront investment. We also see demand in our own work: hospitals in our installed base asking for paediatric coverage, the Beginnings Fund's neonatal rollout, and Queen Elizabeth Central Hospital asking to expand its use of IMPALA. RP's model shows continuous monitoring to likely be extremely cost-effective. We believe this evidence is enough to answer the question of whether patient monitoring is a cost-effective intervention in low-resource settings. However, funders often tell us to come back with an RCT, which we estimate would need around 72 hospitals across three countries, US$5 million and three to four years.
Additionally, as RP also notes, a conventional RCT is hard to run for this intervention. It can't be blinded, because health workers know when monitors are on the ward. It has to randomise whole hospitals, because the central patient overview and the positive impact on workload affects care for every child on the ward, not only those attached to a monitor. Death is a relatively rare outcome, and baseline neonatal mortality across our sites ranges from 4.3% to 27.8%, which makes a trial harder to power. Ministries are also reluctant to keep hospitals in a control arm for years while neighbouring hospitals have monitoring. So is an RCT needed before scaling? If the effect is real, waiting three to four years for trial results means children in hospitals that could have had monitoring go without it. Last but not least it is important to emphasize that continuous monitoring is already considered standard of care in most parts of the world, so perhaps it is less about if it works but rather to what extent it should be prioritized amidst many other priorities and gaps.
Our take. The opportunity cost of waiting is too high. Decision-relevant gaps are not whether patient monitoring is effective. We see gaps in areas like: heterogeneity across settings, durability beyond 24 months, and post-discharge outcomes. We recognise that funding IMPALA before an RCT is a higher-risk bet. In a recent post Jack Lewars argues that donors don't always need an organisation to have run its own RCT.[12] They can instead look for monitoring, evaluation and learning plans calibrated to the intervention, with outputs, outcomes and accurate costs measured against targets set in advance. He is also clear that this makes a grant riskier, and his first test is whether the intervention itself has RCT evidence, which continuous monitoring in these settings does not yet have. We think our evidence meets his middle-ground standard, and we are working to close the remaining gap. We do think a stepped-wedge RCT, led by the Amsterdam Institute of Global Health and Development (AIGHD), is still a valuable long-run design, but nested in a funded rollout as we scale, not before. We see a trial as one route to stronger evidence; further real-world evidence studies, such as the planned evaluation of our installed base, are another.
Our questions
How much weight would you put on this report from Rethink Priorities? Do you see critical gaps in the analysis (positive or negative)? How much more evidence and what kind would you like to see to say with confidence that IMPALA is a cost-effective health intervention?
If you were a funder, what is the biggest concern you would need answered before supporting IMPALA?
Where does an organisation like GOAL 3 fit in the EA global health landscape? We can think of reasons EA funders might hesitate: our hybrid social enterprise and foundation model, a hardware and health-system intervention rather than a commodity like bed nets, and open questions about who should pay at scale. Which of these, if any, matter most, and are there others we're missing?
Next steps
Let us know what you think
We present what we have for feedback from the community. We are genuinely interested in the questions presented above, and want to hear what you have to say on it. Both us and RP will be available to answer questions in the comments.
Talk to us
We'll be at the organisation fair at EA Global New York, and I'm joining a panel at EAGxBerlin. On Wednesday November 4 at 16:00 CET (10am EST) we will host a webinar together with Rethink Priorities to discuss the report further (sign-up here). You can also email [email protected] to set up a call.
Conflict of Interest
GOAL 3 commissioned and paid for the report; RP had full editorial independence over its conclusions.
This post is published by Niek Versteegde, Founder and CEO of GOAL 3. It is in my interest to promote IMPALA both from a business and impact perspective.
AI Disclaimer
AI (Claude & ChatGPT) were used to help with formatting, spell check and improving general flow. All text was checked before posting.
I want to read more about IMPALA, where can I do that?
Our team has spent the past months making our work, and how we do it, open for anyone to read in depth. You can also hear from us directly: join our next webinar (sign-up here), or email [email protected] and we'll set up a 1:1 call.
Alongside sharing RP's report, we're using this post to officially open theGOAL 3 Public Knowledge Hub on Notion. The Start Here page is the best entry point. The links below take you straight to what you're most likely looking for.
TheEvidence Summary Dashboard goes claim by claim. For each step in our theory of change, it sets out what we claim, which sources support it, how strong we think the evidence is and which questions remain open.
If you want to understand the IMPALA product:
Our product: IMPALA covers the IMPALA Monitor, the IMPALA Platform, the IMPALA Care service model, and how IMPALA differs from other monitoring options.
The costs of IMPALA breaks down the cost of an implementation for a hospital.
If you want to understand the organisation:
Our organization explains where GOAL 3 came from, the problem we work on and our theory of change.
How we work explains our hybrid structure as a social enterprise with a foundation, and our governance.
Absorption capacity sets out how much additional funding we could put to use, financially, operationally and strategically.
Risks, limitations and safeguards are where we set out what could go wrong and how we manage it. If you only have time for one page after this post, we'd suggest this one.
The hub is a living resource, and we update it as new studies come in. If something is missing, unclear or wrong, please tell us in the comments or at [email protected].
Kruk et al. (2018). Mortality due to low-quality health systems in the universal health coverage era: a systematic analysis of amenable deaths in 137 countries. Lancet. https://pubmed.ncbi.nlm.nih.gov/30195398/
Rakers et al. (2024). Cautiously optimistic: paediatric critical care nurses’ perspectives on data-driven algorithms in low-resource settings—a human-centred design study in Malawi. https://link.springer.com/article/10.1186/s44263-024-00108-8
The ninth estimate, from the neonatal unit at Rwamagana Hospital in Rwanda, was essentially flat: a 0.1 percentage-point increase in mortality (95% CI −1.7 to +1.5 pp, p = 0.90), from an uncontrolled before-after comparison. RP interprets this as noise around no effect rather than a sign of harm. Rwamagana also had the lowest baseline neonatal mortality of the neonatal units in our studies, 4.3% before IMPALA, compared with 5.6–27.8% at the other sites, which left little room for a measurable reduction. We think this is the most likely explanation, and it suggests IMPALA's effect may be smaller where outcomes are already relatively good at baseline.
Both discounts are subjective judgements, and RP notes that others could reasonably reach different values. The 60% internal validity discount is in line with how GiveWell treats non-randomised evidence, and given that our evidence is entirely non-randomised, we see it as a reasonable, if cautious, choice. We think the external validity discount is more likely to be too harsh. The studies took place in routine wards across three countries, and IMPALA now runs in more than 60 hospitals in six countries, although RP was only able to assess effects in a small number of them and says it would revisit the discount as evidence from more settings comes in.
Ogero et al. (2018). An observational study of monitoring of vital signs in children admitted to Kenyan hospitals: an insight into the quality of nursing care? Journal of global health. https://doi.org/10.7189/jogh.08.010409
At the end of the day, the Ben et al. study is of low quality and questionable integrity. RP's subjective discounting and limited engagement with the actual validity of the study, its statistical methods and its confidence intervals obscures how unreliable this study seems. Therefore, this independent review from RP should not change much with respect to our confidence in Impala's effect size. Better, actually independent research is needed before we can estimate the effectiveness of the intervention with any certainty.
A few of the issues with the study, beyond what has been raised by RP:
tl;dr: subjective discounting is not a replacement for critically engaging with evidence quality. The 3x discounting can give the reader the impression that the researchers are being very conservative, but a 12% effect size is still moderately large in absolute terms and large relative to other interventions. It is huge given that most of the effects are from spillover, and mostly covers a reduction in deaths from children not actually monitored by the device. So I warn any readers against accepting the framing that a 0.5-1pp mortality reduction across the entire hospital is a very conservative estimate.
Thanks Martijn. This is the kind of critical reflection we hoped for when we opened this up, and I’m grateful you took the time to go through the paper, its supplement and the review history in this much detail. These are important points, and they add nuance to how the evidence should be read.
On the conflict of interest: you are right. Our commercial interest should have been declared from the first submission, and it wasn’t until the final version. That was an oversight because we did not handle the submission process and we assumed it was done, not a deliberate choice, but it remains a mistake and I share responsibility for it as a co-author. Readers should weigh the paper with that in mind. The same applies to the RP report: GOAL 3 commissioned and paid for it.
I do want to respond to one word: “integrity.” I understand why the late COI declaration raises that question, and that one is on us. But I’d ask you to separate it from the rest of the work. The study was designed, run and analysed by researchers from six independent academic institutions in Malawi and the Netherlands, with GOAL 3 staff contributing to data collection and writing, not the analysis. The full results were published, including the HDU result that was not significant and the increase in crude HDU mortality, and the data are available for replication. Questioning the methods is fair and useful, and I will aim to get the answers as soon as possible. Questioning the integrity of the researchers, many of whom have no stake in IMPALA, is a different claim, and I don’t think that does justice to all the people who have contributed. I’d appreciate it if you’d reconsider that framing.
On the statistical points: I’m a clinician, not an epidemiologist, unfortunately the methods in this study (TMLE, the matched sensitivity analysis, how the estimates are defined) are beyond what I can judge with confidence. I don’t want to explain/defend something I can’t fully assess. I’ll ask the lead authors to respond to your points on the HDU estimate, the matched analysis, the case-mix adjustment and the data breakdown at St Luke’s, and I’ll share their answer here. The paper says anonymised data are available for replication, and I’d support an independent reanalysis. I’ll also ask Rethink Priorities to respond to your point on how they weighed this study, since that question is fairly theirs to answer.
Some broader context. I agree the Zomba HDU result can’t carry weight on its own, and that the ward-level results depend on spillover, which this study did not test directly. That’s why it may help to look at the rest of the evidence:
We understand our evidence is not super-strong, eight of nine mortality estimates point in the same direction, but only some are statistically significant. None are randomised, and most were co-authored by GOAL 3 staff. A consistent direction makes chance less likely, but it doesn’t remove biases the studies share, such as before-after confounding. So my honest summary is that the evidence points towards a benefit but does not establish its size, and your critique makes clear the uncertainty is wider than a single discounted figure suggests. That’s also why we still want an independent, prospective controlled evaluation alongside scale-up, as discussed with Nick above.
Thanks again. This is exactly the kind of scrutiny that makes the evidence base better and helps to improve our understanding as well where we have blind spots and need to make additional steps, which was also one of the key reasons to ask RP to assess our evidence.
I hope the process will also help to build your confidence in us and that we are as truthful and transparent as we can be in search of improvement and sharpening our reasoning, evidence (including interpretation) and expanding our impact.
Regards,
Niek
Thanks for the reply Niek. Perhaps I will have time to respond more in depth later but let me just say an important bit:
I said the study and the way it has come about is of questionable integrity. It wasn’t an assessment of the integrity of the people involved.
That being said, we shouldn’t just assume everyone involved in the study to have acted under complete integrity either, given the COI. We have no visibility on the extent to which COI’d authors were involved in editorial decisions during writing, methodological decisions during analysis and data-presentation as well as operational decisions during the study’s actual implementation. In all of these, I’m more concerned about bias (as distinct from lack of integrity), so that’s a good reason to keep matters of personal integrity out of it.
One more addition --> we have planned to use all feedback and insights from this Forum to further sharpen our own evidence overview which we have created on notion. So our own interpretation and representation of the evidence will be improved. If you are interested, please have a look. All feedback (positive and negative) is welcome and will be used to further improve the overviews. https://goal3knowledgehub.notion.site/public-knowledge-hub?pvs=143
Specific recommendations or suggestions for further research (beyond a trial as we already have planned that but will need funding first) are also much welcomed.