A) We want it to be representative so only selecting referral hospitals does not do the job as it will not inform a ministry of health what will happen in rural areas. The challenge is that hospitals vary quite a bit, so you have to match them first for certain characteristics and then randomly assign them. The larger the variation the more hospitals you will need, hence we came at 72 through our power calculations. The challenge is most funds only go up to 2 or 3 million USD
B) I think you are right from where we are today, but there have been many moments where it was likely we would not survive. We have our own development team, registrations to maintain all requiring significant investments. If it takes too long, likelihood of dying in the valley of death will increase. There are very few innovators who are actually successful in public health systems for this reason.
Scaling slow also has high opportunity costs. We have made calculations based on our growth projections (hospitals reached) and our impact per hospital (mortality data). I can not share these now, but let's use a fictive scenario that is close to our estimates:
Let's assume Rethink Priorities assessment is right and each hospital using IMPALA will save 7 newborn lives per year and 12,2 pediatric lives per year. If we grow according to our plans we will reach~1000 hospitals by 2030 and IMPALA will help avert ~19,200 deaths every year.
If this is delayed and only 500 hospitals are reached, this means a difference of ~9,600 lives saved in a single year and cumulatively a lot more. At the levels of our own results this would be 4-5 times higher. Of course I also understand it could be significantly lower (but as rethink priorities said even at 1-2% reduction it would still be a good investment). The longer we wait the higher the opportunity costs will be.
My point is not that we should release all breaks and all funding should go to us to scale. However I think it is more than justifiable to invest significantly in a large(r) scale program with thorough evaluation, which could yield results a lot faster and generate more decision critical evidence.
As a few follow-up question to you (and the rest of the forum): What study results (number of sites, quality etc), not from a trial, would convince you and why? If only a trial can convince you why is that? What evidence does it deliver that routine evaluation at scale could not?
C) In any research setting there will be research staff and a lot more data collection, this by itself has a huge effect on how care is provided. Moreover not all sites will be suitable for it, creating a significant selection bias. Therefore the performance in (optimal) trial circumstances often has limited meaning for what it means in real life. Moreover variations across countries, will likely influence success of the intervention. So generalizability is also limited (Why would someone in West Africa, South-East asia or the middle east believe the trial from East Africa) Real world evidence solves a large part of this, whether if it is implementation research or derived from the real world evaluation. We look at each installation for usage, adoption and impact on health workers stress and burnout, where possible (if funding allows) we look at impact on mortality. While the quality of the evidence may be lower, it says a whole lot more about what is actually happening within the specific hospital/country and provides opportunities to intervene (either through us, or the hospital itself) Seeing this over multiple hospitals in one country will help a lot more with decision making then a (potentially outdated) trial from 5000km away (as a figure of speech).
Again, not opposing a trial here just wonder if it is the right instrument for the stage we are at. Just to provide a different perspective. A 5 million implementation program, would allow me to go to 200-300 hospitals in different countries and perform a thorough monitoring and evaluation program in parallel which can look at before-after differences, but also at differences between other hospitals. Those results would be available within 1 year after implementation, so likely 2 years after a grant was awarded.
D) Don't think Claude is accurate. Without a verifiable source on the mortality drop that answer is pretty useless, beyond that it is not wise to assume a conclusion that this is common based on one site. Data quality is notoriously bad in this setting and there is severe under reporting. We have solved this by diving into individual patient files hand by hand (not us, but our research partners)
The study you cite is hardly comparable in my view - general wards versus ICU/HDU - High resource settings only - high levels of staffing and no lack of monitoring equipment - Only adults - Vast majority is surgical patients - total number of patients only 1284 over all studies combined - relative rare occurance of the events making them statistically underpowered to proove significant impact
Moreover if you dive into their discussion you also find that over multiple systematic reviews all findings are consistently positive favoring monitoring, yet not significant. That is relevant to our discussion, because one of the key conclusions is that even at 1-2% mortality reduction IMPALA would still be cost-effective against any bar.
Hi Nick, thanks for your message. Happy to jump on a call myself as well to dive deeper feel free to email ([email protected]) or text via the chat function of the forum (if it has that)
as a first quick reflection/response, hopefully I have more time later this week but here are a few reflections, and apologies for not following your exact numbering.
A) we have been trying over the past 4 years to get trials funded (EDCTP, givewell, EU funding, Gates) without success. There are some ways to make it lower cost (lowest we got was about 2,5M, but that also reduces quality and strength of evidence). In between time we have done what we could through different smaller sources of funding to get the best evidence possible. Agree by the way that blinding is not needed and not relevant and that a large effect actually comes from the behavior change due to its clear presence on the ward. Also randomization is not an issue as long as control sites also get the intervention.
B) trials can be great, but if we wait for those results to come out before scaling we would likely not survive as an organization. This is not unique for us, but for all innovations and a major bottleneck for innovative solutions in general. Moreover, 3 years of waiting to scale and finding out that the results are same, would create a huge amount of deaths not averted, happy to provide more precise estimates at a later stage. Or plea is not to skip the trial, our plea is to do it in parallel.
C) While I like trials I actually think real world evidence is better or at least as important. With monitoring and many other interventions it is not about if they work, but how do they work in a very complex environment that is under-resourced and understaffed. In other words, if I would be the minister of health, I would value evidence from 100% of 50 sites in a real world setting a lot higher than the evidence from the intervention under trial circumstances. Just to be clear --> we are doing all this (we are currently running a controlled interrupted time series in multiple countries and multiple other impact evaluations are still ongoing) and still want to do the trial if we can get the funding for it.
D) A trial as the definitive answer to answer the question if something should be scaled is not rational from a perspective of levels of uncertainty. In my view the lower bound of the margin of certainty just needs to be above (or close to) the treshhold because if it is above that level it would not be rational to say we need a trial first before scaling it up, because the statistics already say it is highly unlikely that it is. I sometimes use the picture at the bottom to describe that (NB if evidence changes the direction of the arrow may also change). Of course it is good to discuss if you agree to the estimated impact and the margins of uncertainty to that which Rethink priorities found, because I do think that is important to agree on (or disagree and find ways to close the gap)
E) too salesy --> I think you may underestimate how important the technology, service-model and business model are compared to anything else that is out there. There are dozens if not hundreds of changes we have done and are continuously doing to improve the intervention (at GOAL 3 level) and at the facility (to improve adoption and usage at facility level) all embedded in the business model which allows us to do continue to do this over time. We do not describe this here in this post to sell or brand our solution, but also to make clear that just putting a monitor down will not have the same effect.
Last but not least: it is also a product of passion and enthusiasm that comes from building 8+ years towards this solution. I just can't be fully objective in how I describe our solution. Firstly because it is the answer to the problems I experienced myself when working in the field, secondly I think there are a lot of lessons to learn from our success that can be used by others. I hope you can accept that (and maybe even appreciate it ;-) )
F) thanks again. Really appreciate the open discussion. Would be great to connect
Apologies for a belated response. You are correct that there are quite large seasonal influences. So we matched everything with the respective months in the same period to compensate for it. 2 weeks ago we also got the individual patient level data. This data shows that the post-intervention group was actually on average in a worse condition than the pre-intervention group. If this is taking into account the mortality reduction is close to 50%.
Sorry for a delayed response. But indeed you are right abou this. However, the costs at the hospital staff for availing staff is actually very low. Typically it is 20-30 people for 1 day, the costs are negligible in relation to the 10 year total costs. Moreover in-service training is part and parcel of every nurses job and part of the hospitals responsibilities. That is why we have not counted it so far.
In the coming months we expect to have a proper assessment of the impact on workload and costs and can more clearly describe it as a benefit. Initial results from one hospital indicate that IMPALA is leading to a significant cost-reduction. Will keep you posted about developments.
I guess then it's just my title: In God we trust, but all others must bring data. It just feels frustrating sometimes, also because people actually don't say they don't believe you. So if you don't call me a liar, then why don't you help or support me?
I would really like it if a more systematic way of identifying, testing and scaling successful innovations was available.
My post are definitely not intended to complain by the way. I am blessed with a great and supportive team and have great partners as well. It's a genuine interest to understand, learn and improve.
Actually we think these are all costs. To clarify a bit: we are operating as a social enterprise. The prices mentioned here would be what the hospital/NGO/government pays. Implementation is done with local staff which is quite affordable, after implementation everything is done by hospital staff. There is only limited need for support and maintenance which is all done by local teams which keeps costs low. Because (assuming we will reach sufficient volume) there are margins on the product and services this will ultimately pay for all international staff including mine.
Hospital staff time/costs goes down after the intervention, because the system automates repetitive tasks. Moreover we see a 10% shorter admission time, which will also have a positive impact on workload.
Soon we will publish more results showing that both costs for the health system and the patient will go down after the intervention. Based on what we see so far it is even very likely that our intervention is net cost saving over time.
We didn't include that in this analysis because we did not yet have the results and it again complicates it further.
Thanks a lot for your candid feedback. This is exactly what I hoped to get from this forum.
Honestly speaking it has been quite a struggle to think what could be the right tone and what information should be included and what not. What we do is not that simple and the context in which we operate isn't either. I guess you have captured my failure to strike the right balance well with your feedback.
What might help to better understand what we do is our product video. I didn't include it before because I don't want people to feel like I am here to promote my work. Curious if you feel it should be included.
Regarding the costs remark, actually everything is included in the costs that we presented here. You can find a detailed breakdown with rationale at the bottom of this document from which I took the screenshot below. If we can achieve sufficient scale (>5000 devices) we can deliver and sustain it at the costs presented below and likely for sifnificantly less if we achieve a bigger scale.
Unfortunately I do not have time to rewrite and organize everything right now, but hopefully I can make some improvements over the weekend. If you are open to it, I would gladly receive your feedback before updating the post.
PS. over the next days I will try to add a bit more background about the solution and our evidence, where we think it needs to be strenthened. I was a bit reluctant to put too much information in, because I thougt people might not read it.
Hi Ian, thank you for your thoughtful feedback. We fully agree that independent evaluations and randomized controlled trials (RCTs) are an important tools for understanding impact. However, we must also acknowledge that RCTs come with significant costs, long timelines, and often produce results that don’t fully translate into real-world settings. Our experience has shown that the complexity of low-resource healthcare environments requires adaptable, on-the-ground solutions that are continuously tested and refined in real-time.
That’s why, at GOAL 3, continuous evaluation and learning are embedded into our approach, aimed at Real World Evidence. We feel that waiting for a one-off trial to tell us whether IMPALA works is not appropriate. Instead, we are committed to assessing our effectiveness at every site in the real-world, following up over time, and ensuring year-on-year improvement. This approach allows us to make iterative and localized changes based on the needs of health workers and patients, ensuring we create a sustainable impact that lasts. Our goal is to see a measurable improvement not just immediately after implementation but as a consistent part of the health system's development.
Of course we do want this to be evaluated independently, to also assess unanticipated negative and/or positive impact from the intervention. We know we are biased and we understand that we are not the best evaluators. That is why we leverage the Founders Pledge grant to ensure independent research institutes can do this evaluation across different settings. This will strenthen the evidence on effectiveness across different settings with a larger poule (7) hospitals, over time (>1year follow-up) while also assessing differences.
On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency. The slowness of decision-making in global health is an enormous barrier to innovation. Many life-saving solutions are delayed because innovators and startups simply don’t have the time or resources to wait for trial results. During COVID-19, we saw how innovations were fast-tracked to address an urgent crisis. It feels unfair that the 5 million pediatric deaths every year are not considered urgent enough to prioritize similarly fast solutions. Basically the risk of unanticipated consequences should be balanced against the potential positive impact and use that to navigate decisions about
Ultimately I agree with the process and I should not complain after receiving a grant from Founders Pledge to support us in building the evidence base that is needed. Its just that I am frustrated because it feels like a continuous fight to move things forward or even survive, while deep in my heart I know we are right and that every delay will hinder us from preventing deaths that are unneeded.
A) We want it to be representative so only selecting referral hospitals does not do the job as it will not inform a ministry of health what will happen in rural areas. The challenge is that hospitals vary quite a bit, so you have to match them first for certain characteristics and then randomly assign them. The larger the variation the more hospitals you will need, hence we came at 72 through our power calculations. The challenge is most funds only go up to 2 or 3 million USD
B) I think you are right from where we are today, but there have been many moments where it was likely we would not survive. We have our own development team, registrations to maintain all requiring significant investments. If it takes too long, likelihood of dying in the valley of death will increase. There are very few innovators who are actually successful in public health systems for this reason.
Scaling slow also has high opportunity costs. We have made calculations based on our growth projections (hospitals reached) and our impact per hospital (mortality data). I can not share these now, but let's use a fictive scenario that is close to our estimates:
Let's assume Rethink Priorities assessment is right and each hospital using IMPALA will save 7 newborn lives per year and 12,2 pediatric lives per year. If we grow according to our plans we will reach~1000 hospitals by 2030 and IMPALA will help avert ~19,200 deaths every year.
If this is delayed and only 500 hospitals are reached, this means a difference of ~9,600 lives saved in a single year and cumulatively a lot more. At the levels of our own results this would be 4-5 times higher. Of course I also understand it could be significantly lower (but as rethink priorities said even at 1-2% reduction it would still be a good investment). The longer we wait the higher the opportunity costs will be.
My point is not that we should release all breaks and all funding should go to us to scale. However I think it is more than justifiable to invest significantly in a large(r) scale program with thorough evaluation, which could yield results a lot faster and generate more decision critical evidence.
As a few follow-up question to you (and the rest of the forum): What study results (number of sites, quality etc), not from a trial, would convince you and why? If only a trial can convince you why is that? What evidence does it deliver that routine evaluation at scale could not?
C) In any research setting there will be research staff and a lot more data collection, this by itself has a huge effect on how care is provided. Moreover not all sites will be suitable for it, creating a significant selection bias. Therefore the performance in (optimal) trial circumstances often has limited meaning for what it means in real life. Moreover variations across countries, will likely influence success of the intervention. So generalizability is also limited (Why would someone in West Africa, South-East asia or the middle east believe the trial from East Africa) Real world evidence solves a large part of this, whether if it is implementation research or derived from the real world evaluation. We look at each installation for usage, adoption and impact on health workers stress and burnout, where possible (if funding allows) we look at impact on mortality. While the quality of the evidence may be lower, it says a whole lot more about what is actually happening within the specific hospital/country and provides opportunities to intervene (either through us, or the hospital itself) Seeing this over multiple hospitals in one country will help a lot more with decision making then a (potentially outdated) trial from 5000km away (as a figure of speech).
Again, not opposing a trial here just wonder if it is the right instrument for the stage we are at. Just to provide a different perspective. A 5 million implementation program, would allow me to go to 200-300 hospitals in different countries and perform a thorough monitoring and evaluation program in parallel which can look at before-after differences, but also at differences between other hospitals. Those results would be available within 1 year after implementation, so likely 2 years after a grant was awarded.
D) Don't think Claude is accurate. Without a verifiable source on the mortality drop that answer is pretty useless, beyond that it is not wise to assume a conclusion that this is common based on one site. Data quality is notoriously bad in this setting and there is severe under reporting. We have solved this by diving into individual patient files hand by hand (not us, but our research partners)
The study you cite is hardly comparable in my view
- general wards versus ICU/HDU
- High resource settings only - high levels of staffing and no lack of monitoring equipment
- Only adults
- Vast majority is surgical patients
- total number of patients only 1284 over all studies combined
- relative rare occurance of the events making them statistically underpowered to proove significant impact
Moreover if you dive into their discussion you also find that over multiple systematic reviews all findings are consistently positive favoring monitoring, yet not significant. That is relevant to our discussion, because one of the key conclusions is that even at 1-2% mortality reduction IMPALA would still be cost-effective against any bar.
Hi Nick, thanks for your message. Happy to jump on a call myself as well to dive deeper feel free to email ([email protected]) or text via the chat function of the forum (if it has that)
as a first quick reflection/response, hopefully I have more time later this week but here are a few reflections, and apologies for not following your exact numbering.
A) we have been trying over the past 4 years to get trials funded (EDCTP, givewell, EU funding, Gates) without success. There are some ways to make it lower cost (lowest we got was about 2,5M, but that also reduces quality and strength of evidence). In between time we have done what we could through different smaller sources of funding to get the best evidence possible. Agree by the way that blinding is not needed and not relevant and that a large effect actually comes from the behavior change due to its clear presence on the ward. Also randomization is not an issue as long as control sites also get the intervention.
B) trials can be great, but if we wait for those results to come out before scaling we would likely not survive as an organization. This is not unique for us, but for all innovations and a major bottleneck for innovative solutions in general. Moreover, 3 years of waiting to scale and finding out that the results are same, would create a huge amount of deaths not averted, happy to provide more precise estimates at a later stage. Or plea is not to skip the trial, our plea is to do it in parallel.
C) While I like trials I actually think real world evidence is better or at least as important. With monitoring and many other interventions it is not about if they work, but how do they work in a very complex environment that is under-resourced and understaffed. In other words, if I would be the minister of health, I would value evidence from 100% of 50 sites in a real world setting a lot higher than the evidence from the intervention under trial circumstances. Just to be clear --> we are doing all this (we are currently running a controlled interrupted time series in multiple countries and multiple other impact evaluations are still ongoing) and still want to do the trial if we can get the funding for it.
D) A trial as the definitive answer to answer the question if something should be scaled is not rational from a perspective of levels of uncertainty. In my view the lower bound of the margin of certainty just needs to be above (or close to) the treshhold because if it is above that level it would not be rational to say we need a trial first before scaling it up, because the statistics already say it is highly unlikely that it is. I sometimes use the picture at the bottom to describe that (NB if evidence changes the direction of the arrow may also change). Of course it is good to discuss if you agree to the estimated impact and the margins of uncertainty to that which Rethink priorities found, because I do think that is important to agree on (or disagree and find ways to close the gap)
E) too salesy --> I think you may underestimate how important the technology, service-model and business model are compared to anything else that is out there. There are dozens if not hundreds of changes we have done and are continuously doing to improve the intervention (at GOAL 3 level) and at the facility (to improve adoption and usage at facility level) all embedded in the business model which allows us to do continue to do this over time. We do not describe this here in this post to sell or brand our solution, but also to make clear that just putting a monitor down will not have the same effect.
Last but not least: it is also a product of passion and enthusiasm that comes from building 8+ years towards this solution. I just can't be fully objective in how I describe our solution. Firstly because it is the answer to the problems I experienced myself when working in the field, secondly I think there are a lot of lessons to learn from our success that can be used by others. I hope you can accept that (and maybe even appreciate it ;-) )
F) thanks again. Really appreciate the open discussion. Would be great to connect
Hi Toby,
Apologies for a belated response. You are correct that there are quite large seasonal influences. So we matched everything with the respective months in the same period to compensate for it. 2 weeks ago we also got the individual patient level data. This data shows that the post-intervention group was actually on average in a worse condition than the pre-intervention group. If this is taking into account the mortality reduction is close to 50%.
Regards,
Niek
Hi Ian,
Sorry for a delayed response. But indeed you are right abou this. However, the costs at the hospital staff for availing staff is actually very low. Typically it is 20-30 people for 1 day, the costs are negligible in relation to the 10 year total costs. Moreover in-service training is part and parcel of every nurses job and part of the hospitals responsibilities. That is why we have not counted it so far.
In the coming months we expect to have a proper assessment of the impact on workload and costs and can more clearly describe it as a benefit. Initial results from one hospital indicate that IMPALA is leading to a significant cost-reduction. Will keep you posted about developments.
I guess then it's just my title: In God we trust, but all others must bring data. It just feels frustrating sometimes, also because people actually don't say they don't believe you. So if you don't call me a liar, then why don't you help or support me?
I would really like it if a more systematic way of identifying, testing and scaling successful innovations was available.
My post are definitely not intended to complain by the way. I am blessed with a great and supportive team and have great partners as well. It's a genuine interest to understand, learn and improve.
Hi Ian (and John),
Actually we think these are all costs. To clarify a bit: we are operating as a social enterprise. The prices mentioned here would be what the hospital/NGO/government pays. Implementation is done with local staff which is quite affordable, after implementation everything is done by hospital staff. There is only limited need for support and maintenance which is all done by local teams which keeps costs low. Because (assuming we will reach sufficient volume) there are margins on the product and services this will ultimately pay for all international staff including mine.
Hospital staff time/costs goes down after the intervention, because the system automates repetitive tasks. Moreover we see a 10% shorter admission time, which will also have a positive impact on workload.
Soon we will publish more results showing that both costs for the health system and the patient will go down after the intervention. Based on what we see so far it is even very likely that our intervention is net cost saving over time.
We didn't include that in this analysis because we did not yet have the results and it again complicates it further.
Dear John,
Thanks a lot for your candid feedback. This is exactly what I hoped to get from this forum.
Honestly speaking it has been quite a struggle to think what could be the right tone and what information should be included and what not. What we do is not that simple and the context in which we operate isn't either. I guess you have captured my failure to strike the right balance well with your feedback.
What might help to better understand what we do is our product video. I didn't include it before because I don't want people to feel like I am here to promote my work. Curious if you feel it should be included.
Regarding the costs remark, actually everything is included in the costs that we presented here. You can find a detailed breakdown with rationale at the bottom of this document from which I took the screenshot below. If we can achieve sufficient scale (>5000 devices) we can deliver and sustain it at the costs presented below and likely for sifnificantly less if we achieve a bigger scale.
Unfortunately I do not have time to rewrite and organize everything right now, but hopefully I can make some improvements over the weekend. If you are open to it, I would gladly receive your feedback before updating the post.
PS. over the next days I will try to add a bit more background about the solution and our evidence, where we think it needs to be strenthened. I was a bit reluctant to put too much information in, because I thougt people might not read it.
Hi Ian, thank you for your thoughtful feedback. We fully agree that independent evaluations and randomized controlled trials (RCTs) are an important tools for understanding impact. However, we must also acknowledge that RCTs come with significant costs, long timelines, and often produce results that don’t fully translate into real-world settings. Our experience has shown that the complexity of low-resource healthcare environments requires adaptable, on-the-ground solutions that are continuously tested and refined in real-time.
That’s why, at GOAL 3, continuous evaluation and learning are embedded into our approach, aimed at Real World Evidence. We feel that waiting for a one-off trial to tell us whether IMPALA works is not appropriate. Instead, we are committed to assessing our effectiveness at every site in the real-world, following up over time, and ensuring year-on-year improvement. This approach allows us to make iterative and localized changes based on the needs of health workers and patients, ensuring we create a sustainable impact that lasts. Our goal is to see a measurable improvement not just immediately after implementation but as a consistent part of the health system's development.
Of course we do want this to be evaluated independently, to also assess unanticipated negative and/or positive impact from the intervention. We know we are biased and we understand that we are not the best evaluators. That is why we leverage the Founders Pledge grant to ensure independent research institutes can do this evaluation across different settings. This will strenthen the evidence on effectiveness across different settings with a larger poule (7) hospitals, over time (>1year follow-up) while also assessing differences.
On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency. The slowness of decision-making in global health is an enormous barrier to innovation. Many life-saving solutions are delayed because innovators and startups simply don’t have the time or resources to wait for trial results. During COVID-19, we saw how innovations were fast-tracked to address an urgent crisis. It feels unfair that the 5 million pediatric deaths every year are not considered urgent enough to prioritize similarly fast solutions. Basically the risk of unanticipated consequences should be balanced against the potential positive impact and use that to navigate decisions about
Ultimately I agree with the process and I should not complain after receiving a grant from Founders Pledge to support us in building the evidence base that is needed. Its just that I am frustrated because it feels like a continuous fight to move things forward or even survive, while deep in my heart I know we are right and that every delay will hinder us from preventing deaths that are unneeded.
Apologies all, I accidentally posted this without the content at first. Still getting used to the platform.