Thanks for this great response

My only real quibble is with this statement "It seemed unlikely to us that improving monitoring truly has no effect, even if the real mortality reduction is much smaller than observed." I agree with this statement. But how unlikely? You have made the call here that no effect is not just unlikely, but that there is a well under 5% chance that monitoring has no effect. This is where I disagree. Given other null results when RCTs were done on monitoring (in different contexts), and an inconsistent history of before and after study results, I think it would have been better to include null in your confidence interval. I didn't do the research so I have little clue but I would maybe have guestimated the probability of no effect in the 10%-30% range given these priors.

I might have discounted further on the point estimate, but this is perhaps a "how short is a piece of string" situation where you just have to make subjective calls so fair enough here - and you did discount a lot already.

"the RP report does not claim this is the most cost-effective global health intervention." Point estimates matter a lot as this is your best estimate as to the effect. This an interesting philosophical question, but with your eye-popping point estimate and a confidence interval that doesn't include zero effect, I think there is a decent argument that your math at least does seem to claim this is one of most cost-effective global health interventions on earth, even if that isn't your intent. 

Yep completely agree with this from a comms perspective generally, although on the EA forum here I think it was fair enough to lead like they did.

Wow these guys are taking some punches graciously from the rest of us and @MichaelPlant weighs in with a curveball ha.

My main comment here is to reiterate that this is a completely standard thing which most GHD people do, largely because the biggest player GiveWell uses it as their standard. 

My take here is that there are different degrees of "removal" as it were, all with different levels of understanding. We can't who's subjectivity it's just how much we are willing to live with. 

You could classify these differently but I'm my mind there's at least 3 degrees of this

1. Even within DALY measurement there is enormous subjectiity. How does hearing loss compare with itching compare with the death of a child?
2. Comparing across human realms like your example here of income vs health.
3. Comparing across cause areas line animal welfare vs human welfare, where we can't even talk with animals to get their subjective opinion. 

Personally I can liver with the end of the first 2 but currently 50/50 on whether it's worth trying to quantify number 3.

There are also these people who would rather look at this thing called wellbeing which solves  this comparison-between-human issue at least, but I'm not sure if they are really serious.

These are great answers. I do agree that your intervention is more likely to reduce mortality than other montoring interventions for your compelling reasons. I don't think that makes the studies RP and I cited completely irrelevent though and do need to inform our prior to some extent.

My reason for saying I don't agree with RPs confidence intervals are
1) There isn't compelling prior evidence to anchor on
2) Your study design - Interrupted time series in general is great, but confounding can still be huge especially with all the different factors that can affect hospital mortality (no. of admissions, sickness of patinets coming in, seasonality, hospital management quality, staff quality etc.)

I don't think RP's assessment is not "correct" as such. They are doing the best they can on a broad set of assumptions with the evidence you have. Their confidence intervals though in theory give your intervention an under 5% chance of the real effect of your intervention being zero - and I just don't think we can ever be 95% sure from before and after studies in complex settings like a hospital. This statement is based on a number of other occasions where before-and-after results in the development world didn't translate to RCT findings. I would suspect other development experts would agree that. I think 

And if there is even an over 5% chance that your results were due to confounders and chance, then the confidence interval should still include no effect - if that makes sense.

Do you need to convince Minsitrys of Health here? I would imagine they will already think the monitoring machines are great and are keen to sign up. The research is to convince funders and international bodies like the WHO. IMO any positive result from a big RCT, even just in the biggest hospitals would be fine to generalise to smaller hospitals. Like you said if there's any difference detected at all, cost-effectiveness will be very high and unlikely to degrade to meaningless levels at smaller hospitals. The sample size you need is big, so for pragmatism perhaps it would be best to go for the largest, easiest hospitals.

To your overall point, I don't agree that RP's assessment is a reasonable benchmark midpoint on mortality and I don't agree with their confidence intervals. Claiming to be something like the most cost-effective global health intervention on the planet is a bold claim based on the evidence you have so far. In fact there's an argument that confidence intervals should include zero given the lack of RCT evidence around monitoring devices.

 I agree with you if we assumed RP's assessment is sound, then your case would be compelling. On a pragmatic front, lets assume that you and RP are right about this, and the current evidence is compelling enough for scale. Even if you're in the right I think you'll struggle to convince institutional funders of this, because they want to hold ambitious scale plans to the same rigorous standards as other orgs. Most other orgs have RCTs or almost-as- trong evidence showing that what they do can save lives. With the anthropalypse though perhaps you can convince some more risk-taking folks.

On the RCT, I think from an implementation perspective an RCT seems managable as you expand. If you're telling me can go to 200 hospitals in 2 years, then I think you can manage the RCT on that path. Yes it will be harder than your other suggested monitoring but not astronomically harder. Otherwise you will be bigger in 3 years time and funders will still ask the same questions. I get your timing question - it would be OK to wait a year or two but it will need doing at some stage. What if you bundled RCT funding together with the scale funding and bite the bullet that you might roll out to half the number of hospitals?  

On the study I cite - is there a better one you would suggest as the closest to your work? I found 2 pediatric RCTs in low-income countries which looked at different types of monitoring and found no mortality difference. None of these studies not directly comparable, but best practise is often to look at the the best studies we have on a similar topic, and that was the best I could find (it was the one RP quoted too). On the trends in the data I agree, but without statistical significance its pretty weak. 

I think it is an important note that there is likely to be skepticism also due to a mixed history of evidence on patient monitoring. Outside of a number of negative results in adults and kids, famously after big studies continuous fetal monitoring doesn't seem to reduce neonatal mortality which still boggles my mind a bit. I know all of these are very different to what you do, but we still need whatever is the closest reference point, and I think unfortunately it might make the RCT more important than for other intevention. Our priors might be more negative here than for other interventions.
 

Really proud of this response love it! After a slow couple of weeks responding to the flurry of EA press, it's great to see some positive, plain English responses over the last week, and this might be one of the most important.

Also I think it's becoming more clear how important GiveWell and global health is for the reputation and public view of AI safety work. They are an almost unarguable anchor for do-most-good thinking that helps the public take the AI claims more seriously.

Nice one, some quick replies and lets set up a call. So great you are so passionate about this, and I for one think that your org should be well funded and an RCT should be done.

A) Sorry you can't get the RCT funded. Have you tried the new DIV? You're right randomising at hospital level makes it really tough. I feel like you probably need national referral hospitals for the required power. From Claude if you had hospitals roughly with 1,000 admissions a month at 3% mortality, looking for a 20% reduction in mortality you'd need 4-5 massive (national referral) hospitals, or maybe 2 massive ones and 15-20 smaller? 72 feels like too much but I obviously haven't looked into the details, just asked Claude. As your current results seem to show 30ish percent mortality reduction this seems on the difficult but feesable end?

B) I think you are in a great position continue to grow steadily if not scale without better evidence and survive - your organisation is still so young there's plenty of room to grow and then scale. You are in a pretty good position for philanthropy to scale the setup costs (rather than government) if an RCT shows it is this amazing.

C) This one confuses me I don't really understand what you mean by "Real world evidence". Your RCT should ideally be basically just what you do anyway, so wouldn't that be real world evidence? The problem is that you're looking at these mortality drops as evidence enough, where I would see them more as higher level M&E of your program. Your own M&E can never touch an external study for importance.  Orgs like ANSH used mortality drops as a marker for their progrram, but they already know the intervention works from RCTs. I just dont' think it can fly for before/after studies or your own M&E to be the evidence which carries you to scale.

D) I like your chart a lot and I agree with the concept completely! But I don't agree with the levels of uncertainty RP cites, although at least they are pretty big. Given the variation even in your own data, relying on before/after studies, the nature of the effect of these kind of interventions to reduce over time (I think a 7 year horizon is too long for this CEA), and the general trend of hospital mortality improving over time I would put your error bars much wider. Claude told me Kamuzu hospital mortality dropped fom 9% to 3% between 2012 and 2015 of its own accord, so these kind of drops seem not unusual for a bunch of reasons. Also Claude and RP cited a bunch of studies in Western pediatric wards which showed benefits in before/after studies, but then when high quality RCTs were pooled the benefit disappeared. I would have anchored on the best quality research we have on the topic much more heavily than RP did, even though its in a different context. I think if the evidence isn't clear in high income countries, it increases the burden for you somewhat.

https://doaj.org/article/2d05aad6031749878cb5b302f931473f
 

Hey this is super cool!

I think this could be a great intervention, but I disagree on the RCT front. I strongly believe that in the medium to long-term, this intervention should, and will only go to scale after high quality RCTs.

Yours and RP's reasons for not doing it aren't compelling to me. I'm almost a bit confused.. This might actually be one of the easier RCTs to do in the development world. If I was a funder, I would be asking why you aren't in the middle of a big RCT at the moment, or at least why you aren't getting started on one. I agree with the funders.

"Funders often tell us to come back with an RCT, which we estimate would need around 72 hospitals across three countries, US$5 million and three to four years." If this is what's needed, then yeah do it. You can get the 5 million - have you asked GiveWell?  But I'm skeptical you really need 72 hospitals. Mortality rates are high enough in massive hospitals I would imagine if you are in national referral hospitals in capital cities, or similar, then this number would drastically drop. Maybe you are considering a bunch of smaller hospitals? 

From your article
"conventional RCT is hard to run for this intervention. It can't be blinded, because health workers know when monitors are on the ward. It has to randomise whole hospitals, because the central patient overview and the positive impact on workload affects care for every child on the ward, not only those attached to a monitor. Death is a relatively rare outcome, and baseline neonatal mortality across our sites ranges from 4.3% to 27.8%, which makes a trial harder to power. Ministries are also reluctant to keep hospitals in a control arm for years while neighbouring hospitals have monitoring. So is an RCT needed before scaling? If the effect is real, waiting three to four years for trial results means children in hospitals that could have had monitoring go without it. Last but not least it is important to emphasize that continuous monitoring is already considered standard of care in most parts of the world, so perhaps it is less about if it works but rather to what extent it should be prioritized amidst many other priorities and gaps.

From RP
"Note that a traditional randomized controlled trial may be genuinely hard to run in this context: a monitoring system cannot be disguised as a placebo, and it may change how the whole ward operates. For example, staff know when the monitors are present, and the intervention is bundled with training and supervision. The observed effect size therefore reflects both the device plus the attention surrounding it (i.e., the Hawthorne effect^[5]), 


1. Your current evidence is fairly weak, and low on the evidence ladder. Before and after studies are notorious for being unreliable because there are so many other factors involved.  Hospitals are always improving their systems and there are many reasons why mortality could drop. Seeing consistency between studies is encouraging but evidence needs to be better.

2. Blinding is not very important - the staff seeing the monitors is part of the intervention no? I'm not sure why RP and you think this is a problem? If it changes how the whole ward operates that is part of the benefit of the intervention.  In fact that might be where you main benefit is! I'm confused about this comment by RP here and not sure what point they are making.... The training and supervision front could be a big deal though, especially when you believe your main impact to be ward spillovers. That's why I think training and supervision can't really last more than a few months, then you need to largely leave the hospital be with the monitors with minimal supervision and see if mortality reductions are sustained over 2-3 years. This isn't that long an RCT!

4. Randomising whole hospitals isn't an ethical issue. Ministries will not refuse to allow hospitals being in a control arm (they are getting free monitors) and the evidence isn't nearly good enough yet to argue an RCT is unethical. If the difference in mortality between hospitals is big after year 2 say, the study will be triggered to stop for ethical reasons anyway.

5. The range of mortality rates doesn't make powering a study harder I don't think? (might be missing something) Its low mortality rates and low sample sizes that might be the issue here. But there are lots of hospitals where multiple kids die every week (I have one 500 meters from my home) If you are in the busiest hospitals where 150+ kids die every year, then I would think you wouldn't need neadly that amount (I might Claude this later). If you need to do a multi-country RCT so be it, that seems likely needed.

6. The urgency argument seems unrealistic. Because of the low level of evidence intervention isn't "obviously" better than other things we could spend money on. I don't think your case is nearly strong enough here to make a "bypass the RCT" argument. I think actually even framing it like this could make people lose confidence in the intervention. It feels more like a sales pitch than a good argument.

7. What is considered "Standard of care" in rich countries is close irrelevant in healthcare interventions in low-income countries. I don't think it should even be a factor really. There are 100 "standard of care" things you could introduce to a pediatric ward in Malawi (Better antibiotics, Dr. nos, nursing ratios, frequency checks, infection control, blood cultures, blood gases etc.). This doesn't mean you don't study their benefit in a low-income setting.

As an unrelated point, I feel like the presentation of the intervention here might be too branded. The intervention is a monitoring device, yet you are at pain to emphasise how your intervention is different from other devices. If your plan is to scale through government, its the monitoring system that matters not the branding. It might be better even to brand machines as "Ministry of Health" machines and think of yourself as a support org as much as an innovation org? This is a tricky one though I know.

I absoutely love this intervention - I'd be keen to get on a call and chat about it even if you might be. Sadly someone ffrom your org was pegged to meet me at EAG New York, but I can't go because of travel restrictions due to ebola :(