Hey this is super cool!

I think this could be a great intervention, but I disagree on the RCT front. I strongly believe that in the medium to long-term, this intervention should, and will only go to scale after high quality RCTs.

Yours and RP's reasons for not doing it aren't compelling to me. I'm almost a bit confused.. This might actually be one of the easier RCTs to do in the development world. If I was a funder, I would be asking why you aren't in the middle of a big RCT at the moment, or at least why you aren't getting started on one. I agree with the funders.

"Funders often tell us to come back with an RCT, which we estimate would need around 72 hospitals across three countries, US$5 million and three to four years." If this is what's needed, then yeah do it. You can get the 5 million - have you asked GiveWell?  But I'm skeptical you really need 72 hospitals. Mortality rates are high enough in massive hospitals I would imagine if you are in national referral hospitals in capital cities, or similar, then this number would drastically drop. Maybe you are considering a bunch of smaller hospitals? 

From your article
"conventional RCT is hard to run for this intervention. It can't be blinded, because health workers know when monitors are on the ward. It has to randomise whole hospitals, because the central patient overview and the positive impact on workload affects care for every child on the ward, not only those attached to a monitor. Death is a relatively rare outcome, and baseline neonatal mortality across our sites ranges from 4.3% to 27.8%, which makes a trial harder to power. Ministries are also reluctant to keep hospitals in a control arm for years while neighbouring hospitals have monitoring. So is an RCT needed before scaling? If the effect is real, waiting three to four years for trial results means children in hospitals that could have had monitoring go without it. Last but not least it is important to emphasize that continuous monitoring is already considered standard of care in most parts of the world, so perhaps it is less about if it works but rather to what extent it should be prioritized amidst many other priorities and gaps.

From RP
"Note that a traditional randomized controlled trial may be genuinely hard to run in this context: a monitoring system cannot be disguised as a placebo, and it may change how the whole ward operates. For example, staff know when the monitors are present, and the intervention is bundled with training and supervision. The observed effect size therefore reflects both the device plus the attention surrounding it (i.e., the Hawthorne effect^[5]), 


1. Your current evidence is fairly weak, and low on the evidence ladder. Before and after studies are notorious for being unreliable because there are so many other factors involved.  Hospitals are always improving their systems and there are many reasons why mortality could drop. Seeing consistency between studies is encouraging but evidence needs to be better.

2. Blinding is not very important - the staff seeing the monitors is part of the intervention no? I'm not sure why RP and you think this is a problem? If it changes how the whole ward operates that is part of the benefit of the intervention.  In fact that might be where you main benefit is! I'm confused about this comment by RP here and not sure what point they are making.... The training and supervision front could be a big deal though, especially when you believe your main impact to be ward spillovers. That's why I think training and supervision can't really last more than a few months, then you need to largely leave the hospital be with the monitors with minimal supervision and see if mortality reductions are sustained over 2-3 years. This isn't that long an RCT!

4. Randomising whole hospitals isn't an ethical issue. Ministries will not refuse to allow hospitals being in a control arm (they are getting free monitors) and the evidence isn't nearly good enough yet to argue an RCT is unethical. If the difference in mortality between hospitals is big after year 2 say, the study will be triggered to stop for ethical reasons anyway.

5. The range of mortality rates doesn't make powering a study harder I don't think? (might be missing something) Its low mortality rates and low sample sizes that might be the issue here. But there are lots of hospitals where multiple kids die every week (I have one 500 meters from my home) If you are in the busiest hospitals where 150+ kids die every year, then I would think you wouldn't need neadly that amount (I might Claude this later). If you need to do a multi-country RCT so be it, that seems likely needed.

6. The urgency argument seems unrealistic. Because of the low level of evidence intervention isn't "obviously" better than other things we could spend money on. I don't think your case is nearly strong enough here to make a "bypass the RCT" argument. I think actually even framing it like this could make people lose confidence in the intervention. It feels more like a sales pitch than a good argument.

7. What is considered "Standard of care" in rich countries is close irrelevant in healthcare interventions in low-income countries. I don't think it should even be a factor really. There are 100 "standard of care" things you could introduce to a pediatric ward in Malawi (Better antibiotics, Dr. nos, nursing ratios, frequency checks, infection control, blood cultures, blood gases etc.). This doesn't mean you don't study their benefit in a low-income setting.

As an unrelated point, I feel like the presentation of the intervention here might be too branded. The intervention is a monitoring device, yet you are at pain to emphasise how your intervention is different from other devices. If your plan is to scale through government, its the monitoring system that matters not the branding. It might be better even to brand machines as "Ministry of Health" machines and think of yourself as a support org as much as an innovation org? This is a tricky one though I know.

I absoutely love this intervention - I'd be keen to get on a call and chat about it even if you might be. Sadly someone ffrom your org was pegged to meet me at EAG New York, but I can't go because of travel restrictions due to ebola :(

Thanks for writing this up! It's rare to get an inside look at how funders handle the practical challenges of allocation, and I think you had a lot of interesting ideas. 

However, the various approaches you discussed seem odd to me in that they all involve taking the cost-effectiveness of the intervention and probability of its successful implementation and converting them into some kind of score, then allocating funding proportionately to that score. That approach seems like it misses something fundamental, which is that an EV-maximizing grant portfolio should be one that equalizes the marginal cost-effectiveness (including value of information) of the last dollar given to each recipient (note that there's some further complexity if you have flexibility in how much funding to give this year vs. hold onto for future years, since one should also want to equalize the marginal cost-effectiveness of grant dollars over time). Starting from that goal, it seems clear that it would be worth scoring organizations on how quickly cost-effectiveness would be likely to fall off, based on total addressable problem size, management capacity etc. That's not to say that one can in practice compute exact marginal cost-effectiveness curves and perfectly solve this optimization problem, but I do think it's worth bringing that marginal mindset to evaluating each grant.

I'd expect the portfolio that resulted from thinking hard about equalizing marginal cost-effectiveness to look something like:

• Give a lot to the organizations where the first dollar has high expected cost-effectiveness, filling them until they hit serious diminishing returns. (e.g. H, G, J)
• Give grants just large enough to get significant information value to organizations that have high impact conditional on successful implementation but low probability of success, since the value of information there is very high (e.g. B)
• Give nothing to organizations that have low impact conditional on successful implementation and low probability of success, since the value of information there is low (e.g. C, K)
• Give the remainder to organizations in the middle, or do something else with it (hold it for future years, give to an outside option that can absorb lots of dollars at a high cost-effectiveness floor like GiveDirectly)

This looks a decent amount like edge, but it's skewed somewhat away from the middle tier. For example, I'm skeptical that edge's $1.9m grant to F makes a lot of sense, given that F's EV is much lower than the top tier of organizations and the information value if it works is still low. I'd expect the recommendations to differ more from edge in cases where organizations had very different capacities to absorb funds, which this example doesn't really capture.

Really proud of this response love it! After a slow couple of weeks responding to the flurry of EA press, it's great to see some positive, plain English responses over the last week, and this might be one of the most important.

Also I think it's becoming more clear how important GiveWell and global health is for the reputation and public view of AI safety work. They are an almost unarguable anchor for do-most-good thinking that helps the public take the AI claims more seriously.