Comments
This is extremely cool, you should all be very proud of yourselves
This is extremely cool, you should all be very proud of yourselves
HealthLearn provides accredited, engaging, mobile-optimized online courses for health workers in Nigeria and Uganda. We focus on lifesaving clinical skills that are simple to implement.
Our recent evaluation of the HealthLearn Newborn Care Foundations course showed significant improvements in birth attendants’ clinical practices and key birth outcomes. Early initiation of breastfeeding, strongly linked to reduced newborn mortality, improved significantly in the evaluation.
After applying large (>10X) discounts, we estimate the course is ~24 times more cost-effective than GiveWell’s cash transfer benchmark. We are uncertain about the precise magnitude of impact, but a sensitivity analysis suggests that the program is cost-effective under a wide range of plausible scenarios.
Our already-low unit costs should decline as we scale up. This is likely to increase or at least maintain the program’s cost-effectiveness, even if the impact per trainee is lower than our current point estimate. We also earn revenue by hosting courses for another NGO, which covers a portion of our core team costs and increases cost-effectiveness per philanthropic dollar spent.
We have identified key uncertainties in evidence strength, sustainability of clinical practice change, and intervention reach. We plan to improve our monitoring and evaluation to assess these uncertainties and develop more precise estimates of impact per trainee. We will continue our work to improve and scale up the Newborn Care Foundations course, while also developing new courses addressing other gaps in clinical practices where impactful interventions are needed.
HealthLearn is an AIM-incubated nonprofit that develops and provides engaging, accredited, case-based, mobile-optimized online courses for health workers (HWs) in Nigeria and Uganda. This includes one HealthLearn course (Newborn Care Foundations) and two courses (focused on epidemic preparedness and hypertension diagnosis and management) from another NGO. HealthLearn is distinct from many other digital health solutions (for example, data collection tools, chatbots, or clinical decision support apps) in the sense that our intervention is specifically designed to deliver in-service professional development for facility-based health workers who are using their own devices.
While this post focuses on our newborn care work in Nigeria, our overall objective is to develop a highly scalable tool applicable across clinical areas and countries, targeting critical skills to save lives.
The Newborn Care Foundations course teaches the basic care that every newborn should receive, following Nigeria’s national guidelines and international evidence-based best clinical practices. The training places special emphasis on practices with a proven impact on key health outcomes (newborn and maternal mortality). For example, in every uncomplicated delivery, HWs should:
It’s worth noting that skin-to-skin care in the first hour after birth is not the same as kangaroo care, which should be provided specifically to low birthweight infants over several days or even weeks. Our course covers kangaroo care and other perinatal practices, but it mainly emphasizes the universal care that every newborn should receive in the first 90 minutes after birth. If you’re interested in learning more about kangaroo care, we’d recommend checking out the excellent work of ansh!
We recently concluded our first clinical practice evaluation, which demonstrates that health workers significantly improve two of the three practices and outcomes enumerated above after taking the course, while the third, delayed cord clamping, already had very high compliance at baseline.
Although there are a number of limitations to our evaluation (described in more detail here), these results provide us with an opportunity to update our estimates of HealthLearn’s cost-effectiveness.
Before we delve into the modeling, we want to clarify that our objective here is to assess whether HealthLearn has been plausibly cost-effective to date. While the modeling suggests that the program has been cost-effective, we do not think a model (on its own) can be considered “proof.”
To build our model, we had to make a number of simplifications and assumptions.
Even after discounting, our model suggests the program is ~24 times more cost-effective than the GiveWell benchmark of pre-2024 cash transfers (comparable to the cost-effectiveness of recognized top charities in global health). A simple sensitivity analysis suggests that the program is more cost-effective than cash transfers under a broad range of scenarios (and discounts) that we consider to be plausible. The results of this simplified model are in overall alignment with a more fully featured probabilistic impact model.
We account for the total lifetime costs of HealthLearn (from founding to November, 2024) in the above analysis. However, there is a wrinkle: HealthLearn earns revenue by hosting and supporting courses for another NGO. Some of that revenue goes towards the time we invest to work closely with them to develop and improve their courses, and the rest contributes to the “shared services” costs of developing and maintaining the online platform that hosts all of these courses. The above estimates are “per philanthropic dollar” (that is, per dollar donated to HealthLearn that we have spent to date). We think it is most appropriate to assess cost-effectiveness this way, but a more conservative approach would measure cost-effectiveness per dollar spent on all of HealthLearn, in which case we would estimate our cost-effectiveness to date to be ~9 times that of the GiveWell benchmark.
This modeling exercise and sensitivity analysis highlight what we consider to be the three key uncertainties associated with our impact evaluation. These are, in order of cost and complexity to evaluate (with the first being the most expensive and challenging to address):
We have applied discounts to the current model due to these uncertainties. They also provide a roadmap for future evaluation work.
One of the major benefits of a tech-enabled intervention is excellent economies of scale. While our program does have certain fixed costs (primarily personnel) that are required for our operations, the marginal cost of each additional trainee is comparatively low. The overall cost-effectiveness of our intervention is driven by two key factors: impact per trainee and cost per trainee (Figure 2).
Notably, we are much more uncertain about the impact per trainee than the cost per trainee, which we can calculate directly from our platform analytics and financials. The average philanthropic cost per trainee is ~$14 to date, whereas marginal cost per trainee is a mixture of simple unit costs (e.g., WhatsApp and SMS messaging fees, costs of acquisition) and personnel costs that are more step-like (e.g., the costs of hiring additional software engineers). As we scale up, we anticipate that we will be able to drive down marginal cost per trainee to as low as $1. As marginal costs decline, the intervention will still be cost-effective even if the impact per trainee is substantially lower than the (already discounted) impact we estimate in our current evaluation.
This relationship between cost and impact per trainee is a linchpin of our strategy. While it may be challenging to measure a precise impact per trainee, if we can establish a plausible range of impacts per trainee while scaling up and keeping unit costs low, we will increase our confidence in the cost-effectiveness of the intervention. Cost per trainee is a simple “North Star metric” that we will use in our planning and ongoing evaluation.
This analysis also highlights a key challenge we face in evaluating our program: the program as a whole can be very cost-effective even with a relatively small benefit per trainee, so future evaluations of endline impacts (such as reductions in newborn mortality) may require complex study designs and large sample sizes. In this sense, the program may have more in common with policy and mass media interventions than other “direct” global health programs.
We believe the HealthLearn platform could become a cross-cutting tool for improving clinical practice at scale. These impact estimates only account for improvements in early initiation of breastfeeding, but other checklist indicators also improved in our evaluation. It is likely that this increases the impact of the Newborn Care Foundations course, but at this time we do not feel we have sufficient evidence to directly model the cost-effectiveness of other clinical practice improvements. Similarly, the courses we host for other NGOs are likely to increase the overall impact of the platform, but we think that these benefits should primarily be attributed to those organizations rather than HealthLearn and we do not consider them here.
Looking beyond newborn care, there are clinical practices we could cover in future courses that have proven impacts and suboptimal levels of adherence. Notably, most cases of diarrhea are effectively cured with lifesaving and inexpensive oral rehydration therapy, but only 60-70% of children in Nigeria seen in primary health centers for diarrhea receive this treatment. While there are excellent interventions to address this problem at the community level, we’re not aware of programs that effectively and scalably improve the diarrhea diagnosis and management practices of facility-based health workers. We believe that much of the potential impact of our approach could come from developing more high-impact clinical training that covers a variety of topics.
We have addressed the key uncertainties identified in this modeling exercise by applying discounts. While we think this is appropriate given the small scale of our evaluation and gaps in the available research literature, the true impact may be higher than these discounted estimates imply. To give two examples: this model does not account for clinical practice changes that last for more than a year, and the effect of training may be larger for users outside of our evaluation cohort due to lower baseline knowledge. We hope that future evaluations will allow us to assess these uncertainties. Finally, there are potential risks or harms that could stem from online training (e.g. taking HWs away from their work, replacing other training, or accidentally creating confusion about a key concept that reduces adherence to best practice), but we didn’t account for these in our model.
Unlike many other clinical practices, the skills of coaching the mother on breastfeeding and leaving the baby in skin-to-skin care for the first hour after birth do not require special commodities or equipment. In our evaluation, we observed a low baseline level of early breastfeeding and low adherence to the practices that support it. This aligns with a recent study indicating that just 36% of neonates in Nigeria begin breastfeeding within an hour of birth. Another analysis finds that only ~55% of neonates in 35 sub-Saharan African countries were encouraged to breastfeed within the first hour after delivery. In its assessment of facility-based (in-person) health worker training in maternal and neonatal health, GiveWell found that such programs are likely to be cost effective, but also have “very heterogeneous training content.” The specific outcome of early breastfeeding may be a simple, underappreciated, and shared mechanism contributing to the benefits of these newborn care training programs. We wonder whether concerted efforts to improve early breastfeeding via mass media campaigns, policy initiatives, supportive supervision, and/or health worker training might effectively save many lives.
The association between early initiation of breastfeeding and lower mortality has been studied in many contexts and with large pooled sample sizes, but this evidence is observational: babies who begin breastfeeding earlier are at lower risk of mortality. It seems plausible (even likely) that healthier neonates are more likely to begin breastfeeding earlier, which raises concerns about reverse causality. However, we think the available evidence is consistent with a direct benefit of early breastfeeding. For example, one study found that early initiation of breastfeeding had similar mortality effect sizes even when unwell and premature babies were excluded. In our modeling, we only considered the mortality reduction associated with breastfeeding within the first hour as opposed between 1-24 hours after birth. Babies who begin breastfeeding >24 hours after birth have an even greater risk of mortality (and, we suspect, are somewhat more likely to have underlying health problems that cause the delay).
In a rapid search of the literature, we were not able to find experimental studies that directly assess the impact of behavioral determinants of early breastfeeding (such as coaching the mother and continuous skin-to-skin care) on mortality. Experimental studies do show that early skin-to-skin care increases exclusive breastfeeding (feeding only breastmilk for the first six months of life), and other studies suggest that exclusive breastfeeding reduces newborn mortality. Nonetheless, more research is needed to better assess the causal relationship between early initiation of breastfeeding and neonatal mortality.
Our modeling indicates that the Newborn Care Foundations course has been plausibly cost-effective to date, even when we apply large (>90%) discounts. The course could potentially be much more cost-effective at scale. The results directly inform our strategy for the next year. We plan to:
Get in touch at: [email protected]
This work would not have been possible without the leadership and support of:
Very cool, and thank you for such a well considered cost-effectiveness model.
Do you have modelling for the potential scale of this intervention? How many healthcare workers are there in settings where this will be counterfactually useful?
Thanks for the question.
Nigeria was selected in part because it's one of the most promising places for this training to deliver impact. We've identified more than a dozen other countries that have low rates of early initiation of breastfeeding, with a population of health workers > 1 million in just those countries.
You can get a sense yourself of the number of health workers in Africa in Table 1 of this WHO report. Our initial digging reveals some promising countries for our intervention outside of Africa as well. Our sheets related to this are not super polished, but we're confident the ceiling is pretty high - we hope to be training > 1 million health workers every year when we're operating at scale.
Our analysis above is focused on one particular practice, but our course(s) can be re-configured fairly easily to focus on different practices in different contexts. If a country has great adherence to early initiation of breastfeeding, there may still be a strong case for us hosting a course there that instead focuses on whichever is the next most promising thing.
Some of these other clinical practices might have different impacts than EIBF, but low unit costs at scale should make a lot of other interventions look good in a CEA, too. Our course-completer cost in the CEA was ~$14, but we expect that may come down a lot as we scale up 🤞.
Writing ~10 months later and the unit costs are ~7x lower than when I wrote the above. See here. I'm hopeful that we can continue to cut costs with greater scale!