Hi, thanks for writing this, I thought this was great. I agree with the limitations section.
Some nitpicks:
- Can you say a bit more about how you adjust for counterfactual impact? 5km is like an hour of walking? Are you halving this? - Out of curiosity, how difficult would it be to subsidize transport instead?
Thanks so much for the comment and the questions. I really appreciate you reading this and thinking about it, especially given you are so engaged in longtermism stuff. Loved the skepitical braindump on existential risk from AI.
Not Nitpicks at all
1) This is the biggest weakness in our calculation. We use the global burden of disease data, because it estimates the average DALY burden of everyone who gets a disease (say malaria) in Uganda whether they were treated early lateor not at all. OneDay Health centers treatthe people who previously had the worst healthcare access in Uganda, so we assume that our treatment could remove at least that nationwide average DALY burden. You couldargue that this still is likely to overestimate the benefit of treating patients, but you could argue the opposite direction too. Without RCT data I think this is the best we can do at the moment - I couldn't think of a better way to estimate the DALYs we might avert for each disease and also others in big orgs like PSI have used this approach before which is reassuring.
5km can seem like a short distance, but when you or your child are sick it can be very difficult to walk that far. For what it's worth, the WHO uses this 5km threshold as a key accessibility indicator in countries like Uganda where walking is the only option for the vast majority of rural subsistance farmers. Also for context only 5-10% of Ugandans live in these healthcare black holes so the majority of people are in a better situation.
5km is only an indicator too - most of our patients live further than this from a government health center as well. It's also a very complicated ecosystem as many government health centers don't have enough medications so people have to buy the medication anyway even when you reach there. Distance isn't the whole story, I simplified somewhat for this post
2) Subsidising transport would be (close to) impossible for a variety of reasons, mostly the first 2 I thnk
- How would you decide who to give the money to? Over the phone it would be impossible to know who was really sick and needed money. The system would likely fall down pretty quickly as everyone claimed to be sick. I might be strawmanning your proposed system though if so let me know! Give directly only works because they give unconditional money to most people in a given area.
- Many people in "Healthcare black holes" (only the most remote 5-10% of Uganda) don't have good access to phones or cellphone reception which would be necessary for any kind of transport program.
- Motorbike transport means are not easily available in some of these areas (not the most important barrier, and motorbike accessibility is improving all the time).
Moreover, a transit-subsidy model would overload the already under-resourced government health centers with additional patients compared to what they would have if ODH did not exist at all. So you'd need to account for the likely increased costs (due to, e.g., medications that patients had to purchase from private sources) and worsened care quality for all patients in those facilities.
In contrast, the existence of an ODH health center should reduce the demand on government health centers, which may well create some benefits for other users of those facilities. Nick's model does not attempt to capture those benefits.
Some more information about accessibility in Uganda is available here although the most important map is hard to read. ODH works in Northern Uganda, where the access to healthcare is significantly lower than near Kampala.
1) Ok, so let me try to rephrase and then you can tell me whether this makes sense
Per here, for an individual malaria patient, you are calculating the impact as:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working[1].
And then we are both thinking, well, this should really be something like:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working × Chance that the treatment was counterfactual.
But then you are pointing out that your term for "Badness of malaria" is actually the "Burden of disease of malaria", which is actually how bad malaria is, but given that some of the patients are already receiving treatment. So in the original estimate, you didn't have the counterfactual adjustment, but in exchange your "Badness of malaria factor" was too low.
So that's my paraphrasing this so far. Do you think it's mostly correct?
Then, what my intuition tells me one could do would be to try to:
a) Make the "Badness of malaria" factor be "Badness of untreated malaria" and model the "Chance that the treatment was counterfactual" factor explicitly. But then this wouldn't capture all the benefits, so then maybe have some upwards adjustment for quality of care?
b) If gettting estimates for the badness of malaria is somehow too difficult, keep the burden of disease number, but add the counterfactual adjustment, and note that this is explicitly an underestimate.
At this point though, the counterfactual adjustment might get really gnarly:
perhaps the most severe cases would have counterfactually been treated more often? Unclear how much this could reduce your estimate of impact.
One could also model downstream effects, e.g., maybe making it less urgent for the Ugandan gvmt to have more hospitals? But then freeing its capacity to do other things, etc.
The more factors you start to consider, the less this would be an apples-to-apples comparison against, say, GiveWell's estimate of AMF. Like, they have some adjustment for what would happen if AMF didn't exist, but do I buy those estimates? Not sure.
One could also just have the non-counterfactually adjusted numbers, compare to the non-counterfactually adjusted AMF numbers, and leave it as an exercise to the reader to input the relative counterfactual adjustment[2]. Idk, maybe this is too cute.
My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment]. Anyways, hope that these thoughts are midly useful, though obviously I'm missing lots of context and looking at this from a really abstracted perspective.
E.g., if the non-counterfactually-adjusted impact for ODH is X, and the non-counterfactually adjusted impact for AMF is Y, and you are choosing between the two, you don't actually have to calculate the counterfactual adjustments for both, you can just estimate the ratio R of "how much more counterfactual is ODH than AMF", and then see if X * R > Y. As I said, maybe too cute.
Your paraphrasing is amazing (probably better than my original post). I just fear you know my brain a bit better than I do. Are you the first GAI? I also don't feel like your analysis is that abstracted at all - your points seem quite concrete actually.
One small correction I might make is that most Ugandans who get malaria would get treatment, not just a few. We target the 5-10% of places which are really remote and getting treatment is difficult - that's what where here for.
It's an interesting idea to do "Badness of untreated malaria" x "Chance the treatment was counterfactual". This is a cleaner method than what I did that's for sure. The first issue with this is that I'm not sure we have a clear data point for badness of untreated malaria (although I can look into this more). Obviously impossible to study now and we need to rely on older data.
The chance of treatment (yes or no) is counterfactual would be more realistic to find, but is very black and white when really there's a lot more too it than getting treatment or not. Quality of care is important - but perhaps even more important like @Ray_Kennedypointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.
On your counterfactual adjustments (love it)
The most severe cases would be more likely to get treated yes (interesting point never thought of ths) - but as a counterpoint as already discussed early treatment is really important. Often (not always) severe malaria is a direct result of inadequate, late or non-treatment.
Yes there's definitely the counterfactual of the government doing less because NGOs are doing the work. A given thing which should probably warrant a dscount with basically any NGO program! As a caveat here, the Ugandan government made the active decision to spend their resources upgrading current facilities rather than building more in remote places. In the last 10 years the government has opened 0 (or maybe 1) new health center in Northern Uganda. Also of course like many other NGOs we have the dream that perhaps someday a government could see the value in our model and potentially take it over or roll out a version of their own. So this could be a tiny point in our favour (see my bias kicking in again...)
(Side note) Givewell seems quite rough on AMF on this front, positing a 50% chance that other funders would come in if they weren't there in some countries. This feels overconservative to me - what might well be happening is that Global Fund (or others) instead fund more nets in other places instead of working where AMF are working
The only comment I didn't really understand was the difference between where the vale comes from.
"My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment]."
I might well be missing something, but better quality of care and more "convenient" treatment (meaning people get earlier treatment) both avert deaths and save DALYs, just like getting treatment vs. not getting at all does. So doesn't it all play into the same value proposition?
I might well be missing something, but better quality of care and more "convenient" treatment (meaning people get earlier treatment) both avert deaths and save DALYs, just like getting treatment vs. not getting at all does. So doesn't it all play into the same value proposition?
See, me missing context matters here. I was imagining that the most pessimistic scenario would be that:
ODH provides treatment for malaria which is faster, nearer & more convenient
but patients would have otherwise gotten the same treatment, just later, further away and paying more costs to get it
So the value of ODH wouldn't be the value of the treatment, it would be the value of making it more convenient
But as you point out ("Quality of care is important - but perhaps even more important like @Ray_Kennedypointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.") you can't just neatly separate getting faster care from getting better care. There are some fun things you could do with distributions, i.e., explicitly model the benefit as a function of how fast you get treatment, and then estimate the counterfactual value as
∫∫ (Value of getting treatment in h hours - Chance of having otherwise gotten treatment in (h + x) hours instead × Value of getting treatment in (h + x) hours) dx dh
(where the double integral just means that you are explicitly estimating the value of each possible pair of x and h and then weighing them according to how likely they are)
But I think this would be overkill, and only worth coming back to do explicitly if/when ODH is spending a few million a year. Still they might add some clarity if we don't do the calculations. Anyways, best of luck.
On the counterfactual of the government potentially doing less, I speculate that it would be politically difficult for the government to copy ODH's business model under which 2/3 of total costs are covered by patient fees. Specifically, my understanding is that user fees for public healthcare were dropped in the early 2000s, although as a practical matter the public system isn't always free. Reinstating official fees only in certain areas probably wouldn't fly well politically. So the government would likely have to spend several times what ODH does to set up the same health centers, and that is probably relevant to assessing the odds that it might counterfactually do so.
Thank you so much for the writing.
I am Ugandan and I fully understand the situation.
From what I have observed over time, the Health facilities may be there at whatever distance, and patients can juggle around the transport to reach them. The biggest challenge is that the Centres do not have medicine and other medical tools or equipment, so, patients will not travel to get there and be frustrated by lack of medicine!
I also wonder, you say you are based in Northern Uganda and want to reach out to other East African countries...why so soon?
Even in Central, East, West and Southern Uganda, there are similar rural areas facing the very same if not worse situations.
Is it not possible to go to other areas even before you go to other East African countries?
Thanks Annette those are amazing questions and comments wow!
Yes the government health centers have many problems including medications out of stock which I agree is an enormous problem We operate far from government though, and we're not in the business of fixing the government system right now - hundreds of other NGOs are trying as you know.
You are definitely right that there are other parts of the country with many healthcare black holes. we are already in eastern Uganda and moving into central soon with 5 health centers there at the moment. In western Uganda though there are not as many healthcare bleack holes. Government coverage is better. It's not to say there aren't areas in WEstern far from government - just not as many as in the North and East
The worst situation is actually in the North east, Karamajong and Pokot area, but the population there is so low and people so poor that our sustanable OneDay Health model wouldn't work there. Perhaps we could do an even more subsidised model there in future thogh because the healthcare situation really isn't good there..
Great to see analysis like this on the forum, and I would love for more charities to try to lay out their impact like this.
I'm struggling a bit to get my head around this bit and wonder whether an alternative approach might work better (or maybe I'm just misunderstanding it): "We used the average Global burden of disease DALY burden per patient in Uganda to estimate the DALY benefit of treating individual patients. This average includes everyone who suffered from each disease in Uganda, whether they were treated correctly, poorly or not at all. This accounts somewhat for what might have happened if we hadn’t treated the patient and avoids the counterfactual of assuming that patients would have not been treated without us."
I think the main pathways for impact from your model are (please add if I've missed something): 1) reaching patients who otherwise would have missed out on care 2) improved timeliness of care (probably quite a big deal for malaria, maybe less so for family planning) 3) improved quality of care vs. alternative (unclear whether you are claiming this or not, I could easily believe this is a big factor if the alternative is faith healers or traditional medicine but less so if you think govt clinics are similar standard)
Estimating the proportion of 1) is crucial I think.
One way of generating more evidence would be a baseline of careseeking frequency from health facilities before you establish your centre. If it is 2 visits/year/family before and 4 visits/year/family after - that gives you a reasonable sense of how much additional access you are providing. It sounds like you might already have some of that data too.
So (made up numbers) if we were just thinking about malaria patients... say there are 10 per month, we could assume that 5 of those are 'additional' ones vs. counterfactual with no facility-> 100% of your treatment benefit is counted. The other 5 would have gotten slower care/poorer quality care -> 50% of your treatment benefit is counted.
Patients will also benefit from reduced travel cost. I think you could model that as just the equivalent to a givedirectly donation with no overhead probably. Time savings for patients could be substantial also, I imagine for rural people who need to plant crops/harvest this could be a big factor.
Thanks Ray. I think it's really valuable for smaller orgs like us to try and calculate our potential impact even with all the flaws!
Yes you're exactly right with those 3 points driving our impact. I think improved quality (which includes common complete misdiagnosis in many settings) and timeliness might be nearly as important in driving impact as serving those who would have missed out on care completely. Its not like those who don't get treated are likely to die, the human body is an incredible thing - without treatment we heal ourselves most of the time for most diseases, even malaria. Treatment Quality, prompt treatment and getting any treatment at all are all important impact factors
Sorry about the poor explanation - that's my bad I should have done better. The average DALYs incurred by a Ugandan with any given disease seemed the best measure available at this time, as it takes the average DALYs per person of whole spectrum of people who get that disease. From those who got no treatment at all to the majority who would get treatment. It's one of the few ways I could think of to get I'm very open to other ways of calculating DALYs averted per individual patient . At the time neither PSI or myself could think of a better one.
Measuring careseeking behaviour is a good thought, we have considered measuring this (we don't right now). One of the issues is that variability of malaria prevalence is so high that it can confound the data. For example let's say the first year there's a high malaria season and they visit healthcare 8x a year, then the second year is low malaria and they only visit 4x. It looks like careseeking behaviour is worsening but it's just that there's ess malaria. Obviously we could try and control for this using regional malaria data but it ain't easy. Also how do we account for going to a drug shop and buying a few pills? Does that count as accessing healthcare? There is much depth to these things.
Also decreased careseeking behaviour can even be the opposite, a sign of better health in the community. If an ODH health center had been doing good work treating patients well and the community is getting generally healthier, they will need to visit the facility less often. If people are treated poorly on the other hand the could end up coming back 5 times for the same condition. It's complicated that's for sure but I still think looking at healthseeking beahviour could have value!
I like your idea of 100%, 50% benefit etc and I might hit you up about that for futre analysis. We stlil run into that same problem though that we still need to decide what 100% benefit actually means in DALYs. We still need to pull that from somewhere - the problem described above that we currently use the GBD DALYs per person as a proxy for. Our current approach kind of does take this into account in a blunt and flawed way, as the average patent in Uganda takes into account whole spectrum of patient treatment (High quality, late, not at all)
Yes saving money a big factor and I like your idea of modelling it perhaps using a givedirectly model. I even thought about trying to include those benefits in the analysis, but it seemed like a lot both to do and present all at once. WE should definitely do this soon!
Thanks for the explanation, definitely agree that there are some big limitations on looking at careseeking behaviour in that way. No perfect solution but possibly excluding malaria cases as they are so seasonal would be appropriate, or if you can collect baseline data for a year then you can compare month to month.
I think existing cost-effectiveness studies might be something you can mine to get to DALY/case... for instance, this study here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8757489/#!po=51.5625
suggests that in their intervention, treating an additional 124 cases of diarrhoea = saving almost 5 DALYs (if my quick skim of table 3 is right). That's modelled I think, but might be a good additional datapoint.
One of the things I like about modeling direct/indirect patient financial benefits is that it should allow a high-confidence lower bound for ODH's effectiveness. Let's unrealistically and uncharitably assume that all of ODH's patients would have counterfactually sought care at a more distant facility, and would have received as-timely and as-good care. In that case, the net benefit to patients includes avoided travel costs + loss-of-time costs + other-care costs (the other care is supposed to be free, but often isn't in practice for various reasons), less fees paid to ODH.
In that pessimistic scenario, we should also include estimated (but difficult to calculate!) benefits to those served by the public health care system, because ODH's existence diverted tens of thousands of patients a year from that underresourced and underfunded system. (Unless ODH scaled to multiple times its size, I doubt its existence would funge overall public health-care spending in Uganda. It's just too small to affect a nationwide budget given how political processes generally work.)
A somewhat more realistic, but still probably awfully conservative, model would credit ODH (say) 25% of the DALYs in Nick's model and 75% of the income effects in the model described above. Because the DALY source includes both treated and untreated cases, such a mixed model would assume more than 75% of patients would be counterfactually treated.
Doing a really quick BOTEC, that would yield 3.425 DALYs plus the [75% of net economic effects from the financial-only model, less $87.50[1]] for a donor/grantor spend of $137.50. If we assume $3 savings per patient in the financial-only model (based on Nick's travel estimate of $3 + time value of money and other-care costs equal to the ODH fee), that yields us a patient savings of 112.5 patients (75% of average volume) * $3, or $337.50. Subtracting the $87.50 yields $250. If you think those savings are equal in value to GiveDirectly cash transfers, you could model that at 1.81X GiveDirectly + getting the DALYs (and benefits to non-ODH patients from relieving the burden on nearby government health centers) for "free." (I note that this is primarily intended to demonstrate the idea of a mixed-benefits model only, not to make assertion about a reasonable lower-bound estimate for ODH.)
I'm probably doing this wrong, but GiveWell's moral weights seem to suggest a DALY is morally "worth" about doubling the consumption of a person for two years. Given the median income in Uganda is $804, averting a DALY would be roughly equal to increasing consumption by $1600, which would make averting 3.425 DALYs roughly as effective as increasing consumption by $5480. I don't mean to assert that ODH is 41.67X GiveDirectly on this conservative model ($5480 from DALYs + $250 from economic savings / $137.50 donor spend), but this quick analysis does suggest that the bulk of the value in the mixed-model described above comes from the "free" components and not the hypothesized 1.81X GiveDirectly economic effect.
In addition to finding ODH's cost-effectiveness analysis very promising, one thing I like about its value proposition is how concrete it is to non-EAs. It may be even more legible than GiveDirectly, because the good of "treating kids and others for malaria and other stuff" is perhaps even more obvious than "giving money to poor people" (the effectiveness of which requires at least some additional context to show that the recipients spend money in ways that substantially furthers their long-term welfare).
I think talking about organizations like ODH can be a great introduction to charity effectiveness for a broad non-EA audience, in certain low-bandwidth environments, or with certain people. You have identifiable beneficiaries with a very high probability of benefitting from the intervention (and even stories on ODH's website), so you don't have to get your audience to become comfortable with the idea of more abstract benefits right away. You can offer very concrete examples of what the donor's money does -- e.g., $1.11 subsidizes the treatment of three patients who would have otherwise not received treatment or would have had to spend substantially more to receive treatment at a distant facility, $87.50 subsidizes the operating costs of a health center for a whole month, etc.
I think getting a donor to rigorously consider effectiveness for the first time is a major win, and I like the idea of having options like ODH that make that as psychologically easy for the donor as possible. I suggest the feedback loop between cognition and action is more two-way than our theories of change sometimes recognize, and so having options that make it easy for donors to make a first effective donation is important.
In the interests of full disclosure, I not only offered comments on Nick's post but also made a donation to OneDay Health US last month to stand up a new health center. In addition to concluding that the case for cost-effectiveness was good, I felt that both the EA and non-EA ecosystems don't have a lot of good funders for most micro-sized global health non-profits. And I thought I had a few comparative advantages to evaluating the case for a nanogrant here.
Thank for writing this and explaining your model to the wider EA community, as well as taking your time to explain the limitations of your calculations, that’s very thoughtful and can inspire an interesting discussion in a global health community here or elsewhere.
And I am happy to hear that Lafiya Nigeria’s analysis has been helpful!
I wish you best of luck with this project and I would be keen to work closer to assess the family planning benefits of your organisation, I know that Lafiya Nigeria can learn a lot from your vast experience and we can address some of the impact assessment limitations together.
to estimate burden at the location of ODH centres and analyse based on expectation of shifting mortality rate* closer to other regions where there are closer health care centers.
A few notes:
MAP does incorporate proximity to health care center in their modelling but I do not know how to download in a format I can use (only see TFF option).
Proximity to health care obviously correlated with urbanisation, which reduces the malaria burden itself, but you can probably largely control for this off incidence or prevalence rates.
Obviously less helpful approach to non-malaria burden, but I think estimations of improvement could be generalised across other health issues.
This modelling (or at least a prior form) is a core contributor to the GBD data.
If you think there is possible merit here, I am happy to discuss.
*can be converted to DALYs, although this might be an issue if we are to consider "lives saved" in the same way as GiveWell (i.e. not using DALYs). I have not quite got my head around if/how this is an issue yet.
Thanks so much Scott - is this the project you are working on?
I'll message you about this . There's a geomapping project I've never managed to get going along this line which I think could be hugely powerful, incorporating this kind of data and others such as distance to maternity centers, vaccination rates etc. to form an overall "neglectedness map" that can help NGOs and government target the most neglected areas, rather than roll out projects fairly randomly.
Yes, I think there is immense value in looking for practical and cost efficient ways to provide universal primary healthcare, even if like you say we are not as cost-effective as hoped. Many seem have given up on solving the problem of proximal, comprehensive primary care in remote places. I feel like the hope is that community health workers can treat a proportion of the population in the meantime, while countries develop and urbanise to the point that this is no longer necessary- but that's a whole nother discussion
We need to be using more data based methods in OneDay health like this - this malaria map is pretty amazing I didn't know about it! We could definitely do a more accurate analysis using this - no question.
Also I'm going to check out your post from a week ago. I didn't see it at the time.
I am not working on MAP. That is a project mostly funded by the Bill & Melinda Gates Foundation. That post I made a week ago was just intended as a potentially-interesting description, however as I mention there MAP estimates drive both WHO and GBD estimates. I was also surprised to only recently find out about MAP given that role, and their own slick site.
I have acquired some general knowledge about malaria doing volunteer research for SoGive (to whom I am grateful). Outside of that, I am now reading up on the An. stephensi threat to Africa, but I would stop short of calling that a project. If you have a malaria-related question you want answers to that doesn't involve advanced math, there is a reasonable chance I can help.
Hi Nick, your work sounds really great. I do want to get your view on whether it is potentially more cost-effective to (a) run centres which people come to, vs (b) send community health workers out into villages, rotating through on a regular basis. For (a) the benefit is you don't have to expend resources and time on identifying who is sick; for (b) the benefit is you get to reach out to people who are sick but who are too ill/poor to travel long distances to get treatment & medication; this is especially true for chronic issues which can be detectable any time and whose management can be fairly light touch. The latter is an issue that experts I've been talking to seem fairly worried about, and I would value your opinion on how.
TLDR even for villages which may not warrant a centre per OneDay's metrics, would it be potentially be valuable to rotate nurses through on a semi-regular basis?
Thanks Joel! Your community health worker question is a common one! I'm a big community health worker fan, and the movement is even part of the inspiration for ODH. But there are a number of common misconceptions about what they actually do
1) Community health workers might serve an area of 500 households (or more). Even the most active community health worker can't get around more than 50 housholds in a day (and the data shows the norm is more like 5-20 households). When you have malaria or pneumonia, you need help in the first 24 hours of illness and the illness can often get very bad within 1-2 days. So in reality most people visit a VHT for treatment - the VHT doesn't visit them, it's just not possible. This myth of the VHT actively case finding most of their work is persistant though...
2) Community health workers are not usually very cost effective. Monitoring them, supplying them and paying them to do a part time job works out as being surprisingly expensive. The orgs I've look at (Lastmile health and living goods) including all project costs seem to treat patients at between 4 and 10 dollars per patient (we are between $1-$2 depending on how you calculate), but it's hard to tell as I haven't seen these orgs release these numbers.
3) Community health workers only usually treat kids under 5 for malaria, pneumonia and diarrhoea - it would be medically irresponsible to get them treating others as they have minimal training. They don't treate adults for malaria. Yes these childhood illnesses are the highest DALY burden conditions, but they still can't treat the majority of patients.
I'm not sure you mean by chronic issues? Community health workers rarely treat chronic conditions, they treat malaria, pneumonia and diarrhoea in under 5s and sometimes (rarely) provide antenatal care. Maybe in urban places they are involved in NCD care, but certainly not in remote rural placee
There are a lot of good reasons for community health workers t be doing part of this job. Filling the gap of lack of trained personal (like Uganda), health education and getting as much malaria treatment out there as possible. There are many RCTs proving their worth But we still need more cost effective solutions, and high quality universal helathcare in remote places.
I think Joel's question and your response raise interesting issues about the most cost-effective ways to meet (at least more of) the health-care needs of people in places too small/remote to support a ODH health center. I can think of three general possibilities:
One, improve transit for bringing sick people to a center, such as by providing a bike (with small pedicab trailer) for a trusted community member to lend out to those needing to transport friends or family members to an ODH or other health center. The obvious problem there is that the bike might disappear or be diverted to other uses.
Two, create some sort of lightweight stationary version of ODH health centers for communities that cant support a full one -- employing a part-time, non-nurse community health worker with a sharply limited scope of practice (e.g., malaria, pneumonia and diarrhoea in under 5s).
Three, put a nurse on a motorbike with a small trailer to create a mobile, semi-lightweight version of an ODH health center (e.g., two hours a day in each of three locations, with two hours of transit time). The cost per patient would obviously be higher than ODH's model due to lower patient volume per nurse, travel expenses, and likely higher facility costs per patient.
I suspect all of those require more donor subsidy per patient than ODH's current model. The potential advantage would be that the treatment provided may be even more counterfactual than what ODH currently provides.
I'm not Nick, but it might be helpful to clarify what you mean by "community health workers" -- your TL;DR suggests you may be referring to nurses. (ETA: Nick's comment above explains the difference in scope of practice.)
Wikipedia (usual disclaimers apply) describes Uganda's levels of public non-hospital care as follows:[1]
The lowest rung of the district-based health system consists of Village Health Teams (VHTs). These are volunteer community health workers who deliver predominantly health education, preventive services, and simple curative services in communities. They constitute level 1 health services. The next level is Health Center II, which is an out patient service run by a nurse. It is intended to serve 5000 people. Next in level is Health Center III (HCIII) which serves 10,000 people and provides in addition to HC II services, in patient, simple diagnostic, and maternal health services. It is managed by a clinical officer. Above HC III is the Health Center IV, run by a medical doctor and providing surgical services in addition to all the services provided at HC III. HC IV is also intended to provide blood transfusion services and comprehensive emergency obstetric care.
This is a crosspost from the new Animal Welfare Alignment Newsletter by Anima International. You can subscribe on Substack if you are interested in following these efforts. Audio reading also available on Substack.
The goals of this post are to:
1. Raise a question I see as crucially important to the goal of aligning AI to animal welfare...
“How long have you been v*g*n?”
This is one of the most common icebreakers at animal protection events. It’s a baseline assumption, and it mostly holds true: if you’re out advocating for animals not to be tortured or abused, realistically these days you are v**n, or close. And it makes for good conversation. It seems fairly safe to assume when you meet strangers.
But this assumption is hurting the movement in a way which we don’t always notice: someone new comes into the sp...
AI Use Note: Main body text entirely human written. Claude (Opus 4.8) helped develop models of animal life histories in the appendix.
Cross-posted from Good Structures.
Executive Summary
* Animal advocates sometimes make claims like “there are X of this animal...
TLDR: Operating basic health centers in remote rural Ugandan communities looks more cost-effective than top GiveWell interventions on early stage analysis - with huge uncertainty.
I’m Nick, a medical doctor who is co-founder and director of OneDay Health (ODH). We operate 38 nurse-led health centers in healthcare “black holes,” remote rural areas more than 5 km from government health facilities. About 5 million Ugandans live in these healthcare black holes and only have bad options when they get sick. ODH health centers provide high-quality primary healthcare to these communities at the lowest possible cost. We train our talented nurses to use protocol based guidelines and equip them with over 50 medications to diagnose and treat 30 common medical conditions. In our 5 years of operation, we have so far treated over 150,000 patients – including over 70,000 for malaria.
Since we started up 5 years ago, we’ve raised about $290,000 of which we’ve spent around $220,000 to date. This year we hope to launch another 10-15 OneDay Health centers in Uganda and we're looking to expand to other countries which is super exciting!
If you’re interested in how we select health center sites or more details about our general ops, check our website or send me a message I’d love to share more!
Challenges in Assessing Cost-Effectiveness of OneDay Health
Unfortunately, obtaining high-quality effectiveness data requires data from an RCT or a cohort study that would cost 5-10 times our current annual budget.[1] So we've estimated our impact by estimating the DALYs our health centers avert through treating four common diseases and providing family planning. I originally evaluated this as part of my masters dissertation in 2019 and have updated it to more recent numbers. As we’re assessing our own organisation, the chance of bias here is high.
Summary of Cost-Effectiveness Model
To estimate the impact of our health centers, we estimated the DALYs averted through treating individual patients for 4 conditions: malaria, pneumonia, diarrhoea, and STIs. We started with Ugandan specific data on DALYs lost to each condition. We then adjusted that data to account for the risk of false diagnosis and treatment failure (in which case the treatment would have no effect). We then added impact from family planning. Estimating impact per patient isn’t a new approach. PSI used a similar method to evaluate their impact (with an awesome online calculator), but has now moved to other methods.
Inputs for our approach
Headline findings
For each condition, we multiplied the DALYs averted per treatment by the average number of patients treated with that condition in one health center in one month. When we added this together that each ODH health center averted 13.70 DALYs per month, predominantly through treatment of malaria in all ages, and pneumonia in children under 5.
ODH health centers are inexpensive to open and operate. Each health center currently needs only $137.50 per month in donor subsidies to operate. The remaining $262.50 in expenses are covered by small payments from patients. Many of these patients would have counterfactually received treatment, but would have incurred significantly greater expense to do so (mainly for travel). In addition, about 40% of patient expenses were for treating conditions not included in the cost-effectiveness analysis.
We estimate that In one month, each health center averts 13.70 DALYs and costs $137.50 in donor subsidies. This is roughly equivalent to saving a life for $850, or more conservatively for $1766 including patient expenses. However, there is huge uncertainty in our analysis.
The Analysis
Measuring Impact by Estimating DALYs Averted in Individual Patients
Before doing our assessment, we searched academic literature to find previous estimates of DALYs averted per patent treated in similar contexts. PSI had done the most work in this area.
To estimate the impact of ODH health centers treating individual patients for these four conditions, we used a similar evaluation to PSI - a linear DALYs averted model.
We used the average Global burden of disease DALY burden per patient in Uganda to estimate the DALY benefit of treating individual patients. This average includes everyone who suffered from each disease in Uganda, whether they were treated correctly, poorly or not at all. This accounts somewhat for what might have happened if we hadn’t treated the patient and avoids the counterfactual of assuming that patients would have not been treated without us. That said, OneDay Health treats patients in remote regions of Uganda who have very poor access to healthcare, so our treatment is likely to be more impactful than for treating the average Ugandan. I’m aware thatthis is probably the weakest link in our modelling efforts.
DALYs averted treating patients for malaria, Pneumonia and STIs
Our model accounts for the estimated false-diagnosis and treatment-failure rates to estimate DALYs averted per patient treated. We estimated these rates from relevant peer-reviewed literature. Based on the model for treatment of malaria in children under 5, each treatment averts on average .179 DALYs per patient treated:
We used similar models for these other treatments (See Appendix 1 for diagrams)
Malaria treatment over 5 - 0.131 DALYs averted per treatment Pneumonia treatment under 5 - 0.375 DALYs averted per treatment Pneumonia treatment over 5 - 0.036 DALYs averted per treatment Diarrhoea treatment under 5 - 0.021 DALYs averted per treatment
Although the benefit of treatng diarrhoea is low, this is consistent with PSI’s assessment and also my experience that diarrhoea is no longer a leading cause of morbidity and mortality in rural Uganda. We assumed that treatment of diarrhoea in over-5s averted 0 DALYs.
DALYs averted per patient for treating STIs were also unexpectedly small (0.00038 DALYs per patient (only 3.33 disability-adjusted life hours!). We aren't sure why the Global Burden of Disease weighed STIs so minimally. We believe our model significantly underestimates the benefits of STI treatment, especially since STIs are associated with miscarriage, stillbirth and even neonatal death.[2]
DALYs averted for Family Planning
For family planning, We cheated and sponged off Lafiya Nigeria’s calculations (thank you and they are amazing!). We discounted by an arbitrary 30%, guessing that the benefits of FP in the rural Nigerian area they work are probably higher than in remote rural Uganda. Lafiya Nigeria delivered 2400 injections, averting an estimated 503 DALYs, meaning they averted an estimated 0.21 DALYs per patient treated. Discounting by 30% that yields a figure of 0.146 DALYs per treatment for our assessment. This calculation does not incorporate all potential benefits of family planning such as positive environmental impacts. While some ODH nurses insert long-term implants for family planning, they insert too few at this stage for the result to be significant.
Calculating DALYs Averted Each Month by Each ODH Health Center
For each condition, we determined the average number of patients treated every month at each health center using data collected throughout 2022 in our 38 ODH Health Centers. We multiplied the average monthly patient volume in each category by the DALY figures described above to calculate the average DALYs averted per ODH health center per month. We estimate that each ODH health center averted 13.70 DALYs through treatment of the four conditions and family planning. The bulk of the DALYs were averted through malaria treatment (10.30 DALYs) and treatment of pneumonia in under-5s (2.625 DALYs). Specific calculations are in the chart below:
Condition Treated or Intervention
Under-5s treated monthly
DALYs averted per under-5 treated
Over-5s treated monthly
DALYs averted per over-5 treated
Total DALYs averted
Malaria
28.2
0.179
40.1
0.131
10.30
Pneumonia
7
0.375
2
0.036
2.70
Diarrhoea
7
0.021
1
0
0.15
STI
N/A
N/A
7.3
0.00038
0.00
Injectable FP
N/A
N/A
3.8
0.146
0.55
Total
13.70
Our current model does not capture benefits that accrue to patients treated for any other condition. About 40% of all patients are treated for a range of other conditions like skin infections, urinary tract infections, and high blood pressure. One example of a potentially high impact intervention is provision of antibiotic injections for severely ill patients while referring them to more definitive care.
Finally, our model does not account for the psychosocial benefits of having quality, reliable healthcare close to home, as residents no longer have to worry about lack of accessible care when they or their children become ill.
Costs
Donors or grants fund startup costs of $3000 per health center, as well as 25% of operating costs ($87.50 per month). Launch costs include furniture, initial medication supply, tests, a solar unit, and initial rental. Ongoing operating costs are around $350 a month. At present, patients pay 75% of ongoing costs. Three years ago, our health centers were closer to 100% locally funded but a combination of COVID, inflation and general poor economic conditions here in Uganda has reduced average monthly patient volume and our revenues.
Our model conservatively estimates that each health center will operate for an average of 5 years, and prorates the startup costs over a 5-year period. Thus, we allocate $50 of the startup costs to each month. ODH health centers are designed to be a permanent fixture in the community, but we will close a health center if the patient volume is too low and it appears that the community doesn’t place a high value on the health center. So far we have closed 8 of 46 health centers opened (15%). This means total cost of operation is $400 monthly
In calculating the benefits of each health center, we did not include any benefits from the 40% of patient visits that address other medical conditions. Rather than accounting for these visits on the benefit side, we have deducted $105 (i.e., 40% of patient expenditures) from the patient-cost figure. This adjustment ensures we incorporate only patient costs that are related to treatment for the specified medical conditions. Based on patents’ willingness to pay for these visits, we assume that patients receive at least $105 of value from them. So with this assumption we adjust monthly cost is $400- $105 = $295
A major reason patients visit OneDay Health centers is because care at an ODH facility costs less than transport to distant facilities. The average cost to a patient of visiting an ODH health center is $1.50, while the average round trip transport cost to the nearest government health facility is $3.00. In our patient survey many patients expressed gratefulness that ODH health centers saved them money. OneDay Health may therefore cost saving for patients or at least cost neutral.
Therefore if we assume that visiting a OneDay Health center is at least cost neutral for patients, we can account for only the donor contribution to operating the facility. The total donor/grantor cost per health center per month is $137.50 ($87.50 + $50),
Overall Cost Effectiveness
We then divide the total monthly operational cost by total DALYs averted, to find our cost per DALY averted.
Final Cost-Benefit calculations
So our more conservative estimated cost per DALY averted including patient costs 1. 295 / 13.7 = $21.5 per DALY averted 2. Assuming an 82 year lifespan, equivalent to $1766 per life saved.
Assuming accessing our health centers is cost neutral for patients, accounting for Donor funds only
137.5 / 13.7 = $10.0 per DALY averted
Equivalent to $820 per life saved.
Uncertainty
The level of uncertainty here is high as we made many assumptions and don't have direct RCT or other longitudinal evidence from communities served by ODH health centers.
In an attempt to quantify uncertainty, I performed a probabilistic Monte Carlo simulation on a previous version of this model to explore what happened when our inputs were varied. In order to perform the simulation, distributions were assigned to important data inputs, based on the nature of available data. These included costs, GBD incidence, DALY data, and the positive predictive values of diagnosis. During my masters thesis, I had help from a health economist and embarrassingly I can’t remember how I did the analysis. I haven’t run the simulation again with these latest figures but when I did, 1000 simulations produced a range of impacts between half and double the original point estimate. I think this analysis massively underestimates the uncertainty, but include it to both acknowledge the enormous uncertainty that does exist and to show that at least I tried ;). I’m keen to learn about better ways to quantify uncertainty here.
Limitations
This analysis carries a number of limitations, which are likely to bias the results in various ways.
Limitations and effects more likely to cause overestimate of effectiveness
Most interventions show reduced cost-effectiveness as the intervention is more fully studied - this is an early stage analysis.
This analysis is based on 5 year old global burden of disease data, with the burden of malaria and pneumonia likely to have reduced since then
This analysis was performed by ODH’s co-founder and director (that’s me folks), increasing the risk of bias
Limitations and effects that could contribute to inaccuracy in either direction
Absence of high-quality evidence on the effectiveness of the ODH intervention, such as from an RCT or cohort study
Potential inaccuracy of the DALY data used
Potentially inaccurate assumptions about counterfactual patient outcomes
Limitations and effects more likely to cause underestimate of effectiveness
Calculated benefit from STI treatment is unreasonably low, and does not account for effect on miscarriage, stillbirth, and neonatal death
Analysis excludes 40% of patients treated for other conditions
Potential saving benefits to the community are not properly accounted for here
Assumption that patient fees will only support 75% of operating expenses may be pessimistic, given current challenging macroeconomic climate in Uganda
Could ODH be a highly cost-effective intervention?
This analysis provides some evidence that ODH may be a high-impact charity. Each of our health centers treat on average 1800 patients a year and avert an estimated 166.4 DALYs. This is equivalent to saving about 2.25 lives (range = 1.07 to 4.50) for a donor/grantor cost of $1,650. Although these figures are highly uncertain, our ODH remote health center intervention may remain competitive on cost-effectiveness even if these estimates are several times too optimistic.
Thanks so much for taking the time to read this, I’m super keen for feedback either in the comments or direct messaging - on how to improve this analysis, and any other thoughts on ODH in general.
Huge thanks to @Jason for huge help in providing insights and editing, @Klau Chmielowska from Lafiya Nigeria for allowing us to sponge of their data and to @Lizka for encouraging me to go ahead with it!
Appendix: Details of DALY Analysis for Other Conditions
One could perform an RCT similar to the Ugandan Living Goods study, randomising remote communities to either receiving an ODH health center or not. In addition to the cost, it would be difficult for the study to capture a full range of health outcomes.
Under GiveWell’s moral weights as of 2020, the value of averting one neonatal death from syphilis is 84 times as large as the value of doubling consumption for one person for one year. The value of averting a stillbirth (one month before birth) is 33.4 times as large.
Hi, thanks for writing this, I thought this was great. I agree with the limitations section.
Some nitpicks:
- Can you say a bit more about how you adjust for counterfactual impact? 5km is like an hour of walking? Are you halving this?
- Out of curiosity, how difficult would it be to subsidize transport instead?
Thanks so much for the comment and the questions. I really appreciate you reading this and thinking about it, especially given you are so engaged in longtermism stuff. Loved the skepitical braindump on existential risk from AI.
Not Nitpicks at all
1) This is the biggest weakness in our calculation. We use the global burden of disease data, because it estimates the average DALY burden of everyone who gets a disease (say malaria) in Uganda whether they were treated early lateor not at all. OneDay Health centers treatthe people who previously had the worst healthcare access in Uganda, so we assume that our treatment could remove at least that nationwide average DALY burden. You couldargue that this still is likely to overestimate the benefit of treating patients, but you could argue the opposite direction too. Without RCT data I think this is the best we can do at the moment - I couldn't think of a better way to estimate the DALYs we might avert for each disease and also others in big orgs like PSI have used this approach before which is reassuring.
5km can seem like a short distance, but when you or your child are sick it can be very difficult to walk that far. For what it's worth, the WHO uses this 5km threshold as a key accessibility indicator in countries like Uganda where walking is the only option for the vast majority of rural subsistance farmers. Also for context only 5-10% of Ugandans live in these healthcare black holes so the majority of people are in a better situation.
5km is only an indicator too - most of our patients live further than this from a government health center as well. It's also a very complicated ecosystem as many government health centers don't have enough medications so people have to buy the medication anyway even when you reach there. Distance isn't the whole story, I simplified somewhat for this post
2) Subsidising transport would be (close to) impossible for a variety of reasons, mostly the first 2 I thnk
- How would you decide who to give the money to? Over the phone it would be impossible to know who was really sick and needed money. The system would likely fall down pretty quickly as everyone claimed to be sick. I might be strawmanning your proposed system though if so let me know! Give directly only works because they give unconditional money to most people in a given area.
- Many people in "Healthcare black holes" (only the most remote 5-10% of Uganda) don't have good access to phones or cellphone reception which would be necessary for any kind of transport program.
- Motorbike transport means are not easily available in some of these areas (not the most important barrier, and motorbike accessibility is improving all the time).
Moreover, a transit-subsidy model would overload the already under-resourced government health centers with additional patients compared to what they would have if ODH did not exist at all. So you'd need to account for the likely increased costs (due to, e.g., medications that patients had to purchase from private sources) and worsened care quality for all patients in those facilities.
In contrast, the existence of an ODH health center should reduce the demand on government health centers, which may well create some benefits for other users of those facilities. Nick's model does not attempt to capture those benefits.
Some more information about accessibility in Uganda is available here although the most important map is hard to read. ODH works in Northern Uganda, where the access to healthcare is significantly lower than near Kampala.
1) Ok, so let me try to rephrase and then you can tell me whether this makes sense
Per here, for an individual malaria patient, you are calculating the impact as:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working[1].
And then we are both thinking, well, this should really be something like:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working × Chance that the treatment was counterfactual.
But then you are pointing out that your term for "Badness of malaria" is actually the "Burden of disease of malaria", which is actually how bad malaria is, but given that some of the patients are already receiving treatment. So in the original estimate, you didn't have the counterfactual adjustment, but in exchange your "Badness of malaria factor" was too low.
So that's my paraphrasing this so far. Do you think it's mostly correct?
Then, what my intuition tells me one could do would be to try to:
a) Make the "Badness of malaria" factor be "Badness of untreated malaria" and model the "Chance that the treatment was counterfactual" factor explicitly. But then this wouldn't capture all the benefits, so then maybe have some upwards adjustment for quality of care?
b) If gettting estimates for the badness of malaria is somehow too difficult, keep the burden of disease number, but add the counterfactual adjustment, and note that this is explicitly an underestimate.
At this point though, the counterfactual adjustment might get really gnarly:
One could also just have the non-counterfactually adjusted numbers, compare to the non-counterfactually adjusted AMF numbers, and leave it as an exercise to the reader to input the relative counterfactual adjustment[2]. Idk, maybe this is too cute.
My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment]. Anyways, hope that these thoughts are midly useful, though obviously I'm missing lots of context and looking at this from a really abstracted perspective.
Or, chance that the treatment works × magnitude of the improvement, if we are being punctillious.
E.g., if the non-counterfactually-adjusted impact for ODH is X, and the non-counterfactually adjusted impact for AMF is Y, and you are choosing between the two, you don't actually have to calculate the counterfactual adjustments for both, you can just estimate the ratio R of "how much more counterfactual is ODH than AMF", and then see if X * R > Y. As I said, maybe too cute.
Your paraphrasing is amazing (probably better than my original post). I just fear you know my brain a bit better than I do. Are you the first GAI? I also don't feel like your analysis is that abstracted at all - your points seem quite concrete actually.
One small correction I might make is that most Ugandans who get malaria would get treatment, not just a few. We target the 5-10% of places which are really remote and getting treatment is difficult - that's what where here for.
It's an interesting idea to do "Badness of untreated malaria" x "Chance the treatment was counterfactual". This is a cleaner method than what I did that's for sure. The first issue with this is that I'm not sure we have a clear data point for badness of untreated malaria (although I can look into this more). Obviously impossible to study now and we need to rely on older data.
The chance of treatment (yes or no) is counterfactual would be more realistic to find, but is very black and white when really there's a lot more too it than getting treatment or not. Quality of care is important - but perhaps even more important like @Ray_Kennedy pointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.
On your counterfactual adjustments (love it)
The only comment I didn't really understand was the difference between where the vale comes from.
"My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment]."
I might well be missing something, but better quality of care and more "convenient" treatment (meaning people get earlier treatment) both avert deaths and save DALYs, just like getting treatment vs. not getting at all does. So doesn't it all play into the same value proposition?
See, me missing context matters here. I was imagining that the most pessimistic scenario would be that:
But as you point out ("Quality of care is important - but perhaps even more important like @Ray_Kennedy pointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.") you can't just neatly separate getting faster care from getting better care. There are some fun things you could do with distributions, i.e., explicitly model the benefit as a function of how fast you get treatment, and then estimate the counterfactual value as
∫∫ (Value of getting treatment in h hours - Chance of having otherwise gotten treatment in (h + x) hours instead × Value of getting treatment in (h + x) hours) dx dh
(where the double integral just means that you are explicitly estimating the value of each possible pair of x and h and then weighing them according to how likely they are)
But I think this would be overkill, and only worth coming back to do explicitly if/when ODH is spending a few million a year. Still they might add some clarity if we don't do the calculations. Anyways, best of luck.
On the counterfactual of the government potentially doing less, I speculate that it would be politically difficult for the government to copy ODH's business model under which 2/3 of total costs are covered by patient fees. Specifically, my understanding is that user fees for public healthcare were dropped in the early 2000s, although as a practical matter the public system isn't always free. Reinstating official fees only in certain areas probably wouldn't fly well politically. So the government would likely have to spend several times what ODH does to set up the same health centers, and that is probably relevant to assessing the odds that it might counterfactually do so.