Sjlver

245Joined Mar 2021

Bio

Jonas loves his wife, being in nature, and exploring interesting worlds both fictional and real. He uses his bamboo bike daily to get around in Munich. During the day, he writes software to track bednets for AMF. At night, he enjoys playing Ultimate and dancing.

Comments
77

Thanks for the thoughts!

I think we are getting closer to the core of your question here: the relationship between cases of malaria (or severe malaria more specifically) and deaths. I think that it would indeed be good to know more about the circumstances under which children die from malaria, and how this is affected by various kinds of medical care.

The question might partially touch upon SMC. Besides preventing malaria cases, it could also have an effect on severity (I'm thinking of Covid vaccines as an analogy). That said, the case for SMC (as I understand it) is that it's an excellent way to prevent malaria infections. This is what the RCTs measure, and this is where its value comes from.

To answer the question, I believe it would be more helpful to do research into malaria as an illness, rather than doing an SMC trial replication. I continue to think that the evidence base for SMC is good enough. You have doubts since "most published research findings are false", but "most published research findings" might be the wrong reference class here:

  • It includes observational studies, surveys, and other less reliable methods; here, we have RCTs.
  • It includes all published studies, also those with small samples and effect sizes. Here, we have >7 trials, >12k participants, and the effect (SMC's reduction of malaria episodes) is >6 standard deviations away from zero.
  • It includes studies with effects that are multiple causal steps away from the intervention (e.g., deworming improves income) and have many confounding factors. Here, we are measuring the effect of a malaria medication on malaria, with clearly-understood underlying mechanisms.

You also ask about the settings in which SMC is rolled out. There is no specific answer here, since SMC is often rolled out for entire countries or regions, aiming to fully cover all eligible children. More than 30 million children received SMC last year. In their cost-effectiveness analysis, GiveWell looks at interventions by country and takes a number of relevant factors into account, such as the "mortality rate from malaria for 3-59 month olds".

In general, malaria fatality (deaths per case) is trending downwards a bit, due to factors such as better access to medical care, better diagnosis, better education of parents, and certainly many others. It could make sense to make this explicit when doing a cost-effectiveness analysis.

I'd expect GiveWell to be mindful about these things and to have thought of the most-relevant factors. I don't think additional RCTs would lead to large changes here.


Regarding the post-script about AMF: We are fortunate to have a board of trustees and leaders that think a lot about high-level questions and trends, both those closer to AMF's work (e.g., resistance to insecticides used in nets) and those more peripheral (e.g., the impact of new vaccines). There is also good and regular communication between GiveWell and AMF. As for myself, the day-to-day preoccupations are often much more mundane ;-)

Looks like I can confirm this. Relevant passages from Cissé et al (2006):

The study was designed to measure Malaria, not deaths:

The primary outcome measure was a comparison of the occurence of clinical malaria between children in the two study groups.

Children with positive malaria tests received treatment:

Malaria morbidity was monitored through home visits every week and by detection of study children who presented at one of three health centres in the study area. At each assessment, axillary temperature was measured, and if it was 37.5C or greater, or if there was a history of fever or vomiting during the previous 24 h, a blood film was prepared. Results of the blood film examination were usually available within 2 h. Antimalarial treatment was given when appropriate according to the national guidelines: chloroquine as firstline treatment, quinine or sulfadoxine-pyrimethamine as second-line treatment in cases of failure of treatment with chloroquine, and injectable quinine for cases with persistent vomiting or severe malaria. Study children received iron supplementation if they presented at a health centre with an illness suggestive of anaemia, pale mucosae, or both.

I'll still think more about this... but here we have at least a lead towards better understanding of low death numbers in SMC trials.

I appreciate the thoughts! I'm going to think about this more thoroughly... but here's a quick guess about the low death numbers:

These trials involved measuring malaria prevalence in children. Presumably, children with a positive result would then get medication or be referred to a health center. Malaria is a curable disease, so this approach would save lives. Unfortunately, it's also quite likely that the child would not receive appropriate treatment in the absence of a diagnosis, due to lack of knowledge of the parents, distance to health facilities, etc.

Anyway, it's just a quick guess. Might be worth checking if the studies describe what happened to children with positive test results.

The Right-Fit Evidence group provides good resources related to this post. They publish guidance on what types of evidence implementers should collect to demonstrate and monitor the impact of their programs.

Notably, different types of evidence are ideal depending on the stage of a program. In the beginning, when there is lots of uncertainty about an intervention, a randomized controlled trial is great. At a later stage, when the program is scaling to many recipients, it is more important to monitor the program and ensure that the implementation is done well.

In the case of SMC, millions of children receive treatments. A wealth of monitoring data is collected, much more than could be obtained in an RCT. Even though that data isn't randomized or controlled, its quantity might make up for these deficits and allow us to determine whether SMC works with sufficient confidence.

More information can be downloaded on the Right-Fit Evidence website. And here's an introduction to their framework.

That seems fair. I agree that my request for an estimate is a big, maybe even unreasonable, request.

I asked because I am wondering if there really is enough reason to doubt the results of existing SMC trials. If I understand your post correctly, your main worry is not about actual errors in the trials; we don't have concrete reasons to believe they are wrong. Indeed, the trials provide high-quality evidence that SMC reduces malaria cases, including severe cases.

Your worries seem to be that (1) studies are underpowered to quantify reduction in malaria deaths. I'm not sure if that is a big problem, given that there are clear causal links between malaria cases and malaria deaths. (2) The trials did not follow the new best practices that we've identified since they were published. This indeed makes the trials less reliable than we would wish for, but I'm not sure whether the problem extends to a meta-analysis of seven trials.

For all these reasons, I keep wondering: how strongly do you really believe these results are wrong? And by how much? Even some rough answer would be OK here... and I'm sure it would also help GiveWell when they evaluate this post.

Yes, absolutely.

As far as I can tell, that type of RCT indeed is not being done. I don't know much about research on SMC specifically, but Givewell reports the following quote of Christian Lengeler, author of Cochrane Review of insecticide-treated bed nets:

To the best of my knowledge there have been no more RCTs with treated nets. There is a very strong consensus that it would not be ethical to do any more. I don't think any committee in the world would grant permission to do such a trial.

Thanks for these thoughts!

A question: How large do you expect the effects of such a replication to be? Maybe you could estimate "a study of cost would lead to a change if effect size of with probability " for some instances of . That would help to estimate whether the study would, in expectation, be worth more than one life saved per 5000 dollars.

And an observation: I think it would be very difficult to get ethical approval for such a study. SMC is (according to current knowledge) an amazing intervention. Any controlled trial would require a control group that does not receive SMC, nor other interventions that could act as confounding factors. Think about it... you'd expect the study to cause ~10 additional preventable child deaths in the control group, just so it can measure an effect! It might be more feasible to make comparison studies between different types of SMC, but of course these don't directly answer your question.

Jonas here, AMF software engineer.

Thank you for your research! I would really like to publish more of AMF's PDM data to enable this kind of work. Unfortunately, we have to prioritize how we spend our time in the small AMF team, and this task hasn't made it to the top yet.

If you were interested in doing a more in-depth analysis (and have the time required for this) it might be good to let Rob (our CEO) know. This can help in prioritizing this type of task.

(disclaimer: I work for AMF, but this is my personal opinion)

Yes, we have to prioritize. No, life quality seems the wrong metric for prioritization.

A few practical responses to the challenge first: AMF funds bednets at the scale of countries or provinces, that is, a few million nets at a time. This allows for efficient distributions that leverage economies of scale. Prioritization takes a number of factors into account, such as malaria prevalence (which might have an effect on the bednet use rate). Life quality metrics are currently not a factor for prioritization, as far as I know.

Re the challenge: I think you'd have to consider that life quality goes up as a result of the intervention. It might go up more in areas that start from a lower baseline than in areas where people are already better off. It's important to note that bednets (or, for that matter, most interventions) are not purely life-extending interventions. They have a variety of other benefits:

  • Economic improvements: Fink and Masiye 2015 found that free bednets increased the average annual harvest value for a farmer by $76, about 12% of the group’s average annual harvest value at baseline.
  • Education improvements: since children cannot go to school when they are ill
  • General welfare improvements since people don't have to care for sick children, don't have to mourn them or organize funerals so often, ...
  • Health improvements from averting long-term effects of malaria infections

A final question: you keep using net-negative lives in your examples, yet people are accepting help, voluntarily. This seems to indicate net-positive life quality. How do you think about that?

Related job ad, but not by the forum team; feel free to remove if not appropriate

The Against Malaria Foundation is close to Finnish time zones and completely remote. It currently has employees in the UK, Germany, and South Africa. One employee is working part-time due to parenting. AMF is hiring for several roles.

Load More