This is really excellent work, Joey! It seems like replicating your research / providing feedback on interventions is fairly high in expected value terms. If there are any EAs that have helpful knowledge on this, I'd encourage them to do so.
This is excellent!
This looks really great! I was wondering is there any chance that the conference call will be recorded? 7PM PST is a bit late for those of us in GMT, and I would be super keen to hear more about what you and your team have been up to!
Also I am probably just wrong about this, but isn't there a fairly strong correlative link between higher IQ and higher income? Shouldn't we expect that the IQ benefits of salt iodization would lead to increased economic outcomes given the correlation between these two traits? Or is the rough idea that we shouldn't expect to see that correlation in the context of the developing world?
It will indeed be recorded and posted on our website, sorry about the time zones its hard to accommodate everyone outside of normal working and sleeping hours given the spread of the EA movement.
So re-IQ it was really tricky, your right there is pretty good evidence of developed world income and IQ correlation but generally we found metrics like this did not connect as strongly to the same result happening in the developing world as we would have expected and found a lot less evidence for a connection in the developing world. Additionally there was quite a bit of data on the lack of correlation with IQ and subjective well-being and health in the developing world which further worried us. We can see reasonable people making different judgement calls on how to weight off these factors.
Awesome- thanks for clarifying! Looking forward to listening to the recording.
We are unsure how to value IQ increases as we cannot find strong connections between increased IQ and other things, like happiness, income, or improved health in the third world.
I agree that the connection between IQ and wellbeing is weaker than between health and wellbeing, but it seems a bit extreme to effectively treat it as zero.
Moreover, it seems to be in tension with academic consensus among psychologists and economists, who think IQ matters. Here's a survey on job performance: https://80000hours.org/2013/05/intelligence-matters-more-than-you-think-for-career-success/
Put it this way: if I could increase your IQ by 20 points for $1000, would you accept?
After completing our first two month phase of research across thirty intervention areas, we have ruled out twenty-two intervention areas, leaving eight broad intervention areas left. We have now turned those into numerous specific charity ideas that we will dive deeper into with further research.
Each of these intervention areas were researched for up to forty hours (using this outline) to determine whether our team could and should create an effective charity in that area. While we have ruled out these twenty-two interventions at this stage for our team going forward, we still think many of these interventions could be worth considering as donation targets or for other teams to found charities in. Also, some interventions were ruled out largely because they already had an exceptionally strong charity recommended by GiveWell. Additionally, all ratings were relative to other intervention areas on our list, and the interventions that we ruled out are likely still unusually cost-effective and evidence-based relative to average global health programs.
Below we attempt to provide a quick summary of the research we have conducted and published including links to more detailed information. This is not meant as a full justification or explanation so much as to give a sense of our views on the weakest and strongest of interventions we looked at. Over time we will be posting many blog posts explaining our thinking and outlining our research on interventions that did make the cut. We will also eventually start posting our ideas about which concrete charities we think may be high value and why.
For each of the intervention areas we are no longer putting researcher time into we wrote a brief paragraph covering some of the reasons why we think they don’t offer as strong specific charity options as the interventions that we are still considering.
Bednet Distribution: We think bednet distribution is strong in a lot of ways (GiveWell’s full report) but we also think there are already multiple strong charities working in this area (2) and do not see a need to add additional effort. The Against Malaria Foundation is both strong and has considerable room for funding (3), so we believe it is better to support them rather than start another charity in the space.
Polio eradication: We did not feel we could create and scale a charity to be useful in this area fast enough to be productive given the current trends in Polio prevalence. Currently only two countries remain where the disease is endemic, Afghanistan and Pakistan (4), and many areas in the most need are arguably conflict zones which likely makes the logistics of such a program very complex.
River blindness: Our main concern with river blindness was its evidence base (5), and although we reviewed some other interventions that were similarly weak in evidence base, we feel river blindness did not have enough redeeming features, such as flexibility, logistical ease or a lack of crowdedness to make up for this weakness. Furthermore, prevalence rates seem to be decreasing (6), and there are already a few other organizations active in the area.
Immunization to prevent maternal and neonatal tetanus: We’re concerned most about counterfactual scalability as there seems to be a limited number of cases (7) and a positive trend with large and recent successes (8). Increasing immunization coverage in order to prevent maternal and neonatal tetanus also seems more logistically difficult and less cost-effective than our top interventions.
Mass Drug Administration to combat Lymphatic filariasis (MDA LF): MDA LF is much less cost-effective (unpublished CEA estimate) compared to other intervention areas we looked at. Since the program is a mass drug administration rather than a targeted approach, the prevalence has to be above a certain rate for it to be cost-effective. The microfilaria prevalence is too low for MDA LF to be cost-effective in India (9). Infection rates in other countries are also not sufficiently high. The majority of those infected never exhibit symptoms (10), of those who do, only a small percentage develop severe symptoms that cause large problems like social ostracization and depression. Furthermore, crowdedness in this intervention is fairly high. There are already eight charities active in the area, and the crowdedness in the areas of greater microfilaria prevalence is especially high. The Indian government claims to cover 85% (11) of the country with preventative medication. While there is a problem of people not taking the medication once receivedthey receive it (12), this is not a straightforward or cost-effective problem to solve.
Malaria treatment: Malaria treatment interventions are very difficult logistically and don’t score strongly enough in any other area to overcome this weakness. While individual cost-effectiveness estimates varied a lot, we are inclined to think malaria treatment is less cost-effective than the top interventions we are considering. This area also does not appear neglected, as there are quite a few strong organizations already working on malaria treatment (13).
Unconditional Cash Transfers (UCTs): Much like bednets,UCTs are strong in a lot of ways (14) but we think GiveDirectly is already a very strong charity working in this area that has considerable room for funding (15) and focus should be on expanding GiveDirectly rather than creating a new organization. We also feel as though we could likely get better cost-effectiveness from a conditional cash transfer charity if attached to the right condition, even if it was just a labeled condition (LCT).
Tuberculosis case finding and first-line treatment: This was one of the most logistically complex interventions we looked at. Diagnostics looked hard, treatment looked hard, and preventing defaulting looked very hard (16). The length and unpleasantness of treatment causes many of those with tuberculosis to default. Although it scored mid-range in most other areas, except for scoring low on scalability (17), we felt it was beyond our current skill level to found a charity in this area.
Trachoma control: There are a few different ways to do trachoma control methods but we found problemsmajor flaws with them all. The recommended SAFE (Surgery, antibiotics, face washing and environmental control) strategy could not compete in cost-effectiveness with our top interventions (DCP2) and some sections of it had very weak evidence (18). The most promising and cost-effective part of SAFE was surgery but it seems logistically complicated and hard to scale with few benefits to outweigh the flaws.
Prevention of mother-to-child transmission of HIV: The biggest factor against this intervention was that there seem to be competent bodies working in this area. For instance, globally 73% (19) of pregnant women living with HIV had access to antiretroviral medicines to prevent mother-to-child transmission of HIV to their babies in 2014; and new HIV infections among children were reduced by 58% from 2000 to 2014. Additionally there is a charity called New Incentives working in this area in a very similar way to what we think would be best.
Therapeutic Zinc Supplementation (TZS): TZS has reasonably strong evidence it reduces the duration and severity of childhood diarrhea yet limited evidence it reduces mortality. There is some reason to be concerned that the weaklimited data for mortality reduction evidence means a new large RCT may show that TZSthe mortality reduction rates are overstated for TZS (20). Since mortality reduction is important for the TZS cost-effectiveness of TZS, this is a significant concern. Additionally, it’s worth noting that we feel TZS would be best done alongside the administration of oral rehydration salts.
Provision of antiretroviral therapy to treat HIV/AIDS: Antiretroviral therapy is noticeably less cost-effective than our other interventions, (21) though opportunities may exist that will decrease the costs involved with this intervention (22) many of these had increased logistical difficulty. HIV/AIDS is also quite crowded and well-funded relative to other intervention areas we looked at (23).
Deworming: Deworming has possibly higher cost-effectiveness (24) than other interventions, but there are already two outstanding deworming charities, Evidence Action and SCI and we saw little reason to create a third. Furthermore, we were discouraged by this intervention’s somewhat thin evidence-base.
Oral Rehydration Therapy (ORT): We like ORT in a lot of ways, however, it is quite crowded as a numerous major international global health players are already involved in trying to increase ORT usage rates (25). These efforts have been ongoing in some places for decades, often without much success (26). This gives some reason to believe scale-ups may be difficult. Further, while the cost of ORT materials per treatment is very low, the cost of actually implementing a successful program, accounting for delivery costs and increasing compliance rates, is less clearly established and could be quite high (26, 27). There might be a opportunity for an innovative approach in this area but we did not see any in our research that seemed promising enough for inclusion into our next stage of research.
Pneumonia Treatment: Treating pneumonia was a surprisingly neglected area (28) given the scale of the problem (29,30) and the evidence-base was mid to weak (31). Overall it was stronger than several other areas we considered. The most promising solution to this seems to be improving the existing healthcare infrastructure, including providing practitioners with decision aids for diagnosing and treating pneumonia (30). Ultimately, we decided that other similar healthcare interventions were stronger picks and that pneumonia treatment could be ruled out.
Hand Washing: Although there was some decent evidence hand washing is quite effective at reducing diarrhea, there was only weakly evidenced or relatively not cost-effective hand washing interventions (31). It’s a recurring behavior change as well as a soap/clean water distribution problem (32). There might be a opportunity for an innovative approach in this area but we did not see any in our research that seemed promising enough for inclusion in our next stage. However, we can see hand washing being an effective part of a larger mHealth or mass media program.
Breastfeeding: Overall we like a lot of elements of breastfeeding and it’s possibly very cost-effective (33). However, breastfeeding carries some of the same challenges as hand washing and all other behavior changes. It’s difficult logistically, particularly in scaling to areas with different cultural factors or reasons for low breastfeeding rates. Furthermore, the area seems crowded and given large increases in breastfeeding rates in a short time period based on recent government efforts (34), there seems to be a significant chance that a government program would make a large difference in this area that would swamp charity efforts before we could become established. We can imagine this being an effective part of a larger mHealth or mass media program.
Folic acid: The primary strengths of this intervention are its evidence, metric focus, and ability to be synced with other food fortification or supplementation programs. There’s quite strong evidence for folic acid fortification and supplementation reducing neural tube defects (NTDs) (35) and many countries have seen a significant drop in their NTD rates after mandating folic acid fortification (unpublished GW folic acid review). Additionally, folic acid fortification could be paired with iron fortification because they both are often added to flour. The weaknesses of this intervention are its scalability, side effects and cost-effectiveness. Scalability is weak with only 300,000 NTDs per year globally including the roughly 70 countries that already mandate folic acid fortification. Furthermore, only 20-60% of NTDs can be prevented with fortification, though that number is likely higher in countries with higher baseline rates of NTDs. Another weakness of folic acid fortification, however, is the list of possible and documented side effects (36) . Folic acid fortification and supplementation also has some minor negative side effects (on anti-malarial drugs, vitamin B12 deficiency and cancer). Nevertheless, we think this very easily could be part of a strong micronutrient or fortification program although we do not see it as strong enough to found a charity focused on folic acid.
Salt iodization: Salt iodization was one of the hardest interventions for us to evaluate. It scored middling to high in almost every area and this would normally put it in our top interventions. However the benefits of salt iodization are almost entirely IQ increases (37) and we are unsure how to value IQ increases as we cannot find strong connections between increased IQ and other things, like happiness, income, or improved health in the third world. We are broadly utilitarian so we ultimately care about what leads to greater well-being in the world.
Charities that collect or generate information and data: This was one of the hardest interventions to evaluate and we are notleast confident in our conclusion on it. There does seem to be some shortage of data, information, high quality RCTs and replication studies. However, assigning the marginal value of creating information in a given area was very difficult. We looked at a few different ways of comparing research to direct work in terms of expected value and could not find a reference group that we felt was representative. Many cost-effectiveness analyses were focused on first world research and a few were based on clear outliers and thus unrepresentative.
Ultimately, it came down to a question of priority. We felt as though there were already some strong organizations in this area (Eg, IDI, DCP, CC, JPAL, IPA and 3IE) and it seemed like there was more of a funding gap than a talent gap. We saw only a few charities trying explicitly to address the gap between research and implementation and feel as though this is needed more in the poverty world right now. Additionally, we felt we would get some of the benefits of a research organization by running an RCT on the direct intervention we find most promising.
Whilst researching in India we have also written up a couple of more soft and subjective experiences, such as, our reflections on a slum tour and our thoughts on the treatment of animals in India. We hope to write a couple more of these posts which will include our reflections on a visit to a local clinic and some more general lessons.
We also recognize that our internal research is outpacing our current external publication of that research and that people may have questions and feedback that could help us do better research. While we do intend to write up more over the coming months, we thought the easiest way to bridge the gap in the meantime was to hold an open research call about our research. More details on how to join our research call.
Subscribe to our research blog if you want to get updates more often, although we will also post major updates on the EA forum.
This is really great. I found myself most often understanding and agreeing with your rationale for eliminating and ranking interventions. Very excited to see what comes of this.
I find it somewhat interesting that a few interventions seem to have been eliminated due to the complications associated with creating behavior change, and yet two of the four highest rated are in behavior change. What distinguishes those two from the others that were rejected?
In your summary spreadsheet image, I don't see a category for neglectedness. Is that incorporated into 'other factors'? I have a weak, non-evidence based impression that there's been a lot of movement on malnutrition recently and was looking to check that with your findings.
How do you plan on implementation? Is there a write-up somewhere? Reading about it being your first times in a developing country makes me naturally skeptical of your ability to implement in the field.
How much time/effort is Charity Science focusing on this? Just wondering how it ranks with other priorities and if there was specific donor interest / influx that allowed for greater resourcing to add this to your priorities.
I encourage you to explore outside of India. Poverty looks and feels very different in every place I've experienced and interacted with it, and I feel like I learn more each time. Some places with extreme poverty are much easier to live in than India too, and so you may not experience the productivity losses you felt there.
Complexity of behavior changes was a negative point on something, but if something was sufficiently cost effective, flexible, etc, it could still end up as one of our top interventions
We did originally have a separate section for neglectedness but we combined it into scalability part way through our research. We found many areas that had many orgs working in an area (thus low neglectedness) but the problem was so large or wide spread there was still plenty of room for effective charities. We found that combining the total amount of unfilled room into a counterfactual scalability score ended up being more intuitive for us.
Implementation will vary alot depending on the area we end up working in, but in all cases we will be hiring and partnering with locals and domain experts to run an effective charity. I think being skeptical of charities until they prove themselves is a good and rare epistemic habit, although I did not find developing world experience correlated particularly well with the high impact charities I interviewed.
Charity Science is basically a separate organization from Charity Entrepreneurship, although they are both under the broader brand (a bit like GWWC and 80k). They both have their own dedicated staff and budgets, although I spend some hours on both. We have had pretty good donor interest in both projects. I would say the big thing that allowed CE to happen was our senior staff deciding it was the highest value thing they could put time into.
I definitely believe that there is a lot of range in poverty. We saw huge differences in different cities in India. I imagine that we could keep learning almost indefinitely from different places. We do plan on getting a better sense of the developing world once we have a specific location picked where we think it would be highest impact to run the pilot for our charity.
I would be really interested to hear what countries you think would have more mild productivity losses and in what ways they differ from India.
Thanks for all of the clarifications / explanations. This definitely helps provide more context and understanding to the story of how this originated and is progressing.
"I would be really interested to hear what countries you think would have more mild productivity losses and in what ways they differ from India."
Southeast Asia (Thailand, Laos, Vietnam) is probably the 'easiest' place to work that is relevant to the interventions we often look at. Very strong internet, low costs, easy to travel, usually clean, and usually very high accessibility to high quality, safe, and tasty food. (Some of this likely applies to parts of Malaysia, Philippines, and Indonesia as well, but I'm not sure).
My (limited) experiences in East Africa had both positives and negatives in terms of productivity v. India. It was very difficult to find strong internet, but the ease of life, cleanliness, and quality of food (esp. Ethiopia) exceeded what I most often found in India.