Review of Education Interventions and Charities in Sub-Saharan Africa

Huwelium

Review of Education Interventions and Charities in Sub-Saharan Africa

Huwelium

1 min readFeb 25, 2019

Comments 14

Sorted by

New & upvoted

aog

I really like the education review, it seems like a great introduction to the literature on effective education interventions. And it's even better that you'll be reviewing health interventions soon, given that they seem generally more effective than education, both in terms of certainty and overall impact.

But I would still have strong confidence that GiveWell's top charities all have significantly higher expected value than the results of this investigation, for two reasons.

First, GiveWell has access to the internal workings of charities, allowing them to recommend charities that do a better job of achieving their intervention. This goes as far as GiveWell making almost a dozen site visits over the past five years to directly observe these charities in action. There's just no way to replicate this without close, prolonged contact with all the relevant charities.

Second, GiveWell simply has more experience and expertise in development evaluations than someone doing this in their free time. It's fantastic that you all are working with these donors, and your actions seem likely to have a strong impact. But GiveWell has 25 staff, a decade of experience in the area, and access to any relevant experts and insider information. It's very difficult to replicate the quality of recommendations that come from that process. Doing the research yourself has other benefits: it increases engagement with the cause, it teaches a valuable skill, etc. But when there's a million dollars to be donated, it might be best to trust GiveWell.

If the donors want an intervention that's both certain and transformative, GiveDirectly seems like an obvious choice.

Kirsten

I generally agree with the above. I love GiveWell. However, I think doing your own charity evaluation has more benefits than just learning skills and becoming more engaged. A couple of extra benefits, off the top of my head:

-Doing your own charity evaluation means you can challenge GiveWell when you think they've gotten something wrong.

-Encouraging people other than GiveWell to do charity evaluation means we're in a better position if GiveWell ever stops performing to its current standards (eg if 2-3 key staff members left at the same time).

-Investigating a particular area in depth which GiveWell hasn't spent much time on recently, like education, could give the community access to useful information (maybe one of these charities is more effective than we think; maybe this list helps us pick a charity to donate to on behalf of our teacher friend; maybe it provides useful advice for EA Quebec's donors!)

aog

Good point, I wasn't fully considering that. I think Michael Plant's recent investigation into mental health as a cause area is a perfect example of the value of independent research - mental health isn't something . While I still think it's going to be extremely difficult to beat GiveWell in i.e., evaluating which deworming charity is most effective, or which health intervention tends to be most effective, I do think independent researchers can make important contributions in identifying GiveWell's "blind spots".

Mental health and education both could be good examples. At this point, GiveWell doesn't recommend either. But they're not areas that GiveWell has spent years building expertise in. So it's reasonable to expect that, in these areas, a dedicated newcomer can produce research that rivals GiveWell's in quality.

So I'd revise my stance to: Do your own research if there's an upstream question (like the moral value of mental suffering, the validity of life satisfaction surveys, or the intrinsic value of education) that you think GiveWell might be wrong about. Often, you'll conclude that they were right, but the value of uncovering their occasional mistakes is high. Still, trust GiveWell if you agree with their initial assumptions on what matters.

Ben Kuhn

In addition to Khorton's points in a sibling comment, GiveWell explicitly optimizes not just for expected value by their own lights, but for transparency/replicability of reasoning according to certain standards of evidence. If your donors are willing to be "highly engaged" or trust you a lot, or if they have different epistemics from GiveWell (e.g., if they put relatively more weight on models of root-level causes of poverty/underdevelopment, compared to RCTs), I bet there's something else out there that they would think is higher expected value.

Of course, finding and vetting that thing is still a problem, so it's possible that the thoroughness and quality of GW's research outweighs these points, but it's worth considering.

saulius

Have you seen Founders Pledge report on Women's Empowerment? Seems relevant.

Huwelium

I hadn’t seen it. Will definitely check it out. Thanks Saulius!

Sanjay

I spotted this sentence: "GiveWell (2018) for instance does not assign any intrinsic value to education itself, nor to improved attendance or test scores." Do you have a source for this? (As far as I'm aware, I don't think they have ever said that.)

Huwelium

Yes, you can check out this webpage: https://www.givewell.org/international/technical/programs/education

Here's a relevant excerpt:

" Evidence of effect on outcomes (such as income, health, or social outcomes) rather than outputs (such as increased time in school or improved test scores). There are a number of variables that can be used to measure the effects of education interventions, and we place significantly more emphasis on the effects on some variables than others. We distinguish between 'outputs' of education interventions, namely whether they increase time in school or student learning (measured by test scores),3 and the effect of education interventions on people's life 'outcomes', including employment, earnings, health outcomes, fertility, and marriage. We do not place much intrinsic value on increasing time in school or test scores, although we think that such improvements may have instrumental value.4 The majority of experimental studies of the effects of education interventions focus on the effects on time in school and test scores. However, we place far more emphasis on a few recent studies that estimate the effects of education programs on life outcomes, such as earnings and rates of fertility and marriage among young women and girls."

Aaron Gertler 🔸

I've spent the last 1.5 years working closely with a small private foundation that makes large donations to causes in the global development sector. They have similar goals to the people who approached EA Quebec, and they've been happy so far with the donations I've helped them to make. I'd be happy to share some of the things I've learned about giving recommendations to specific donors.

I've reached out to the author in a separate message. If anyone reading this has found themselves in a similar position and wants to talk, please send me an email!

Wayne Chang 🔸

Hi Huwelium, thanks so much for your post! I’m also advising someone on highly cost-effective interventions, so I found your thoughtful analysis to be very interesting. My question relates to your cost effectiveness estimates vs GiveWell’s. Based on GiveWell’s spreadsheet, their modeling of DDK (2017) places that program’s cost effectiveness at 0.5x – 2.5x GiveDirectly’s. Their modeling of Bettinger et al (2017) places that program’s at 0.2x – 1.4x GiveDirectly’s. Both of these estimates are for consumption effects only and excludes non-pecuniary benefits like reduced teenage pregnancy. This seems most comparable with your document's cost-effectiveness estimates, which are based on income effects only. However, for Pratham, you conclude its cost effectiveness is 20x - 200x GiveDirectly’s.

I’m having trouble undertanding how your estimates are one to two orders of magnitude different from GiveWell’s. I'm probably missing something important so I was wondering if you've attempted a reconciliation. Any clarification on assumption differences and their relative importance would be very much appreciated. Thanks so much!

Huwelium

Hi Wayne,

Thanks for your post. I would love to get in touch and compare notes on research for advising donors. I’ll try to reach you via this site’s messaging.

Sorry for the very late reply (I don’t get alerts when someone posts here). I believe the difference comes simply from the wide range of cost effectiveness of education interventions. As mentioned in the Google doc, “Rachel Glennerster mentions in an 80000 Hours podcast that good interventions typically deliver at least 1 learning adjusted year of schooling (LAYS) per 100 USD spent, with some interventions delivering about 10-30 LAYS per 100 USD, and the best delivering up to 460 LAYS per 100 USD.”

For Pratham, the info I found suggested roughly 1.7 to 27.6 extra years per 100 USD. Assuming an increase in income of 8.8% for each extra year of schooling, this means an increase in income of about 15% to 243% per 100 USD donated. Comparing to DDK 2017, GiveWell cites a 24% increase in income for 541$ spent, so 4.4% increase in income per 100$ spent.

I don’t know if this helps? I think the basic explanation is that there is a very wide range in effectiveness of education interventions, and that Pratham seems to be higher in this range than DDK, say.

Huwelium

Thanks Aidan, Aaron and Khorton for your comments - much appreciated!

I definitely agree that GiveWell does excellent work, and we are indeed thinking of including a GiveWell charity (or more) in our final recommendation to the donors, which will probably include a few charities rather than just one. As Aidan mentioned, GiveDirectly seems like it might be the best fit with the donors’ criteria, among GiveWell’s top charities (some of their standout charities might also be a good fit).

Regarding education interventions, GiveWell did not have any recommendations, so using their recommendations was not an option. We could have recommended against donating in this sector based on GiveWell’s review, but we didn’t do that primarily because the donors assign intrinsic value to better education, while GiveWell does not (so here it’s more a question of values rather than expertise).

Kirsten

I believe GiveWell have said that schistosomiosis deworming can lead to better school attendance and performance, although I think the evidence is mixed. Probably worth looking into.

Huwelium

We’re planning to look into early childhood interventions (including things like deworming, improved nutrition, etc.) separately. Having said that, we had put deworming aside since it’s basically presented by GiveWell as “low probability of high impact” which didn’t appeal to the donors. But as you say, we should review what the evidence is in terms of education impact. I’ll add that to my to do list.

Comments