Comment Permalink

Hi Wayne,

Thanks for your post. I would love to get in touch and compare notes on research for advising donors. I’ll try to reach you via this site’s messaging.

Sorry for the very late reply (I don’t get alerts when someone posts here). I believe the difference comes simply from the wide range of cost effectiveness of education interventions. As mentioned in the Google doc, “Rachel Glennerster mentions in an 80000 Hours podcast that good interventions typically deliver at least 1 learning adjusted year of schooling (LAYS) per 100 USD spent, with some interventions delivering about 10-30 LAYS per 100 USD, and the best delivering up to 460 LAYS per 100 USD.”

For Pratham, the info I found suggested roughly 1.7 to 27.6 extra years per 100 USD. Assuming an increase in income of 8.8% for each extra year of schooling, this means an increase in income of about 15% to 243% per 100 USD donated. Comparing to DDK 2017, GiveWell cites a 24% increase in income for 541$ spent, so 4.4% increase in income per 100$ spent.

I don’t know if this helps? I think the basic explanation is that there is a very wide range in effectiveness of education interventions, and that Pratham seems to be higher in this range than DDK, say.

See in context

Review of Education Interventions and Charities in Sub-Saharan Africa

by Huwelium

Feb 25 20191 min read 14

14

Global health & developmentEducation

Frontpage

Two anonymous donors have approached Effective Altruism Québec for donation recommendations. The initial mandate given by the donors was to find one or more donation opportunities to reduce human misery as much as possible, with a focus on interventions that are potentially transformative and empower people to improve their lives durably. In line with what the donors are thinking, we are assuming this goal will involve aid to people living in extreme poverty in the developing world. The donors are particularly interested in helping people in Sub-Saharan Africa, so we are focusing on this region.

The intent is to donate 1 million Canadian dollars starting in 2021, probably donating 200k per year for five years. It’s possible that other such donations will follow afterward.

We tried clarifying with the donors exactly how to interpret the “transformative” criterion and what this translates to in terms of outcomes to maximize. This criterion remains fuzzy for the donors themselves, and indeed their preferences are not entirely fixed. However, the criterion is linked to empowerment and durable positive change, as opposed to short-term interventions which may have no durable impact.

The donors’ initial thought was to focus on education, so we have started our work by performing a review of this sector, including charity recommendations, which you can find here: https://docs.google.com/document/d/1-JzmsKJFHPq3j1vAypy8yZM7NbGRco6e_S5con3TOTI/edit?usp=sharing

We would welcome any comments on this, either here or directly in the Google doc.

We would also welcome your feedback on the "transformative" / empowerment criterion: do you think this can be meaningfully defined, or not? If so, are there particular interventions to reduce extreme poverty that you think are most likely to satisfy this criterion? And finally, how do you think GiveWell top charities stack up against this criterion?

14 Reactions

Comments14

Sorted by

New & upvoted

Click to highlight new comments since: Today at 6:35 AM

aogFeb 26 20198

I really like the education review, it seems like a great introduction to the literature on effective education interventions. And it's even better that you'll be reviewing health interventions soon, given that they seem generally more effective than education, both in terms of certainty and overall impact.

But I would still have strong confidence that GiveWell's top charities all have significantly higher expected value than the results of this investigation, for two reasons.

First, GiveWell has access to the internal workings of charities, allowing them to recommend charities that do a better job of achieving their intervention. This goes as far as GiveWell making almost a dozen site visits over the past five years to directly observe these charities in action. There's just no way to replicate this without close, prolonged contact with all the relevant charities.

Second, GiveWell simply has more experience and expertise in development evaluations than someone doing this in their free time. It's fantastic that you all are working with these donors, and your actions seem likely to have a strong impact. But GiveWell has 25 staff, a decade of experience in the area, and access to any relevant experts and insider information. It's very difficult to replicate the quality of recommendations that come from that process. Doing the research yourself has other benefits: it increases engagement with the cause, it teaches a valuable skill, etc. But when there's a million dollars to be donated, it might be best to trust GiveWell.

If the donors want an intervention that's both certain and transformative, GiveDirectly seems like an obvious choice.

KirstenFeb 26 20198

I generally agree with the above. I love GiveWell. However, I think doing your own charity evaluation has more benefits than just learning skills and becoming more engaged. A couple of extra benefits, off the top of my head:

-Doing your own charity evaluation means you can challenge GiveWell when you think they've gotten something wrong.

-Encouraging people other than GiveWell to do charity evaluation means we're in a better position if GiveWell ever stops performing to its current standards (eg if 2-3 key staff members left at the same time).

-Investigating a particular area in depth which GiveWell hasn't spent much time on recently, like education, could give the community access to useful information (maybe one of these charities is more effective than we think; maybe this list helps us pick a charity to donate to on behalf of our teacher friend; maybe it provides useful advice for EA Quebec's donors!)

aogFeb 27 20194

Good point, I wasn't fully considering that. I think Michael Plant's recent investigation into mental health as a cause area is a perfect example of the value of independent research - mental health isn't something . While I still think it's going to be extremely difficult to beat GiveWell in i.e., evaluating which deworming charity is most effective, or which health intervention tends to be most effective, I do think independent researchers can make important contributions in identifying GiveWell's "blind spots".

Mental health and education both could be good examples. At this point, GiveWell doesn't recommend either. But they're not areas that GiveWell has spent years building expertise in. So it's reasonable to expect that, in these areas, a dedicated newcomer can produce research that rivals GiveWell's in quality.

So I'd revise my stance to: Do your own research if there's an upstream question (like the moral value of mental suffering, the validity of life satisfaction surveys, or the intrinsic value of education) that you think GiveWell might be wrong about. Often, you'll conclude that they were right, but the value of uncovering their occasional mistakes is high. Still, trust GiveWell if you agree with their initial assumptions on what matters.

Ben KuhnFeb 27 20197

In addition to Khorton's points in a sibling comment, GiveWell explicitly optimizes not just for expected value by their own lights, but for transparency/replicability of reasoning according to certain standards of evidence. If your donors are willing to be "highly engaged" or trust you a lot, or if they have different epistemics from GiveWell (e.g., if they put relatively more weight on models of root-level causes of poverty/underdevelopment, compared to RCTs), I bet there's something else out there that they would think is higher expected value.

Of course, finding and vetting that thing is still a problem, so it's possible that the thoroughness and quality of GW's research outweighs these points, but it's worth considering.

sauliusMar 2 20196

Have you seen Founders Pledge report on Women's Empowerment? Seems relevant.

HuweliumMar 2 20192

I hadn’t seen it. Will definitely check it out. Thanks Saulius!

SanjayMar 10 20193

I spotted this sentence: "GiveWell (2018) for instance does not assign any intrinsic value to education itself, nor to improved attendance or test scores." Do you have a source for this? (As far as I'm aware, I don't think they have ever said that.)

HuweliumMar 25 20192

Yes, you can check out this webpage: https://www.givewell.org/international/technical/programs/education

Here's a relevant excerpt:

" Evidence of effect on outcomes (such as income, health, or social outcomes) rather than outputs (such as increased time in school or improved test scores). There are a number of variables that can be used to measure the effects of education interventions, and we place significantly more emphasis on the effects on some variables than others. We distinguish between 'outputs' of education interventions, namely whether they increase time in school or student learning (measured by test scores),3 and the effect of education interventions on people's life 'outcomes', including employment, earnings, health outcomes, fertility, and marriage. We do not place much intrinsic value on increasing time in school or test scores, although we think that such improvements may have instrumental value.4 The majority of experimental studies of the effects of education interventions focus on the effects on time in school and test scores. However, we place far more emphasis on a few recent studies that estimate the effects of education programs on life outcomes, such as earnings and rates of fertility and marriage among young women and girls."

Aaron Gertler 🔸Feb 26 20193

I've spent the last 1.5 years working closely with a small private foundation that makes large donations to causes in the global development sector. They have similar goals to the people who approached EA Quebec, and they've been happy so far with the donations I've helped them to make. I'd be happy to share some of the things I've learned about giving recommendations to specific donors.

I've reached out to the author in a separate message. If anyone reading this has found themselves in a similar position and wants to talk, please send me an email!

Wayne Chang 🔸Apr 16 20192

Hi Huwelium, thanks so much for your post! I’m also advising someone on highly cost-effective interventions, so I found your thoughtful analysis to be very interesting. My question relates to your cost effectiveness estimates vs GiveWell’s. Based on GiveWell’s spreadsheet, their modeling of DDK (2017) places that program’s cost effectiveness at 0.5x – 2.5x GiveDirectly’s. Their modeling of Bettinger et al (2017) places that program’s at 0.2x – 1.4x GiveDirectly’s. Both of these estimates are for consumption effects only and excludes non-pecuniary benefits like reduced teenage pregnancy. This seems most comparable with your document's cost-effectiveness estimates, which are based on income effects only. However, for Pratham, you conclude its cost effectiveness is 20x - 200x GiveDirectly’s.

I’m having trouble undertanding how your estimates are one to two orders of magnitude different from GiveWell’s. I'm probably missing something important so I was wondering if you've attempted a reconciliation. Any clarification on assumption differences and their relative importance would be very much appreciated. Thanks so much!

HuweliumMay 6 20193

Hi Wayne,

Thanks for your post. I would love to get in touch and compare notes on research for advising donors. I’ll try to reach you via this site’s messaging.

HuweliumFeb 27 20192

Thanks Aidan, Aaron and Khorton for your comments - much appreciated!

I definitely agree that GiveWell does excellent work, and we are indeed thinking of including a GiveWell charity (or more) in our final recommendation to the donors, which will probably include a few charities rather than just one. As Aidan mentioned, GiveDirectly seems like it might be the best fit with the donors’ criteria, among GiveWell’s top charities (some of their standout charities might also be a good fit).

Regarding education interventions, GiveWell did not have any recommendations, so using their recommendations was not an option. We could have recommended against donating in this sector based on GiveWell’s review, but we didn’t do that primarily because the donors assign intrinsic value to better education, while GiveWell does not (so here it’s more a question of values rather than expertise).

KirstenFeb 27 20192

I believe GiveWell have said that schistosomiosis deworming can lead to better school attendance and performance, although I think the evidence is mixed. Probably worth looking into.

HuweliumFeb 27 20193

We’re planning to look into early childhood interventions (including things like deworming, improved nutrition, etc.) separately. Having said that, we had put deworming aside since it’s basically presented by GiveWell as “low probability of high impact” which didn’t appeal to the donors. But as you say, we should review what the evidence is in terms of education impact. I’ll add that to my to do list.