Comment Permalink

One thing to note about the bounds of the FP cost-effectiveness estimate is that they aren't equivalent to a 95% confidence interval. Instead they've been calculated by multiplying through the most extreme plausible values for each variable on our cost-effectiveness calculation. This means they correspond to an absolute, unimaginably bad worst case scenario and an absolute, unfathomably good best case scenario. We understand that this is far from ideal: first, cost-effectiveness estimates that span 6+ orders of magnitude aren't that helpful for cause prioritization; second, they probably overrepresent our actual uncertainty.

On TaRL specifically, the effects seem really good--whether or not we can get governments to implement TaRL effectively seems to be where most of the uncertainty lies.

See in context

Million dollar donation: penny for your thoughts?

by Huwelium

Jun 16 20201 min read 24

30

Donation writeup

Frontpage

Two anonymous donors approached me about two years ago for donation recommendations. The donors' intent is to donate 1 million Canadian dollars starting in 2021, probably donating 200k per year for five years. The donors are particularly interested in helping people in Sub-Saharan Africa. They also have a special interest in education but are open to considering other types of interventions.

I have been working on this project for the past two years with help from members of the Québec Effective Altruism community.

At this stage, I have produced a report (see link to Google Doc below) with an overview of our work and recommendations to the donors. Prior to presenting them with this report, I would welcome your feedback! In particular, I would be interested in your thoughts on:

-Which of the charities selected as potential recommendations (Section 2.3, Table 1 of the report) do you think best correspond to the donors' mandate (see Mandate section)?

-Are there other charities or organizations that we may have missed that you think would better correspond to the donors' mandate?

-Do you have any suggestions for publicizing this report so that it can help other people trying to advise donors?

Also, feel free to add any other comments directly in the Google Doc, and to make any other suggestions. I would also be open to setting up a meeting with people interested in discussing this further.

Link to report: https://docs.google.com/document/d/1LvyjRVDotlBMBf3nrN3RLRkUAHOyrTmabw9TuPAs88s/edit#

30 Reactions

Mentioned in

18EA Forum Prize: Winners for June 2020

Comments24

Sorted by

New & upvoted

Click to highlight new comments since: Today at 11:28 AM

CarlShulmanJun 16 202028

My preference within this genre would be for something with more leverage and greater expected impact at the expense of local linearity. I'd especially call out the Center for Global Development, which has a history of policy wins that I think justify its budget many times over, and Target Malaria for work getting gene drives deployed to eliminate vector-borne diseases such as malaria. I'd prefer one dollar to these over multiple dollars to AMF, or the recommendations in the report.

HuweliumJun 16 20203

Thanks for your feedback Carl, and for the links to the Open Phil reports on Center for Global Development and Target Malaria. I will check them out. One thing I have to keep in mind though is the donor's preferences/mandate. While these are somewhat fuzzy/open-ended, they did not seem very interested in policy advocacy or interventions where there isn't yet clear evidence of impact.

Kayuri_BhimaniAug 20 202010

Hello!

While I appreciate the insights in the report, I believe that you have, too soon, ruled out considering organizations that work to support people in launching small and medium enterprises that help create jobs for low income individuals and subsequently reduce poverty.

The global Hand in Hand network is a partnership of NGOs who are united by a shared commitment to poverty alleviation through women’s economic empowerment and micro-enterprise development. Launched in 2003 in southern India, their work has trained over 3 million people – 90 percent women – and supported the creation of over 2.9 million enterprises, improving the lives of over 16 million people through strengthened incomes.

They have worked in East Africa since 2010, launching in Nairobi, and expanded into Tanzania in 2018. They also had a three-year joint program in Rwanda with CARE, which concluded in 2016.

The donors you are working with are interested in evidence of impact – and Hand in Hand’s work in Eastern Africa has been consistently validated by independent evaluations. The structures, enterprises, and gains to knowledge and confidence made their projects are sustained far longer than the projects themselves. To this end, I believe they are well worth your consideration for presenting recommendations of worthy NGOs supporting poverty alleviation in sub-Saharan Africa.

1. A 2016 review of a project in Kenya funded by the Swedish government – which created 14,000 jobs – found “the effect of the project in reducing the number of people living below Ksh 3,000 (US $30) has been phenomenal”, with a reduction from 56 percent of members below this poverty level, to only 15 percent. At the same time, the proportion of members earning US $100 or more each month jumped from nine to 27 percent, with retail and services resulting in average monthly sales of US $217 and $191 respectively.

You can download the full report here: http://www.handinhandinternational.org/hand-in-hand-a-centre-of-excellence-says-review/

2. As part of an ongoing partnership with Visa Inc. in Kenya, the firm 60 Decibals undertook a rapid response survey polling 170 past project members two years after their project’s closed. Some relevant findings are that:

- Hand in Hand’s training is useful. 95 percent of respondents were still using it in their business.

- Hand in Hand’s training improves people’s lives. 95 percent saw improvements in their quality of life after completing our training. Bigger incomes were the main reason why.

- Hand in Hand goes where other NGOs don’t. 92 percent of respondents said there was no alternative to Hand in Hand where they lived.

- Given a choice, they prefer Hand in Hand. Among respondents who had an alternative, 85 percent said Hand in Hand was better.

You can download the full report here: http://www.handinhandinternational.org/95-of-hand-in-hand-members-report-improved-quality-of-life/

3. Again in Kenya, a midline report in a project funded by IKEA Foundation found poverty had been “drastically reduced” for project members at halfway point, with the widespread use of climate resilient practices being a driving factor.

You can download the full report here: http://www.handinhandinternational.org/poverty-drastically-reduced-for-hand-in-hand-members-report/

4. In Rwanda, those groups receiving training from Hand in Hand earned an average 65 percent higher profit than comparison groups.

You can download the full report here: http://www.handinhandinternational.org/10-5-million-opportunities-leveraging-savings-groups-with-business-training-from-hand-in-hand/”

I am happy to provide further details upon request and look forward to connecting further.

Kayuri Bhimani, Friends of Hand in Hand US Office

samcartJun 18 20209

Hi! Writing as a Research Advisor at Founders Pledge and formerly staff at J-PAL. I really enjoyed reading this report and was very impressed by the detail, clarity, and brevity (30 pages isn't short, of course, but this could've been much wordier - it's very to-the-point). Some specific things I found particularly exciting:

(1) The purpose of this report and the goals/values of the donors were made very clear at the beginning and referenced throughout the document. This is surprisingly easy to leave out, especially when writing for a familiar audience (like Effective Altruists) whom we may expect to have identical worldviews to our own.

(2) It was great to see your rationale for selecting your top interventions, as well as the reasons you chose not to select others from Table 1. It helped provide a complete picture of how and why you arrived at your conclusions.

(3) It was interesting (with no positive nor negative connotation - just truly interesting!) to see the GiveWell-evaluated health interventions included in the category of early childhood development. I've never seen them framed in that way, but it made sense to me while reading.

Other organizations you could consider looking into:

(1) BOMA Project, Trickle Up, and BRAC all implement the Graduation approach on the African continent.

(1a) BRAC is, of course, the creator of the approach and a very impressive and evidence-focused organization. I think it would be worth getting in touch with BRAC to see if donors of this size could earmark support to the Ultra-Poor Graduation Program in Uganda.

(1b) BOMA Project implements a version of the Graduation approach in a few countries, with most of its work in Kenya. I don't know them well, but they're often speakers at conferences with sessions on the Graduation approach and related social protection programs.

(1c) Trickle Up also implements the Graduation approach on the African continent and elsewhere. I have a positive impression of them from a few meetings with their M&E team, but don't have deep knowledge about their programs so can't recommend for or against.

(2) BRAC's Empowerment and Livelihoods for Adolescents program is also something that might be of interest. It's been studied with RCTs in Uganda and Sierra Leone (the latter study is really interesting as it took place during the Ebola outbreak) and found to have positive results on income and education levels. I've never seen a cost-effectiveness analysis, but the SL RCT notes that the program "has proved to be scalable and cost-effective across countries" (p. 11).

(3) J-PAL's Innovation in Government Initiative (IGI). I'd guess this was left out because it fits into the "evidence-based policy" category, but the competitive nature of the fund means that the projects it supports are more like TaRL Africa's work (specific, discrete technical assistance to meet policymaker demand) than generic advocacy. IGI is the recommended charity from Founders Pledge's report on evidence-based policy, and fits the "transformative" criterion in a slightly different way - enabling the transformation of entire government programs that are designed to improve lives.

A few notes and questions I had while reading:

(1) Did you assess try to assess organizations’ track record/strength when making these recommendations? From my experience, both TaRL Africa and Village Enterprise are exceptionally strong, evidence-based organizations with strong leadership and implementation on the ground. It could strengthen your recommendation to include a note about their commitment to evidence use and generation, local partnerships, etc.

(2) For Village Enterprise in particular, it would be good to mention the Sedlmayr, Shah, and Sulaiman (2018) study that directly compared VE's program with a cost-equivalent cash transfer in Uganda. This can help bolster your reasoning for selecting them over GiveDirectly (although the cash only results were really noisy and hard to interpret - I hope an updated paper will be released sometime soon).

(3) I was confused about your cost-effectiveness estimates for TaRL until I looked at the linked spreadsheet. The intervention can vary along a number of axes, and both the costs and impacts will vary based on the local context and population served, so I wasn't sure what a cost-effectiveness estimate for the TaRL approach meant. I see from the spreadsheet that your cost estimates came from the studies with Pratham in India - I think it would help to clarify that in the document itself.

HuweliumJun 20 20203

Hi samcart,

I'm glad you liked the report :) Thanks for the detailed and constructive feedback! Note: One reason that it isn't too long is that other reports preceded it (on education, early childhood development, small and medium enterprise support, rural electrification and long-term effects of cash transfers). This final report tries to summarize things.

Regarding the other organizations that implement the graduation approach in Africa, I did indeed look into them (see the report on cash transfers for details). BRAC might have been a good choice, but they run a very wide range of programs, and it was alot harder to figure out what would be the net impact of even a targeted donation to their graduation approach program (see e.g. this GiveWell post on BRAC). I was also influenced by discussions with Founders Pledge who also looked into these different charities and ended up recommending Village Enterprise.

As for J-PAL's Innovation in Government Initiative, I personally find it very interesting. I don't know if it would be a good fit with the donors' preferences. They will be going over the comments here, though, so they'll get a chance to see some of the other options that have been proposed.

Responses to the questions you posted:

1)Yes, I did try to assess organizations' strength and track record, though admittedly that was harder to do than going over RCTs and literature. It was part of the motivation for recommending TaRL Africa and Village Enterprise. I'll look over the text again and see if I can make that clearer. Mind you, all the other organizations that made it to Table 1 also seem to have a strong track record. I partly relied on the fact that many of these had been vetted by GiveWell and Founders Pledge.

2) and 3) I'll do that, thanks for the suggestion :)

HuweliumJun 29 20201

samcart, just a quick note that TaRL Africa is (or was) part of J-PAL's Innovation in Government Initiative (or its predecessor, the Government Partnership Initiative). See these links:

https://www.povertyactionlab.org/blog/1-16-18/increasing-use-data-and-evidence-real-world-policy-stories-j-pals-government

https://www.povertyactionlab.org/blog/8-8-17/zambia-scale-teaching-right-level-program-1800-schools

HuweliumJan 25 20217

Epilogue: I am happy to report that the donors have decided today to donate 1 million dollars (CAD) over the next five years to support Teaching at the Right Level, Africa. This initiative is helping to change education systems in African countries so that students in primary school can learn to read, write and count (which is sadly not the case right now, for the most part). Since writing my report, Teaching at the Right Level, Africa has also been reviewed and recommended by Founders Pledge.

I thank all of you for your thoughts and comments and am very excited to have been able to contribute to this :)

tomweinJun 16 20204

Thanks for your work on this. One charity to consider is Raising Voices, which I think fits their mandate exceptionally well (and which I also think should be considered by the wider EA community).

Full disclosure: I'm a former employee of Raising Voices.

RV has a program to prevent violence against children in schools. It is well-evidenced - there are now more than 25 peer-reviewed papers, including RCTs and cost-effectiveness evaluations, available on their VAC prevention program. They are based in Uganda, and their theory of change explicitly focuses on empowerment. They currently reach 750 schools across Uganda, and are currently scaling up across the country. Their work also has the potential for wider impact, since they lead several global coalitions. This sum would be significant in the context of their annual budget. Preventing violence in schools is good in itself, but there is good evidence that it relates to positive physical health, mental health, educational and economic outcomes.

HuweliumJun 16 20201

Thanks for the recommendation, tomwein. Could you provide a link to the RCTs or other experimental evidence?

tomweinJun 17 20203

Of course - this is the main RCT: https://www.thelancet.com/journals/langlo/article/PIIS2214-109X(15)00060-1/abstract

HuweliumJun 29 20201

tomwein, I just looked through the RCT. The reduction in violence is indeed significant and promising. In terms of test scores, though, they report "There was no evidence that the intervention had an impact on any educational test scores". Test scores and income were the main metrics we were focusing on for education interventions, so this intervention doesn't seem like a good match to the donors' preferences at this stage. From reading the RCT, it seems that studies about violence prevention in school are quite novel and that the evidence base is still thin, so again this wouldn't match well with the donors' wish to support interventions with strong evidence of effectiveness. Though obviously it's very valuable to be gathering evidence given the lack of previous studies.

[anonymous]Jun 18 20201

Perfect, thanks!

Stephen ClareJun 18 20203

Nice work! This seems like a pretty great overview of our current understanding of a whole range of international development interventions, at least on the micro end of the spectrum. Useful not just for your donors, but the community as a whole.

Two quick points. First, in your appendix you write that it would be interesting to see a rigorous evaluation of Jeff Sachs' Millenium Village Project. DFID did fund a big evaluation of that project which returned pretty negative results. Marginal Revolution discusses that here, plus there's a paper by Gelman et al. cited at the end of the MR post.

Second, I do think there's a bit of tension between the part where you write "the donors are comfortable with impact within the next 1 to 20 years or so, but not keen on options like R&D and policy advocacy which may or may not have an impact at all" and your recommendation of TaRL. As you note, it's completely possible that J-PAL's work to scale up TaRL across Sub Saharan African will fail. The line between policy advocacy and policymaking support is pretty blurry.

I don't want to discourage you from recommending TaRL because I think it's great! And of course you can't evalute everything. But I'd be interested in your thinking about what differentiates TaRL advocacy from advocacy for better health or macro policies, which you've largely excluded.

HuweliumJun 20 20202

@smclare: I'm glad you liked the report :) I definitely hope it can be helpful to others, since alot of work went into it! If it can save others some time, that would be great!

I'll check out the Marginal Revolution post on Millenium Villages and see if I can include a few sentences about that in the report.

As for TaRL Africa, alot of what they are doing is directly implementing TaRL in different African countries with partners and trying to ensure that the scale up there is a success. So I don't think of it as being mainly advocacy. You're right that there's no guarantee that it will work. I think there is a tension between the donors' goal of backing things that have been shown to work and having a long term and transformative impact, since the evidence on long term impacts is generally lacking (beyond say 7 years or so at most, in the case of the graduation approach). I tried to find a middle ground between their different goals. One of the reasons I included various charities in Table 1 was a recognition that different interpretations of their goals might lead to different recommendations.

samcartJun 18 20202

Initially, I had a similar thought while reading this - my understanding is that, at the outset, a large part of TaRL Africa's work focused on advocating for the TaRL approach and identifying partners who would be interested in adapting and implementing the approach in their contexts. However, TaRL Africa's work does seem to include technical assistance to local implementers who have committed to using the approach. Their work is described as "supporting policymakers and practitioners to set achievable goals, use teaching-learning practices that are at the level of the child rather than being dictated by a rigid age-grade curriculum, set up hands-on, on-site mentoring systems to support teachers to deliver effectively, and promote measurement strategies that lead to action" (from the Co-Impact page on their grant to TaRL Africa).

From reading about it in more detail, it seems that TaRL Africa's approach is substantially different from generic policy advocacy. It seems that TaRL Africa is working with a defined set of committed partners to achieve shared goals as effectively as possible. Their work in Zambia has already been rolled out to 1800 schools, and their partnerships elsewhere suggest that their probability of success is relatively high.

Of course, the cost-effectiveness of any donation will depend on the new activities that it enables, so it would be a good idea to check in with TaRL Africa (if you can) about what a donation of this size/timing would allow them to achieve.

Disclaimer: I work with Stephen at Founders Pledge and previously worked at J-PAL, including work on government partnerships that included the TaRL Zambia project.

HStencilJun 16 20201

Because you're recommending Village Enterprise, I'd also flag TechnoServe, which runs similar programs and is the top-rated poverty alleviation charity by ImpactMatters. It's worth noting that ImpactMatters only evaluated (I believe) TechnoServe's Impulsa Tu Empresa program, which operates exclusively in Latin America, but the organization runs analogous programs in Sub-Saharan Africa. Obviously, those might not be similarly cost-effective, but a prima facie basis for making that assumption (rather than the opposite, for instance) isn't obvious to me.

HuweliumJun 29 20202

@HStencil, TechnoServe does seem like an interesting organization. There is definitely some similarity to Village Enterprise, but Village Enterprise is focused on the graduation approach while TechnoServe runs a broader range of programs including support to small and medium enterprises. I was also very interested in small and medium enterprise support, but my review of the evidence on this suggests that it's inconclusive whether such programs have had success at creating jobs or reducing poverty. Of course, it's entirely possible that TechnoServe's programs do indeed do this. If so, that would be great! ImpactMatter's evaluation does suggest this, but as they point out the evidence is not very strong: it's based on a pre-post analysis conducted by TechnoServe themselves, response rates post-intervention were low and enterprises that did not experience increased revenues were excluded from the analysis (see excerpt from ImpactMatter's evaluation below).

Excerpt from ImpactMatter's evaluation report:

"Impulsa Tu Empresa’s beneft/cost ratio is based on data collected by TechnoServe on businesses’ gross revenues before and after participating in the program. We subtract from these figures the growth we assume businesses would have experienced had they not in fact participated in the program. However, only a small number of firms responded to TechnoServe’s surveys conducted two to three years after they completed the program. TechnoServe also did not count businesses that had lower revenues after the intervention than before the intervention, biasing upward the benefit/cost ratio"

HStencilJun 29 20201

Thanks so much for looking into this and posting your read of the research! I'm glad I now have a clearer sense of how these two types of interventions compare to one another. The flaws you noted in TechnoServe's internal evaluation are certainly quite concerning, and I'm glad someone brought them to my attention.

[anonymous]Jun 18 20201

Thanks for the tip, HStencil :) I will check them out!

bfinnJun 16 20201

Looks good from a quick read-through.

To make an obvious point, as relevant information (including about new charities/causes) will presumably improve a lot over the next 5 years, there seems a case for updating your recommendation annually rather than the donors committing upfront to donating 5 years' worth to particular charities (if that was the idea).

Depending to some extent on whether a 5-year commitment is essential for the programmes being donated to. If it is, a middle way might be to commit upfront to donating for 5 years subject to the programmes achieving XYZ goals, to be independently assessed each year.

Another obvious point (which you mention of course): the extremely wide range of the TaRL and salt iodization cost-effectiveness figures, from far below 1 (Founders Pledge estimate) to far above, would give me concerns as a donor that these are poorly understood interventions.

tomweinJun 16 20202

Multiyear commitments have a particularly high value to charities, especially when they can be used for operational support. They allow charities to take more risks, act more directly in line with their mission, and spend less time on report writing. They are much rarer in the sector.

HuweliumJun 16 20201

I think the donors do indeed intend to commit for 5 years, for the reason tomwein invokes. But of course if new evidence suggests an intervention really isn't having the impact that we expected, or something else that seems much more promising comes along, presumably they could still revisit their commitment on an annual basis.

Regarding TaRL, the intervention has been studied extensively. The main uncertainty is whether and to what extent gains in test scores translate to long-term outcomes like higher income. But since the donors also care about improvements in learning outcomes per se, there is a bit of a hedge here. It just isn't captured in the cost-effectiveness analyses, which only incorporate effects on income.

Stephen ClareJun 18 20201

On TaRL specifically, the effects seem really good--whether or not we can get governments to implement TaRL effectively seems to be where most of the uncertainty lies.

HuweliumJun 20 20201

@smclare Thanks for giving some background on the Founders Pledge cost-effectiveness scenarios. For TaRL, I'm surprised that you say that the optimistic scenario is the unfathomably best case scenario. Even in that scenario, impacts are assumed to last 20 years, and the impact of test scores improvements on earnings does not use the most optimistic cases mentioned in the Founders Pledge education report. It seems fathomable impacts could last a whole career (say 40 years). As you can see from my cost-effectiveness estimates for TaRL, my unfathomably best case scenario is significantly more optimistic than the one from Founders Pledge (I included in the cost-effectiveness spreadsheet a worksheet using the worksheet from Founders Pledge as a starting point, but with my own scenarios in there). And in both cases, we only include the impact on income. It seems quite plausible that education would have impacts beyond that which aren't taken into account.