Background

As part of the activities of EA Israel, we recently completed our first cost-effectiveness analysis of a program run by an Israeli charity. We analyzed Challenge 22, a program promoting a vegan diet that is carried out by Animals Now. The full report is available here.

Why did we write this report?

Here are some of our motivations:

1. While we were uncertain about the competitiveness of this program compared to the world’s most effective charities helping to reduce animal suffering, we thought it was worth exploring. As a local EA group in Israel, we expected to have a comparative advantage in assessing the cost-effectiveness of the program (by avoiding the need to bridge over cultural differences).
2. We hope this report will help Israeli donors who are interested in donating to effective Israeli charities to give more effectively. While we encourage donors to consider giving to global effective charities (such as those recommended by GiveWell or ACE), many Israelis feel strongly about giving locally. We believe that in some cases it is wiser to just give such donors what they want, helping them to find effective local charities rather than preaching to them about the merits of impartiality. In the long run we hope they will eventually come to absorb more of EA’s core values and someday consider giving to the world’s most effective charities, regardless of nationality.
3. We anticipate that many of the Israeli donors who prefer giving locally also prefer cause areas that help people (rather than animals). We hope this report will strengthen our reputation among Israeli charities and philanthropists, allowing us to form more collaborations and write similar reports for charities in other cause areas as well.
4. Our long-term goal is to promote standards of cost-effectiveness and impact evaluation in Israeli and Jewish philanthropy, a vision we promote through other projects as well (see for example our Cause-specific Effectiveness Prize). The reasons we think this is a worthwhile goal to pursue are:
   • There are billions of dollars going to Israeli NGOs every year. Having this huge sum of money allocated even slightly better can have a very large direct impact.
   • If we want to promote the ideas of EA and make them more mainstream, we need to acknowledge how strange they sound to the average person. The idea that one should consider giving their money to distribute bed nets in Africa sounds very alien to the average Israeli. On the other hand, making the claim that one should consider giving to Israeli charities that are transparent and have strong evidence for high impact is much more intuitive to people. We think it is common-sense enough to have a real chance to take root and become mainstream in the coming years. As mentioned in point (2), we see the promotion of cost-effectiveness standards as a stepping stone on the way to accepting EA values more broadly.

For your convenience, we copy-pasted the executive summary of our report and present it at the remainder of this post.

We would appreciate any type of feedback!

Introduction

This report analyzes Challenge 22, an international program carried out by the Israeli charity Animals Now. The program promotes veganism and reduction in meat consumption by encouraging individuals to take a pledge to change their diet for 22 days. Throughout the challenge and afterwards, participants have access to social support through Facebook groups aimed at helping them to follow through with the pledge, and to sustain their dietary changes in the long run.

Cost-effectiveness estimates

The primary evidence we rely on for evaluating the cost-effectiveness of the program is a survey study published in 2019 (following participants who had signed up to the program during 2018). By integrating the results of this study with estimates on the scale and budget of the program (provided by Animals Now), we estimate that every 1 ILS of funding going into the program corresponds to a reduction in meat consumption of somewhere between 1 to 12 portions of meat for an average participant in the program over the course of the first 7 months after taking the pledge. In US dollars, we estimate that every $1 corresponds to somewhere between 3 to 40 portions of meat (based on the exchange rate of October 2020 - 1 ILS for 0.29 USD).

While we only have solid evidence for dietary changes of participants following the first 7 months after taking the pledge, there are positive indications that these changes are robust and could persist for substantially longer time periods. Specifically, there is no significant difference between the dietary change observed after 7 months to that observed after 1 month. This suggests that the reduction in meat consumption could actually be substantially larger than the aforementioned estimate. If we assume, for example, that dietary changes following the program last 5 years on average, we would get that every 1 ILS of funding going into the program corresponds to a reduction in meat consumption of 9 to 100 portions of meat. 

Limitations and uncertainties

These cost-effectiveness estimates are subject to some important limitations of our analysis, which should be kept in mind when judging the impact of the program:

1. The numbers above describe the estimated change in meat consumption for participants who have joined the program, not direct causal effects. As the underlying study is observational, we don’t have sufficient evidence to prove a causal link. However, we believe that the results look encouraging, and causal effects seem plausible.
2. The underlying study surveyed only Israeli participants, which comprise only about 10% of the program. As stated in the report, there are marked differences distinguishing the Israeli program. Our lower-bound estimate incorporates the assumption that the average magnitude of dietary change for non-Israelis is no less than 35% of the change for Israelis. Much of the wide uncertainty range in our final estimate (with a 12-fold factor between the lower to the upper bound) is due to this limitation.
3. The report assesses the total magnitude of dietary changes per ILS based on data observed in the past. An important aspect of the program is that it is rapidly changing and evolving. While these changes are generally positive (reflecting attempts to scale the program and make it more cost-effective, with some indications of success), it also means that it is hard to extrapolate what the future cost-effectiveness of the program will be. Also, this report does not thoroughly tackle the question of how much room for more funding the program has.
4. There are some important limitations of the survey study on which our estimates for reduction in meat consumption are based. Most substantial is its reliance on self-reports from participants, which might be prone to social desirability or memory biases. Other, more minor potential limitations are detailed in the report.

On top of these limitations, there are other uncertainties which we have already factored into our analysis, and which have led to the wide uncertainty range in our final estimate. A more exhaustive summary of the limitations of our analysis can be found at the 8th chapter of the report.

Another point that should be kept in mind is that Challenge 22 is only one of several other activities pursued by Animals Now. In terms of budget, it comprises about 20% of the charity. We haven’t analyzed the cost-effectiveness of the other programs.

As a result of all the many limitations of the analysis mentioned above, we encourage the reader not to take the numbers we have come up with too literally. Rather, this cost-effectiveness analysis should be interpreted as a rough estimate pointing at a general order of magnitude for the potential impact of the program.

Conclusion

Despite all of the mentioned limitations and uncertainties, in the context of Israeli philanthropy this program is quite exceptional in terms of measurement, transparency standards and evidence for positive impact. 

The full report is available here.