by Avi Norowitz
Animal Charity Evaluator’s (ACE’s) 2016 cost-effectiveness estimates indicated that corporate campaigns to improve the welfare of farm animals by The Humane League (THL) and other organizations was a highly cost-effective intervention. Although ACE’s 2017 estimates still show high levels of cost-effectiveness for corporate campaigns, the 2017 estimates do indicate a notable decline from 2016. In the case of THL, some of this difference disappears after adjusting for some changes in ACE’s assumptions. From a quantitative perspective, the remaining difference appears to have been caused by a large increase in budget combined with a smaller increase in equivalent years of suffering spared. This post also considers some qualitative considerations that may help explain the apparent changes in cost-effectiveness. This post ends with some reasons to put limited weight into this analysis.
Animal Charity Evaluator’s (ACE’s) 2016 cost-effectiveness estimates indicated that corporate campaigns to improve the welfare of farm animals was a highly cost-effective intervention. ACE’s 2016 cost-effective estimate for The Humane League (THL), for instance, estimated that THL’s corporate campaigns during most of 2016 had the equivalent benefit of sparing ~57 animal years of suffering per dollar.  This is in line with ACE’s 2016 estimates for other organizations, with THL performing somewhat better than other organizations.
ACE’s 2017 cost-effectiveness estimates indicated lower levels of cost-effectiveness for corporate campaigns, with THL’s cost-effectiveness for their 2017 corporate campaigns declining to ~7 equivalent years of suffering spared per dollar. While ~7 equivalent years of suffering spared per dollar still places THL’s corporate campaigns among the most cost-effective programs according to ACE’s cost-effectiveness estimates, it does represent a substantial decline from ~57. This post investigates the reasons for the apparent decline in the cost-effectiveness of THL’s campaigns between ACE’s 2016 and 2017 cost-effectiveness estimates.
Although I’m interested in the cost-effectiveness of corporate campaigns and related work in general, this post focuses on THL because of data limitations and time constraints.
ACE’s cost-effectiveness estimates: 2015-2017
To make comparisons over multiple years easier, I reproduced the “campaigns” section from ACE’s 2015, 2016, and 2017 cost-effectiveness estimates of THL into a spreadsheet in Google Sheets.  On the metric of “equivalent years of suffering spared per dollar,” THL’s corporate campaigns peaks to a high in 2016 and then declines to a low in 2017.
So, why does cost-effectiveness appear to decline in 2017? To investigate this, I generated a few more graphs. First, the following graph shows the “equivalent years of suffering spared,” without taking into consideration cost. The graph indicates a decline of this metric between 2016 and 2017.
Second, the following graph shows THL’s annual budget on corporate campaigns, according to ACE’s cost-effectiveness estimates. This graph shows very large increase in THL’s annual budget for corporate campaigns in 2017.
The above graphs suggest that the an apparent decline in “equivalent years of suffering spared” and a large increase in costs both contributed to the apparent decline in cost-effectiveness in the estimates. As discussed in the following sections, however, there is more to the story.
Changes in ACE’s assumptions in 2017
Upon further investigation, I found that much of the apparent differences are explained in changes in some of the assumptions in ACE’s cost-effectiveness estimates.
- In 2017, ACE added a new parameter for the “Probability that groups will follow through on the pledges that they made,” and assigned a value of 0.75 for THL. Since this didn’t exist in the 2015 and 2016 estimates, those estimates implicitly assume a probability of 1.00.
- In 2015 and 2016, ACE estimates that the “Proportional improvement in welfare due to cage-free policies” is ~0.10. That is, moving one hen from a battery-cage to a cage-free facility reduces 10% as much suffering as preventing the hen from existence. In 2017, ACE reduced this estimate further to ~0.05.
As demonstrated in the following section, applying these changes to 2015 and 2016 causes a decline in cost-effectiveness for these years, and also reduces the apparent decline from 2016 to 2017.
ACE’s cost-effectiveness estimates, adjusted with 2017 assumptions
I think ACE would probably agree that the assumptions described above should apply similarly to 2015 and 2016. Therefore, I’ve adjusted the estimates applying the 2017 assumptions to the previous years as well. With these adjustments, the apparent decline in cost-effectiveness between 2016 and 2017 still remains, but is smaller.
With these adjustments, we also see an increase in “equivalent years of suffering spared,” in contrast to the decrease we saw before these adjustments.
The adjustments, however, do not affect the spending, which substantially increased between 2016 and 2017. This increase in spending substantially outweighs the increase in “equivalent years of suffering spared” in the adjusted estimates.
What factors might explain year-by-year changes in ACE cost-effectiveness estimates?
2016 represented an unusual year of US cage-free wins
The following graph from Lewis Bollard at the Open Philanthropy Project shows an extraordinary increase in 2016 in the number of hens covered by US corporate cage-free pledges, representing more than ~80% of the US egg supply.
Given the unprecedented scale of these pledges, it may be expected that the years before and after 2016 appear less cost-effective on ACE’s cost-effectiveness metrics.
ACE cost-effectiveness estimates don’t capture the importance of earlier groundwork
For more than a decade, THL and other organizations have been laying the groundwork for the cage-free pledges in 2016. It’s likely that much of this groundwork, which included smaller, early wins, was necessary for the 2016 pledges to materialize. Therefore, attributing to the pledges in 2016 to the costs in 2016 may cause the 2016 work to appear more cost-effective than it actually was, while making earlier years appear less cost-effective than they really were.
Moving forward, THL and other organizations are working on securing pledges from corporations to improve the welfare for broiler chickens in the US and layer hens outside the US. While the US cage-free pledges cover most of the 300 million layer hens in the US, representing a very large number of animals, these numbers are still small in comparison to the ~1.1 billion broiler chickens in the US and the ~6 billion layer hens elsewhere. If THL and other organizations are able to secure pledges for a large proportion these animals over the next few years, it might be correct to attribute much of these pledges to the groundwork done in earlier years, including the smaller wins in 2017.
The large budget increase in corporate campaigns from 2016 to 2017 represents a large capacity building effort. It’s expected that capacity building may show limited benefits in the short term as employees are hired and trained, even if they yield large benefits in the long term. One consideration against this view, however, is that THL also substantially increased their corporate campaigns budget between from 2015 to 2016, and this increase was associated with a large increase in apparent cost-effectiveness.
Reasons to put limited weight into this analysis
This analysis represents an attempt to understand the apparent changes in cost-effectiveness, but there are multiple reasons not to put too much weight into it.
ACE’s acknowledges the limitations of their cost-effectiveness estimates
In regard to their cost-effectiveness estimates, ACE writes:
We want to be very clear: our cost-effectiveness estimates are approximations, they involve uncertain quantitative estimates, and they are subject to bias and error. They are not the only factor we consider when we evaluate interventions and charities—so our readers should interpret them carefully.
We do not advise our readers to accept our CEEs unquestioningly.
Comparisons between years could be misleading
ACE makes no claim that comparing cost-effectiveness estimates between years will lead to meaningful results, and there are reasons for concern that such comparisons could be misleading.
The adjustments I made for 2015 and 2016 were based on the two obvious changes I observed in ACE’s assumptions. However, there may have been other less obvious changes that affected these estimates. As an example, there’s some ambiguity about what time period each cost-effectiveness estimate covers, and which aspects of the estimates are based on past data vs. future projections. If estimates for different years vary in their approach in these areas, this may reduce the accuracy of these comparisons.
Different metrics would lead to different results
Although I’m focusing on the metric “equivalent years of suffering spared,” ACE also provides a different metric of “equivalent animals spared.” I prefer to focus on the former because I regard it as more morally relevant, but others might disagree. Focusing on the metric of “equivalent animals spared” would lead to a very different story about changes in cost-effectiveness, in which the changes are overwhelmingly dominated by the pledge THL secured in 2016 from United Egg Producers to end the culling of day old male chicks (a.k.a. “in ovo sexing”).
 ACE prefers to provide cost-effectiveness estimates as ranges rather than expected values, but for simplicity I’m just considering the expected values.
 The 2016 estimate will likely show similar but not identical results for “Equivalent years of suffering spared per dollar by Campaigns.” This occurs because each refresh at Guesstimate runs multiple Monte Carlo simulations, producing varying results each time.
 I’ve collected “equivalent years of suffering spared” for corporate campaigns and engagement for multiple organizations, and you can find a graph of this in this spreadsheet. I considered including this graph here, but decided against it because I worry that without following it up with an adjusted version, some readers might be misled.
 For 2015, the budget costs include university and Meatless Monday campaigns, so the cost-effectiveness of corporate campaigns may be understated.
 The original 2016 and 2017 Guesstimate estimates will likely show similar but not identical result for “Equivalent years of suffering spared per dollar by Campaigns.” This occurs because each refresh at Guesstimate runs multiple Monte Carlo simulations, producing varying results each time.
 Estimates are from this spreadsheet by Lewis Bollard at the Open Philanthropy Project. The layer hen estimate is from the “Top 15” sheet, subtracting the 300 layer hens in the US. The US broiler chicken estimate is from the “US States (2015)” which is extrapolated from USDA data and probably more accurate than the FAO data in the “Top 15” sheet. Although ~9 billion broilers chickens are slaughtered each year in the US, these animals are slaughtered after ~47 days so only 47/365 of these animals are alive at a given time.