Concerns with ACE research

Summary

I outline some concerns about ACE’s research. I show that some of ACE’s older research is of low quality, and should be removed from the website - ACE’s new Research Director agrees with this. More importantly, ACE’s research on the impact of corporate campaigns is flawed, and consequently ACE’s research does not provide much reason to believe that their recommended charities actually improve animal welfare. This is not a criticism of ACE’s recommended charities. I conclude with some thoughts about how ACE could improve and note a cause for optimism, as a new Research Director arrives.

***

ACE’s research has been criticised in the past, most notably in a December 2016 blogpost by Harrison Nathan. ACE’s research has improved since then with some of the most serious problems being resolved. For example, their November 2017 leafleting report was of a good standard, and was a major improvement on their previous leafleting report, which was of poor quality. However, I have examined ACE’s research in 2018 and 2017 and believe that it still contains some serious flaws.

Evaluating the impact of animal charities is generally more difficult than evaluating the impact of charities carrying out direct health interventions because evidence is sparse and much hinges on difficult questions about animal sentience. Consequently, ACE’s research team faces a harder problem than GiveWell’s. However, this point notwithstanding, ACE’s research still falls short of what we should expect. The problems concern the failure to remove low quality older research and, more importantly, the reasoning for judgements about the effectiveness of corporate campaigns.

1.       Old research

a.       Many of ACE’s older intervention reports are of low quality, and should have been removed from ACE’s website.

b.      ACE now acknowledges in private that these reports are of low quality and for the most part does not rely on them in its charity reviews, but in many cases ACE fails to inform reader of this fact.

2.       Corporate campaigns

a.       ACE does not have up to date research of sufficient quality on the welfare effects of corporate campaigns.

b.      ACE also does not check whether their recommended charities are genuinely causally responsible for the corporate policy successes that they claim.  

The problems I discuss here pertain only to the reviews of The Humane League (THL) and Animal Equality, as I have not had time to look into their research on the Good Food Institute. This piece is in no way a critique of ACE’s recommended charities.

I publish this critique firstly in the hope that it will encourage an improvement in standards at ACE; secondly, that it will encourage external scrutiny of ACE research going forward; and finally that it will provide information relevant to individual donors. In August 2018, ACE appointed a new Research Director, Toni Adleberg, who should not be held responsible for the mistakes discussed here. My interactions with Ms Adleberg and other members of ACE’s current research staff have been very positive, and I am optimistic that there will be improvements in ACE’s research in the future.

Disclosure: I interned at ACE for a few months in 2015. I am currently an unpaid external research consultant for ACE, and have thus far carried out about 2 hours of work for them in this role. ACE reviewed this piece prior to publication. The views expressed here are my own, not those of Founders Pledge.

1. ACE’s view on the impact of their recommended charities

Before we begin, it is useful to distinguish three platforms in which ACE presents its views:

1.       Intervention reports

2.       Cost-effectiveness analyses

3.       Current all-things-considered view expressed in charity reviews

The view expressed in each these three things are often different. For example, the view expressed in the intervention report on investigations is different to the view expressed in the cost-effectiveness analyses of investigations. ACE is also at pains to point out that their cost-effectiveness analyses are only supposed to give a very rough picture of the cost-effectiveness of their charities, and they say that cost-effectiveness analyses play “only a limited role in our overall opinions of which charities and interventions are most effective”.[1]

With this clarified, we can now outline ACE’s current view on THL and Animal Equality have impact. The interventions pursued by these two organisations can be divided into grassroots outreach and corporate outreach. As I am defining the term, grassroots outreach includes things like leafleting, online ads, social media outreach, humane education, and undercover investigations.

Corporate outreach accounts for the majority of the modelled impact of both charities in the cost-effectiveness analyses of THL and Animal Equality: for THL, ~90% of the modelled impact is from corporate outreach, and ~10% from online outreach; and for Animal Equality, >90% of the modelled impact is from corporate outreach. This seems consistent with ACE’s all-things-considered view of which interventions are likely to be impactful. ACE says that “[corporate campaigns] can be highly impactful when implemented thoughtfully”.[2] ACE seems to be pessimistic about grassroots outreach, saying:

“THL works to effect change through several different kinds of outreach, including leafleting, online ads, and education. While there is little evidence available about the effectiveness of these interventions, we do not currently recommend the use of leafleting or online ads as we suspect that they are not as effective as some other means of public outreach.”[3]

Thus, ACE’s view as of August 2018 is that grassroots advocacy has close to no effect, though ACE does estimate that THL’s online outreach is beneficial. I discuss this in section 2b.

2. Grassroots outreach

In this section, I discuss ACE’s research on the impact of grassroots outreach. Due to lack of data on the impact of most forms of grassroots outreach, in their cost-effectiveness analyses, ACE estimates the impact of most forms of grassroots outreach – humane education, investigations and Animal Equality’s i360 video outreach –  by assuming that they were X times as effective as leafleting. For example, we would, the argument goes, intuitively expect humane education to be ~5 times as cost-effective as leafleting, and leafleting spares X animals from factory farming, so humane education spares ~5x animals from factory farming. 

Prior to November 2017, ACE believed that leafleting produced fairly substantial benefits. However, in November 2017, ACE published a new leafleting report, stating that their view on the impact of leafleting had changed. They now believe that leafleting has tiny impact, and is in fact very slightly harmful. However, ACE continues to infer the impact of other forms of grassroots outreach from the impact of leafleting, using a scaling factor. Thus, because they judge humane education to be ~5 times as effective as leafleting and they estimate leafleting to be harmful, they now estimate that the mean effect of humane education is 5x as harmful as leafleting. It is not clear that continuing to use a scaling factor makes sense when the sign of the effect has changed, but that is perhaps open to debate. They continue to use this approach for modelling the impact of humane education, investigations, and Animal Equality’s video outreach.

Setting the specifics of the cost-effectiveness analysis to one side, as mentioned, ACE’s apparent current all-things-considered view is that all forms of grassroots outreach (with the possible exception of online ads) have close to zero effect. I agree that grassroots outreach is likely to have very small effects because securing dietary change through advocacy appears to be very difficult, with the best data suggesting that the percentage of veg*ns has barely increased since the 1990s, as discussed in this excellent recent ACE blog.  

2a. Investigations

Of THL and Animal Equality, only Animal Equality carries out investigations. The Feb 2016 intervention report on investigations on ACE’s website evaluates the impact of investigations in a different way to the cost-effectiveness analyses of Animal Equality, and produces a different estimate of their impact. The quality of the intervention report is low. One major concern is that the impact calculations are done in a table, rather than in a spreadsheet or Guesstimate model, making it difficult to understand important inputs, calculations and outputs.

Direct suffering avoided per intervention unit

An ‘intervention unit’ is the measure of the length of an intervention. For investigations, 1 intervention unit is 1 investigation. A unit of suffering is a year of farmed captivity, or equivalent, averted.

ACE provides pessimistic and realistic (but not optimistic) estimates of ‘direct suffering per intervention unit’. No explanation is given for these figures and they are actually the same as the ‘indirect suffering avoided per intervention unit’. Assuming that direct and indirect suffering averted are intended to be different, this suggests a mistake that should have been noticed in a review of the page. Here is a screenshot (I have added the column labels from a separate screenshot); these numbers appear without any explanatory calculations.

(Screenshot: 15th August 2018, about 95% of the way down the page)

Indirect suffering avoided per intervention unit

This is a screenshot of the relevant part of the table on ‘indirect suffering avoided per intervention unit’ (again, I have added the column labels from a different screenshot further up the page):

(Screenshot: 15th August 2018, about 95% of the way down the page)

There are three problems with this. Firstly, ACE estimates the reach of a campaign stating:

‘A person does not have to be directly contacted by a staff member in order to be “reached”. They must merely encounter the campaign in some capacity, including living under a legal jurisdiction being targeted by a legislative campaign.’

The condition specified in the second sentence is clearly inappropriate: obviously, not everyone who lives in a legal jurisdiction targeted by a campaign should count as being reached by the campaign.

Secondly, the optimistic estimate cannot be correct and is therefore not a reasonable rendering of ‘optimistic’. It implies that each undercover investigation makes 2.8% of those reached vegetarian or vegan. If, as estimated in the Feb 2016 intervention report, each campaign will reach ~9m people, this would imply that each campaign will create ~250,000 new vegetarians or vegans. Since Animal Equality runs 20 investigations a year,[4] this would imply that they alone will create ~5m new vegetarians or vegans per year exclusively through their undercover investigations. It is somewhat difficult to estimate how many vegetarians there are in the US because accurately tracking dietary habits is challenging and people tend to lie about their animal product consumption. While up to 6% of people in the US self-identify as vegetarian or vegan, but studies that ask people whether they ate meat or fish on two consecutive 24 hour periods show that the vegetarianism rate remains below 1.5%. See this ACE blog for more. Even on the higher estimate based on self-identification, the estimate of the impact of investigations cannot be correct.

Thirdly, no explanation is provided for the pessimistic or realistic estimates; as you can see in the above screenshot, headline figures appear without any calculations. This is concerning. When I asked ACE about this in 2017, they said that the basis for the figure was another page on the impacts of media coverage on meat demand, which is not referenced at that point on the cost-effectiveness analysis of undercover investigations.[5]

I pointed all of this out to ACE in September 2017, and they acknowledged that this was a mistake, but the page has still not been altered as of August 2018. I think this page should be immediately removed. In conversation with ACE’s new research team, they have told me that they were already considering removing these old intervention reports before I sent them this report, which is an encouraging sign.[6]

2b. Online ads

As I mentioned above, in ACE’s cost-effectiveness analysis, online ads are responsible for around 10% of THL’s impact.  There are two main problems with ACE’s treatment of online ads.

Firstly, it models the effect of online ads in a different way to how it models other similar forms of grassroots outreach. As discussed above, ACE uses the ‘leafleting scaling factor’ approach for all other forms of grassroots outreach. It does not use this approach for online ads, despite the fact that it seems that whatever rationale justifies using the ‘leafleting scaling factor’ approach for other forms of grassroots outreach must also apply to online ads. For example, Animal Equality’s i360 video outreach seems functionally similar to online ads, and yet a different approach is taken for each.

Secondly, in its cost-effectiveness analysis of THL’s online ads, ACE relies on an August 2016 intervention report on online ads. This report is highly opaque and it is very unclear how ACE arrived at certain crucial figures, such as number of vegetarians created per click on an online ad. Here is the crucial section in the cost-effectiveness estimate of online ads:

 Source: intervention report, online ads (as of 10^th August 2018)

The reasoning presented for these numbers is limited, and because the numbers are presented in a table rather than a spreadsheet, it is not clear how they are combined together.

When I asked about this in October 2017, ACE said the researcher who made the calculation has left and that the factors bearing on it were not weighted in a formal way that could be publicly explained.[7] Indeed, in the comprehensive review of Animal Equality, ACE says “There is little evidence, thus far, about the effectiveness of online outreach...”[8]

In summary, ACE has used figures which it cannot publicly explain and which it now rejects, to estimate the impact of one of its recommended charities

3. Corporate campaigns

In their cost-effectiveness analyses, ACE estimates that corporate campaigns account for the vast majority of the impact of their recommended charities. It also appears to be ACE’s all-things-considered view in its charity reviews that corporate campaigns are one of the highest impact interventions. For example, in its review of THL, ACE says of interventions aiming to influence industry “We find that these interventions can be highly impactful when implemented thoughtfully”.[9]

However, ACE’s research on corporate campaigns is flawed in four ways. These flaws are the most important ones I have found, given how important corporate campaigns are in ACE’s current research.

1. In the November 2014 intervention report on corporate campaigns, ACE did not carry out a comprehensive literature review of the evidence on the welfare effects of cage-free systems. They relied only on a paper by De Mol et al (2006), which seems quite badly flawed, as discussed in a report by the Open Philanthropy Project published in September 2017 (specific section of the report on De Mol et al (2006)).[10] The evidence on the welfare benefits of cage-free systems is much more unclear than suggested in ACE’s assessment of corporate campaigns.
2. ACE no longer endorses the conclusions in its intervention report on corporate campaigns, but does not inform the reader of this on the intervention report page or anywhere in the reviews of its recommended charities. The intervention report on corporate campaigns apparently plays no role in the cost-effectiveness estimate of corporate campaigns or in the charity reviews, but this is not explained to the reader. This leaves a major justificatory gap in ACE’s research. In its cost-effectiveness analyses, ACE estimates that their mean estimate of the “proportional improvement in welfare due to cage-free policies” is ~0.05,[11] but provides only a one sentence explanation for this estimate. This crucial figure should be defended with multiple page report and review of the science on the welfare effects of cages.
3. ACE have been aware of the problems with the De Mol et al (2006) paper at least since September 2017, and told me in October 2017 that revising the corporate campaigns intervention report was a high priority for the coming year.[12] As of August 2018, this report has still not been updated and ACE have told me that it will very likely be updated in early 2019.[13] The report has now not been updated for almost four years. This progress seems much too slow given how important corporate campaigns are in ACE’s evaluations, and according to conventional wisdom in effective animal advocacy.
4. In its review of THL’s and Animal Equality’s corporate outreach, ACE relies only on the charities’ self-reported corporate policy successes, which it then discounts by an arbitrary uncertainty factor: ~0.4 for both Animal Equality and THL. ACE does not check with third party news sources, experts or with the companies themselves on whether the claims of the charities are accurate.[14] This seems to me like basic due diligence that one should carry out when assessing corporate advocacy.

Conclusion

ACE’s research has improved somewhat over the last year or so, but serious problems remain. I believe that at present, donors interested in animal welfare would learn little from ACE’s research. Some of the problems (those regarding grassroots outreach) chiefly concern the failure to remove older poor quality research and to adequately communicate their views to consumers of their research. Other problems (those regarding corporate outreach) are highly relevant to crucial judgements about whether ACE’s recommended charities have any impact at all, or are even harmful (which is possible on some apparently reasonable views on the effects of cage-free systems).

1.       Grassroots outreach

a.       ACE believes that grassroots outreach has small to zero impact, which seems right.

b.      However, ACE’s intervention reports on many forms of grassroots outreach are of poor quality and should have been removed from ACE’s website.

c.       ACE does not communicate their current view of the quality of some of these reports, nor does it explain that the conclusions therein diverge from their current view.

2.       Corporate outreach

a.       Corporate outreach accounts for most of the recommended charities’ impact in ACE cost-effectiveness analyses and in their all-things-considered view of the impact of their charities.

b.      In spite of that, ACE still does not have a high quality up to date review of corporate outreach or of the welfare effects cage-free systems.

c.       ACE does not independently check the claims of charities about their corporate policy successes.

To be clear, I do not think that these problems can be explained by the fact that cost-effectiveness analyses for animal interventions are extremely uncertain and should not be taken overly literally. The main problems pertain to research questions absolutely crucial to figuring out whether animal charities are beneficial or harmful, which have not been adequately answered by ACE.

As I mentioned at the start of this brief, ACE research has improved over the last two years, and I am optimistic that with the arrival of their new Research Director, Toni Adleberg, ACE’s research will improve at a faster pace in the future. Going forward, I think the main improvements would come from:

1.       Removing the older lower quality intervention reports from the ACE website. As I have mentioned, it is a positive sign that ACE were already considering doing this before I sent them this report.

2.       Producing a report on the welfare effects of corporate campaigns and independently checking charities’ claimed corporate policy successes.

3.       Regularly updating intervention reports and charity reviews to reflect ACE’s current views, rather than updating them wholesale every >1 year.

I should also add that I do not think it would be wise for ACE to rely less on quantification in cost-effectiveness analyses in the future. Quantifying the impact of animal charities is very difficult, but without quantifying impact, comparisons between charities become much more difficult.