Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just haven't gotten around to it.

I think GIF's impact methodology is not comparable to GiveWell's. My (limited) understanding is that their Practical Impact approach is quite similar to USAID's Development Innovation Ventures' impact methodology. DIV's approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGO's impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. It's true that GiveWell Top Charities are much more likely to survive without GiveWell's help but this leads to my next point.  

GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). They've been funding a range of interventions that aren't Top Charities and in many cases, are very early, with GiveWell support being critical to the NGO's survival. According to GiveWell's All Grants Fund page, "As of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so there's a high likelihood that donations to the All Grants Fund will support a top charity grant." This suggests that in GiveWell's own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldn't account for the majority of the fund. 

When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isn't), it's plausible their innovations have more impact and thus are more 'cost-effective'. But we are talking about leverage now, so again, I don't think the methodologies are directly comparable.

Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.

Thanks for your response, Joel!

Stepping back, CEARCH's goal is to identify cause areas that have been missed by EA. But to be successful, you need to compare apples with apples. If you're benchmarking everything to GiveWell Top Charities, readers expect your methodology to be broadly consistent with GiveWell's and their conservative approach (and for other cause areas, consistent with best-practice EA approaches). The cause areas that are standing out for CEARCH should be because they are actually more cost-effective, not because you're using a more lax measuring method.

Coming back to the soda tax intervention, CEARCH's finding that it's 1000x GiveWell Top Charities raised a red flag for me so it seemed that you must somehow be measuring things differently. LEEP seems comparable since they also work to pass laws that limit a bad thing (lead paint), but they're at most ~10x GiveWell Top Charities. So where's the additional 100x coming from? I was skeptical that soda taxes would have greater scale, tractability, or neglectedness since LEEP already scores insanely high on each of these dimensions.

So I hope CEARCH can ensure cost-effectiveness comparability and if you're picking up giant differences w/ existing EA interventions, you should be able to explain the main drivers of these differences (and it shouldn't be because you're using a different yardstick). Thanks! 

Hi Joel, I skimmed your report really quickly (sorry) but suspect that you did not account for soda taxes being eventually passed anyway. So the modeled impact of any intervention shouldn't be going to 2100 or beyond but out only a few years (I'd think <10 years) when soda taxes would eventually be passed without any active intervention. You are trying to measure the impact of a counterfactual donated dollar in the presence of all the forces already at play that are pushing for soda taxes (how some countries already have them). This makes for a more plausible model, and I believe is how LEEP or OpenPhil model policy intervention cost-effectiveness (I could be wrong though).

New phrasing works well!

Got it. But I think the phrasing for the number of animals that die is confusing then. Since you say "100 other human [sic] would probably die with me in that minute," the reference is to how many animals would also do during that minute.  I think what you want to say is for every human death, how many animals would die, but that's not the current phrasing (and by that logic, the number of humans that would die per human death would be 1, not 100).

I'd suggest making everything consistent on a per-second basis as smaller numbers are more relatable. So  1 other human would die with you that second, along with 10 cows, etc.

Thanks for writing this! The very last sentence seems off. Did you mean to say every second (instead of minute)? Also, the number of farm animals that die every second should be 1/60 (not 1/120) of that in the “minute” table above.

This last sentence was quite shocking for me to read. It’s sad…but very powerful.

Minor suggestion: in your title and summary, please just write out "10 k" as 10,000. No need to abbreviate when people may be unsure that it's actually 10,000 (given that it's such a large difference). 

I agree with Michael that concrete examples would be very helpful, even for researchers.  A post should be informative and persuasive, and examples almost always help with that. In this case, examples can also make clear the underlying logic, and where the explanation can be confusing. 

For example, let's think about investing in alternative protein companies as a way to tackle animal welfare. Assume that in a future state where lots more people eat real meat (bad world state), the returns for alt-proteins in that state are low but cost-effectiveness is high. This could be because alt proteins have faced lower rates of adoption (low returns) but it's now easier to persuade meat eaters to switch (search costs are now low since more willing-switchers can be efficiently targetted). The opposite situation is true too. In a good future state with few meat-eaters, alt protein returns are high but cost-effectiveness is low. So this scenario should put us in your table's upper left quadrant (negative correlation btw/ World State and Cost-Effectiveness + negative correlation btw/ Return and Cost-Effectiveness).

This example illustrates how some of your quadrant descriptions may be  confusing or even inappropriate:

1. "Underweight investment": I agree with this one since to have a greater EV, you want investments with a positive correlation between returns and cost-effectiveness. This isn't true for alt proteins here, so you should avoid them.
2. "Divest from evil to do good": I don't think this makes sense because alt proteins are not "evil" (but you should avoid them given the scenario).
3. "Mission leveraging": I was quite confused initially because I was assuming that the comparison is to no investment at all. If so, then investing in alt proteins can lead to an ambiguous impact on volatility (depending on the relative magnitude of return changes versus cost-effectiveness changes). It could in fact be mission hedging (with an improvement in the bad state) if the low returns end up producing more total good because of the state's high cost-effectiveness. However, I eventually realized that the comparison is to a fixed grant within the animal welfare space (although this was never made explicit in the post and may not be what most people would assume). If so, then indeed this is always mission leveraging since a positive correlation between the world state and returns does ensure lower volatility.

So as you can see, an example makes clear where table descriptions may be inappropriate and where a clearer description can be helpful. It also makes more concrete what various correlation signs mean and how to think about them.

This post (and the series it summarizes) draws on the scientific literature to assess different ways of considering and classifying animal sentience. It persuasively takes the conversation beyond an all-or-nothing view and is a significant advancement for thinking about wild animal suffering as well farm animal welfare beyond just cows, pigs, and chickens.

Thanks for the clarification, Owen! I had mis-understood 'investment-like' as simply having return compounding characteristics. To truly preserve optionality though, these grants would need to remain flexible (can change cause areas if necessary; so grants to a specific cause area like AI safety wouldn’t necessarily count) and liquid (can be immediately called upon; so Founder's Pledge future pledges wouldn't necessarily count). So yes, your example of grants that result "in more (expected) dollars held in a future year (say a decade from now) by careful thinking people who will be roughly aligned with our values" certainly qualifies, but I suspect that's about it. Still, as long as such grants exist today, I now understand why you say that the optimal giving rate is implausibly (exactly) 0%.