There is a relatively small comparison class here; we often say we’re focused on “impact-focused” evaluators. Here is our database of evaluators we know of that we might consider in this reference class. In the medium to long run, however, we could imagine there being value in investigating an evaluator completely outside EA, with a very different approach. There could be some valuable lessons of best-practice, and other insights, that could make this worthwhile. I expect we probably won’t prioritise this until we have looked into more impact-focused evaluators. 

Thanks Lauren for your question, and thanks Vasco for helping to answer it! I've replied to the comment under the post on our evaluations that I believe you're referring to, and am happy to elaborate on any part of my answer there (and what's in the report / what Vasco shared) if helpful.

Hi Lauren,

You and other interested readers may want to check this section of GWWC's evaluation of ACE. Here is the part of the section I see as most relevant to your comment:

ACE helpfully — and on very short notice — provided us with private documentation to elaborate on the cases for three of its 2023 charity recommendations. Unfortunately — potentially in part because of time constraints ACE had  — we still didn’t find these cases to provide enough evidence on the marginal cost-effectiveness of the charities to justify relying on them for our recommendations. However, for one of the charities recommended by ACE — The Humane League — we reasoned that a strong enough case would be made by combining ACE’s recommendation with further evidence in favour of the marginal cost-effectiveness of THL’s work on corporate campaigns for chicken welfare:

• A 2018 evaluation by Founders Pledge of THL’s corporate campaigns’ cost-effectiveness at that point in time.
• A 2019 review by Rethink Priorities of the cost-effectiveness of corporate campaigns for chicken welfare more generally.
• A direct referral from Open Philanthropy’s Farm Animal Welfare team — the largest funder in the impact-focused animal welfare space — on THL indeed currently being funding-constrained, i.e. that it has ample room to cost-effectively use marginal funds on corporate campaigns and that there aren’t strong diminishing returns to providing THL with extra funding.

To be clear, there are strong limitations to this recommendation: 

• We didn’t ourselves evaluate THL’s work directly, nor did we compare it to other charities (e.g., ACE’s other recommendations).
• The availability of evidence here may be high relative to other interventions in animal welfare, but is still low compared to interventions we recommend in global health and wellbeing.
• We haven’t directly evaluated Open Philanthropy, Rethink Priorities, or Founders Pledge as evaluators. 
• We have questions about the external validity of the evidence for corporate campaigns, i.e. whether they are as cost-effective when applied in new contexts (e.g. low- and middle-income countries in Africa) as they seem to have been where the initial evidence was collected (mainly in the US and Europe).
• We also have questions about the extent to which the evidence for corporate campaigns is out of date, as the Founders Pledge and Rethink Priorities reports are from more than four years ago and we would expect there to be diminishing returns to corporate campaigns over time, as the “low-hanging fruits” in terms of cost-effectiveness are picked first.

Taken together, all of this means we expect funding THL’s current global corporate campaigns to be (much) less cost-effective than the corporate campaigns in 2016-2017, which were evaluated in those reports. However, we still think funding them is likely highly cost-effective, and the most justifiable charity recommendation we can currently make, based on the available evidence and our limited time. We also think that recommending at least one competitive alternative to the AWF in the animal welfare space — if we transparently and justifiably can — is valuable. We hence decided to make this exception to evaluating only evaluators to make an individual charity recommendation, motivated by our overarching principles of usefulness, justifiability, and transparency.

This is a really insightful question! 

I think it’s fair to characterise our evaluations as looking for the “best” charity recommendations, rather than the best charity evaluators, or recommendations that reach a particular standard but that are not the best. Though we’re looking to recommend the best charities, we don’t think this means that there’s no value in looking into “great-charity evaluators” as you called them. We don’t have an all-or-nothing approach when looking into an evaluators’ work and recommendation and can choose to only include the recommendations from that evaluator that meet our potentially higher standard. This means, so long as it’s possible some of the recommendations of a “great-charity evaluator” are the best by a particular worldview, we’d see value in looking into them.

In one sense, this increases the bar for our evaluations, but in another it also means an evaluator’s recommendations might be the best even if we weren’t particularly impressed by the quality of the work. For example, suppose there was a cause area for which there was only one evaluator, the threshold for this evaluating being the best may well be: they are doing a sufficiently good job that there is a sufficiently plausible worldview by which donating via their recommendations is still their best option (i.e., compared to donating to the best evaluator in another area).

It’s too early to commit to how we will approach future evaluations, however, we currently lean towards sticking with the core idea of focusing on helping donors “maximise” expected cost-effectiveness, rather than “maximising” the number of donors giving cost-effectively / providing a variety of “great-but-not-best” options. 

You might also explicitly state that you don't intend to evaluate great-charity recommenders at least at this time.

As above, we would see value in looking at charity evaluators who take an approach of recommending everything above a minimum standard, but we would only look to follow the recommendations we thought were the best (...by some sufficiently plausible worldview). 

but would recommend making it clear in posts and webpages that you are evaluating best-charity evaluators under standards appropriate for best-charity evaluators

I’d be interested in where you think we could improve our communications here. Part of the challenge we’ve faced is that we want to be careful not to overstate our work. For example, “we only provide recommendations from the best evaluators we know of and have looked into”, is accurate, but “we only provide recommendations from the best evaluators” is not (because there are evaluators we haven’t looked into yet). Another challenge is to not overly qualify everything we say, to the point of being confusing and inaccessible to regular donors. Still, after scrolling through some of our content, I think we could find a way to thread this needle better as it is an important distinction to emphasise — we also don’t want to understate our work!

Hi wes R, I'll answer your questions in this comment!

The impact measurements greatly varied by evaluator. For example, GW makes decisions using its “moral weights” (which primarily measures consumption and health outcomes, but I don’t believe in a way that neatly reduces to QALYs). Meanwhile, HLI uses “WELLBYs”. Other evaluators used different measurements at different times, or relied on subjective scores of cost-effectiveness. You can read more about these in our evaluations (linked to here). 

I’m not sure we have much in the way of a generalised view of which metrics we think should be used or not. In general:

• These metrics should help support making more cost-effective recommendations and grants.
• To the extent they do, we’re happy to see them!
• In some cases, metrics might end up forcing over-precision in a way that is not particularly helpful. In these cases, we think it could be more sensible to take a more subjective approach.

Hope that helps!

Thanks for your question! We explain the general principles we used to choose which evaluator to investigate here, and go into our specific considerations for each evaluator in their evaluation reports.

For FP's GCR Fund compared to LTFF and LLF specifically, some of the main considerations were (1) our donors had so far been donating most to the LTFF, so the stakes were higher there, and (2) Longview was one of the most-named options by other effective giving organisations as an evaluator they weren't relying on yet but were interested in learning more about.

And yes there are other evaluators we've considered and are considering for future evaluations, some of which we mention throughout the reports. See here for an overview of the impact-focused evaluators making publicly available recommendations that we are currently aware of, and which we may consider in our next iterations of this project.

We explain more how we view funds vs charities more generally here.

And for the GWWC cause area funds we answer your question for each individual fund on their page, e.g. here for the Global Health and Wellbeing Fund, under "How does donating to this fund compare to similar giving opportunities?".