In July, Charity Navigator announced their new nonprofit rating system that they call Encompass. This system looks at four “beacons” to determine their rating of each charity. One of these beacons is Impact & Results. At the time, they did not specify how they would evaluate this beacon. Yesterday's latest post from them finally sets down the initial methodology they will use.

Some basic takeaways:

• At the same time as releasing their new rating system, they intend to increase the number of charities they rate from 9,000 to 160,000. Clearly, most of the ratings must be automated to do this, so only a vanishingly small proportion of charities will be evaluated on the basis of Impact & Results. It's not clear how they will prioritize which charities get rated in this beacon in the future, but they're starting with this list of cause areas and they have a sign-up form for charities that want to be evaluated on Impact & Results.
• They will not be comparing causes nor sometimes even intervention types. Their system will only look at cost-effectiveness within a cause area in some cases, and only within a single intervention type for other cases. For example, they may rate the most highly cost-effective charity that provides emergency shelters for the homeless population. There will be no indication whatsoever that a cataract surgery charity scoring 100 points on Impact & Results might be more effective than a Veterans Disability Benefits charity scoring 100 points on Impact & Results.
• Within each cause area, they give four possible scores for Impact & Results: 0 points for charities without publicly available data, 50 points if they provide data but are determined to be inefficient, 75 points if they are found to be effective, and 100 points if they are found to be highly effective. Assuming they successfully find the most highly effective charities, this would give the likely incorrect appearance that the most highly effective charities are only 1/3 better than charities who just barely do better than breaking even. It's also not clear what percentage of charities within a cause area may be simultaneously rated at 100 points in the Impact & Results beacon.
• Within each cause area, they use vastly simplified calculations to determine impact. For example, when it comes to emergency shelters, they assume that all beds are equally good; they disregard counterfactual beds that would be available if the charity not existed there; they give full marks for providing a bed, even if other beds were available at the time; and when determining costs, if the charity doesn't specify what it costs for them to provide the cost of a bed, they instead just use the average cost as reported by HUD’s Housing Inventory Count dataset. While I believe these are humongous assumptions to be making, I don't necessarily think these simplifications are bad considering their goal; if they're serious about analyzing hundreds of thousands of charities, then they have to make simplifications somewhere. [EDIT 16 Oct: Elijah Goldberg of ImpactMatters clarifies in a comment below that this bullet point may be misleading.]

They have noted that they are looking into additional alternative methodologies for the future.

The system that Charity Navigator is using for its Impact & Results beacon was acquired from ImpactMatters, which was previously discussed on the EA Forum.

I have some comments following up on this in this shortform here. (By the way, I wrote that before seeing your post)

So far the outcomes don't seem great to me, but I think there is still room for things to improve. I hope to keep at this.

Thanks for writing this piece.

And good on Charity Navigator for the change. I hope it works out well for them and that more effective charities get more donations as a result.