Effective Altruism Forum
EA Forum

by Luke Freeman 🔸
1 min read 6

48

Effective givingCause prioritizationImpact assessment
Frontpage

According to the announcement on their blog (heard through Catherine Low).

They seem to be acknowledging the importance of cost-effectiveness now:

Why Cost-effectiveness?
Take a simple thought exercise: A program has a limited budget of $100,000 to improve literacy in a community. It can choose between two approaches to do so: one that can boost literacy by a grade level for 100 students and a second that can also boost literacy by a grade level but for 200 students. All else equal, a sensible program administrator would choose the second, as of course it reaches twice as many students. This is a cost-effectiveness decision. We have limited resources and unlimited needs. Cost-effectiveness is a decision tool that makes those resources go further - helping more people in more ways.

However, their criteria still includes:

"Direct" criterion: At least two-thirds of the nonprofit's activities (as measured by percent of total program service expenses) are directly delivered to beneficiaries and reasonable to expect impact measurement for. Many nonprofits work one or more steps removed from beneficiaries, such as by conducting research, advocating for policy change or making grants to other organizations. We do not yet have a method for consistently estimating the impact of these nonprofits, and so have excluded them from the Impact & Results beacon at this time.

Will be interesting to see what the outcomes of this are. It first guess, I imagine it'll be mixed.

48

0
0

Reactions

0
0
Mentioned in
Comments6


Sorted by
New & upvoted
Click to highlight new comments since:
EricHerboso49
0
0

In July, Charity Navigator announced their new nonprofit rating system that they call Encompass. This system looks at four “beacons” to determine their rating of each charity. One of these beacons is Impact & Results. At the time, they did not specify how they would evaluate this beacon. Yesterday's latest post from them finally sets down the initial methodology they will use.

Some basic takeaways:

  • At the same time as releasing their new rating system, they intend to increase the number of charities they rate from 9,000 to 160,000. Clearly, most of the ratings must be automated to do this, so only a vanishingly small proportion of charities will be evaluated on the basis of Impact & Results. It's not clear how they will prioritize which charities get rated in this beacon in the future, but they're starting with this list of cause areas and they have a sign-up form for charities that want to be evaluated on Impact & Results.
  • They will not be comparing causes nor sometimes even intervention types. Their system will only look at cost-effectiveness within a cause area in some cases, and only within a single intervention type for other cases. For example, they may rate the most highly cost-effective charity that provides emergency shelters for the homeless population. There will be no indication whatsoever that a cataract surgery charity scoring 100 points on Impact & Results might be more effective than a Veterans Disability Benefits charity scoring 100 points on Impact & Results.
  • Within each cause area, they give four possible scores for Impact & Results: 0 points for charities without publicly available data, 50 points if they provide data but are determined to be inefficient, 75 points if they are found to be effective, and 100 points if they are found to be highly effective. Assuming they successfully find the most highly effective charities, this would give the likely incorrect appearance that the most highly effective charities are only 1/3 better than charities who just barely do better than breaking even. It's also not clear what percentage of charities within a cause area may be simultaneously rated at 100 points in the Impact & Results beacon.
  • Within each cause area, they use vastly simplified calculations to determine impact. For example, when it comes to emergency shelters, they assume that all beds are equally good; they disregard counterfactual beds that would be available if the charity not existed there; they give full marks for providing a bed, even if other beds were available at the time; and when determining costs, if the charity doesn't specify what it costs for them to provide the cost of a bed, they instead just use the average cost as reported by HUD’s Housing Inventory Count dataset. While I believe these are humongous assumptions to be making, I don't necessarily think these simplifications are bad considering their goal; if they're serious about analyzing hundreds of thousands of charities, then they have to make simplifications somewhere. [EDIT 16 Oct: Elijah Goldberg of ImpactMatters clarifies in a comment below that this bullet point may be misleading.]

They have noted that they are looking into additional alternative methodologies for the future.

The system that Charity Navigator is using for its Impact & Results beacon was acquired from ImpactMatters, which was previously discussed on the EA Forum.

Reply
elijahgoldberg25
0
0

Hi Eric, thanks for your note! Happy to provide some more context on a few things:

  • You're right, the 160,000 include an analysis of finance & accountability that is automated off of 990s. The Impact & Results is not automated. Honestly, the key barrier to "scale" here is smart labor (a team of 3 has been working on this). Certainly in typical EA terms, many of the nonprofits that are analyzed are not the most cost-effective. But we also know that standard EA nonprofits are a fraction of the $300 bil nonprofit sector, and there is a portion of that money that has high intra-cause elasticity but low inter-cause elasticity. Impact analysis could be a way of shifting that money, yielding very cost-effective returns (again, ImpactMatters spent half a million or so last year to rate $15 billion in nonprofit spending. How much did we actually move? Probably not a lot. But hopefully this acquisition changes that, and we'll be running experiments over the next year to figure that out).
    • If anyone is looking to pitch in on the cost-effectiveness analysis, we're looking to build a small volunteer team - more
  • True! But the beauty is (as we see it), that now there is actually a largeish raw dataset that donors can use to apply own weighting & build benefit cost analyses. The barrier to benefit/cost has never been the b/c methodology ... but the raw CEA estimates to feed in
  • I'm not sure what "incorrect" means in this context. Fwiw, we are working on moving to a continuous scale that may address some of your critiques. But I don't think that anyone believes that cardinal rankings are actually that much use in the space.
  • Got to disagree! Sorry, but this is incorrect - maybe not the claim about oversimplification, but your summary of our methodology certainly is. Some nonprofits run an emergency shelter as well as other shelter or housing programs, such as transitional housing or permanent supportive housing. Wherever possible, we exclude from our calculation the value of these non-emergency shelter services as well as the costs of providing them. If the nonprofit has not separated out programmatic costs in this way, we apply a standard cost adjustment. The cost adjustment is calculated using HUD’s Housing Inventory Count dataset. The Housing Inventory Count dataset reports the number of individuals sheltered by each nonprofit on a single night in January, broken out by six types of shelter and housing programs, including emergency shelter. This allows us to calculate the number of individuals sheltered as part of a nonprofit’s emergency shelter program as a percentage of the total individuals it sheltered across all programs. We then multiply the proportion by total programmatic costs, yielding an estimate of costs associated only with the nonprofit’s emergency shelter program. See Reference Manual on Data Analysis for more details on this calculation.
Reply
Nathan Young8
0
0

Thanks for engaging here Elijah and thanks for your hard work. It means a lot to me and I am sure many others here. 

Reply
Aaron Gertler 🔸7
0
0

This was an excellent comment and saved me a lot of time I'd otherwise have spent reading the methodology in full. Thank you for posting it!

Reply
david_reinstein1
0
0

I have some comments following up on this in this shortform here. (By the way, I wrote that before seeing your post)

So far the outcomes don't seem great to me, but I think there is still room for things to improve. I hope to keep at this.

Reply
Nathan Young1
0
0

Thanks for writing this piece.

And good on Charity Navigator for the change. I hope it works out well for them and that more effective charities get more donations as a result.

Reply
More from Luke Freeman 🔸
View more
Curated and popular this week
Paul Present
 ·  · 28m read
 · 
Note: I am not a malaria expert. This is my best-faith attempt at answering a question that was bothering me, but this field is a large and complex field, and I’ve almost certainly misunderstood something somewhere along the way. Summary While the world made incredible progress in reducing malaria cases from 2000 to 2015, the past 10 years have seen malaria cases stop declining and start rising. I investigated potential reasons behind this increase through reading the existing literature and looking at publicly available data, and I identified three key factors explaining the rise: 1. Population Growth: Africa's population has increased by approximately 75% since 2000. This alone explains most of the increase in absolute case numbers, while cases per capita have remained relatively flat since 2015. 2. Stagnant Funding: After rapid growth starting in 2000, funding for malaria prevention plateaued around 2010. 3. Insecticide Resistance: Mosquitoes have become increasingly resistant to the insecticides used in bednets over the past 20 years. This has made older models of bednets less effective, although they still have some effect. Newer models of bednets developed in response to insecticide resistance are more effective but still not widely deployed.  I very crudely estimate that without any of these factors, there would be 55% fewer malaria cases in the world than what we see today. I think all three of these factors are roughly equally important in explaining the difference.  Alternative explanations like removal of PFAS, climate change, or invasive mosquito species don't appear to be major contributors.  Overall this investigation made me more convinced that bednets are an effective global health intervention.  Introduction In 2015, malaria rates were down, and EAs were celebrating. Giving What We Can posted this incredible gif showing the decrease in malaria cases across Africa since 2000: Giving What We Can said that > The reduction in malaria has be
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
LewisBollard
 ·  · 8m read
 · 
> How the dismal science can help us end the dismal treatment of farm animals By Martin Gould ---------------------------------------- Note: This post was crossposted from the Open Philanthropy Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post. ---------------------------------------- This year we’ll be sharing a few notes from my colleagues on their areas of expertise. The first is from Martin. I’ll be back next month. - Lewis In 2024, Denmark announced plans to introduce the world’s first carbon tax on cow, sheep, and pig farming. Climate advocates celebrated, but animal advocates should be much more cautious. When Denmark’s Aarhus municipality tested a similar tax in 2022, beef purchases dropped by 40% while demand for chicken and pork increased. Beef is the most emissions-intensive meat, so carbon taxes hit it hardest — and Denmark’s policies don’t even cover chicken or fish. When the price of beef rises, consumers mostly shift to other meats like chicken. And replacing beef with chicken means more animals suffer in worse conditions — about 190 chickens are needed to match the meat from one cow, and chickens are raised in much worse conditions. It may be possible to design carbon taxes which avoid this outcome; a recent paper argues that a broad carbon tax would reduce all meat production (although it omits impacts on egg or dairy production). But with cows ten times more emissions-intensive than chicken per kilogram of meat, other governments may follow Denmark’s lead — focusing taxes on the highest emitters while ignoring the welfare implications. Beef is easily the most emissions-intensive meat, but also requires the fewest animals for a given amount. The graph shows climate emissions per tonne of meat on the right-hand side, and the number of animals needed to produce a kilogram of meat on the left. The fish “lives lost” number varies significantly by
Relevant opportunities
View more