Simple Tips for Sound Cost-Effectiveness Analysis

In August of 2022, the GiveWell organization announced its “Change Our Mind” contest, which sought essays critiquing the nonprofit’s cost-effectiveness analysis (CEA). These analyses are used to guide organizational decisions about the allocation of philanthropic resources. In response to this call, this essay offers some recommendations to GiveWell about best practices in CEA, along with some warnings about common CEA pitfalls. 

I should start by noting that GiveWell is an impressive organization. I endorse their efforts to make philanthropy more evidence-based. Moreover, CEA is a useful tool for achieving that objective. With that said, there are also some systematic problems that tend to arise in cost-effectiveness analysis. These problems largely fall in two categories:

1. Confusing Discounting Practices – Outcomes in CEAs are often weighted based on moral criteria depending on their timing. Sometimes additional moral weighting/discounting occurs within time periods. These practices conceal the true impacts of interventions, making analysis confusing and not transparent.
2. Opportunity Cost Neglect – Ongoing impacts associated with created or displaced investments are often not accounted for properly in CEAs, resulting in costs incorrectly tabulated.

Both of these issues may be relevant to GiveWell CEAs to varying extents, so next we explore them in more detail.  

Confusing Discounting Practices

The discount rate in CEA does not reflect the rate of return to capital (as is sometimes incorrectly assumed), but rather is an ethical parameter about the relative value of present versus future benefits. GiveWell CEAs, because they include a social discount rate, incorporate some moral weighting. GiveWell is straightforward about this. However, GiveWell also takes care to note that it does “not make charity recommendations solely on the basis of cost-effectiveness calculations.” Thus, the organization considers factors besides cost-effectiveness in its decisions about which interventions to fund—as well it should—however these factors are considered at a separate stage in decision making.

This raises questions about why some ethical criteria are included in CEAs, while other factors are left to be considered outside the CEA framework. By including “moral weighting,” including social discounting, in CEAs, analysts arguably conceal the true impacts of interventions, thereby making analysis less transparent. These practices may undermine the legitimacy of the decision-making process by arbitrarily excluding some important ethical criteria from the economic analysis, while including other ethical concerns. 

A further issue is that a social discount rate converts outcome measures like lives saved or illnesses prevented into a measure of “wellbeing.” Typically, wellbeing in an economic context is thought of as an ordinal measure, i.e., a ranking. However, cost per unit of benefit, which is what cost-effectiveness is, is not a particularly meaningful concept when the outcome measure is ordinal. 

To make this issue more clear, consider an intervention that costs $50 per unit of utility saved. Moving from one unit to two units on the utility scale is not necessarily equivalent to moving from 50 units to 51 units, since distances between utility units are not necessarily equivalent. The entire concept of cost-effectiveness becomes problematic when dealing with ordinal outcome measures. One could argue that CEAs are evaluating cardinal wellbeing changes instead, as this avoids problems associated with ordinality. However, this raises a whole host of other concerns, such as whether the cardinal wellbeing measure is even justified.

Recommendation 1: It would be best if GiveWell analysts simply avoided including ethical criteria in CEAs, leaving economic analysis to focus exclusively on tradeoffs, while allowing ethical concerns to be incorporated into decision making at a separate stage in the funding process.

Opportunity Cost Neglect

GiveWell CEAs make some effort to account for counterfactuals.  However, it is very common in the CEA literature for the opportunity cost of investments to be dealt with incorrectly or to be ignored altogether. For example, one common mistake is to confuse the investment rate of return with the social discount rate (as discussed above). 

Let’s say for example that an intervention results in expenditures of $10,000 made by a donating organization like GiveWell. The cost of the intervention is not $10,000, because in absence of the intervention, some fraction of the $10,000 would likely be invested and grow at some rate of return over time. (Or, perhaps in GiveWell’s case, it would be donated to some other cause, which can be thought of as an investment.) Imagine, for example, a simple case where half of the expenditure would have been invested at a 7 percent annual return, while the other half would have been consumed. 

In that case, the total cost of the intervention is $5,000 in consumption plus $5,000 x SP, where SP is the shadow price of investment. Then  , where t is number of years the investment is held before being cashed in and its returns consumed. 

The opportunity cost of investment must also be accounted for on the “effectiveness” side of the CEA ledger. Let’s say, for example, a health intervention saves the life of an infant. Presumably the infant will then grow up and produce something of value during its adult lifetime. Like with the opportunity cost of monetary expenditures, this benefit can be subdivided into consumption and investment components, and a shadow price should be applied to investment flows to account for investment growth over time. 

Recommendation 2: GiveWell analysts should account for the opportunity costs of investment by dividing monetary flows into their corresponding consumption and investment components, and then applying a shadow price to the investment flows.

After subdividing the financial costs and benefits in this way, and applying a shadow price to the corresponding investment streams, positive savings should ideally be subtracted from negative costs so as to identity the total “net” cost of the intervention. There may still be value in identifying the “gross” cost, since this may be the expenditure amount that is most relevant to a donor. However, the net cost is more relevant from a society-wide perspective. 

Notably, some interventions may even have negative net costs, meaning they save more resources than they cost society. Such interventions are said to be “cost saving” or to have negative cost-effectiveness. These interventions in essence “pay for themselves” and constitute a kind of “philanthropic free lunch.” 

Recommendation #3: GiveWell analysts should prioritize identifying cost-saving interventions, as these will tend to be the most cost-effective interventions .

Conclusion

Value judgments related to ethical criteria should be avoided wherever possible so that CEAs offer a clear representation of the actual economic tradeoffs associated with interventions. Interventions that are not cost-effective may be worth funding for other reasons, such as ethical ones. However, these ethical reasons should be made transparent, not hidden away as technical parameters in CEAs. To the extent possible, GiveWell should remove all ethical parameters from CEAs. 

GiveWell analysts should also be careful to account for the opportunity costs of investment correctly. This generally involves subdividing financial flows into consumption and investment components and then applying a shadow price to the investment streams. Examples of how to do this can be found in a recent article co-authored by this author and Andrew Baxter, as well as in a forthcoming book chapter (see references). 

Because interventions tend to both induce investments (e.g. by saving lives) and offset investments (by displacing investment activity that would happen otherwise), the social costs of interventions will in some cases be negative.  In general, these are the most cost-effective interventions and should be prioritized by philanthropic funders.

I commend GiveWell for its openness to challenge and its search for new and improved ways of conducting CEA. I hope these recommendations proves useful as it proceeds with its mission of  advancing evidence-based philanthropy.

Resources

James Broughel and Andrew Baxter. 2022. “A Mortality Risk Analysis for OSHA’s COVID-19 Emergency Regulations.” Journal of Risk and Financial Management 15(10): 481.  https://doi.org/10.3390/jrfm15100481

Broughel, James and Dustin Chambers. (forthcoming, 2023). “How Risk Analysis Can Improve Wisconsin Regulations and Save Lives.” In Onward Wisconsin: Unleashing Capitalism with Common Sense Public Policy (eds. Russell Sobel and Adam Hoffer).

James Broughel. “The Social Discount Rate: A Primer for Policymakers.” Mercatus Policy Brief (Arlington, VA: Mercatus Center at George Mason University, June 30, 2020).