Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

As noted previously, I’ve been working on improving our broad understanding of the role that philanthropy can play in influencing and informing policy. One of the questions I’ve been focused on is “What is the track record of policy-oriented philanthropy?” Specifically:

• Are there compelling cases in which a major policy change can be partially or fully credited to philanthropic efforts?
• How often have philanthropic efforts succeeded in bringing about change? Often enough to imply a good “return on investment?”

I’ve concluded that answering these questions reasonably well could require an enormous, long-term effort. This is true both because the questions are inherently difficult to answer - many of philanthropy’s claimed impacts on politics are highly long-term and diffuse, such that it is difficult to confidently isolate impact - and because there has been little academic work on the subject.

Below, I first list salient examples I’ve seen in which philanthropy is believed (by some) to have had an important impact on public policy. I think the nature of these examples illustrates some of the challenges with isolating the impact of philanthropy. I then discuss our understanding of the current state of the literature on this topic, and what we would do to make more progress. For the purposes of this post only, “success” is defined as causally impacting public policy, not as having positive social impact, since the goal is to determine how and when philanthropy has been effective in changing policy (the question of how policy should be changed is a different question).

Potential cases of philanthropic impact on public policy

Many of the claimed success stories of policy-oriented philanthropy involve long-term (sometimes 10+ years), diffuse impacts. Pages 4-8 of The Elusive Craft of Evaluating Advocacy, by Steven Teles and Mark Schmitt (both of whom we’ve spoken with - see conversations notes for Teles and Schmitt) give a good feel for this kind of story, and cover some of the cases we’ve frequently heard reference to: the role of Atlantic Philanthropies and others in laying the groundwork for the Affordable Care Act, the role of (presumably philanthropy-backed) church and student groups in “claw[ing] back the application of the death penalty,” the repeal of the “Don’t Ask, Don’t Tell” policy, and impact of the conservative Federalist Society, which is claimed to have magnified the impact of other organizations on law. We excerpt the part about health care reform, as an example:

The effort that culminated in 2010 was the work of decades, including a previous, high-profile failure in the early 1990s, multiple waves of state-based reform and numerous incremental efforts at the national level. Advocates invested hundreds of millions of dollars on initiatives ranging from media initiatives such as encouraging television producers to include stories of the uninsured, multiple coalition-building projects, university- and think tank-based research and well-funded grassroots initiatives. The basic outlines of reform policies were worked out well in advance, in advocacy groups and think tanks, who delivered a workable plan to presidential candidates. Key interest groups who could block reform, such as small business, had been part of foundation-supported roundtables seeking common ground for years. Technical problems had been worked out. And tens of millions of dollars had been set aside as long ago as 2007 for politically savvy grassroots advocacy initiatives targeted at key legislators. After a very long slog, the outcome was the Affordable Care Act.

Another very long-term, diffuse impact that has been cited to us is the case of the Robert Wood Johnson Foundation’s efforts to reduce tobacco use.

• Activities included funding research on the impact of policy on tobacco use (page 13), funding early-career researchers in the hopes of developing a stronger set of professionals focused on the issue (page 17), creating the Campaign for Tobacco-Free Kids (page 17), and indirect support for state-level lobbying (page 20).
• The program started in 1991 and had reached ~$20 million per year by 1993 (page 12), but its goals - changing policy to be more discouraging of tobacco use and ultimately reducing smoking rates, particularly among minors - don’t seem to have shown much progress until after 1996 (see pages 36-38 for policy change indicators, page 26 for smoking rate data).
• Given the number of different initiatives RWJF created and invested in, the number of states it worked in, and the long-term nature of some of its investments (building organizations, funding research and researchers), evaluating the extent to which its funding can be “credited” for reductions in tobacco use could be an extremely challenging project.
• The Foundation spent ~$700 million in this area over time (page 12); if one assumes it reduced the number of smokers by 10 million (its evaluation estimates 5.3-14.2 million, page 28), then the total cost of the program was ~$70 per “person stopped from smoking,” which would compare favorably to our estimate of ~$80 per life-year gained for our top charity if smoking causes one to lose multiple years of life. The point of this calculation isn’t to praise the program - we haven’t vetted these figures and realize there are a substantial number of questions here - but to illustrate that the sort of claims made about policy impact imply potential competitiveness with our top charities.

We have also seen claims of shorter-term, more tangible impact of funders and nonprofits on public policy, though usually at smaller scale. Change Philanthropy goes through several such cases, including a movement in which lawsuits were brought against states alleging that they needed to invest more in education to comply with their own constitutions. The Center for Lobbying in the Public Interest also lists smaller-scale success stories. I’ve heard multiple informal claims that online advocacy groups had tangible, extremely rapid impact on halting (a) passage of the Stop Online Piracy Act (b) construction of the Keystone XL pipeline.

Assessing the ROI of policy-oriented philanthropy

While we’ve seen many claims of impact, none appear easy to assess. Moreover, the most prominent and largest-scale claimed impacts generally appear the hardest to evaluate.

We’ve asked multiple people (including Frank Baumgartner, Mark Schmitt and Steven Teles) whether there is a literature that might help us in this endeavor. It appears that we aren’t going to find any work systematically looking across a large number of philanthropic efforts to change policy, such that we could get an idea of aggregate return-on-investment. We may be able to find case studies on specific policy changes.

We don’t see a clear or easy path to assessing the question, “Can policy-oriented philanthropy have a high enough probability of success to make the cost-effectiveness competitive with our top charities?” We expect our History of Philanthropy project to generate better information on particular, high-profile claimed successes (perhaps via reviewing existing case studies, perhaps via generating new ones focused on the role of philanthropy). If it does, we may consider a more systematic project aiming to catalogue and investigate the policy-oriented work of a representative set of foundations. But it could be a long time before we have useful estimates of the “good accomplished per dollar” of policy-oriented philanthropy, and in the meantime, we will be moving forward on exploring good opportunities within policy-oriented philanthropy.

Reason to be optimistic about ROI

One rough, heuristic-based reason that I’m optimistic about policy-oriented philanthropy (in terms of “impact per dollar”) is that it generally seems like a given amount of money “means more” in the context of policy-oriented work than in other contexts.

• According to IHME, total global health aid is in the range of ~$30 billion a year, with ~$1 billion coming from the Gates Foundation alone.
• Within the U.S., the National Institutes of Health spend ~$30 billion a year on scientific research, and Howard Hughes Medical Institute alone spends ~$700 million a year.
• Total U.S. lobbying spending is about ~$3 billion a year. If a philanthropist spent $1 billion a year on policy-oriented work, this would exceed the combined budgets of the top 18 think tanks in the US according to the Center for Global Development.

There are many reasons that this comparison could be unfair or irrelevant, but it syncs with my general impression that the amount of money it takes to be a “significant player” is smaller in policy-oriented work than in direct aid or scientific research.