A single law or social program in a single country can affect the well-being of tens or hundreds of millions of people. Policy change represents a massive opportunity for impact — but may require different tactics and types of organizations than the effective altruism (EA) community usually supports. In this talk, Norma Altshuler of the William and Flora Hewlett Foundation describes promising strategies for policy work in low- and middle-income countries, as well as practical and moral considerations for people who want to pursue this kind of work.

A transcript of Norma’s talk, which we have lightly edited for clarity, is below. You can also watch this talk on YouTube.

The Talk

If you’re like me and many other members of the effective altruism (EA) community, you spend a lot of time wrestling with this question: What are the most cost-effective ways to improve the lives of the global poor? For the past few years, our community has been wrestling with the role of policy influence in this process. We've recognized that governments spend trillions of dollars to improve the lives of the poor, and we have started thinking about what we can do to help them spend it better. 

As Nathan [the moderator] mentioned, I've had the opportunity to work in EA organizations like GiveDirectly . On the funding side, for a number of years, my job was to figure out how to finance deworming. Now, my work is about trying to change how African governments use data for decision-making.

Today, I'll share my takeaways for the EA community. I'm going to make three points:

 

1. Influencing how governments spend money can be very cost-effective. 
2. However, we can only make this kind of cost-effective impact if we're willing to accept a lot of uncertainty around the value our donation or action will add. 
3. One tool that can help organizations be more effective at having an impact is flexible funding, which decreases transaction costs and helps organizations seize time-sensitive policy opportunities. Of course, we want to make data-driven decisions, so I will suggest some data-driven ways to think about flexible funding.

But first, how do we maximize impact? 

 

Three years ago, Michael Kremer — father of the “randomista” movement in development, Harvard professor, and my old boss — spoke at this conference. He proposed using a pretty simple formula: the probability of an outcome times the net value of that outcome. As I'm sure Michael would agree, we have to consider a lot of things when thinking about net value. Two of those things are impact per dollar and the number of dollars.

So, where are the dollars? 

 

In 2017, GiveWell moved almost $150 million. That money saved and improved lives. But aid [in development assistance, as a whole] is roughly the same number — $147 — but in billions of dollars. There’s a lot of money in charity we can try to influence, but where the numbers get really big is in governments. Low- and middle-income country governments spent about $800 billion in 2017 on health alone — not including social protection, education, and lots of other things that don't require governments to spend money, such as taxing people. So, it's easy to see why GiveWell and many others are taking government spending very seriously. But I’m guessing that some of you, as EAs, are looking at that large amount of money and asking, “What can I possibly add to that enormous dollar amount and the considerable expertise that already exists in the development sector?”

 

I would say EAs add a combination of three things. One is appreciating and understanding evidence. Another is analytical decision-making. And the third is flexibility. I cannot overstate how rare that is. Almost all big money from governments, from aid agencies, and from foundations like mine is siloed — and often siloed by political or donor preferences, which are not necessarily about cost-effective opportunities for impact. The biggest recipients of foreign aid from the U.S. are countries in the Middle East. And that's not because that's where the most poor people live. 

Equally important, EAs can be flexible in how they work. As those of you who have applied for grant applications from governments or big donors know, decisions can take months or sometimes years. There is a lot of paperwork. EAs can choose to be flexible.

I’m going to talk a lot about uncertainty. Let me quickly explain what I mean.

 

First, there can be uncertainty around exactly what the impact was — exactly how much of a reduction in poverty there is for people. Second, there can be uncertainty around our contribution. What did the dollar that you donated do, exactly? 

Let's get into some examples about why this can be cost-effective. You probably know about [Evidence for Action’s] Deworm the World Initiative. 

 

This is an example of one path to scaling impact. In 2017, Deworm the World reached 280 million children. They work closely with governments; through them, governments hand out 50-cent deworming pills. The health effects are such that we see, on average, higher earnings for these children as adults. 

I want to call out two points around uncertainty. First, this is an example of how the EA community already accepts a lot of uncertainty around impact. Our friends at GiveWell know that deworming only has a 1% to 2% chance of having the level of impact in campaigns today that it had in the original randomized trials. And it's still one of their top charities. And second, there has been a lot of uncertainty about Deworm the World's policy influence over the years. When I had the privilege of interning there in Kenya, in 2011, not much was happening. The government had put the program on hold for reasons that were beyond Deworm the World’s control, but they did the hard and unglamorous work of maintaining relationships — of continuing to be useful to government partners. As a result, deworming is now a national event in Kenya.

If we're willing to accept uncertainty, another way we can influence governments is by improving them — by taking what they are already doing and making it more effective. Let's talk about J-PAL in India.

 

The government of India spends $70 billion on social protection programs that are designed to improve life for the poor, but only half of those resources actually make it to poor people. One state, Andhra Pradesh, decided to test a payment system to verify your identity before getting the cash that you are supposed to receive as a transfer. You put your thumb on a meter, it recognizes you, and then you get the cash. As a result, 40% less money got lost. And in this one state, after accounting for the cost of the system — which the government paid for — we saw $40 million saved. J-PAL simply paid for the randomized control test (RCT) with support from the Omidyar Network, and RCTs cost a couple hundred thousand to $1 million, depending on a lot of things.

I hope it is easy to see why this kind of investment in evidence can be very cost-effective. But we have to accept some uncertainty about exactly what role research played. I'm pretty sure that J-PAL’s relationships are what made the difference. In fact, the government cited the research when they expanded the program (which was not an easy thing for J-PAL to convince the government to do). But in any particular case, we never know exactly what difference our advocacy made. There are always a lot of factors. We're proud at the Hewlett Foundation to flexibly support J-PAL, because they have a track record with a lot of these kinds of examples, even if each one comes with a little uncertainty.

Let me go one step further and talk about helping governments set policy. Let me tell you about CRES, a think tank in Senegal. 

 

They recognize what many of you probably know: that tobacco kills 7 million people a year — about 80% of them in low- and middle-income countries. They convinced the government of Senegal to tax tobacco, and they helped the government design a lot of the specifics of the tax regime. The important thing to note is that CRES and Hewlett (as a funder) were not sure it would work. At first, we thought maybe the tobacco lobby would be too powerful. And there are a million other reasons why a good idea like this might not have worked. But it did. And not only that, with the flexible funding CRES had from Hewlett, they had an opportunity to influence ECOWAS [the Economic Community of West African States], which is kind of like the EU of West Africa. As a result, 15 countries have agreed to a binding policy directive to tax tobacco, and CRES is helping them design and implement specific tax policies. 

So, policy advocacy is not that expensive. Sometimes it can be done for tens of thousands of dollars. Generally, it can be done for hundreds of thousands of dollars. And I hope that example showed that under some circumstances, it can be cost-effective.

Next, I’m going to address what I think is the next frontier for EAs: influencing how governments themselves use data to make decisions. 

 

During my time at the Hewlett Foundation, I've gotten to meet with a lot of government officials who genuinely want to use data to improve programs — this is not everyone, but there are pockets of both politicians and civil servants who do. In Ghana, I heard the vice president give a speech about data interoperability. He said that his government needed to invest in data interoperability so that they could better deliver results. They need results to get elected, and the government has started to take action. You can read on Vox about the work they're doing on their next census. They're going to great lengths to try to count everyone, no matter how hard it is, so they can better target programs like their cash-transfer program.

Most of the money and activity are coming from the government itself, or from big aid agencies — donors in the U.K., U.S., and others with much bigger budgets than we have in the EA community. But at the Hewlett Foundation, we found some small ways to be helpful. One involves one of our partners, IDinsight. 

The president of Ghana has set up a new Ministry of Monitoring and Evaluation and wants information on how his top-priority programs are doing. This ministry has significant political clout and the ability to make a difference, but not enough technical capacity yet, which is hard for them to get with all of the government regulations about staffing. That's where IDinsight comes in. They're working on the president's free senior high school education program.

The president wants every Ghanaian to go to high school. They’ve started with little decisions to make that happen. They found, for example, that there are schools with too few teachers right next to schools with more than the required ratio [of teachers to students]. So they reallocated teachers. Those kinds of small, evidence-based wins position IDinsight for bigger, more influential steps. The government is now taking the idea of remedial education — which is one of the most cost-effective educational interventions — very seriously. We're hoping that this positions the government, with IDinsight’s help, to make many more evidence-based decisions over time. Maybe the government will even have more capacity to make its own evidence-based decisions someday, without IDinsight’s help. 

Of course, there are many sources of uncertainty associated with this. As I said, this is truly the next frontier. We're still learning to what extent this kind of work can have a cost-effective impact, but I'm hoping some of you want to go along on this journey. If you take nothing else away from this talk, I hope it's this: that if we accept uncertainty, we have more opportunities for impact.

 

Note that I said “more opportunities” — not “better opportunities.” What you end up doing depends, in part, on your values. How much uncertainty are you willing to tolerate? What are your time horizons? Are you a strict utilitarian, or do you also put value in democratic governments delivering on the priorities of their citizens? Regardless, I'm pretty sure we have cost-effective opportunities, in a strict utilitarian sense, across the spectrum. We need to keep funding what we know works, like cash transfers, and we need to scale that. We need to continue to do deworming. And I hope I've gotten some of you excited about setting and improving policies. We have to accept a lot more uncertainty, but we can also leverage a lot more dollars. I hope some of my examples have convinced you there's potential in policy work.

I want to leave you with a tool that I think will apply regardless of where you end up on the wide spectrum of impact opportunities. 

 

Flexible funding decreases transaction costs and can help NGOs respond to time-sensitive policy opportunities. I know that’s going to be controversial, so let me say it again: Flexible funding decreases transaction costs and can help you respond to time-sensitive policy opportunities. Let's remember Michael Kremer's formula: the probability of impact times the net value of impact. In development, and frankly in life, we take a lot of actions, and only some of them matter. That's what Michael's formula is about. 

Let's start with a non-controversial example. Let's go back to deworming pills. We deworm 280 million kids, and as a result, some of them get much healthier — so much healthier that they make more money later in life. But for a lot of them, there's no impact. They're just the same. And that's okay. The expected value is really high. In fact, it's actually more cost-effective to just deworm kids, rather than test individual kids to see if they have worms and then treat them. Deworming pills are cheap and almost always harmless. And when they have an impact, the impact is big. 

Evidence Action talks to a lot of governments about deworming evidence. Some of those conversations go somewhere and have enormous impact. Sometimes they talk to governments for days, weeks, or years — and those partnerships don't go anywhere. And that's okay, because if we think back to Kremer’s formula, the expected value is high. Now, if Evidence Action had to write a grant application every time they had one of these opportunities, their lives would be a lot harder and they would miss a lot of opportunities. When I was there, I saw times when the government needed help in two weeks. [Funders] can't write a check that quickly. And every hour organizations spend writing a grant application for donors is an hour they're not out achieving real impact in the world.

J-PAL runs the Innovation in Government Initiative, which I think will produce a lot more successes. GiveWell recently gave an incubation grant to them. If I understand correctly, they did so by essentially applying Kremer’s formula. They looked back on the past successes that J-PAL had produced and they concluded that J-PAL was big enough to justify their support. So, J-PAL now has flexibility. But before GiveWell, there was the Hewlett Foundation. My predecessor [at Hewlett] looked at J-PAL — which didn't have quite the track record they have today — but decided that they had enough potential for cost-effective impact to receive unrestricted support.

I want to close with a confession. Six years ago, GiveWell started asking for unrestricted donations. I sent them an email saying, “What, exactly, is my dollar going to do and what, exactly, is your funding gap?” And honestly, they just sent me a quick email back, so I donated to something that had a lot more certainty. Fortunately, other people made different choices and GiveWell went on to have enormous impact. But I see organizations all the time that aren't so lucky. Our evidence-oriented partners have to turn down policy opportunities that are time-sensitive. Or, without flexible funding, they just don't have the space to dream big.

I know that we all are committed to data-driven decision-making. And if I had time I'd tell you a lot more about the analytical process for deciding on flexible support. But I'll just leave you with a simple checklist:

 

* Is there — as in the case of J-PAL — a track record of, or potential for, cost-effective impact? 
* Is there organizational capacity (financial, managerial, operational)?
* Are there clear goals and plans for the organization to assess results? 

The idea might sound radical, but our friends at Good Ventures have made unrestricted grants to a number of organizations by essentially applying the same logic. As we all wrestle with the uncertainty — and opportunity — that comes with policy influence, I hope you won't make the mistake that I made when I decided not to fund GiveWell. I hope you will recognize, as I have, that one way to empower, make change, and advance stewardship is to find evidence-driven organizations, give them what they need to succeed, and leave them to get the work done. Then, we can learn from what they do. Thank you.

Q&A

Nathan Labenz [Moderator]: We have a few minutes for questions. Let’s start with a question that doesn't get asked very often, because people assume that the premise is false, but it seems as if you're suggesting that it's true: Why is there so little money in politics? You're essentially saying that people should spend a lot more resources trying to influence how governments make decisions. And that implies that there's not enough money in politics.

Norma: Yeah, that's a great question. As you saw from those numbers, there's a lot of money with governments and there are lots of NGOs and donors trying to influence how governments act. I guess I would go back to my slide about what EAs add, and I would just note that with this money, it's pretty rare to focus on evidence and be analytical in your decision-making. And it is super-rare to have flexibility.

Nathan: It's still very counterintuitive to me that this would be an underexploited opportunity. It flies in the face of what I hear all the time, which is that everybody's lobbying every government to get what they want. And you're saying that's not the case — that it's a green field.

Norma: It's a great point. A lot of people are lobbying governments to get what they want. A lot of them are people like the tobacco industry. People who are looking at evidence and lobbying governments based on facts and analytical decision-making are, in my opinion, quite rare.

Nathan: And you think uncertainty is what has made it rare?

Norma: My hypothesis is that uncertainty is what has kept it rare in this particular community. I think there are a lot of factors that affect others. One is that there aren’t that many people in the world who find data, analytics, and evidence-based decision-making sexy. Another is that the line to results is less direct than it is to some other things. Another is that there are not a lot of people who appreciate and understand evidence. And finally, I think we have work to do to help governments appreciate evidence. There are places where they do — in Ghana and other places. And governments increasingly have sway over what donors provide to them.

Nathan: Can you share what evidence looks like in a context where you can't eliminate all of the uncertainty, but you're trying to figure out whether a government interaction helped or not? 

Norma: We commissioned a report years ago to look at this. Essentially, the metaphor they came up with is that it's like a foreign intelligence analyst trying to put a lot of information together. But it was a pretty qualitative approach. 

Now, we can quantify that. For example, with J-PAL in India, as I mentioned, the government actually cited J-PAL’s study. That is a good sign, although not decisive. J-PAL and partners like us can go and talk to people in and around government who know what happened. 

Usually, when I have those kinds of conversations, people point out that there were a lot of factors. There's never just one thing that influences policy. But you can pretty quickly get a sense of who is influential and who isn't. And often, you can unpack the story behind specific interventions and assign a numerical probability to it.

Nathan: What concrete steps would you recommend that people take in order to become involved in policy opportunities in developing countries?

Norma: I'm wondering if this question is about career paths or about giving opportunities. I'll start with giving opportunities. One of the things I hope you take away is that the organizations I mentioned can be resources for you. Starting with things that you know can be helpful — for example, trying to influence malaria spending, including helping governments use data about malaria for decision-making. This community already knows a decent amount about malaria. Unpacking that can be useful. I'm not here to pitch specific charities, but I will tell you that all the ones I mentioned are underfunded, and I think that the EA community can learn a lot from them in the course of donating.

Nathan: Let's consider the career side of the question, too, because I do think there's a lot of people in the audience who are thinking about career decisions.

Norma: I would say that being really humble about what can happen from here in the U.S., and what can happen in low- and middle-income countries, is important. One thing I like to think about is my marginal impact on an organization that I work in. There are some talented people who want to work in government and there are others who don't. But thinking about what you add, compared to the person who would have been hired in your place, is also really instructive. 

Nathan: Could you give some advice on how to identify opportunities to “set [policy] or improve it” that are likely to succeed? In other words, what should people be looking for as indicators that they can make a change? 

Norma: My strategy is to find people who are closer to the problems than I am and empower them to try to find those opportunities. And even if you're not prepared to see things through a fully unrestricted support lens, you can focus on things like innovation funding, which gives people time to identify opportunities. 

I think you really need to know the local context. You also need to find organizations that are embedded and know the local context themselves. So, in the tobacco example I gave, that organization succeeded because they were in Senegal, knew and had relationships with lots of people there — including civil society organizations — and knew and understood the local politics. How the organization is positioned is more important than what the specific policy is.