I'm reposting this from the CGDev site, as I thought it might be interesting to EA folks (thanks to Ryan Briggs for the suggestion). For the short version, here's a twitter thread.
--
In the 2000s, cost-effectiveness analysis said it was a bad use of money to send antiretroviral drugs to low-income countries—drugs that ended up saving millions of lives.
Twenty years ago, in the same State of the Union speech in which he made the case for invading Iraq, George W. Bush asked Congress for $15 billion over five years for an ambitious new plan to pay for antiretroviral drugs for two million AIDS patients in Africa and the Caribbean.
The President’s Emergency Plan for AIDS Relief, or PEPFAR, went on to become probably the most celebrated American foreign aid program since the Marshall Plan. An evaluation by the National Academy of Sciences estimates PEPFAR has saved millions of lives (PEPFAR itself claims 25 million). Impacts on total mortality rates across fourteen African countries were visible within just the first few years of the program (see figure 1). Separate research suggests the rollout of antiretrovirals, of which PEPFAR was a major component, explained about a third of Africa's economic growth resurgence in the 2000s.
Figure 1. Adult mortality in PEPFAR focus and non-focus countries (from Bendavid et al 2012, JAMA)
But at the time, some economists balked. The conventional wisdom within health economics was that sending AIDS drugs to Africa was a waste of money. The dominant conceptual apparatus economists use to evaluate social policies—comparative cost-effectiveness analysis, which focuses on a specific goal like saving lives, and ranks policies by lives saved per dollar—suggested America’s foreign aid budget could’ve been better spent on condoms and awareness campaigns, or even malaria and diarrheal diseases.
“Treating HIV doesn’t pay”
In a now infamous op-ed published in Forbes in 2005, before PEPFAR’s impacts were well documented, Brown University economist Emily Oster declared that “treating HIV doesn’t pay.” “It is humane to pay for AIDS drugs in Africa,” she wrote, “but it isn’t economical. The same dollars spent on prevention would save more lives.”
In fairness to Oster and others, the phrasing here is important. Her argument was not that African HIV patients’ lives weren’t worth the cost—that retroviral drug prices exceeded the “value of a statistical life”, as economists might phrase it—but rather that if we take the budget as fixed, and the prices as fixed, the money could do more good if spent on other health programs.
Oster wasn’t alone. While her delivery was perhaps deliberately provocative, her basic reasoning reflected a broad professional consensus, which viewed antiretrovirals through the lens of comparative cost-effectiveness analysis, and deemed them middling to poor value.
A systematic review published in the Lancet in 2002, just as the Bush administration was privately plotting the PEPFAR announcement, found that in terms of saving “disability-adjusted life years” or DALYs, "a case of HIV/AIDS can be prevented for $11, and a DALY gained for $1” by improving the safety of blood transfusions and distributing condoms, whereas “antiretroviral therapy for adults, cost several thousand dollars per infection prevented, or several hundreds of dollars per DALY gained."
Figure 2. Cost-effectiveness of interventions related to high-burden diseases in low- and middle-income countries (from Disease Control Priorities in Developing Countries, 2nd ed., 2006)
In the 2006 edition of its authoritative volume on Disease Control Priorities in Developing Countries, the World Bank looked at 60 different health interventions to address “high burden diseases”, i.e., diseases like HIV/AIDS, TB, and malaria that were killing large numbers of people (see Figure 2). Evaluated on cost-effectiveness, antiretroviral treatment, the core of PEPFAR’s multi-billion dollar strategy, ranked 45th out of 60, far behind investments to prevent diarrhea and malaria, and behind eight other investments to reduce HIV/AIDS mortality.
Overall the World Bank review concluded antiretrovirals were “moderately cost-effective” in the easiest settings, costing between $350 to $500 per disability-adjusted year of life saved, and “significantly poor value” in more difficult contexts where drug adherence was often poor.
Here at the Center for Global Development, my colleague Mead Over, who helped draw the World Bank’s attention to the AIDS epidemic in the 1990s and served on PEPFAR’s scientific advisory board, emphasized the need to stop future transmission. “We cannot do everything,” Over wrote in 2010. “If we accept that AIDS patients currently on treatment have an entitlement to continued support, President Bush’s call to scale-up treatment likely means less money for the rest of PEPFAR’s mandate—which is to prevent infection so that fewer people will need treatment in the future."
Lesson #1. What persuaded the White House was evidence of feasibility and efficacy, not cost effectiveness
For all of its early public advocates, from U2 frontman Bono to conservative Senator Jesse Helms, PEPFAR was sold on moral grounds, not material interests.
But moral arguments don’t exist in an empirical vacuum, so it’s interesting to look back at the kind of empirical evidence the White House used to underpin its moral case. Nobel-winning psychologist Harold Varmus interviewed some of the participants in those original PEPFAR planning meetings to understand what kind of evidence mattered, and noted:
“all agree that two kinds of testimony helped to convince the president’s representatives that the program should go forward. First, the evidence from those who had worked in the field that antiretroviral drugs and preventive measures could be deployed effectively and at reasonable cost, even in very poor settings, like Uganda and Haiti. Second, perhaps even more powerful, the visual demonstrations of how the new drug regimens could restore health—several participants alluded to this as ‘the Lazarus effect.’”
In short, evidence on feasibility and efficacy won the case.
Intuitively, it’s easy to see how these types of evidence could trigger moral imperatives. Ask yourself why you didn’t give money to the homeless person you passed on the way to work today, and I suspect your answer has more to do with efficacy (he’ll waste it on booze) or feasibility (my $5 is not going to end homelessness, we need systemic solutions). If you really believed your $5 would help him, and this kind of charity would solve homelessness, would you balk at the cost (I need those $5 for Starbucks)?
Not all problems in global development can be solved by throwing money at them. But money is, effectively, what the White House had, and they happened to land on something that money could in fact buy: “a single pill that could save millions of lives.”
Lesson #2. The budget constraint wasn’t fixed; PEPFAR unlocked new money
Bush was apparently allergic to measuring foreign aid in terms of dollars spent. Instead, the White House would start with health targets and solve for a budget, not vice versa. "In the government, it’s usually here is how much money we think we can find, figure out what you can do with it,” recalled Mark Dybul, a physician involved from the outset who went on to lead PEPFAR. “They were so adamant—because we tried that the first time and they came back and said, 'That’s not what we want.' […] They wanted a bottom-up, what would give us goals, tell us how you’re going to achieve them, and then tell us how much it will cost and we’ll figure out if we can pay for it or not, but don’t start with a cost."
Economists are trained to look for trade-offs. This is good intellectual discipline. Pursuing “Investment A” means forgoing “Investment B.” But in many real world cases, it's not at all obvious that the realistic alternative to big new spending proposals is similar levels of big new spending on some better program. The realistic counterfactual might be nothing at all.
In retrospect, it seems clear that economists were far too quick to accept the total foreign aid budget envelope as a fixed constraint. When in practice, as PEPFAR proved, the total size of the foreign aid budget was very much up for debate.
When Bush pitched $15 billion over 5 years in his State of the Union, he noted that $10 billion of it would be new money. And indeed, looking at the history of American foreign aid, 2003 marked a clear breaking point. In real dollar terms, foreign aid had been essentially flat for half a century at around $20 billion a year. By the end of Bush’s presidency, between PEPFAR and massive contracts for Iraq reconstruction, that number was hovering around $35 billion, where it has stayed since.
Figure 3
If $15 billion sounded like a lot, note that exactly one sentence after announcing that number in his State of the Union address, Bush pivoted to the case for invading Iraq, a war that would eventually cost America something in the region of $3 trillion—not to mention thousands of American and hundreds of thousands of Iraqi lives. Money was not a real constraint.
A broader lesson here, perhaps, is about getting counterfactuals right. In comparative cost effectiveness analysis, the counterfactual to AIDS treatment is the best possible alternative use of that money to save lives. In practice, the actual alternative might simply be the status quo, no PEPFAR, and a 0.1 percent reduction in the fiscal year 2004 federal budget. Economists are often tempted to be quite pessimistic about the prospects of big additional spending, not out of any deep knowledge of the budgeting process, but because holding that variable fixed makes the problem more tractable. In reality, there are lots of free variables.
Lesson #3. Prices also weren’t fixed, and PEPFAR may have helped bring them down
PEPFAR didn’t only show the folly of treating budget constraints as fixed; it should also remind us that costs aren’t permanently fixed either.
Mead Over documents this price decline in his book, Achieving the AIDS Transition. The first HIV drug treatment regimens, including AZT, cost about $100,000 per patient per year circa 1993. By 1997, triple-drug regimens had brought the price down to about $20,000 per patient per year. By the time of his 2003 State of the Union announcement, Bush noted treatment could be done for $300.
That last number was optimistic, but efforts by activists and international organizations continued to push prices down. Surging demand for antiretrovirals led India to begin producing generic HIV drugs in the 2000s, and by 2006 – under pressure from the Clinton Foundation and others – PEPFAR switched to generics.
It’s hard to see how the cost-effectiveness equation has changed, because nobody questions it anymore. By 2018 when the third edition of Disease Control Priorities in Developing Countries was published, antiretroviral treatment for HIV/AIDS was not even on the list of interventions subject to cost-benefit analysis, except for expanding access to people with low CD4 counts or uninfected partners of HIV patients. Access to antiretroviral drugs for HIV infected patients with high CD4 counts has transformed, in the space of twelve years, from too expensive to unquestionable.
There is an obvious analogy to climate change here, and the debate between economists who have traditionally favored carbon taxes and cap-and-trade schemes, versus a newer crop of progressive activists pushing investment in green technology to bend the cost curve. The latter has proved to be a political winner in the US with the passage of Biden’s Inflation Reduction Act including a raft of subsidies for clean energy and electric car manufacturing. But the path of investing in innovation rather than Pigouvian taxation and trade-offs may prove to be better economics as well. Analysts have consistently underestimated the speed of progress in battery technology, for instance. Given the pace of change, and the possibility that it is endogenous to our investment decisions, just choosing to tighten our belts looks crazy.
You don’t have to give up on utilitarianism or cost-effectiveness altogether to recognize these frameworks led economists astray on PEPFAR—and probably some other topics too
In a memorable passage from Tracy Kidder’s 2003 biography of Dr. Paul Farmer – who helped sell the Bush White House on the PEPFAR plan – Farmer’s team at Partners in Health in Haiti grapples with how to treat a boy named John in one of their rural clinics who is dying of a rare form of cancer. John’s only chance for survival is a medevac flight from Port-au-Prince to Boston, at a cost of $18,500, followed by $100,000 in treatment.
A colleague asked Farmer, “what circumstances would keep them from bringing John to Mass General.”
“‘No circumstances,’ Farmer said. ‘It’s his only chance.’
‘What will I say if I’m asked why we’re doing this?’
‘That his mother brought him to us,’ said Farmer. ‘And we’re doing everything we can to help him.’”
It’s cinematically perfect. We all want to be Paul Farmer in that story. But it’s a frustrating story analytically, because it gives us few guiding criteria to work with. Even Farmer acknowledged trade-offs at times. He wavered on the decision to send John to Boston for days. And Partners in Health’s standard operating procedure was not to medevac all their patients to Mass General.
The story of PEPFAR is not like the decision to medevac a patient from Haiti. It’s not that hard-headed economists need to be more soft-hearted, and save young John. Economists got PEPFAR wrong analytically, not emotionally, and continue to make the same analytical mistakes in numerous domains. Contrary to the tenets of the simple, static, comparative cost-effectiveness analysis, cost curves can sometimes be bent, some interventions scale more easily than others, and real-world evidence of feasibility and efficacy can sometimes render budget constraints extremely malleable. Twenty years later, with $100 billion dollars appropriated under both Democratic and Republican administrations, and millions of lives saved, it’s hard to argue a different foreign aid program would’ve garnered more support, scaled so effectively, and done more good. It’s not that trade-offs don’t exist, we just got the counterfactual wrong.
–
Thanks to Ranil Dissanayake, Victoria Fan, Amanda Glassman, Javier Guzman, Susannah Hares, Charles Kenny, and Mead Over for comments and discussions.
It's not clear to me that the core point of the essay goes through. For instance, the same amount of money as applied to malaria would also have helped many people, driven down prices, encouraged innovation—maybe the equivalent would have been a malaria vaccine, a gene drive, or mass fumigations.
i.e., it seems plausible that both of these could be true:
At the risk of over-simplification: I think you should at least discount the expected value of all those alternative health mega-projects by (a) the probability that congress would've funded them, and (b) the probability they would've worked as intended at scale. If the first probability was, say, 10%, and the latter maybe 25%, numbers get pretty small pretty fast. (Of course, PEPFAR only scores 100% and 100% in retrospect. But it seems it would've still scored quite high circa late 2002.)
Longer version...
The first point seems to be saying that we should factor in the chance that a program works into cost-effectiveness analysis. Isn't this already a part of all such analyses? If it isn't, I'm very surprised by that and think it would be a much more important topic for an essay than anything about PEPFAR in particular.
The second point, that people should consider whether a project is politically feasible, is well taken. It sounds like the lesson here is "if you find yourself in a situation where you have to recommend either project A or B, and both are good, but A is better than B, but if you do activism for A it still won't happen, but if you do activism for B your input will push it over the edge into happening, do activism for B." I agree with this as far as it goes, but there seem like some important caveats:
Good points. Agree that "always go for a big push instead of incrementalism" is waaayyyy too simple and sweeping a lesson to draw from PEPFAR. Also, three cheers for not lying. I think the World Bank was right not to suppress its data on the low cost-effectiveness of ARV drugs circa the mid-2000s. But in retrospect, I think people drew bad policy conclusions from that data.
My piece above is largely a plea for a little bit of intellectual humility and introspection on the part of the cost-effectiveness crowd (of which I'm often an active participant). If we find our analytical framework leads us to oppose the core of PEPFAR (i.e., the distribution of free ARVs in poor countries), then we ought -- in the face of its historic achievements -- to pause and reflect a bit, rather than shrugging and insisting "yeah, but I could've done better."
Your first paragraph gets at one reason why I think that kind of confidence isn't credible. Successful trials don't automatically translate into successful national or multi-national, multi-billion dollar programs. So we should be more skeptical that something that scored higher on cost-effectiveness terms ex ante would've converted into a massive global policy win. I genuinely don't think cost-effectiveness analysis, at least as practiced in global development/global health, really takes those questions of 'scalability' and implementation feasibility seriously enough. As a result, we confuse stuff that really could be done a million times over if money was available with stuff that looks cheap but faces much deeper political and organizational hurdles.
When we find something that works and really can scale with an influx of cash, we should run with it. With the benefit of hindsight, PEPFAR seems to fit that description. And, coincidentally, setting aside broader philosophical debates to stick with this concrete case, PEPFAR happens to be up for reauthorization in congress this year...
I noticed that much of the political tractability discussion has focussed on counterfactuals involving other diseases (e.g. malaria) but I'm more confused why the government prioritised treatment over prevention.
Oster's article argued that prevention would be a better bet than treatment and that was written in 2005 when Congress had approved the budget and the prices of the drugs had come down. You also highlight the importance of, "the evidence from those who had worked in the field that antiretroviral drugs and preventive measures could be deployed effectively and at reasonable cost, even in very poor settings". (emphasis added)
So both options had evidence of feasibility and efficacy, the money was already there, and it seems like, at that time, they didn't know which option would scale better or get cheaper quicker.
I'm struggling to see what Oster got wrong. Was the decision to prioritise treatment over prevention mostly driven by the emotional appeal of the 'Lazarus effect'?