This essay was jointly written by Peter Hurford and Marcus A. Davis.

Note that because of technical length restrictions on the EA Forum, this essay is the third part of three parts: Part 1, Part 2, and Part 3. To see all three parts in one part, you can view the article on our research site. This article contains all the appendices and endnotes referenced from Part 1 and Part 2.

Previous articles also include an analysis of how beneficial vaccines have been, an analysis of how much it costs to roll-out a vaccine, how much it costs to research and develop a vaccine, and how long it takes to research a new vaccine. However, this series is structured so that starting at Part 1 should be all you need to do.

Appendix A: What do other articles say?

To compare our findings to other articles, we looked at some other articles that we could find comparing the cost-effectiveness of different vaccines[9]. Comparing across all these studies gives further confidence in our estimates and understanding of model assumptions and model uncertainty.

Here’s what we end up concluding:

For details of the individual studies, see below:

A.1.) Cost-effectiveness of HPV - Fessenfeld, Hutubessy, and Jit (2013)

Fesenfeld, Hutubessy, and Jit (2013) did a meta-analysis and found a cost-effectiveness of $146 to $34,382 per QALY in inflation-adjusted international dollars for the HPV vaccine (see Table 1) based on studies in Thailand and Mexico[10]. Our model for HPV finds a cost-effectiveness of $370 - $1600 / DALY. Our estimate seems to fit snugly within their wide range.

But why does Fesenfeld, Hutubessy, and Jit (2013) have such a large range with such large values? Part of the reason for the large difference in estimations between sources and between the estimates given above are significant differences in cost estimation rather than fundamental methodological problems, as in these studies cost per HPV series estimates range from $49 to $511. In our model, if we use the more worst-case scenarios of cost per vaccine of $23 in the developing world, the cost-effectiveness of roll-out would be about $1964 per DALY while $49 per series yields an estimate of over $4000 per DALY.

Since prices in the developed world can be as high as $400, the price per vaccine used would, naturally, significantly change the cost-effectiveness. Indeed, in the United States, cost-effectiveness has been benchmarked between $4,894 to $18,447 per QALY (adjusted for inflation between 2005 dollars to 2017 dollars) by Chesson, Ekwueme, Saraiya, and Markowitz (2008) and benchmarked at $31,441/QALY (after conversion from 2001 dollars) by Sanders and Tiara (2003). If our model were scaled to have a $400 roll-out cost per person, it would agree with this, suggesting ~$35K/DALY.

Thus our model matches with the same parameters, but we don’t find the same need to be as uncertain about our assumptions of cost per dose.

A.2.) Making Markets for Vaccines - Levine, Kremer, & Albright (2005)

Levine, Kremer, & Albright (2005) look at the cost-effectiveness of encouraging vaccine development for malaria, HIV, and tuberculosis. They explore the idea of creating an “advanced market commitment”, which is a legal commitment to purchase vaccines at a certain preset price once that vaccine is researched and developed, incentivizing that research to happen by providing a return that could be reasonably guaranteed and known in advance.

They estimate a $3B market in 2004 dollars would be necessary ($4B in 2018 dollars), assume that the cost of capital (interest rate) of 8%, discount DALYs at 3% per year, assume three doses of the vaccine will provide protection for five years, assume vaccines are 60% effective, and assume a vaccine price of $15 per vaccine and delivery costs of $0.75 per dose ($2.25 total) in 2004 dollars ($23 per person in 2018 dollars). The malaria vaccine is then modeled as returning $15/DALY in 2004 dollars, or ~$20/DALY in 2018 dollars. Levine, Kremer, & Albright (2005) released a spreadsheet model of their calculations.

On one hand, our estimates are very close, which is reassuring. Our models suggested malaria had a return of $23 - $52 / DALY if vaccinating 60% of SSA and $11 - 18/DALY if vaccinating to eradication. These estimates are notably close to the $20/DALY given by Levine, Kremer, & Albright (2005). The $3B market size suggested by Levine, Kremer, & Albright (2005) is also shockingly close to the $3.17B we estimated would be needed to research, develop, and roll out a malaria vaccine.

On the other hand, our methodologies are different, and this is worth discussing in light of model uncertainty. Levine, Kremer, & Albright (2005) are modeling vaccinating 200M people in targeted areas. This is much larger than our SSA 60% scenario, which is modeled for ~96M children under 5 in the first year plus another ~22M children a year for every year after, and much smaller than our eradication scenario modeled for 675M people. Our model does not take into account an advanced market commitment, models for a range of estimates for efficacy, and assumes a roll-out cost of $22 per person. When we adjust our model to factor in the price, delivery costs, efficacy, and vaccinated population as Levine, Kremer, & Albright (2005), we get $12 - $19 / DALY, which is more optimistic than Levine, Kremer, & Albright (2005) projects.

Malaria Vaccine Cost-Effectiveness

A.3.) Modeling the Cost-effectiveness of Introducing the RTS,S Malaria Vaccine Relative to Scaling up Other Malaria Interventions in Sub-Saharan Africa - Winskill, Walker, Griffin, and Ghani (2017)

Winskill, Walker, Griffin, and Ghani (2017) set out to compare introducing the malaria vaccine to other malaria treatments like distributing bednets. We make a similar comparison in Appendix C. They look directly at the large phase III trial of the RTS,S malaria vaccine across 11 sites in Africa, which vaccinated ~15,500 children. Per Winskill, Walker, Griffin, and Ghani (2017)’s numbers, the vaccine had efficacy defined by a range between 13.7%-46.9% and cost $20 per child to administer. Given the observed transmission levels, they concluded an incremental cost-effectiveness ratio (ICER) of $44–$279 per DALY.

They also found a ICER per DALY of $27 ($8.15-$110/DALY) for long-lasting insecticide-treated nets (LLINs), $143/DALY (range $135-$150/DALY) for indoor residual spraying (IRS), and $68/DALY (range $62-$75/DALY) for seasonal malaria chemoprevention. By comparing ranges, distributing bednets came out ahead, but note that this may not hold as diminishing marginal returns are experienced by scaling up.

Malaria Vaccine Cost-Effectiveness

Again, while our estimates look very different, it comes down to assumptions. When we adjust our model to factor in the price, delivery costs, efficacy, and vaccinated population as Winskill, Walker, Griffin, & Ghani (2017), we get $54 - $230 / DALY, which is very similar to their range. Winskill, Walker, Griffin, & Ghani (2017) made similar assumptions as us with regard to vaccine cost, but only considered four years of benefits (compared to us considering twenty), used a technically more accurate but much less optimistic vaccine effectiveness rate (we assume the rate would get better over time), uses a significantly smaller sample population (just that of the actual Phase III trial) and does not consider factoring in any fixed roll-out costs or R&D costs. Together, all of these changes in assumptions, except the last one, greatly increase the cost-effectiveness estimate.

Essentially, Winskill, Walker, Griffin, & Ghani (2017) is looking at the cost-effectiveness of rolling out the malaria vaccine as it exists right now, whereas we are looking at the cost-effectiveness of developing and rolling-out a future malaria vaccine that would likely reach higher efficacy targets more in line with existing vaccines and established cut-offs for minimum efficacy needed to have a viable vaccine.

A.4.) Country-specific Predictions of the Cost-Effectiveness of Malaria Vaccine - Galactionova, et al. (2017)

Galactionova, et al. (2017) models the existing RTS,S malaria vaccine and finds a median cost-effectiveness ratio of $188 per DALY averted, though with a very wide range of range of $78 - $22,448 / DALY. Assumptions made by Galactionova, et al. (2017) were similar to that of Winskill, Walker, Griffin, and Ghani (2017).

The range is so wide because of the inclusion of four countries where the predicted impact is highly imprecise due to a limitation of data. When those countries are excluded, the range is greatly reduced to between $115 and $220 / DALY (Galactionova, et al., 2017, section 3.2). This smaller range fits snugly within our model with parameters adapted from Winskill, Walker, Griffin, and Ghani (2017).

Malaria Vaccine Cost-Effectiveness

A.5) Cost-Effectiveness Analysis of Vaccinating Children in Malawi with RTS,S Vaccines in Comparison With Long-Lasting Insecticide-Treated Nets - Seo, Baker, & Ngo (2004)

Seo, Baker, and Ngo (2004) use data from the RTS,S trials and do a sensitivity analysis, varying time horizon, discount rates, vaccine efficacy, and price. While they do not directly state a confidence interval resulting from this sensitivity analysis, they headline a figure of $145.03 / DALY but suggest a range of $8 - $329.78 / DALY.

Malaria Vaccine Cost-Effectiveness

The assumptions are similar to Winskill, Walker, Griffin, and Ghani (2017), which ends making the range given by Seo, Baker, and Ngo (2004) a close match for the range constructed from the parameters from Winskill, Walker, Griffin, and Ghani (2017) – the maximums of the range are nearly the same. However, Seo, Baker, and Ngo (2004) is more optimistic because they consider possibilities such as $1/dose ($4/person roll-out cost, or ~18% of what we project in our model) in their sensitivity analysis.

A.6.) Public Health Impact and Cost-Effectiveness of the RTS,S/AS01 Malaria Vaccine: A Systematic Comparison of Predictions from Four Mathematical Models - Penny, et al. (2016)

Penny, et al. (2016) is of particular importance because its estimates form the basis for much of the GAVI (2016) report on malaria that we rely on for a lot of our models. Like the previous articles we considered, Penny, et al. (2016) looks at the data from the RTS,S trials. However, unlike these previous articles, they attempt to extrapolate the data from this trial to future cost-effectiveness using multiple models. This matches much closer to what we intend to do, but still doesn’t fully match our assumptions of large potential future increases in vaccine efficacy following further R&D spending prior to achieving licensing and large-scale roll-out of a future vaccine.

Focusing on their four-dose model, their model captures ten years of benefits and uses a declining exponential decay curve of efficacy with a mean of ~6% - ~25% over those ten years, depending on the model used (see Figure S2.1). They also vary the price between $8-$40 per fully vaccinated child. Integrating these efficacy and price figures into our model for a population of 100,000 vaccinated children over ten years gives us a result of $28 - $270 / DALY, which is a slightly wider range but captures all the values found from Penny, et. al. (2016)’s range.

Malaria Vaccine Cost-Effectiveness

A.7.) Rotavirus Vaccination: Cost-Effectiveness - Atherly, et al. (2009)

Atherly, et al. (2009) looks at the cost-effectiveness of rotavirus across most of the GAVI-eligible countries using a demand forecast model was used to predict adoption, and then modeling health outcomes of a hypothetical birth cohort in the target population, and finds increasing cost-effectiveness of distributing the rotavirus vaccine over time, ultimately resulting in a cumulative figure of “$43 per DALY averted during 2008-2025”. This estimate is within the range we give for our estimate, despite differences in methodology and context.

Rotavirus Vaccine Cost-Effectiveness

A.8.) Cost-effectiveness and Economic Benefits of vaccines in Low- and Middle-Income Countries: A Systematic Review - Ozawa, et al. (2012)

Ozawa, et al. (2012) does a meta-analysis of vaccine cost-effectiveness, finding 44 articles that reported costs per DALY averted. Data available in Table 1 gives a variety of ranges for many vaccines – the ones that overlap with vaccines we considered include HIV, HPV, malaria, and rotavirus.

While the HIV range does overlap some, Ozawa, et al. (2012) points to much more pessimistic HIV ranges than our estimate. It’s hard to figure out why without digging into the underlying study, but one likely reason is simply context – the HIV estimates in Ozawa, et al. (2012) come from Thailand, instead of SSA.

The HPV estimates from Ozawa, et al. (2012) frustratingly only give upper bounds, and while they are generally more optimistic than our range, all but one of them do fit in our range.

The malaria estimates from Ozawa, et al. (2012) are nicely clustered around our estimate. The reason why they vary is because each estimate has been constructed from a different assumption about vaccine price per dose. When you take the estimate that matches our projection of price per dose at $4-5/dose, you end up with the estimate that is $31 - $49/DALY, which is an incredibly close match to our actual estimate of $21 - $49 / DALY.

Ozawa, et al. (2012) cites 105 different rotavirus cost-effectiveness estimates from around the world, so we don’t cite them all in the table. Instead, we picked the estimates that were specifically for SSA countries or for lower and middle-income countries (LMIC) generally. For those who had point estimates that varied by dose, we changed them to a range. The first two estimates from SSA countries were very close to our estimate and the two estimates for LMIC generally overlapped with our estimate but were more pessimistic. This is likely because LMIC generally are wealthier and have a lower disease burden than SSC specifically.

A.9.) Dalton (2014)

Dalton (2014) models “the marginal, ex-ante cost-effectiveness of funding medical research” as a “problem of unknown difficulty” (Cotton-Barrat 2014a, Cotton-Barrat 2014b), which assumes a discrete, fixed outcome (e.g., the vaccine is made) will be achieved after an unknown amount of resources (e.g., money, staff time, randomized controlled trials) are used. We model this discrete jump from 0 to 1 as a continuous probability curve over time and then estimate this probability curve with a lognormal distribution.

When applying this to funding for neglected tropical diseases, Dalton (2014) considers additional funding as “bringing forward the date of technological discovery, and therefore shifting forward the roll-out of the vaccine/drug” to the point of total eradication and that therefore “the gain of shifting forward vaccine discovery by a year equals the current annual disease burden”. Dalton (2014) finds a median impact of 13.9 DALYs/$1000 ($71.94 / DALY).

To compare with our work, we both modeled for HIV and malaria specifically. Dalton (2014) calculated these to be 11.2 DALYs / $1000 ($89.29 / DALY) for HIV and 22 DALYs / $1000 ($45.45 / DALY) for malaria (see calculations in their associated spreadsheet). On the other hand, our models suggested malaria had a return of $23 - $52 / DALY if vaccinating 60% of SSA and $11 - 18/DALY if vaccinating to eradication and HIV was $210 - $690 / DALY for 60% of SSA and $300 - $1600 / DALY for eradication.

Dalton (2014) was solidly in our 60% SSA range for the malaria vaccine, but optimistic compared to both our ranges on HIV vaccine. (It was within the same range as our “typical” vaccine too, but that’s a very big range.) Where might that discrepancy lie?

Dalton (2014)’s analysis is a completely different kind of model than our analysis, so a direct comparison is difficult. Notably, Dalton (2014) notes that the “[c]osts of distribution were not accounted for”, which for complete eradication of a disease is very considerable. Our estimate is this would add roughly $3.6B-$5B for vaccinating 60% of SSA for malaria, $13B-$18B for eradicating malaria, $19B-$100B for vaccinating 60% of SSA for HIV, and $180B-$960B for eradicating HIV.

On the other hand, Dalton (2014) also models the spending on research and development for malaria using data from GFinder (2013), which includes all R&D spending – including not only vaccines, but also other drugs, microbicides, diagnostics, and basic research. Given that Dalton (2014) intended to model the returns of research as a whole this seems appropriate but reduces the ability to compare meaningfully to our work to model the cost-effectiveness of vaccine R&D in particular.

Furthermore, GFinder (2013) also uses a total for malaria vaccine research that is higher than the total documented in GAVI (2016) that we rely on for our estimate. In total, we assume that a malaria vaccine could be created with an investment of $605M, whereas GFinder (2013) assumes a total spend of over $3B[12], and Dalton (2014) extrapolates to a total R&D spend of over $67B on malaria, over 100x our estimate for malaria R&D costs and over 4x our estimate for all costs for malaria eradication.

Given that in our model, for HIV the R&D costs are much more prominent than for malaria, the underestimation in roll-out costs and the overestimation in R&D costs by Dalton (2014) relative to our model would explain the discrepancy.

A.10.) Dalton (2016)'s Novel Model Adapted from Karnofsky (2014)

Dalton (2016) produces a novel model based on Karnofsky (2014)’s work. This model assumes that the entire DALY burden of a disease will eventually be eliminated and that new research will be responsible for 50% of this reduction. The benefits would have a lag of 50 years and require discounting at a 3% annual rate, but benefits will grow with population growth. Applying this gave a mean of $51.91/DALY for HIV and $44.14/DALY for malaria (see spreadsheet here).

As Dalton (2016) admits, this is a relatively simplistic model. Like Dalton (2014), this model is hard to compare to our model as it works very differently. The dynamics are also very similar – like Dalton (2014), Dalton (2016) does not directly consider vaccine roll-out costs, except insofar as this may be responsible for assuming that new research is only responsible for 50% of the total reduction. Moreover, Dalton (2016)’s novel model, like Dalton (2014), uses data from GFinder (2013) that may be misleading for the particular application.

Appendix B: Forecasting Cost-Effectiveness for Ebola

Based on the cost-effectiveness modeling in this essay, we could consider using our model to forecast the cost-effectiveness of the Ebola vaccine, which is currently under active development and relatively unknown. We previously found the Ebola vaccine to likely cost ~$1.5B in R&D and to become licensed in ~2022, but we do not know much about the relevant roll-out costs. We’ve also estimated Ebola vaccination to potentially avert 92,685 DALYs per year if rolled out to 60% of the relevant SSA population, assuming the average DALY burden of the last five years, and to prevent ~308,900 DALYs given these assumptions if Ebola were eradicated. Assuming the much higher burden seen during the 2014 outbreak of ~1.3M DALYs per year, ~388,000 DALYs could be prevented by vaccinating 60% of the relevant population[13].

Based on our model, we can create guesses for Ebola based on a variety of potential costs of the Ebola vaccine. We can also vary how focused the population targeted for vaccination is. A broad distribution of the Ebola vaccine may be highly unrealistic. Merler, et al. (2016), for example, advocates for “ring vaccination,” which targets (in some combination) those who have been in direct contact with someone infected, contacts of contacts, and the population in geographic rings around them. The population in the Henao-Restrepo, et al. (2016) Ebola trial was vaccinated on this basis and smallpox was eradicated in part due to such measures (Strassburg, 1982). Also, as Merler, et al. (2016) mentions, future Ebola outbreaks in different regions could have a different basic reproduction number (the number of cases an infected person generates, on average, over their infection period), implying a different approach to be taken. It’s unclear to us what this suggests about the population needed for eradication.

We assume that our above calculations are right, that the vaccine will be 50% effective[14], and that there is a $100M fixed cost associated with developing vaccination infrastructure.

Ebola Eradication Cost-effectiveness

APPENDIX C: COMPARING TO AGAINST MALARIA FOUNDATION

GiveWell has pretty thoroughly vetted the Against Malaria Foundation and found it to create the equivalent value of saving the life of a child under 5 for $764 to $3057 (average $2087) (GiveWell, 2018). While GiveWell would not want us crudely converting this figure to DALYs, we need to do so to have a remotely comparable analysis, so we consider saving the life of a child under 5 to allow them to live to the typical life expectancy of SSA – 59 years total, or an additional 53 years – at full health, which would be 53 DALYs averted. Thus, AMF would be at ~$15 - $58 per DALY averted (average $39).

Scenario 1: Vaccinate 60% of the relevant populations in SSA

Scenario 2: Eradicate the disease completely

If you take this analysis literally, donating to AMF may be more attractive on an average $ / DALY basis than helping eradicate HPV, HIV, or Ebola, but not compared to eradicating smallpox, malaria, or measles. Donating to AMF also emerges better than helping research, develop, and roll-out all vaccines we examined that stop short of eradication except for malaria smallpox, and rotavirus.

However, the estimate of the malaria vaccine compared to donating to AMF are substantially close and the range overlaps. Thus the conclusion could be very sensitive to our particular assumptions. Notably, the current academic literature suggests that increasing bednets is more cost-effective than rolling out existing malaria vaccines (Penny, et al., 2016; Winskill, Walker, Griffin, & Ghani, 2017).

While the comparison between bednet distribution and the malaria vaccine is interesting, it is unclear how meaningful this comparison is. It’s worth noting that the calculations for AMF are likely more robust, which could favor AMF further if a Bayesian adjustment framework were used (for examples, see Karnofsky (2011) and Dickens (2016)).

Additionally, charities themselves, like a charity set up to fund and distribute a malaria vaccine, could differ in marginal cost-effectiveness from the abstract intervention in a way that would undermine the comparison further and make AMF more or less appealing.

Appendix D: Cost-Effectiveness of GAVI

GAVI, a global organization that helps distribute vaccines, is considered to be very cost-effective. A full analysis of the cost-effectiveness of GAVI is greatly beyond the scope of this article, as the use of “leveraged funding” and the difference between marginal vs. total costing approaches, makes the cost-effectiveness estimate vary greatly from that of individual vaccines (see our discussion of assumptions).

However, we can look at three sources: Francis (2010) quotes GAVI as averting 5M deaths against $3.7B in funding, for a cost-effectiveness of $740 per life saved. Lob-Levyt (2011) quotes GAVI as averting 5.4M vaccine-related deaths against $4491M in vaccine-related spending or $831.67 per life saved. Lastly, a GAVI press release quotes GAVI as saving 4M lives against $3.7B or $925 per life saved. When we take these estimates literally, and consider a figure of 30 DALYs per life saved [6], and compare this on a table to our estimates of vaccines, GAVI comes out more cost-effective than the distribution for all of the vaccines except smallpox at 60% of SSA.

The fact that these estimates of GAVI appear more cost-effective than the individual vaccines is not unbelievable, due to massive use of “leveraged funding” and the use of vaccines that may well be very cost-effective that we did not model (e.g., the meningitis A vaccine, the pneumococcal vaccine).

Endnotes