Thanks for sharing your critique of our recent grants with Open Philanthropy for technical support units (TSUs). We really appreciate this thoughtful pushback! We've recommended (and are considering) a number of grants to help respond to the current situation with cuts in US foreign health assistance. So, getting critiques like yours is helpful since it encourages us to pause and consider whether we’re making the right tradeoffs in these grants. While we share some of your perspectives on the uncertainties of this work, we're still excited about our decision in this case.

While this grant’s impact is particularly uncertain, we see this as a difference in degree, not kind, compared to other grants we recommend. Most of our funding still goes to Top Charities - proven programs backed by strong evidence and our cost-effectiveness analysis. But we also recommend opportunities through the All Grants Fund. The goal of this fund is to find and fund what we believe are the highest-impact uses of marginal dollars, even when those opportunities are riskier or harder to model. This grant fits squarely in that approach. We’ve funded technical assistance from the All Grants Fund before, alongside grants that are uncertain for other reasons. For example, sometimes we're trying to generate new evidence, while at other times we're recommending high expected value bets even when we know we’re unlikely to get a definitive answer on their impact.

We agree that the evidence base for TSUs is thin. In general, we think it’s challenging to evaluate technical assistance programs because 

• particular programs might not have very convincing controls or comparators and
• technical assistance packages can vary widely so aggregating results from many programs might not be that informative, even if they’re targeting the same outcome

So even if the review that Nick cites had found good evidence for past TA programs, we still might not feel sure that it would generalize to the TSUs we recommended funding. 

But as discussed above, we don’t consider high uncertainty to be a dealbreaker in grants funded from the All Grants Fund. We still think it can be worth funding TA (see, e.g., our maternal syphilis grants) and we’re very interested in building up our ability to learn about programs like this over time. (We’re working on a project looking back on a subset of technical assistance grants we’ve funded, but don’t have a publication date for that yet.) 

While we don't have detailed theories of change, we still think it's plausible that TSUs could be impactful. We are excited about this grant because we think it could help governments to make difficult prioritization and program adaptation decisions in countries affected by US government funding freezes and cuts. We expect that the details of how this could look will vary by country and so we don’t feel confident that any particular mechanism will cash out in impact. But for example, we think TSUs could help governments to: 

• Adapt programs away from more expensive models implemented with USG support to more cost-efficient models (e.g., by integrating disease-specific verticals or moving from international to local implementing organizations)
• Target programs (e.g., to certain regions or populations) based on cost-effectiveness analyses
• Crowd in philanthropic or other funding by identifying funding gaps 

While we think the above examples are plausible, we agree that the theory of change for these programs is not tightly specified. However, we spoke with senior Ministry of Health officials in each country about this grant, and overall governments voiced support and demand for the proposed TSUs and were eager to have CHAI and PATH's support on this work. We also think both organizations are well-placed to support this work, as both have supported on malaria-specific TSUs in the past, have teams with specific focus on health systems and health financing, and have established relationships with the governments they’re supporting. 

Budget - Thank you for sharing your estimates - this is helpful for us as we continue to update how we review budgets. We’ll share a high level budget breakdown for this grant with our public grant write-ups (which are coming - see below!). One quick clarification (which wasn’t clear in the podcast) is that costs for Nigeria reflect support in seven states as well as national support. 

Outside of that, we think that the higher budget reflects both higher salaries and a higher non-salary budget share (to account for travel, coordinating stakeholder engagement, and support from global technical teams). Our understanding is that salaries are set based on globally-benchmarked salary ranges and localized equity adjustments to account for organizational equitable pay standards and differential cost of living across different geographies. A portion of the compensation costs is also due to benefits (such as health insurance) that may be standard to each location.

Learning - We agree that we should try to learn about the impact of these grants and also agree with commenters and Nick’s revision that an RCT isn’t an appropriate strategy. We’ve asked CHAI and PATH to track and report out on the following. 

• Process indicators (staff hiring, government engagement, TSU recommendations made and implemented);
• Case studies of how government prioritization decisions are made;
• Promising funding opportunities identified through the TSU work;
• Contexts where cost-effectiveness considerations are or aren't helpful for government decision-making. 

We’ll attempt to triangulate these reports through speaking with other stakeholders, though we expect we’ll still have substantial uncertainty about impact given the lack of counterfactuals. 

No CEA and grant write up. These are coming! We typically have a lag between making grants and publishing our write-ups, but wanted to share about this grant sooner because we’ve received a lot of interest in our response to the funding cuts. We expect to publish pages for CHAI and PATH (including a rough BOTEC) by the end of June. 

Urgency. We see the urgency here as being specifically related to governments’ needs to adapt to frozen or cut US health assistance: we heard when investigating this grant that governments were already beginning this planning process and that lighter touch versions of the support offered by TSUs were already being provided on nights and weekends by CHAI staff in certain countries. We also think this kind of grant is inherently uncertain and it didn’t seem likely that we’d reduce that uncertainty by spending additional time investigating. So, with apparent demand for support at the time and since we didn't think waiting would lead to a better decision, we chose to recommend funding relatively quickly.

Hi Nick,

Thanks for reaching out and for your interest in these grants. We're currently working on grant write-ups that we plan to publish soon, but unfortunately, we don't have anything available to share just yet.

We want to share a few thoughts that might help clarify our approach.

Our research incorporates potential downsides or unintended consequences, such that our recommendations take into account factors like potential side effects. Most (if not all) of the information in Leif Wenar’s WIRED piece is something that anyone could find on our website. However, we believe some of what is published in the piece is misleading or inaccurate. Here is a lightly edited version of the material that GiveWell sent to WIRED in response to a request for comment on Leif Wenar's piece. 

Hi Ian, thanks for sharing this article – our team wrote up some notes on how this topic intersects with our work, and I (Isabel Arjmand writing on behalf of GiveWell) thought it might be useful to share here.

Like the Bloomberg editorial board, we're concerned about stalling progress in the fight against malaria, but we're skeptical that quality issues with PermaNet 2.0s have influenced this progress as much as the article suggests.

All things considered, we believe that malaria nets are, and have been, highly effective in reducing malaria burden. The Against Malaria Foundation had first shared the studies highlighted above with us in 2020, and the claims in the Bloomberg article have prompted us to do some additional research.

Based on the work we've done so far, we aren't convinced that decreased net quality is primarily responsible for malaria resurging in Papua New Guinea. So far, we see this as a milder negative update on nets than the article would indicate, in part because we think these tests of net quality may not be a perfect proxy for effectiveness in reducing cases and in part because we no longer fund PermaNet 2.0s (for unrelated reasons). At the same time, renewed interest in the evidence around PermaNet 2.0 quality is a nudge for us to prioritize further work to understand net quality control in general.

More detail on the implications of this research for GiveWell's work

While we no longer fund PermaNet 2.0s because we now fund newer types of nets instead, they make up roughly 20% of nets we've funded historically. The studies referenced in the Bloomberg article looked at nets distributed in Papua New Guinea and indicate that the post-2012 PermaNet 2.0s perform worse on certain efficacy tests. We aren't sure how well those efficacy tests serve as a proxy for malaria transmission (e.g. mosquitoes in these tests could be impaired from the exposure to insecticides even if it isn't sufficient to kill them). We're also skeptical that changes to the formulation of PermaNet 2.0s were the key driver of increased malaria cases in Papua New Guinea. During this time, we think other factors like insecticide resistance and shifts in biting patterns likely played a meaningful role (as highlighted in this paper). That said, we see these studies as a negative update on the effectiveness of those nets.

We did a quick back-of-the-envelope calculation (so this is more illustrative than fully baked, at this point):

• Assuming the insecticide treatment on PermaNet 2.0s was 80% less effective after 2012 would make those nets look 30-50% less effective overall than we'd previously modeled. That's because we model roughly 30% of the benefit of nets as coming from the physical barrier in the absence of insecticide resistance, and we already discount the effectiveness of nets like PermaNet 2.0 because of insecticide resistance. We would guess that with further work, we'd estimate that 80% is on the pessimistic side of things (which would put the overall impact on net efficacy at the low end of our 30-50% range, or lower).
• Then, assuming that similar issues don't apply to other nets (which could be wrong – we plan to look into this more), our overall nets grantmaking would look roughly 5-10% less cost-effective than we'd previously estimated, since PermaNet 2.0s are around 20% of our historical nets distributed. That proportion has varied over time. In 2018, all of the nets we funded were PermaNet 2.0s; now, we fund newer types of nets instead.

While concerns specific to PermaNet 2.0s don't directly affect our future allocation decisions, this issue does raise more general concerns about quality control for nets. Ideally, we would have prioritized more work in this area in the past. We're planning to learn more about quality control processes and we also want to better understand how others in the malaria field are thinking about this.

We've haven't explicitly modeled diminishing returns in this way. Most of the opportunities we consider are for specific pre-defined gaps, so they're more discrete than something you can really scale in that continuous type of way.

We don't select/structure our grants such that we necessarily think the "last dollar" or marginal dollar to that grant is 10x cash. For example: if there was a discrete $5M funding opportunity to support a program in a specific area, we might model the cost-effectiveness of that opportunity as say, 15x overall, but there wouldn't be any particular reason to think the 'last dollar' was more like 10x. Generally, when it comes to funding discrete opportunities (e.g. vaccination promotion in a certain state in Nigeria), we don't tend to think about the value of the first versus last dollar for that discrete opportunity, because we're often making a binary decision about whether to support the program in that area at all. Hope this clarifies! 

Thanks to Vasco for reaching out to ask whether GiveWell has considered: 

• e-Government procurement (benefit-to-cost ratio of 125)
• Trade (95)
• Land tenure security (21).

GiveWell has not looked into any of these three areas. We'd likely expect both the costs and the benefits to be fairly specific to the particular context and intervention. For example, rather than estimating the impact of reduced tariffs broadly, we'd ask something along the lines of: What is the intervention that can actually e.g., lead to a reduction in tariffs? On which set of goods/services would it apply? Which sets of producers would benefit from those lower tariffs? And thus, what is the impact in terms of increased income/consumption?

We think there's a decent chance that different methodologies between Copenhagen Consensus Center and GiveWell would lead to meaningfully different bottom line estimates, based on past experience with creating our own estimates vs. looking at other published estimates, although we can't say for sure without having done the work.

Thanks SoGive for the post! We wanted to share some of GiveWell's current thinking around malaria vaccines in case it's helpful. We also wrote a report on RTS,S in 2022 here and have recommended a couple grants for vaccine rollout and research.

On a cost-per-person-reached basis, we agree ITNs and SMC are superior to either of the two WHO-approved malaria vaccines. However, we think there's less of a differential in cost-effectiveness than this post implies, for a number of reasons:

• The difference in all-in delivery costs is probably less substantial: We think it roughly costs $6 to deliver an ITN to a household^[1], and roughly $7 to provide a child with a full course of SMC. (See also impact metrics here). We estimate the total costs for fully immunizing a child range from $23-$43, depending on the choice of vaccine.^[2] 
• We would approach the comparison for the reduction in malaria incidence differently: Using the effect size from the Pryce et al meta-analysis for ITNs (which consists mostly of trials that lasted one year) and comparing it with the incidence reduction observed after 12 months in RTS,S/R21 trials is not as straightforward as is seems:
  • The reduction found in Pryce et al has to be considerably updated in light of recent developments, notably new types of nets (PBO, dual active ingredient) and increased insecticide resistance. 
  • The RTS,S Clinical Partnership trial includes results from a follow-up four years after the start of the intervention and finds a reduction in clinical malaria from 28% (3 dose group) to 36% (4-dose group) at endpoint. Our malaria team recommends these results rather than the earlier snapshots, as they are less noisy.
  • Our best guess is that R21 and RTS,S are similarly effective at preventing malaria. Available evidence suggests that short-term effectiveness is broadly similar between RTS,S and R21. No data has been published for the impact of R21 on malaria incidence over the long run (>20 months after the first dose). Because the short-term outcomes are broadly similar, and both vaccines employ the same mechanism to induce an immune response in the vaccinated person^[3], our best guess is that four doses of R21 probably offer similar levels of protection as four doses of RTS,S over the long run.
  • We believe the apparent difference cited in the post is most likely due to the different setup between trials (Datoo 2022, unlike the RTS,S trial, was carried out in a seasonal setting). Note also that results from a phase III trial on R21 are now available (Datoo 2023 (preprint)).
• The duration of protection differs: Due to factors such as attrition and physical decay, we currently estimate an ITN to provide between 1.2 - 2.0 years of effective protection. As indicated above, we estimate that malaria vaccines offer around 30% protection over four years. 

There are, of course, additional factors that need to be taken into account to get a full picture (for example, what coverage levels are achievable for each intervention?). However, our current best guess is that even with those included, nets (and SMC) will be more cost-effective than malaria vaccines - just not by an order of magnitude.

1. ^{^}

   Costs per child reached are much higher, roughly $15-$26.

2. ^{^}

   The price per dose of RTS,S was $9.30 in 2022, and estimates for R21 indicate a price per dose of $3.90. We expect that, on average, 70% of children who received three doses will also get a booster shot, which implies vaccine costs per child between $14-$37. The best costing estimates for the delivery of the doses suggest around $9 per child.

3. ^{^}

   “The leading malaria vaccine in development is the circumsporozoite protein (CSP)-based particle vaccine, RTS,S, which targets the pre-erythrocytic stage of Plasmodium falciparum infection. It induces modest levels of protective efficacy, thought to be mediated primarily by CSP-specific antibodies. We aimed to enhance vaccine efficacy by generating a more immunogenic CSP-based particle vaccine and therefore developed a next-generation RTS,S-like vaccine, called R21. The major improvement is that in contrast to RTS,S, R21 particles are formed from a single CSP-hepatitis B surface antigen (HBsAg) fusion protein, and this leads to a vaccine composed of a much higher proportion of CSP than in RTS,S.” Collins et al. 2017, “Abstract”

Hi Vasco,

Thanks for your comment! To clarify, our funding bar being 10x cash doesn't mean that every grant we make will be to things that are 10x cash – it means that we'll generally fund all of the programs we find that are above 10x, and not the ones that we estimate to be below 10x (with caveats that sometimes we will/won't make grants that fall on either side of that line for other reasons not captured in the CEA, e.g. learning value). You can read more on how we make funding decisions here.

Many of the grants we make are above 10x, including a fair amount in the 10-20x range (like this recent CHAI grant – we estimate delivering the program is ~17x cash, not counting the evaluation grants). Using Against Malaria Foundation (AMF) as an example, we fund net distribution campaigns in specific geographic regions that meet our 10x bar (see this grant made to AMF in January 2022 that supports net distribution campaigns in three Nigerian states). Theoretically, if we evaluated six states for net distribution campaigns and only four states met our criteria to be above 10x, we would only fund those four and not the other two, and the average cost-effectiveness across those 4 states would be higher than 10x.

Hi Sanjay - thanks for the close read! You're right that Figure 3 should read 95%, not 90% - we're working on correcting the figure and will update the post ASAP. Thanks again!