How well did EA-funded biorisk organisations do on Covid?

50

A few minutes later, he said:

[...] Be alert, wash your hands a lot where you're in areas where there's a lot of common touch surfaces you might be touching, avoid sick people as much as you can, but there's not much else you can do at this point.

He also said wearing masks would be going overboard in response to a question if people should buy masks. He said he travelled internationally "yesterday" (which would have been February 9th if the video was uploaded the day of the lecture) and didn't wear a mask. He said he saw people wearing masks with their noses out or with masks around their neck (implying it wouldn't be effective to tell people to wear masks?) and also that it's uncomfortable to wear an N95 for too long, so he wouldn't recommend the general public to wear a mask unless sick (in which case "maybe" they should wear a surgical mask).

I think his prediction and advice should probably be judged negatively and reflect poorly on him / Center for Health Security, but I'm not sure how harshly he/ CHS should be judged.

Edit: Also, at 42:10 he said "I do think that it's not containable in any country, it just appears to be so now." I think this was also wrong, since clearly some countries have managed to avoid major outbreaks.

15

He said he travelled internationally "yesterday" (which would have been February 9th if the video was uploaded the day of the lecture) and didn't wear a mask.

This seems totally okay to me, FWIW. In most places (e.g., London or the US), it would have seemed a bit overly cautious to wear masks before the end of February, no?

I think his prediction and advice should probably be judged negatively and reflect poorly on him / Center for Health Security, but I'm not sure how harshly he/ CHS should be judged.

I generally agree with that, but it's worth noting that it was extremely common for Western epidemiologists to repeat the mantra "you cannot do what Asian countries are doing; there's no way to contain the virus."

IanDavidMoss

37

It's actually worse than that. As I discovered when researching COVID giving opportunities for the FRAPPE donor group last year, Johns Hopkins experts explicitly recommended against wearing DIY masks in early March (a position reversed by the end of the month) and were not discouraging people from pressing ahead with travel plans as late as March 6. Sanjay had a phone call with them about a year ago in which he confronted them about these reversals, and they offered a sort of half-hearted defense.

I don't have any inside information about why CHS made the choices it did, but my naive view is that I agree with your comment that mistakes like these should reflect poorly on CHS. CHS's core competency may be more in the area of pandemic preparedness than dealing with the pandemic once it's already here, but their experts were quoted in the media a TON last spring and had significant ability (= responsibility) to shape the public conversation about COVID, particularly in the US. And yet lots and lots of people far less credentialed than CHS epidemiologists had correctly figured out by the first week of March that it was smart to wear a mask and to avoid being around others more than was absolutely necessary. It was left to pop-up initiatives led by non-medical experts like #Masks4All to upend the conventional wisdom about masks that had been propagated by the WHO and CDC. I feel like CHS ought to have been well positioned to challenge the prevailing narrative and was instead getting in the way at a time when it really mattered.

8

"And yet lots and lots of people far less credentialed than CHS epidemiologists had correctly figured out by the first week of March that it was smart to wear a mask"

Not sure how much this is an answer - as I said in a different response, the question isn't whether CHS was right (much less right about one specific thing,) but whether they did better overall than the other policy-influencing organizations.

21

Fwiw, my vague memory is that some other people at CHS, including Tom Inglesby (the director) did better than Adalja. I think Inglesby's Twitter was generally pretty sensible though I don't have time to go back and check. I'd guess that, like most experts, he was too pessimistic about travel restrictions, though. Maybe masks, too?

26

I do think CHS should get some credit for arguing for taking pandemic response very seriously early on. For example, I think Tom had some tweets arguing for pulling out all the stops on manufacturing more PPE in January 2020.

Note - I'm a bit biased since I was working on biorisk at Open Phil the first time Open Phil funded CHS.

18

I think the key question for CHS, or even Amesh specifically, is whether they outperformed other organizations, and by how much - i.e. were investments in CHS positive contributions to response overall. And they / he could have done almost arbitrarily bad in February, and still overall have vastly outperformed the alternatives.

And I think that it's clear that CHS's work was impactful in increasing preparedness in the US, even if the level of preparedness was far short of what was needed.

(Note that I have personal biases about this, and know many of the people at CHS.)

17

To be fair, the Johns Hopkins Center isn't just Adalja. I'm not aware of the list of things they do, but for instance, they kept an updated database in the early stage of the virus outbreak that was extremely helpful for forecasting!

10

If you're referring to what I think you are, it was a different group at Hopkins

2

Oh, you're probably right then!

6

In case anybody's curious: https://coronavirus.jhu.edu/map.html

13

Adalja also confidently predicted the infection fatality rate for the rest of 2020 to be around 0.6% (on the Sam Harris podcast) despite thinking the virus can't be contained (if true, this would have led to more ICU beds and oxygen shortages in lots of places). In reality, the IFR was more like 0.9% or higher for countries like the US and UK. Probably it was lower for countries with younger demographics, but I don't even think Adelja was basing his estimates on that.

(TBC, this isn't as big a mistake compared to other statements or compared to Ioannidis who completely disgraced himself throughout 2020 and ongoing, but I find it worth pointing out because I remember distinctly that, at the time when Adalja said this, there was a lot of fairly strong evidence for higher IFRs, including published estimates. I thought 0.6% seemed [edit]hard to defend, though I don't remember how much he flagged that there's a substantial chance it's significantly higher. Importantly, it would have been higher than it actually turned out to be, if Adalja had been right about "the virus can't be contained.")

RyanCarey

8

Was the prediction for infection fatality rate (IFR) or case fatality rate (CFR)? And high-income or all countries? Globally, the CFR is 2% (3.7M/173M), but the IFR is <0.66%, because <1/3 of cases were detected.

4

IFR (but back in February/March 2020, a lot of people called everything "CFR"). I think he was talking about high-income countries (that's what my 0.9% estimate for 2020 referred to – note that it's lower for 2020+2021 combined because of better treatment and vaccines). I'd have to look it up again, but I doubt that Adalja was talking about a global IFR that includes countries with much younger demographics than the US. It could be that he left it ambiguous.

Here's the Sam Harris podcast in question; I haven't re-listened to it yet.

2

Seems unlikely that <1/3 of all cases were detected at this point, since the recent outbreaks had far higher detection rates than the initial ones.

RyanCarey

4

Ah. If global IFR is worse than rich-countries' IFR, that seems to imply that developing countries had lower survival rates, despite their more favourable demographics, which would be sad.

10

It's still unclear, and the developing world detection and survival rates are a bit uncertain. I think you could probably get a decent approximation by looking at test positivity rates and testing volume compared to death rates over time in different countries, but I'm not going to put together the model to do it.

We're doing something related with IFR estimates by age at 1DaySooner, but using seroprevalence data, i.e. only where there is really good data for the estimate. I don't have results of that yet.

8

that seems to imply that developing countries had lower survival rates, despite their more favourable demographics, which would be sad.

This isn't impossible because there seems to be a correlation where people with lower socioeconomic status have worse Covid outcomes, but I still doubt that the IFR was worse overall in developing countries. The demographics (esp. the proportion of people age 70-80, and older) make a huge difference.

But I never looked into this in detail, and my impression was also that for a long time at least, there wasn't any reliable data.

From excess deaths in some locations, such as Guayaquil (Ecuador), one could rule out the possibility that the IFR in developing countries was incredibly low (it would have been at least 0.3% given plausible assumptions about the outbreak there, and possibly a lot higher).

24

A few opinions:

I think John Hopkins' advice has generally been well respected but that is just rumour on the grapevine, I cannot say exactly why I think that.

Nuclear Threat Initiative's Global Health Security Index said "international preparedness for epidemics and pandemics remains very weak" which seems correct. But the index also put the UK and the USA as the most prepared which seems incorrect (or at least gives a reason not to trust very shallow global indices). Eg yes the UK had a plan, tick box – but turns out it was not very good. Yes the UK did emergency exercises, tick box – but turns out it did not update plans based on the exercises. Etc

Not on your list but in this interview from Feb 2020, 80,000 Hours and the Future of Humanity Institute get a lot correct (eg need for social distancing) but they do both seem to disagree with the case for most travel bans, which seems incorrect in hindsight. See the intro where this is discussed.

Also not a bio org but EA Funded Our World in Data has done a good job on COVID data gathering and presentation.

9

One feature of the things that me and William have picked up on is that early on (say in Feb 2020 or earlier) advice coming from very respectable organisations was relatively poor.

I don’t think this should be seen as evidence that these organisations did badly (maybe a bit that they were over-confident) but that this was a very difficult situation to do things well in.

Max_Daniel

14

I don’t think this should be seen as evidence that these organisations did badly (maybe a bit that they were over-confident) but that this was a very difficult situation to do things well in.

I somewhat agree, but I think this point becomes much weaker if it was the case that at the same time when these organizations were giving poor advice some amateurs in the EA and rationality communities had already arrived at better conclusions, would have given better advice, etc.

I didn't follow the relevant conversations closely enough to have much of an inside view on how strongly the latter is true, but my impression is that many people in the EA/rationality communities (including ones who did follow the conversations more closely) think it's true. Even I am aware of some data points that seem to suggest such a conclusion (e.g., some conversations I remember).

FWIW, at least given what I know I think I find this less compelling as a 'vindication of EA/rationalist epistemics' or whatever than some other people seem to - I think the lessons we should learn depend on a number of additional things which I'm currently uncertain about:

Are we comparing like with like rather than cherrypicking? (I.e., comparing anecdotes of flawed advice from 'expert organizations' to anecdotes of ex-post correct advice from EAs/rationalists?)
- Ideally I'd want to know about the full distribution of views among 'expert organizations' and the full distribution of views among 'amateur EAs/rationalists who spent time looking into COVID things', and then compare those.
What about some other relevant groups? E.g., how "well" did pharma/vaccine companies do? What about academics developing tests? It does seem like at least some of these groups started taking the novel coronavirus pretty seriously in January already.
What should we think about the 'external validity' of EA/rationalist successes? Some people did well in predicting how COVID would play out, what recommendations are good, etc., in the environmental condition of "having significant spare time and being able to post and discuss views with like-minded people without facing a lot of other constraints". Would they still have done well if they had been in the environmental condition "expert making a public statement that needs to make sense to the general public" plus whatever other incentives these experts & organizations were subject to?
- (The above wording might sound like I'm thinking it was "easier" for EAs/rationalists to arrive at correct conclusions or give good advice when making statements we would hear about. However, in fact, I'm unsure about the "net effect" of incentives and other environmental conditions. Similarly, I don't mean to suggest that 'external validity' is necessarily poor, just that it seems worth thinking about before drawing strong conclusions.)

However, no matter the answers to these questions, your claim to me still sounds too generous to these organizations.

12

Hi, Yes good point, maybe I am being too generous.

FWIW I don’t remember anyone in the EA / rationalist community calling for the strategy that post-hoc seems to have worked best of a long lock-down to get to zero cases followed by border closures etc to keep cases at zero. (I remember a lot of people for example sharing this note which gets much right but stuff wrong: eg short lock-dock and comparatively easy to keep R below 1 with social distancing)

Pablo

8

long lock-down to get to zero cases followed by border closures etc to keep cases at zero.

Can you say more about the strategy you have in mind and give examples of the countries that implemented it? It's hard to judge whether EAs or rationalists advocated it from your characterization, which seems to build into it the relevant success conditions. Lots of countries had long lockdowns and strict border closures, but failed miserably at bringing cases down to zero or anything remotely approaching zero. Furthermore, whether this approach was best all-things-considered should obviously also take into account the enormous costs associated with these measures (in all of the countries that implemented long lockdowns and strict border closures).

Tsunayoshi

21

weeatquince's is sharing a widely held view, i.e. that eradication is superior to containment in health and economic outcomes, see e.g. this analysis. The idea itself is plausible, since a successful lockdown allows complete reopening of the internal economy afterwards.

Sample size is however small, especially when it comes to non-island countries. I only know of two non-island countries that seriously went for eradication coupled with border closures, namely Vietnam and Israel. Israel gave up at one point when cases started to rise (which is why it is listed among the containment countries in the analysis above) , but Vietnam succeeded (although it had to restrict travel heavily domestically as well). Personally, I believe it is a suboptimal strategy for non- authoritarian, non-island countries.

I think their original point stands though, that EA/rationalists did not seem to entertain the idea of eradication enough, but probably neither did biorisk organizations last year.

Pablo

5

Thanks for the clarification.

I feel that this discussion is not framed correctly. Yes, successful eradication is superior to containment in health and economic outcomes. This is a pretty weak claim that lots of people can agree with who otherwise differ considerably in their policy proposals. But the original claim was that EAs and rationalists hadn't advocated for long lockdowns and border closures, and that this was relevant for retroactively assessing their performance. The plausibility of the latter claim must be evaluated by considering all the countries that implemented long lockdowns and border closures, and not just the tiny minority that were successful in attaining (near-)eradication by adopting those measures.

I took a quick look at the study you shared. Their analysis compared covid deaths, GDP growth and lockdown stringency in two groups of OECD countries during the first twelve months of the pandemic, and offered this as their original contribution to the study's main thesis that countries which favored elimination had better health and economic outcomes than countries which favored mitigation (the rest of the study is a brief and unsystematic summary of some of the relevant literature). It turns out that the group which supposedly favored elimination consists of just five nations, four of which are islands and the fifth of which (South Korea) shares borders with a single county which has been completely isolated from the rest of the world for decades.

Let's pause for a moment and consider how quickly this kind of evidence would have been dismissed if it had been presented in support of a politically inconvenient conclusion. Yet here it is offered, in the world's most prestigious medical journal, to establish that "elimination, not mitigation, creates best outcomes for health, the economy, and civil liberties".

For what it's worth, I personally have no strong views on how the pandemic should have been handled. (My only strong meta-view is that decisions should have been based on explicit cost-effectiveness analyses, which were surprisingly absent from most policy discussions.) My impression is that EAs and rationalists—to which I would add the Metaculus community—did somewhat better than most experts, but this assessment is based mostly on comparing their performance on simple factual questions or issues involving basic sanity. Here, instead, we are told to downgrade our estimation of how well EAs and rationalists performed because, apparently, they did not advocate for the entire world a policy that was successful in only a handful of geographically highly isolated countries. I don't find that plausible.

4

I don’t think I can help much with answering these questions.

I was thinking of counties like Australia and New Zealand and Taiwan. But whether or not the strategies adopted in these places was actually optimal or best with the available information or applicable to most countries that are not islands or had a high chance of failure – I cannot say!

All I can say is that there is at least one plausible strategy that seems to have worked well in at least some countries and I personally don’t really remember it being discussed within the EA space a year ago.

Feel free to draw what conclusions or analysis you will from that.

the enormous costs

Just to add, I expect (but I might be wrong) that these countries have had lower welfare and economic costs than most other places.

2

I discuss the GHS index at greater length in my answer.

2

Thank you :-)

Max_Daniel

2

they do both seem to disagree with the case for most travel bans, which seems incorrect in hindsight

Can you say more about this, or point me to good resources off the top of your head? I didn't follow the conversation on travel bans, and currently don't have a good sense of what people think about them.

In particular, is the rough basis for "incorrect in hindsight" the impression that travel bans were a potentially essential part of the successful containment strategy of some countries such as New Zealand? Or are there other major considerations as well?

18

My understanding is that travel bans were widely believed to have greater costs than benefits before COVID. There are various quotes along those lines described in the (rather cynical) Lessons From the Crisis post on the topic of border closures.

In February 2020, I believed border closures weren't worth it. I thought they disincentivised countries from being transparent about emerging outbreaks (because said countries would face economic punishment via closed borders) and could only slow down the spread of a disease, not stop it. While I'm still not entirely sure about the relative benefits of open reporting vs. slowed spread, I was definitely underestimating the benefits of the latter. Evidence from Vietnam and New Zealand shows that early and strong international border controls can indeed slow the spread to the point where local outbreaks don't spiral beyond easy containment.

To link you to some resources, a September 2020 Cochrane meta-analysis, Travel‐related control measures to contain the COVID‐19 pandemic: a rapid review, concluded that:

There was insufficient evidence to draw firm conclusions about the effectiveness of travel‐related quarantine on its own. Some of the included studies suggest that effects are likely to depend on factors such as the stage of the epidemic, the interconnectedness of countries, local measures undertaken to contain community transmission, and the extent of implementation and adherence.

The February 2021 paper Evidence of the effectiveness of travel-related measures during the early phase of the COVID-19 pandemic: a rapid systematic review (PDF link) (which I found in the recent Vox article Vietnam defied the experts and sealed its border to keep Covid-19 out. It worked.) concluded that:

The study finds that the domestic travel measures implemented in Wuhan were effective at reducing the importation of cases internationally and within China and that additional travel restrictions were also likely important. The study also finds that travel measures are more effective when implemented earlier in the outbreak.

Max_Daniel

2

Thank you, very helpful!

AnonymousEAForumAccount

20

Great question, and I look forward to following this discussion!

A tangential (but important in my opinion) comment… You write that “EA funders have funded various organisations working on biosecurity and pandemic preparedness”, but I haven’t seen any evidence that EA funders aside from Open Phil have funded biosecurity in any meaningful way. While Open Phil has funded all the organizations you listed, none of them have been funded by the LTFF, Survival and Flourishing Fund, the Centre on Long-Term Risk Fund, or BERI, and nobody in the EA Survey reported giving to any of the organizations.

The LTFF has admittedly made some small biosecurity grants (though as a reference it has granted ~19x more to AI), and FHI (which has relatively broad support from EA and/or longtermist funders) does some biosecurity work. But broadly speaking, I think it’s a (widely held) misconception that EA donors besides Open Phil were materially prioritizing biosecurity grantmaking prior to the pandemic.

20

This is meta, but I'm sort of hoping the recent Long Term Future Fund grant to CSER on "Ensuring the lessons learnt from COVID-19 improve GCR prevention and mitigation in the future" will generate some answers related to this question. I'd be very interested to see some postmortem work within that project on the actions taken by these major biosecurity organizations.

Linch

11

In a throwback to articles like this one, including one critique that tessa summarized as:

Open Phil is offering large grants to a few carefully-selected parties, giving them outsize impact (and a diversity of perspectives and research focuses is good the the field)

I'm also interested in whether Open Phil funded parties did better or worse than either a) existing big players (as opposed to a possibly different bar of being measured "against reality") or b) a more diverse/"democratic"marketplace of ideas among credentialed biosecurity folks.

11

Another slightly tangential but very similar question that came up in conversation I had recently is:

"How well have EA-funded orgs built on the momentum created by the COVID-motivated global interest in GCRs (global catastrophic risks) to drive policy change or other changes to help prevent GCRs and x-risks"

I could have imagined a world where the entire longtermist community pivoted towards this goal and at least for a year or two and focused all available time skill and money on driving GCR related policy change – but this doesn’t seem to have happened much. I could imagine the community looking back at this year and regretting the collective lack of action.

The organisation where I work, the APPG for Future Generations pivoted significantly, kickstarted a new Parliament Committee on risks and I wrote a paper on lessons learned from COVID which had significantly government interest and seems to have driven policy change (writeup forthcoming).

But beyond that there has definitely been some exciting stuff happening. I know:

CSER are starting a lessons learned from COVID project, although this is only just getting started.
FHI staff have submitted a some evidence to parliamentary inquiries (example).
The CLTR (funded by the EAIF) has launched a report on risk (I'm unsure if this was a change in direction or always the plan).
No more pandemics (not funded) was started.

This stuff is all great and I am sure there is more happening – but my general sense is that it is much less than and much slower than I would have expected.

I also loosely get the impression (from my own experience and that of 2-3 other orgs that I have talked to) that various EA funders have been disinterested in pivoting to support lessons learned from COVID focused policy work, some of which could scale up quite significantly, and that maybe funding is the main bottleneck for some of this (I think funding for more policy work is a bottleneck for all of the orgs listed above except FHI).

[Disclaimer – I will be bias given that I pivoted my work to focus on COVID lessons learned and policy influencing and looked for funding for this.]

Linch

9

Measured against reality, a pandemic happened and a lot of people died for preventable reasons, so in at least one sense clearly we/they didn't do good enough. :/

riceissa

7

Vipul Naik asked a similar question near the beginning of the pandemic.

5

The Global Health Security Index looks like a misfire. This isn't directly about performance during the pandemic, but Nuclear Threat Initiative, funded by Open Phil for this purpose (h/t HowieL for pointing this out) and collaborating with the Johns Hopkins Center for Health Security, made the 2019 Global Health Security Index which seems invalidated by COVID-19 outcomes and may have encouraged actors to take the wrong moves. This ThinkGlobalHealth article describes how its ratings did not predict good performance against the virus. The article relies on official death counts rather than excess mortality, but I made that correction and reached similar results.

Looking through the index, there are some indicators which don't make sense, like praising countries for avoiding travel restrictions (which is perverse), praising them for having more ethical regulations against surveillance and clinical trials (which may be ethically justified but is more likely to make it harder to fight a pandemic), and praising them for gender equality (a noble sentiment but not directly relevant to pandemics).

Even cutting some of those dubious measures out, I found the index was not predictive of excess mortality. In general it appears that effective pandemic response is not about preparation and this may have been systematically overlooked by EA efforts and funding recipients in the realm of biorisk.

Some people have also criticized the index for rating China moderately highly on prevention of pathogen release, considering that COVID-19 came from China, but considering that COVID-19 is just one data point of virus emergence or lab leak and that China is a very large country I don't think this is right.

"effective pandemic response is not about preparation"

12

FYI - my impression is that pandemic preparedness is often defined broadly enough to include things like research into defensive technology (e.g. mRNA vaccines). It does seem like those investments were important for the response.

2

Hm, certainly the vaccine rollout was in hindsight the second most important thing after success or failure at initial lockdown and containment.

It does seem to have been neglected by preparation efforts and EA funding before the pandemic, but that's understandable considering how much of a surprise this mRNA stuff was.

9

I think research into novel vaccine platforms like mRNA is a top priority. It's neglected in the sense that way more resources should be going into it but also my impression[1] is that the USG does make up a decent proportion of funding for early stage research into that kind of thing. So that's a sense in which the U.S.'s preparedness was prob good relative to other countries though not in an absolute sense.

Here's an article I skimmed about the importance of govt (mostly NIH) funding for the development of mRNA vaccines. https://www.scientificamerican.com/article/for-billion-dollar-covid-vaccines-basic-government-funded-science-laid-the-groundwork/

Fwiw, I think it's prob not the case that the mRNA stuff was that much of a surprise. This 2018 CHS report had self-amplifying mRNA vaccines as one of ~15 technologies to address GCBRs. https://jhsphcenterforhealthsecurity.s3.amazonaws.com/181009-gcbr-tech-report.pdf

[1] Though I'm rusty since I haven't worked directly on biorisk for five years and was never an expert.

7

I second the impression that it's not that much of a surprise. For example, CEPI was founded with a goal of accelerating vaccine development against the WHO R&D Blueprint priority diseases and according to their R&D webpage:

In September 2017, CEPI requested proposals for vaccine platform technologies that enable rapid vaccine development, elicit rapid onset of immunity, and whose production can be scaled-up quickly to respond to outbreaks of Disease X. CEPI funded three platform technologies: a vaccine printer, molecular clamp platform, and a self-amplifying RNA vaccine platform.

I think it was a surprise that non-self-amplifying mRNA vaccines work as well as they do (mRNA is more immunogenic than predicted, I guess, at least for COVID?). 18 months ago, I don't think I would have bet on mRNA platform vaccines as the future over DNA or adenovirus vaccines.

9

Hello, Thank you for the interesting thoughts. The comments on the GHS index are useful and insightful.

Your analysis of COVID preparation on Twitter is really really interesting. Well done for doing that. I have not yet looked at your analysis spreadsheet but will try to do that soon.

To touch on a point you said about preparation, I think we can take a bit more of a nuanced approach to think about when preparation works rather than just saying "effective pandemic response is not about preparation". Some thoughts from me on this (not just focused on pandemics).

Prevention definitely helps. (It is a semantic question if you want to count prevention as a type of preparation or not). The world is awash with very clear examples of disaster prevention whether it is engineering safe bridges, or flood prevention, or nuclear safety, or preventing pathogens escaping labs, etc.
The idea that preparation (henceforth excluding prevention) helps is conventional wisdom and I think I would want to see good evidence against this to stop believing in this.
Obviously preparation helps in the small cases, talk to a paramedic rushing to treat someone or a fireman. I have not looked into it but I get the impression that it helps in the medium cases, eg rapid response teams responding to terror attacks in the UK / France seem useful, although not an expert. On pandemics specifically the quick containment of SARs seems to be a success story (although I have not looked at how much preparation played a role it does seem to be a part of the story). There are not that many extreme COVID-level cases to look at, but it would be odd if it didn’t help in extreme cases too.
The specific wording of the claim in the linked article headline feels clickbait-y. When you actually read the article it actually says that competence matters more (I agree) and also that we should focus more on designing resilient anti-fragile systems rather than event specific preparation. I agree but I think that designing systems that can make good decisions in a risk scenario is a form of preparation.
I do agree that your analysis provides some evidence that preparation did not help with COVID. I am cautious of the usefulness of this evidence because of the problems with the GHS – e.g. the UK came near top but basically had no plan to deal with any non-influenza pandemic that I have identified.
A confusing factor that might make it hard to tell if preparation helped is that, based on the UK experience (eg discussed here) it appears that having bad plans in place may actually be worse than no plans.
Evidence from COVID does suggest to me that specific preparation does help. Notably countries (E Asia, Australasia) that had SARs and prepared for future SARs type outbreaks managed COIVD better.

So maybe we can say something like:
Prevention definitely helps. Both event specific preparation and generally building robust anti-fragile decision systems are useful approaches but the latter of those is more underinvested in. However good leadership is necessary as well as preparation and without good leadership (which maybe rare) preparation can turn out to be useless. Furthermore bad preparation, such as poor planning, can potentially hinder a response more than no preparation.

Does that seem like a good summary and sufficiently explain your findings.

I am thinking about doing more work to promote preparation so useful to hear if you disagree.

2

Prevention definitely helps. (It is a semantic question if you want to count prevention as a type of preparation or not)

I don't think most people would consider prevention a type of preparation. EA-funded biorisk efforts presumably did not consider it that way. And more to the point, I do not want to lump prevention together with preparation because I am making an argument about preparation that is separate from prevention. So it's not about just semantics, but precision on which efforts did well or poorly.

The idea that preparation (henceforth excluding prevention) helps is conventional wisdom and I think I would want to see good evidence against this to stop believing in this.

Conventional wisdom is worth little when it is the product of armchair speculation rather than experience. If people live through half a dozen pandemics and still have that conventional wisdom then we can have a different conversation.

On pandemics specifically the quick containment of SARs seems to be a success story (although I have not looked at how much preparation played a role it does seem to be a part of the story)

Wouldn't preparation seem to be a part of the story of COVID-19 outcomes given a similarly superficial level of inquiry?

I agree but I think that designing systems that can make good decisions in a risk scenario is a form of preparation

Forget semantics. Did EA funding efforts and recipients design systems that made good decisions about COVID-19? Did anyone who talked about "pandemic preparation" pre-2020 use the term to encompass the design of systems like that?

A confusing factor that might make it hard to tell if preparation helped is that, based on the UK experience (eg discussed here) it appears that having bad plans in place may actually be worse than no plans.

Well you can't just define preparation as "good plans", that's a no-true-Scotsman argument. If you have some way of ensuring that your preparation will be good preparation then it's a different story.

Evidence from COVID does suggest to me that specific preparation does help. Notably countries (E Asia, Australasia) that had SARs and prepared for future SARs type outbreaks managed COIVD better.

That isn't necessarily due to physical preparation, it could easily be intangible changes in the culture and political system, granting that there is in fact a causal connection as opposed to East Asia and Australasia just being better at this stuff.

iirc there was a study which found that American cities that lived through the Spanish Flu (1919) suffered less death early in the COVID19 outbreak. Cannot find the study now but if it's really true then that would be hard to explain through preparation.

Does that seem like a good summary and sufficiently explain your findings.

I'm not sure exactly what anti-fragile means but that doesn't sound right, decision systems in the US/UK for instance didn't fall apart, they were just apathetic and unresponsive to good ideas just like they are for mundane problems that aren't big crises. In other words they calmly kept operating the way they always do.

I don't have reason to believe that there is a positive interaction between good leadership and good preparation. Maybe good preparation and good leadership act more as substitutes for each other rather than compliments.

Not sure it is useful to say 'prevention helps' since we cannot wish away viruses, we can only take measures to attempt to prevent viruses from emerging, and while those measures may be cost-effective it is a different conversation to which I have nothing to contribute.

I would summarize my view by saying that smart actions by government and civil society in the moment make the most difference, and if plans and preparation are to be helpful they will have to be done in careful ways to avoid the failures documented during COVID-19.

4

I don't think most people would consider prevention a type of preparation. EA-funded biorisk efforts presumably did not consider it that way. And more to the point, I do not want to lump prevention together with preparation because I am making an argument about preparation that is separate from prevention. So it's not about just semantics, but precision on which efforts did well or poorly.

I think it actually is common to include prevention under the umbrella of pandemic preparedness. for example, here's the Council on Foreign Relation's independent committee on Improving Pandemic Preparedness: "Based on the painful lessons of the current pandemic, the Task Force makes recommendations for improving U.S. and global capacities to deliver each of the three fundamentals of pandemic preparedness: prevention, detection, and response. " Another example: https://www.path.org/articles/building-epidemic-preparedness-worldwide/

So it might be helpful to specify what you're referring to by preparation.

5

It seems fair to call avoiding travel restrictions a dubious measure in hindsight, but circa 2019 it strikes me as a reasonable metric to put under "compliance with international norms". There was an expert consensus that travel norms weren't a good pandemic response tool (see my other comment) and not implementing them is indeed part of complying with the WHO IHRs.

I am not totally sure that compliance with international norms a good measure of national health security! However, the according to the Think Global Health article you linked on Twitter, even the WHO Joint External Evaluations weren't well-correlated with COVID-19 deaths. (Those evaluations are how the prevention / detection / response capacity are measured in the Global Health Security Index, which then adds measures on health system / compliance with norms / risk landscape.)

-1

Edit: I've reposted this comment as an answer, and am self-downvoting this.

[This comment is no longer endorsed by its author]

6

Note that Open Phil funded this project. https://www.nti.org/newsroom/news/nti-launch-global-health-security-index-new-grant-open-philanthropy-project/

2

[Edit – moved comment to answer above at suggestion of kbog]