Obviously it was a triumph: the fastest vaccine development, approval, and rollout in history (by a factor of 5). We're up to 2.5bn doses in arms, out of say 12bn. And we got several good ones! Huzzah!
Obviously it was a catastrophe: through dumb inaction and a comedy of errors, we squandered the chance to suppress the virus. 4 million people are confirmed to have died of (or with) COVID - and given unbelievable underreporting that might be actually 12 million - and given Delta's momentum 20 million by the end is not unlikely.* This is despite this virus being easy mode: unlike 1918, very few of the deaths were among the young frontline workers keeping the healthcare and delivery systems working; unlike SARS 1, post-viral disability is relatively rare. This is just the present pandemic and doesn't count future deaths from letting the thing become a permanent fixture.
How did we do? How can we even answer that question?
When I talk about whether a given country's response to COVID was a success or a failure, smart friends reply "governments had to balance the tradeoffs, so what looks like failure is really just compromise between multiple objectives (like economic activity)", "it's easy to say the optimal response in hindsight", that "it's difficult to compare different countries because of the different distances from China, wealth, state capacity".
For instance, they think the UK did ok. They can think this because they choose to compare to the average actual response (never mind that the UK is top 20 in deaths per capita). But what would the best possible response look like? What did our institutions stop us from getting?
Vax
Any self-respecting COVID rant must foreground vaccination. It is the solution, where other policies just buy time, or else consume old or disabled people.
We underinvested, and prevented market investment.
The EU paid $14 per Pfizer dose. What was it really worth? The current black market price for Pfizer is about $500. But that's a gross underestimate of the shadow price, since you get almost zero quality assurance or liability from darknet dealers. (You might still get the travel passport, depending on how weak your country's infosec is.) One proper estimate of the per-vaccine social benefit is $6000.
So we should have spent trillions in massive pre-purchase, on every credible vaccine. (The much-praised Operation Warp Speed and its equivalents elsewhere only pre-purchased about 2bn out of the necessary 12-14bn, and did so shockingly late, in August 2020.)
$3000 x 14 bn doses pays for a lot of overtime on microlipid machine assembly (which was the bottleneck on mRNA vaccine supply last year).
(That's assuming that you continue to ban vaccine markets, believing, as you apparently do, that fairness is worth the early death of millions. Another way to fund supply expansion for the global south is to just not get in our own way.)
The rich world defected, duh
16% of the world bought 70% of the vaccines. What force on earth could stop them? None, so we needed the massive supply increases, which were effectively banned.
This was not even good selfishness: it guaranteed the emergence of new strains in the global south.
This is the real evil of the EU procurement. They want to harm their own by delaying 4 weeks, to look strong? Well, that's one thing. But had they done a pre-purchase in March 2020, then global supply could have scaled up, so that the inevitable snatch away from the global south was completely balanced out by expanded supply
What did we do? Overall, about 2bn doses ordered by August 2020, i.e. 5 times too little, 5 months too late.
The strange death of human challenge trials
Probably the biggest mistake was not intentionally infecting vaccinated volunteers. This could be done in 1 month, vs 6.5 months for the ecological trials that the entire world did out of misguided PR ethics. (2.5 is probably more realistic given signups, approvals, and big pharma's slow data analysis and reporting. That's still hundreds of thousands of lives.)
1DaySooner wrote a letter. The world's foremost consequentialist signed. The world's foremost deontologist signed. Two of the most prominent bioethicists in the world signed. 15 Nobellists signed. Dozens of philosophers who otherwise agree on extremely little signed. But they're unethical.
Rarely do I so strongly feel the boot of others on my neck, and humanity's neck.
The one distinctively courageous thing about the UK - the human challenge trials which got 40,000 volunteers - actually eventually started!.. In January 2021, with n=90.
I am extremely puzzled why China or one of the other ahem non-individualist governments didn't do these.
What fraction of all vaccines were wasted?
- by excessive dosing. No sign of the dosing regimes changing despite strong evidence. 50-75%?**
- by bad needle design. 10%-30% more.
- by rigid queueing leading to needless expiry of doses. 1%?
- by not checking anyone for prior antibodies before using a dose on them. Maybe 1.1 bn people have had COVID (confirmed case count * 6.2 factor undercount). So something like a 14% risk of wasting any given dose. Actually not clear if the overhead of testing and delaying vaccine appointments would swamp the saved doses from this. Also supposedly 9% of natural COVID patients don't have antibodies afterwards, and 5% lose them within 6 months(!)
So maybe most of em.
Stuff I'm not counting as waste
- Logistics failures (cold chain faults, delivery errors, fill-to-finish spills, breakages). It's a hard problem! Say 5%.
- The experimental dermal injections which might quintuple supply.
- I won't talk about single-dosing the two-dose vaccines because although it looks like a win (e.g. Pfizer first dose: 94% effect against hospitalization from Delta, second dose 96%) there were credibly thought to be possible tradeoffs re: selection pressure. (No longer?)
The happy timeline: what to compare to
Taking the Moderna vaccine as a model:
- Jan 2020: Beijing doesn't lie and cover-up for 2 weeks. They don't arrest or rectify. The genome goes out Jan 6th 2020 at the latest, 5-10 days early.
- Moderna vaccine (etc) still designed in 2 days, prototyped in a month.
- Feb 2020: Massive pre-purchase of all credible candidates ex ante, before the effectiveness results. Ideally billions each. New factories minted, Huoshenshan-style. Lead time on dozens of new microfluidics machines starts February, conceivably ready by August but maybe not til 2021. Whatever the next limiting factor is, we also identify it and start the lead time in February. Tap the existing spare capacity for 3bn doses.
- I'm going to assume Phase I was already done as quick as possible, in 63 days.
- April 2020: Challenge trial instead of ecological trials for Phase II and III. This would plausibly have taken 2.5 months instead of 6.5 months.
- July 2020: Approval ready to go as soon as Phase III results out. (Unlike say the 3 week FDA delay.)
- Approval sharing between UK / EU / US. Saves the EU three months on AZ.
- New microfluidics machines spun up and debugged at some unknown point between August 2020 and August 2021. (The UK's brand new vaccine factory will end up taking 16 months to build and ramp.)
What does that get us, and at what cost?
We approve in August 2020 instead of December. There were 700,000 confirmed deaths then. In December, when the earliest vaccinations began in our dark timeline, there were 1.4 million. (Again, maybe more like 3m.)
We needed 5 times more vaccines, preferably within 18 months. Was that even possible? One good simulation study looks at what it would take to scale mRNA production from 2bn to 8bn, and says $17bn for the chunky-dose vaccines. This decent study says $36bn and 2 years to scale up. (But that's just the mRNA - they don't seem to include the new microfluidics machines! Not to mention procurement and logistics, i.e. 90% of the total cost.) But add in the dose stretching and zero-dead-volume syringes, and we're done in a year. I am confused by how few experts have produced estimates for this cost - one guy worked on both of the extant estimates I quote above - but then this whole pandemic has been an abyss for cost-benefit arguments.
Here's a terrible guess:
- The UK spent $16.3bn on doses, facilities, R&D and logistics so far. £1.3bn of that was the sticker price of the doses. That got them 267m doses. So naively, 12/0.267 * 16.3 = $732bn for 12bn doses. That would imply totally subsidising 12/0.267 = 44 different vaccines though, which is a bit much even for me. This also includes converting a plant into a vaccine factory, but let's say 44 of those is good. Subtract the $860m R&D subsidy first then, then add 5 lots on: 12/0.267 * (16.3 - 0.86) = $693bn + 5 x 0.86 = $698bn.
- IQVIA estimate a world vaccine spend of $53bn in 2021. That clearly can't be the all-inclusive price. If 90% of costs are not doses but logistics, vaccinators, and procurement then it roughly accords with my guess.
- It's hard to see total charitable spending on R&D and purchase. For once it's probably not too bad to ignore it.
- Again, there were bottlenecks in the manufacturing which would not have allowed for 5 times the money to get 5 times the juice (within a year anyway). But $0.7tn is a decent guess.
- If that looks like a lot, consider what the ongoing stimulus took.
The executive bodies of the world failed to pre-purchase enough, early enough to overcome an unknown number of bottlenecks. The health regulators of the world vetoed challenge trials and in some cases added months of pennywise bickering and font size rejections.
I'm not saying that the above justifies FDA abolition. I'm not even saying that we'd have been better off last year if the FDA, MHRA, SAMR, PMDA, and EMA simply did not exist. (We got lucky that the first vaccines were safe and so effective.) But it does permanently alter the balance of trust for me.
A mandate to cry wolf?
An obvious objection is that we didn't know COVID was going to be COVID in February. Would it really have been optimal during swine flu (2009)?
First off, who's "we"?
Secondly: fine, start it in March and scale down the savings by a quarter.
Third: we certainly knew that we were underprepared for a pandemic. That corrective investment had to happen at some point. Physical plant does depreciate, but over a decade, so helping us deal with the 10-30% risk of another pandemic.
Finally, building 10 more mRNA vaccine factories would be amortized pretty quickly even if no other pandemic happens. Thus, the above is not just hindsight bias or absurd overreaction.
Shut up and calculate
Probably best to follow the script here if you can.
- Average 5 QALYs lost per COVID death, supposedly (seems low).
- We're not getting away without 2 million confirmed deaths (~5 million excess). So say 2 million (~5 million) saved as of now. Then project forward to overall pandemic deaths, which Metaculus has annoyingly stopped predicting so fill in your own number. Mine is 20 million, or 15m saved. So 25 mega QALYs on deaths up to June 2021, or conceivably 75 mega QALYs including all future COVID deaths.
+ - QALYs per averted infection: 1.51?? Way way too high; it counts secondary infections. Table 2 suggest that COVID is 0.43 QALY loss for symptomatic, and 0.5 for the hospitalized. Maybe 1bn averted by the happy path. Median duration of symptoms 27 days. 83% symptomatic supposedly. So 11.5 mega QALYs from symptoms averted.
+ - Long COVID burden. Very unclear. Martin et al use ARDS disutility as their estimate for long COVID: 0.32 QALY loss. This seems like too much but I don't have anything better. 0.14% to 2% risk of permanent damage. So maybe 1.4 mega QALYs per year for the next 5?? years = 7??.
+ - QALY loss from fear, uncertainty, and lockdown. Surprisingly few estimates of how much last year sucked for uninfected people, maybe because it's a tasteless question. I seriously think I have to go with Bryan Caplan's twitter poll(!) At least 2 months less for say half of the world population, so ~151 mega QALYs. (I'm surprised too, but note that this isn't about just not being able to go on holiday: it's about the whole car crash, including the frustration at the response! And deaths would have been far higher without NPIs, so this isn't to say "lockdown was worse than COVID".)
Extremely rough cost per QALY: $2850. Treat this as you would treat working by a very drunk Enrico Fermi. (Run the script and change whatever seems off to you; do parameter sweeps of the dodgy bits.)
Another smart friend dismissed the above argument. He just couldn't believe that politicians would leave such a deal on the table - and so really it wasn't possible to speed it up so much. Sadly his cynicism is pointing in the wrong direction: the last year has to make you realise how incompetent our rulers are. (Not malign.)
Would such people really have been willing to buy a high-variance portfolio of hits-based vaccines, ex ante? to look like they wasted half of the public spend? To risk something for the public good? Yeah, some of them!
Bonfire of the Authorities
Is there any public organisation which can be proud of last year?
Several times (fomites, masks, asymptomatic spread, long COVID, schools, lab-leak) we saw dramatic reversals of global policy, based not on dramatic new evidence, but on shifts in how elite factions decided to interpret the evidence.
I'm not going to enumerate all of the errors, lies, feet dragging, or negative-sum bullshit. For that I can't recommend this Twitter account enough, whose supply of accountability and context is what people think journalists do. Here's a sprinkling:
- WHO and all of public health on masks
- The FDA refused to license any rapid home tests until November because their processes were designed for diagnostic tests (rather than the intended monitoring tests) and had no exception clauses.
- "FDA insiders say the agency and its approximately 17,000 employees were dark for the four-day Thanksgiving holiday, including those working on the vaccine approval."
- The EU's sluggishness, bullying, pandering to risk aversion, and total lack of (short-term) accountability for its own decisions. AZ approval came three months after UK approval, Pfizer three weeks after. Supposedly this was mostly taken up with haggling prices down from crazy low to crazy low. (EDIT: no, that was the procurement process, not the approval process.)
- Behavioural science shown to be unfit for purpose. Systematic and fatal overestimate of risk compensation and "behavioural fatigue". This continues, and will continue.
Many of these processes were run at normal speed; COVID cases weren't reported over the weekend; in some places vaccinations paused for the weekend; pausing research grants; pausing vaccine approval!
It's enough to make a ghost want to hoist the black flag and join the revolt.
The above requires no new science, no new social structures, no public executions. It just needed some courage, needed a small number of people to act as if they were at war, as if trillions of dollars were at stake, as if millions of people were.
This greatly upgrades my estimate of institutional decision-making as a cause, and in particular getting people willing to take (personal reputation) risks as high up as possible, everywhere.
* What about the proper EA units though?: One estimate says 32 million DALYs lost (Jan 2020 - Apr 2021), about the same as malaria. This seems like a serious underestimate to me, probably because they use confirmed deaths (understandable), lowball long COVID (eh), and say nothing about the QALY hit of COVID restrictions or long-term economic scarring(!).
** You might not believe that these things are arbitrary and then locked-in by regulators. Here's another instance: the prescribed "3 week" gap between doses is not even close to optimal. This is a subtler kind of waste: wasting immunity rather than inoculum. Luckily the shortages forced us to fix this one.
EDIT: Did the QALY estimate as a Python script. Not much change.
Note: I'm not a engineer, I'm not an economist, I'm not a pharmacologist, I'm not an epidemiologist. There will be something wrong with the above, no doubt in misunderstanding the nature of vaccine engineering bottlenecks.
I hope it's clear just how many of these numbers are guesses, as opposed to the sanctified guesses of peer-reviewed economics. (Martin et al: "Given the longer timeframe of our model... we felt 0.10 is appropriate".) The QALY estimate could be off by a factor of 10. The key parameter, the microfluidics machine lead time, is based on how long the Braintree facility is taking and little else.
On Human Challenge Trials (HCTs):
Disclaimer: I have been completely plugged out of Covid-19 stuff for over a year, definitely not an expert on these things (anymore), and definitely speaking for myself and not 1Day Sooner (which is more bullish on HCTs)
I worked for 1Day sooner last year as one of the main people investigating the feasibility and usefulness of HCTs for the pandemic. At least back then (March 2020), we estimated that it would optimiscally take 8 months to complete the preparations for an HCT (so not even the HCT itself). Most of this time would be used for manufacturing and approving the challenge virus, and for dose-finding studies. (You give people some of the virus and check if it's enough to induce the disease, then repeat with a higher dose etc.)
I think in a better world, you can probably speed up the approval for the challenge virus, and massively parallize dose-finding to be less lenghty. Not sure how many months that gets you down to, but the 2.5 months for preparation + the actual HCT you assume seem overly optimistic to me. I still think HCTs should have been prepared, but I'm not sure how much speed that would have actually gained us. More details here in the section "PREPARATORY STEPS NEEDED FOR HUMAN CHALLENGE TRIALS" (free access)
There was also some discussion of challenge trials with natural infection (you put people together with infectious people who have Covid-19), which might get around this? But I don't know what came out of that (I think it wasn't pursued further?). Not sure how logistically feasible that actually is. (I think it would at least be more difficult politically than a normal HCT.)
Don't think this changes the general thrust of your post, but wanted to push back on this part of it.
(There's some chance I missed followup work, perhaps even by 1Day Sooner itself, that corrects these numbers, in which case I stand embarrassed :) )
Thanks! That is good knowledge.
This is an important question, because we want to find out what was done right organizationally in a situation where most failed, so we can do more of it. Especially if this is a test-run for X-risks.
There are two examples that come to mind of government agencies that did a moderately good job at a task which was new and difficult. One is the UK's vaccine taskforce, which was set up by Dominic Cummings and the UK's chief scientific advisor, Patrick Vallance and responsible for the relatively fast procurement and rollout. You might say similar for the Operation Warp Speed team, but the UK vaccine taskforce overordered to a larger extent than Warp Speed and was also responsible for other sane things like the simple oldest-first vaccine prioritization and the first doses first decision, which prevented a genuine catastrophe due to the B117 variant. (Also credit to the MHRA (the UK's regulator) for mostly staying out of the way.)
See this from Cummings' blog, which also outlines many of the worst early expert failures on covid, and my discussion of it here:
This plan later went on to succeed and significantly outperform expectations for rollout speed, with early approval for the AZ and Pfizer vaccines and an early decision to delay second doses by 12 weeks. I see the success of the UK vaccine taskforce and its ability to have a somewhat appropriate sense of the costs and benefits involved and the enormous value of vaccinations, to be a good example of how it's institution design that is the key issue which most needs fixing. Have an efficient, streamlined taskforce, and you can still get things done in government.
The other example of success often discussed is the central banks, especially in the US, which responded quickly to the COVID-19 dip and prevented a much worse economic catastrophe. Alex Tabarrok:
I really would like to be able to agree with Tabarrok here and say that, yes, choosing the right experts and protecting them from democratic feedback is the right answer and all we would need, and the expert failures we saw were due to democratic pressure in one form another, but the problem is that we can just look at SAGE in the UK early in the Pandemic or Anders Tegnell in Sweden, who were close to unfireable and much more independent, but underperformed badly. Or China, which is entirely protected from democratic interference and still didn't do challenge trials.
Just saying the words 'have the right experts and prevent them from being biased by outside interference' doesn't make it so. But, at the same time, it is possible to have fast-responding teams of experts that make the right decisions, if they're the right experts - the Vaccine Taskforce proves that. I think the advice from the book 10% less democracy still stands, but we have to approach implementing it with far more caution than I would have thought pre-covid.
It seems like following the 10% less democracy policy can give you either a really great outcome like the one you've described, and like we saw a small sliver of in the UK's vaccine procurement, or a colossal disaster like your impossible to fire expert epidemiologists torpedoing your economy and public health and then changing their mind a year late.
Suppose the UK had created a 'pandemic taskforce' with similar composition to the vaccine taskforce, in February instead of April, and with a wider remit over things like testing and running the trials. I think many of your happy timeline steps could have taken place.
One of the more positive signs that I've seen in recent times, is that well-informed elite opinion (going by, for example, the Economist editorials) has started to shift towards scepticism of these institutions and a recognition of how badly they've failed. We even saw an NYT article about the CDC and whether reform is possible.
Among the people who matter for policymaking, the scale of the failure has not been swept under the rug. See here:
This strikes me as the sort of remark I'd expect to see in one of these threads, which has to be a good sign.
There were some other recent NYT articles which based on my limited COVID knowledge I thought were pretty good, e.g. on the origin of the virus or airborne vs. droplet transmission [1].
The background of their author, however, seems fairly consistent with an "established experts and institutions largely failed" story:
It is interesting that perhaps some of the best commentary on COVID in the world's premier newspaper comes from a former computer programmer whose main job before COVID was writing about tech issues.
(Though note that this is my super unsystematic impression. I'm not reading a ton of COVID commentary, neither in the NYT nor elsewhere. I guess a skeptical observer could also argue "well, the view you like is the one typically championed by Silicon Valley types and other semi/non-experts, so you shouldn't be surprised that if you see newspaper op-eds you like they are written by such people".)
--
[1] What do you do if you want to expand on this topic "without the word limits" of an NYT article? Easy.
Fast rollout would be vaccinating every wiling citizen by July or August 2020. That what you would have gotten if you would have asked who can provide the vaccines by that point, put up prices and get the regulatory barriers out of the way. Stoecker vaccinated himself in March 2020. His company was then selling COVID-19 tests because it was allowed to do so. If it would have been allowed to sell vaccines with regulations that are as easy as selling tests it would likely also have sold those.
That task force might have gotten a D and not an F but calling it a moderately good job is a huge stretch.
Going to take an actual look at this later (my thesis is probably on COVID-19 x IIDM so I'm excited!) but wanted to let you know that I really appreciate the humorous flair in your post!
I'm pretty late to this! Something I notice in this post, which is mentioned in one of the comments, is that it focuses on money and competence as the main barriers to tackling Covid. I would say that the way institutions are set up to make decisions and implement them is a much bigger factor.
The fast timeline described would only be possible in a world where our institutions were set up completely differently. For example, when thinking about how things should have gone, it's easy to say person X and person Y should have talked to each other without realising that they probably didn't have each other's phone numbers, and might not have known each other existed!
We can and should look at how our institutions could have been set up differently to handle Covid better, but that requires some thought too. Institutions that are optimised for one thing are by default not optimised for other things - if we redesigned our institutions to prepare for the next Covid, that might not help us very much in dealing with a super-volcano, or everyday life.
The reason I point this out is because I think this kind of naive analysis is actually very useful - I think it does achieve the goal of showing us where we could have done better - and at its most useful, it highlights weaknesses in systems that can be improved. But I'm concerned that if this kind of analysis were shared without recognizing its limitations it could come across badly.
Agreed - and I agree that "naive" is the right word for it.
I think the main thing I would change about the post (besides the info from Chi which greatly weakens one of the 4 main planks) is better emphasising that it sketches an ideal ceiling which we should expect to sink as details are added. A relatively realistic ideal, but not strong warrant for righteous rage.
Signing off now; thanks to everyone.
"I am extremely puzzled why China or one of the other ahem non-individualist governments didn't do these."
Even in China, politicians and scientists fear public backlash, especially considering how to Chinese eyes the virus seems much more dangerous/lethal than to Europeans, given what happened in Wuhan:
https://www.scmp.com/news/china/science/article/3082570/coronavirus-vaccine-who-issue-guidelines-proposed-use-human
Sure. Challenge trials polled well in the West, but you're right that Wuhan could have been scarifying. Test of this: how did they poll in Lombardia?
My contention is that (in the US and UK at least) bioethicists and policymakers overestimated the controversy, possibly projecting their own misgivings.
I could not agree more with your sentiment, but the "We did ok" side has a point: If there was a much better policy or intervention, why was it done by no country, and no philanthropist? As a country, not much was stopping you a year ago to unilaterally prepurchase tons of vaccines and start manufacturing them. Getting 20 million doses manufactured early is much easier than 2 bn, you do not need to spend time coordinating with others etc., so what happened? From memory:
China only really started to vaccinate its citizens in March (but is doing it really fast now), despite approving the Sinopharm vaccine for EUA in July. Phase 3 data for their vaccines came in at the end of 2020, and seeing how urgent China is vaccinating now, it really does seem like manufacturing was their bottleneck. Russia approved its Sputnik Vaccine in August and started mass production immediately, but appears to only have been able to produce 2 million doses instead of the estimated 30 million by 2020 because of manufacturing problems.
But you do not have to design your own vaccine, you could just prepurchase a comparatively low amount of vaccines. There are enough oil states with no democratic decision making, so why did nobody say to Pfizer: "Here are 100 dollars per dose (9 times what the EU pays you) to start producing them now. The same day you publish your Phase 3 trial results, we expect the doses at our doors so that we can vaccinate our citizens."? E.g. Qatar only has 2 million citizens, so surely they could have procured enough early on, from different manufacturers?
And that is just the procurement/manufacturing side of things. There's also population wide rapid screening tests (AFAIK only Slovakia and Germany), pool testing (China and Rwanda), large, multicenter drug trials (only UK with the RECOVERY trial) as things that seem like extremely low hanging fruits but were neglected almost anywhere, despite strong economic incentives to get things right.
I am noticing my confusion: Are our institutions really so bad at dealing with crises? Or is it much more difficult than it looks to implement changes and react to completely novel situations?
One last guess:
My ideology-of-all-public-officials guess is pretty weak compared to an obvious alternative: simple public-choice herding at the executive level. (200 units instead of a million.)
If governments were each minimising their own reputation loss by (correctly) predicting that they wouldn't be punished for doing what everyone was doing, this could be enough to prevent ~all innovation. As much as you want safety in numbers, you doubly don't want to be the first to risk and lose. No entrainment needed, let alone intentional coordination.
(What could explain Israel's contract being redacted? The dodgy data-sharing agreement? No, that came out. The reputational risk of being seen to have rushed something out in [Autumn 2020]? No: it worked, so why not unredact now?)
I place >60% on the herding belief fwiw, especially if we limit to countries that have enough power to actually shake things (eg China, US, UK, Russia etc).
An additional piece of evidence for this is the degree of correlated beliefs about things like HCTs and genetic enhancement.
I am also very confused. The incentives for politicians to move as fast as they could were so vast.
Besides just vaguely accusing them of lacking courage: Another possibility is a profound entrainment of world elite opinion. One globalised and very narrow Overton window for public professionals. University is the obvious place for this to propagate, but I don't really know. What is its content? "Don't be hasty"? Could a philosophical accommodation really prevent every defection?
(There were some - Hungary vs EU on vaccines, Israel. I actually just tried to find out the date of Israel's Pfizer contract - and it's redacted!)
I think this is currently the strongest objection to the ideal timeline, even if it just points to the space where a concrete objection might well be.
Because philantrophists like Stoecker got sued and fordidden from deploying better interventions.
August 28, 2020: "Production has started at a new plant in Beijing with an annual capacity of roughly 300 million doses. Sinovac has agreed to supply 40 million doses to Bio Farma, an Indonesian state-owned company, between November and March. Sinovac started building the factory in late March and finished the project in July."
So this is in fact a little piece of the happy timeline.
But shouldn't this update our priors towards mostly being on the happy timeline, in the West as well? Given that it took Sinovac/China one year from last March to this March to scale up, and that their vaccines are easier to manufacture than mRNA vaccines, and if we assume high investment from the start in China (so their timeline is close to optimal), it really starts to look like we could not have done much better on manufacturing (because the West does not differ strongly in available doses compared to China)?
I.e. we could have approved a few months earlier, but even in December the UK and the US (I think?) were mostly bottlenecked by supply issues, so an earlier approval should not have changed much by this intuition.
Not mostly happy, I think. China apparently needed a new factory, but other places didn't (to the tune of 3bn wasteful doses or ~12bn real ones).
Also fast approval was only one prong of the fix, along with 2) an order of magnitude more investment, 3) invested much earlier, as pre-Phase I pre-purchases, 4) HCTs, and 5) pivoting away from 80%+ waste as soon as we realise we're doing that.
(HCTs are still relevant here because some of the vaccines have a shelf life < 6 months, and HCTs could thus allow May-June 2020 production to dampen the second or third waves.)
Half a trillion dollars should really make some dent in the known and unknown bottlenecks. Not sure how to shrink my estimate to account for the immovable remainder.
Datum about process over speed in the civil service and civil society (and also about OWID being far more savvy about optics than Cummings):
It's interesting that (being from the Guardian), that article presents the story as being a scandal, with the implication that Cummings was being corrupt.
The last sentence is uncharitable and wildly inaccurate. Do you have any evidence for this? Prior to approval, the contracts had already been finalized, and anyway, it's not the EMA (Europe's FDA equivalent) that is doing the haggling. AZ approval was delayed because of the shoddy quality of their studies (and even more so in the US). Why Pfizer took longer is unknown, but it certainly included, meanwhile leaked, working through problems like this:
It is to the credit of the EMA that they did not publicly release these problems, thus not fuelling vaccine skepticism but instead taking the reputation hit of seemingly needlessly delaying the approval.
Here's my source, "based on dozens of interviews with diplomats, Commission officials, pharma industry representatives and national government aides". Here's another, and another.
The EU's priorities are revealed in the result: 25% - 45% lower prices. They actually sort of brag about it:
It seems to have been a mix of understandable coordination, show of force, price haggling, and liability haggling (which is just a kind of price haggling with extra politics).
I hadn't seen those leaks, thanks. Note that they did plenty to fuel vaccine hesitancy elsewhere. No opinion about attribution to the EMA vs the Commission.
Oh, there is not a shred of doubt that the EU delayed buying the vaccines in order to lower the price, and I agree that this was a disastrous decision that led to supply delays. This is however a separate question from approving the vaccine, which is what my objection was about.
Ah right! Yes, I misspoke; pardon.
You mean not publishing the truth? No I think that's is bad, it's more difficult to counter vaccine skepticism when organizations are not public and transparent, at least I don't see any reason to think not.
Well whatever one may think of it, the EMA had legitimate concerns, and was not merely dragging its feet for negotiation reasons as the OP implied.
I think your happy timeline is still too conservative. Moderna didn't suddenly gain the ability to make a vaccine in 2 days, just before the pandemic began. Realistically they must have had the ability for some fraction of their decade-long existence. In the world where our institutions didn't get in the way as you describe, Moderna would have long since launched mRNA vaccines against other infectious diseases like the flu or malaria. The factories (at least a few) would have already been built and could have been repurposed without massive new investment, the public (or at least public institutions) would already be familiar and somewhat comfortable with the technology, and distribution channels would already be in place.
Novavax shows that the old way of creating vaccines of giving an adjuvant along with the protein (or a subunit) works fine and works with less side-effects. While Novavax does use their own patented adjuvant that needs rare ingridients other adjuvants are possible. Stoeker was likely right when he proposed to vaccinate everyone in Germany with the vaccine he used to vaccinate himself and his family.
We accept mask use based on the precautionary principle and we should have accepted Stoekers vaccine that demostrable produced antibodies on the same principle. While Stoeker only has data for side-effects for hundred patient the fact that the side effects on average are so much lower then that of viral vector and mRNA vaccines suggests that it's likely a safer vaccine (and the fact that Novavax has less side effects supports this).
Suing Stoeker on the allegation of running an unregistered clinical trial on his family was the height of ignorance.
Focusing clinical trial resources on untested and hard to scale up vaccine technology was a very bad idea when it was easy to take existing adjuvants, synthezise the spike protein (/domains of it) and scale up the production.
This is false. Vaccines are not the only possible policy to drive COVID-19 to extinction. While we haven't spend the amount on clinical trials that would be necessary to know for certain the best guess for the effect of prophylactic ivermectin we have at the moment is Low-certainty evidence found that ivermectin prophylaxis reduced COVID-19 infection by an average 86% (95% confidence interval 79%–91%).
The case for challenge trials for prophylactic ivermectin is the same as the case for challenge trials for vaccines.
While taking taffix multiple times per day is more annoying it also is a potential of doing the work of bringing infection risk down. It getting completely ignored instead of running more studies was a policy failure.
While it's popular to say that the lab-leak shift wasn't because of new evidence I think it was because people like Ian Lipkin who published in the beginning of the pandemic that he considers the natural origin the only possible explanation, because they thought that the WIV was doing gain-of-function experiements in biosafety 3 or 4 actually read the papers from the WIV and discovered that they did the experiements in biosafety 2.
While the papers certainly existed beforehand I don't think they were in the expert awareness.
From the very beginning, the top-down systems have worked to help the virus. FDA literally banned hospitals from using their own PCR labs, and forced the nonworking CDC "test" on everyone.
The vaccine companies were and are forbidden from paying to get volunteers from the high-risk groups... a trial in the NY inner-city nursing homes would have been done in a month, while the actual PFE trial using white female volunteers took 35,000 volunteers to get 96 cases... and burned up most of the year doing it.
And of course, the successful Scandi Bio Phase 3 trial results are only in the Turkish drug approval process... and nowhere else, because they use non-pharma molecules. 38% reduction in recovery time as a treatment, and completely safe and GRAS. As a preventive, this protocol could save most of the people in the non-vaccinated countries:
https://onlinelibrary.wiley.com/doi/10.1002/advs.202101222
This post seems to fall victim to the planning fallacy.
One explanation for why people commonly underestimate project completion time is this: people consider a best-case scenario where everything goes smoothly. Like the "happy timeline" in this post. Alas, reality is not like this at all.
The popular solution for countering the planning fallacy is taking an outside view (aka reference class forecasting). In that light, Covid vaccines looks surprisingly good. As you note at the top of your post, Covid vaccines were developed quickly compared to the reference class of vaccines.
I'm very sympathetic to the idea of writing down our collective mistakes and trying to learn from them. It just seems to me that this post contrasts our mistakes with a highly unrealistic, idealized timeline. My guess is that more realistic expectations might lead to better goals for improving the world 🤔
I don't recognise my post in this description. I openly acknowledge that there are bottlenecks., including unknown bottlenecks. I put a 150% interval on the key uncertainty. (I am protected somewhat from Hofstadter's law there by the reference point of the Braintree facility, with its almost known lead time.)
It's not unrealistic to pay weekend overtime or make new weekend hires for regulators, in the biggest health crisis of the century. It's not unrealistic for a single Chinese scientist to just decide on his own to release the genome he already sequenced, two weeks early. It's not unrealistic to decide to stop wasting three-quarters of the precious vaccine, once you realise you are doing so and that the rules you set blindly a year ago were arbitrary choices. Approval sharing between rich countries is not unrealistic. It is not totally unrealistic to imagine that a suitably alarmed establishment might throw whole-number fractions of their wealth at solutions, rapidly; the Manhattan project had much worse odds than any COVID vaccine effort. Challenge trials didn't happen, afaict, because a quite small number of people said so.
To the extent that any of the above were unrealistic, it's for tractable social reasons, and tractable social reasons serve my point about institutional decision-making.
Moreover, while reference class forecasting is a powerful element in prediction, it often fails when equilibria are not efficient. Policy seems like a perfect example. Yudkowsky's updated view is closer to my post than to one-step thought-terminating modesty.
It is important to realise how slow vaccines and all medical innovation is in general. That's why I say it. But it is also important to investigate whether that is for hard-to-change reasons, or not.
If we had used your reference class in March 2020, you would (and we did) predict that the COVID vaccines were going to take 7 years. If this forecast was taken seriously by the wrong people - as it may well have been, given the sluggishness of the "Warp Speed" initiatives! - then a great deal of good would have been, and was, lost.
Here's why the post reminds me of the planning fallacy: When people make flawed plans, these plans don't seem unrealistic. They often consist of detailed steps, each of which is quite likely to succeed. And yet, in most cases, the world takes a different turn and the planned project ends up late and more expensive.
You describe a "happy timeline" that's analogous to such a plan. For it to work, we would have to make many good decisions; many unknown obstacles would have to be overcome; and many novel ideas (like human challenge trials) accepted. None of these is unrealistic when looked at individually. But collectively, it is very unlikely that all these factors could realistically come together to form your happy timeline.
One example to illustrate this: Your post strongly favors vaccines and also attributes enormous costs to lockdowns. I think that this is realistic, but I can think so only in hindsight. In early 2020, it wasn't at all clear that sufficiently strong lockdowns wouldn't bring the pandemic to a manageable level or at least buy us the time we need for vaccine development. Remember the hammer and the dance? Yet, around that same time in your happy timeline, decisions are made to approve and pre-purchase vaccines at high costs. With hindsight, it's easy to say that we should have paid these costs; but at the time, it wasn't obvious at all. The answer to this question wouldn't have been easier to find with better institutional decision-making, either. It was simply a difficult question with no clear answer at that time.
Human challenge trials are a very old idea. Not doing them is the aberration.
Lockdown uncertainty seems moot. I'm not arguing that any lockdown policy should have been different (except that we might have lifted it a few months early if vaccinations were successfully time-shifted). Did anyone think that (realistic, non-Chinese, non-remote-island) lockdowns were an alternative to vaccines - back when we thought vaccines were coming in 2022 or 2027? The UK government seriously thought they could only do lockdown for a month or two. It doesn't add up.
But my real response is: of course the above dates aren't exactly how it would go, of course the cost estimate isn't exactly how much it would take; I'm not god. I'm not even Derek Lowe. But do you really think it would have been slower than what we did? Unless you do, it doesn't seem wiser to me to refuse to estimate - even given that estimation is fallible.
Hmm... Here's how I understand your estimate. Is that a fair summary?
My concern (expressed in the comments above) is mainly that the happy timeline is unrealistic, so the estimate could be off by a large factor, similarly to how the time and cost estimates of our plans are often off by a large factor.
Your estimate is probably still valuable, even if it is imprecise. We can use it to think about whether vaccine development is cost-effective; I reckon 3400$/QALY puts the cost-effectiveness an order of magnitude below effective charities and some orders of magnitude above many other public health interventions. Is that a fair conclusion?
I'd like to take away more from your post than just the estimate, but am not sure at the moment what other recommendations I can take from it...
The estimate undersells long COVID because we don't know how many years of 3.2m QALYs to add, but yes that's roughly it. And yes, I only claim that it was a decent deal, particularly since the funding for it couldn't really have gone on something else.
I freely admit that it could be off by a large factor (see my final paragraph). I would love for someone to come and do a proper Bayesian interval version, which would foreground the uncertainty.
I continue to challenge calling it "unrealistic", on priors, just because it's very uncertain. Last January, a historical baseline would have called it unrealistic to expect completion of vaccine R&D, trials, approvals, and distribution in 11 months. But here we are. I would have you be less sure about what's not possible, or not realistic.
I’m with Sjlver. There’s a lot of hand waving past problems in the happy timeline: eg, how long does it take to train new regulators who are going to work weekends? How quickly do trained regulators working constant overtime burn out? What happens when the Chinese government denies that the genome released early is the right one?
(Also, the invocation of “war time” as a model assumes that wars are run any more efficiently or even urgently. They’re not.)
You seem to be mistaking this for a white paper, or a piece of legislation, or an itemized purchase order for one different timeline please. It is not that. It is instead a thing to measure our situation against, to short-circuit the useless shrugging described in the opening section.
It would be difficult for more seriousness, more money, more personal and institutional courage to not help. I struggle to understand why you are so sure it wouldn't, or, if you do, why you're pointing out that unexpected things happen, on occasion.
In fact the genome was released by one person, and the government didn't say anything [against the genome]. He could have done it a week earlier still.
He seems to make really conservative assumptions though, what do you think was unrealistic? I don't think its unrealistic for the u.s.a to not ban people from testing themselves early in the pandemic, for example.
Challenge trails seems obviously good and not unrealistic.
Couldn't eg India have variolated their entire population and developed a robust process for it within a month in March 2020?
Sounds undoable.
1) unlike challenge trials, which had surprising popular appeal, this looks terrifying.
2) unlike challenge trials, you need to isolate billions of live cultures. Given real world biosecurity this is a nightmare.
3) I don't get the impression that the Indian government could move so fast, even though there are amazing private actors like the Serum Institute.
I don't want to shift the goalposts; my post also relies on a few things being viewed differently. But mine just requires a few hundred elites to get out the way.
Re 1: could it work as opt in? The brave go first?
Re 2: Wait why do you need to do that? Maybe you can have a sick person sneeze into a napkin and immediately rub it on the next person or whatever, so there is no intermediate storage step? Wrt biosecurity, a high percentage of the population was going to get covid / it's an available everywhere already anyway ?
Re 3: Have a private actor do it on govt contract? Competing firms ranked by effectiveness and safety etc
Re goalposts: What are you viewing as the goal post? I think I misunderstood you somewhere. What things are you viewing differently?
Yeah this might be getting out of scope for the discussion but a good variolation technique have no manufacturing, no patents, no buying / selling negotiation. Random people in poor countries with almost no govt could variolate each other. If world order or the science machine breaks down (new pandemic kills all the world leaders or something) then the technique can still be used.
And to get more out of scope, what's the harm reduction of a vaccine? Let's generously say the cost and side effects of getting vaccinated are 1% that of getting the disease itself. (Ignore spread reduction benefits for now.) At what percent would you consider variolation worthwhile as an option: 30%, 10%, 5%, 1%?
1. The UK began 2020 with an unspoken, dubiously voluntary version of this strategy. As the IFR firmed up, the backlash against this was so large that they were forced to deny that they ever intended any such thing.
The goalposts of this post are: "what is the best we could actually do, just changing the opinion of say a few hundred elites?" Challenge trials were popular; I strongly predict variolation is different, and that popularity matters even if 5% of the youngest and maddest volunteer.
2. You missed the absolutely critical, sign-flipping bit: you have to isolate the variolated! Variolation without enforced isolation is harbouring and abetting a lethal pathogen.
Again, this idea goes way past what our current institutions could possibly consider. They would want validation of the pathogen first, and who can blame em. COVID is indeed abundant, but you're massively multiplying the number of intentional virus touchpoints, and when every lab slipup can kill a few hundred people...
3. Can't imagine any private actor touching it with a bargepole.
The nearest possible version of a variolation policy is Hanson's clinics. They look nothing like decentralised napkin passing, and that's a feature.
No clue on threshold, gimme some numbers.
Mm good points. If I were dictator, I would still have a variolation process at least tested and developed for each new pathogen, which could be deployed as an emergency backup.
If by threshold you mean the percentage thing, I would say that at 25% conterfactual harm, variolation is a reasonable option for countries that couldn't manage vaccines, if nobody else was going to give it to them.