In this piece, I argue that the scientific method failed during the Covid-19 pandemic, that it will always fail during rare and extreme events, and that we need other methods to guide our actions in crisis environments.
Before we begin, I want to introduce some definitions and key characteristics of science and of pandemics. This helps us to start from shared ground, but it will be useful later in the piece.
What is Science? Science is a method of intellectual inquiry which uses experimentation, observation, and hypothesis testing to identify fixed characteristics of the natural world.
What is a Pandemic? A pandemic is the spread of a deadly, contagious pathogen which has infected many people across large geographic areas. A pandemic is kind of crisis: a rare and extremely harmful event.
So, why does Science fail in a pandemic? Several reasons…
As a general rule in research, the more data you have the stronger your theories will be, while the less data you have the weaker your theories will be – if indeed you can justify any theories all. Science has strict minimum requirements (which is why we can trust it) and a sufficiently large ‘representative sample’ is one of them.
What is humanity's pandemic dataset? Is it representative? Does it allow us to draw scientific conclusions about pandemics and how they should be managed? Let’s consider the evidence…
How many outbreaks of dangerous pandemic-potential pathogens (‘PPPs’) have there been in recorded history? Quit a lot. If we included every major outbreak in every country, all the way back to the Plague of Athens, we’d have hundreds. So that’s a good start.
For how many of those outbreaks do we have robust data? Bar Covid-19, none (and even that one is debatable).
There were plenty of observations throughout history. The problem is that our ability to collect and store this kind of data was minimal for most of it. We simply didn't have the tech.
Moreover, the germ theory wasn’t confirmed until the late 1800s, so we didn’t even know what the ‘data’ was that we should have been collecting. There are major epidemics which killed millions, and yet we’re still debating which pathogens caused them.
We have scraps of data and, while we can learn from them, we have nothing like enough to reach robust scientific conclusions which can dictate our policy-making today.
If we did want to do robust science on pandemics, we would need comprehensive datasets from dozens of major epidemics – per pathogen! We’re nowhere near that point, and I hope we never get there either. Humanity is not long for this world is we have to live through dozens more Covid-19s before we finally learn how to protect ourselves from PPPs (BTW this is the inevitable conclusion of the 'evidence-based' approach, but we'll cover that another day).
Now the short-term problem: what can we say about the data available to us in the midst of an outbreak and our ability to process it?
Our knowledge of the pathogen will always lag its spread. The first case we identify is rarely, if ever, the first case in the outbreak, so we know from the very start that we are playing catch up. The pathogen will always be spreading somewhere, out of sight, and the official number of cases identified will always the absolute minimum. This data deficit is just a fact of life in epidemic control.
If the pathogen is mutating, then our knowledge will also trail the evolution of its epidemiological characteristics and the symptoms it presents, which further complicates the game of catch up. Since every variable needs its own representative dataset, every mutation puts additional demands on our data collection and processing capacity. In these environments, the variables can multiply faster than the data can be collected, which means science cannot function.
We saw this during Covid, where the pathogen mutated and the strains multiplied. That was a major spanner in the works for the people trying to control the outbreak, but also for the scientists trying to establish its epidemiological features. Then the vaccines arrived, which was good for the people controlling the outbreak but worse again for the scientists, as their data requirements multiplied once more. And all of that was on top of the minimum data needed to stratify the results by country, age, sex, pre-existing conditions, etc.
But even if we did have perfect datasets, the process of scientific research is too slow to make decisions in an epidemic. It takes too much time to collect the data, analyse it, interpret it, publish it, and then get it in front of the relevant policy-makers. By then, the information is already months out of date. It’s hard enough playing catch up, but it’s near impossible when you’re looking in the rear view mirror. And yet, that’s the best that science can offer us.
To be clear, I’m not just saying that science is unsuited to a pandemic, or that it is inadequate and other methods are necessary. I’m saying that much of the time, science cannot function at all!
Scientific research stops being scientific in a pandemic. It cannot function as intended. It cannot do what it is designed to do, much less what we unreasonably expect of it. It’s like a car driving through 5 feet of water: first the engine fails, then the wheels to lose contact with the ground. You don’t have a car anymore; you have some kind of floating wreckage.
The downstream effects of all these problems can be seen in the published academic research.
We have academic papers telling us that masks worked during the Covid-19 pandemic and we have academic papers telling us that they did not (I am deliberately using the word ‘academic’ here, not ‘scientific’).
We have academic papers showing that lockdowns were effective and we have other papers proving conclusively that they were a waste of time.
To this day we have academics telling us that there is “dispositive” evidence that the pandemic began in a wet market in Wuhan, while other papers have already demonstrated that it cannot possibly be so.
Another oddity of the Covid-19 research was that the results tended to match the policy preferences of the authors. Those researchers who supported mask-wearing tended to find that mask-wearing was effective, while those who regarded lockdowns as an assault on civil liberties also found that lockdowns were completely ineffective.
Perhaps my understanding is simplistic or naïve, but if they’re all applying the same scientific method, to the same datasets… shouldn’t they all produce the same results? Even if we allow for a healthy dose of variation, true science should not produce conclusions that are diametrically opposed to each other. Scientists may disagree on the size of the effect, but not on the sign, surely?
There may be other potential explanations for these contradictory results, but I’d argue that the possibility that science cannot function as intended in an extreme, dynamic, low-data environment like a pandemic should be one of them, and it should be sincerely considered by all sincere intellectuals.
Even if we did have exhaustive datasets on pandemics throughout the ages, and even if we did generate comprehensive datasets in real time as the current outbreak unfolded, and even if all of my aforementioned concerns about research integrity could be allayed, we would still be left with the same question which spurred all that data collection and analysis in the first place: what should we DO with that information?
As has been discussed in this parish before, the purpose of science is to make precise statements, but in a crisis like a pandemic, we need to make decisions and take actions. Making statements and making decisions are completely different practices. They requires different skillsets, and the former does not automatically determine the latter.
Science tells us that the pathogen is becoming more contagious. OK, what should we do about that?
Science screams from the rooftops that ‘masks work!’, but again: what should we do with that information? Should we wear masks all day? Even when we're sleeping? The statement of fact does not lead us to a single course of action. Even perfect science still leaves us with the hardest part of the problem unsolved.
Some pieces of information can be decisive in determining our actions (e.g. there is cholera in the water => don’t drink it) but these situations are rare. So even when science is functioning as it intended by providing us with robust statements of fact, we will still need other methods to incorporate that knowledge into decisions, and not least because the environment is constantly changing, so what was true yesterday, may not be so today.
Hence, if we are to safely navigate volatile and dangerous environments like a pandemic, we need to let go of static scientific statement-making and embrace dynamic strategic decision-making instead (aka Risk Management).
If science really held the answers, wouldn’t we have ‘solved’ Covid-19 long ago? There were so many scientists in the media, in the policy discussions, at the highest levels of influence and power, and yet the Covid-19 outbreak still turned into a global catastrophe.
And if pandemics really were amenable to science, wouldn’t we have also reached a scientific consensus on pandemic policies by now? We’ve had centuries of practice, yet as things stand in early 2024, we seem further from consensus than ever. I’d argue that we are less capable of protecting ourselves from PPPs today than we were in 2019, or even 2003.
Let's face the facts: science failed during the Covid-19 pandemic. It could not function as intended and it failed to provide us with the answers and direction we need. Science is essential, but it should be clear now that other methods are required.
Science is fine in calm seas, but Risk Management is needed in a storm. The sooner the medics and campus academics can admit that to themselves, the sooner we can get to work on permanent solutions.
My next piece will be a case study of a decision from the Covid-19 pandemic which was of immense consequence. It will demonstrate that not only does the scientific method fail in a crisis, but that it can lead you into the worst possible decisions.
