In this piece I argue that the scientific method failed during the Covid-19 pandemic, that it will always fail during rare and extreme events, and that we need other methods to guide our decision-making in volatile crisis environments.

Definitions

Just so we're starting on the same page...

What is Science? Science is a method of inquiry which uses experimentation, observation, and hypothesis testing to identify fixed characteristics of the natural world. That point will be important later in this piece.

What is a Pandemic? A pandemic is the spread of a deadly, contagious pathogen which has infected many people across large geographic areas. A pandemic is kind of crisis: a rare and extremely harmful event. The rarity of pandemics will be relevant throughout.

So, why does Science fail in a pandemic? Several reasons…

The Long-Term Data Problem

As a general rule in research: the more data you have, the stronger your theories will be, while the less data you have, the weaker your theories will be – if indeed you can justify any theories all. Science has strict minimum requirements (which is why we can trust it) and a sufficiently large ‘representative sample’ is one of them.

What is our pandemic dataset? Is it representative? Does it allow us to draw scientific conclusions about pandemics and how they should be managed? Let’s consider the evidence…

How many outbreaks of dangerous pandemic-potential pathogens (‘PPPs’) have there been in recorded history? Quit a lot, actually. If we included every major outbreak in every country, all the way back to the Plague of Athens, we’d have hundreds. So that’s a good start.

For how many of those outbreaks do we have robust data? Bar Covid-19, none. (And even Covid-19 is debatable).

One problems is that our ability to collect and store this kind of data was minimal for most of human history. We simply didn't have the tech. Moreover, the germ theory wasn’t confirmed until the late 1800s, so in a way, we didn’t even know what the ‘data’ was. There are major epidemics which killed millions, and yet we’re still debating which pathogens might have caused them.

While we can learn from these scraps of data, we have nothing like enough to reach robust scientific conclusions which can dictate our policy-making today.

If we did want to do robust science on pandemics, we would need comprehensive datasets from dozens of major epidemics – per pathogen! We’re nowhere near that point, and I hope we never get there either. Humanity is not long for this world is we have to live through dozens more Covid-19s before we finally learn how to protect ourselves from PPPs (but this is the inevitable conclusion of the 'evidence-based approach).

The Short-Term Data Problem

Now the short-term problem: what can we say about the data available to us in the midst of an outbreak and our ability to process it? 

Our knowledge of the pathogen will always lag its spread. The first case we identify is rarely, if ever, the first case in the outbreak, so we know from the very start that we are playing catch up. The pathogen will always be spreading somewhere, out of sight, and the official number of cases identified will always the absolute minimum. This data deficit is just a fact of life in epidemic control.

If the pathogen is mutating, then our knowledge will also trail the evolution of its epidemiological characteristics and the symptoms it presents, which further complicates the game of catch up. Every variable needs its own representative dataset, so every mutation puts additional demands on our data collection and processing capacity. In these environments, the variables can multiply faster than the data can be collected, which means science cannot function.

We saw this during Covid, where the pathogen mutated and the strains multiplied. That was a major spanner in the works for the people trying to control the outbreak, but also for the scientists trying to establish its epidemiological features. Then the vaccines arrived, which was good for the people controlling the outbreak but worse again for the scientists, as their data requirements multiplied once more. And all of that was on top of the minimum data needed to stratify the results by country, age, sex, pre-existing conditions, etc.

But even if we did have perfect datasets, the process of scientific research is too slow to make it useful in an epidemic. It takes too much time to collect the data, analyse it, interpret it, publish it, and then get it in front of the relevant policy-makers. By then, the information is already months out of date. It’s hard enough playing catch up, but it’s near impossible when you’re looking in the rear view mirror. And yet, that’s the best that science can offer us.

To be clear, I’m not just saying that science is unsuited to a pandemic, or that it is inadequate and other methods are necessary. I’m saying that, in addition to those points, science cannot function at all! Scientific research stops being scientific in a pandemic. It cannot do what it is supposed to do, much less what we unreasonably expect of it. It’s like a car driving through 5 feet of water: first the engine fails, then the wheels to lose contact with the ground. You don’t have a car anymore; you have some kind of floating wreckage.

The downstream effects of these data problems can be seen in the published academic research.

The Contradictions

We have academic papers telling us that masks worked during the Covid-19 pandemic and we have academic papers telling us that they did not (I am deliberately using the word ‘academic’ here, not ‘scientific’).

We have academic papers showing that lockdowns were effective and we have other papers proving conclusively that they were a waste of time.

To this day we have academics telling us that there is “dispositive” evidence that the pandemic began in a wet market in Wuhan, while other papers demonstrate that it cannot possibly be so.

Perhaps my understanding is simplistic or naïve, but if they’re all applying the same scientific method, to the same data… shouldn’t they all produce the same results? Even if we allow for a healthy dose of variation, true science should not produce conclusions that are diametrically opposed to each other. Scientists may disagree on the size of the effect, but not on the sign, surely?

There may be many different explanations for these contradictory results, other effects at work etc, but I’d argue that the possibility that science simply cannot function as intended in an extreme and dynamic environment like a pandemic should be considered by all sincere intellectuals.

Making Statements vs Making Decisions

Even if we did have exhaustive datasets on pandemics throughout the ages, and even if we did generate comprehensive datasets in real time as the current outbreak unfolded, and even if all of my aforementioned concerns could be allayed, we would still be left with the same question which spurred all that data collection and analysis in the first place: what should we DO with that information?

As has been discussed in this parish before, the purpose of science is to make statements, but in a crisis like a pandemic, we need to make decisions and take actions. Making statements and making decisions are completely different practices, they requires different skillsets, and the former does not automatically determine the latter.

Science tells us that the pathogen is becoming more contagious. OK, what should we do about that?

Science screams from the rooftops that ‘masks work!’, but again I ask you: what should we do with that information – wear masks 24 hours a day? The statement of fact does not lead us to a single course of action. Even perfect science still leaves us with the hardest part of the problem unsolved.

Some pieces of information can be decisive in determining our actions (e.g. there is cholera in the water => don’t drink it) but these situations are rare. So even when science is functioning as it intended by providing us with robust statements of fact, we will still need other methods to incorporate that knowledge into decisions, and not least because the environment is constantly changing, so what was true yesterday, may not be so today.

Hence, if we are to safely navigate volatile and dangerous environments like a pandemic, we need to let go of static scientific statement-making and embrace dynamic strategic decision-making instead (aka Risk Management).

Conclusion

If science really held the answers, wouldn’t we have ‘solved’ Covid-19 long ago? There were so many scientists in the media, in the policy discussions, at the highest levels of influence and power, and yet the Covid-19 outbreak still turned into a global catastrophe.

And if pandemics really were amenable to science, wouldn’t we have also reached a scientific consensus on pandemic policies by now? We’ve had centuries of practice, yet as things stand in early 2024, we seem further from consensus than ever. I’d argue that we are less capable of protecting ourselves from PPPs today than we were in 2019, or even 2003.

Let's face the facts: science failed during the Covid-19 pandemic and it will always fail when it leads in a crisis. It could not function as intended and it failed to provide us with the answers and direction we need. It should be clear now that other methods are required.

Science is fine in calm seas, but Risk Management is needed in a storm. The sooner the medics and campus academics can admit that to themselves, the sooner we can get to work on permanent solutions.

My next piece will be a case study of a decision from the Covid-19 pandemic which was of immense consequence. It will demonstrate that not only does the scientific method fail in a crisis, but that it can lead you into the worst possible decisions.