Thanks to Ray Taylor for posting this video on a EA Facebook page (I can't remember which one). I thought this video was worthy of a Forum page and discussion about the points it raises on  Randomised Control Trials vs other evidence.  The context of the RCTs discussed here is masks for Covid-19 prevention, but I think it can be  generalised to other areas. 

The transcript is taken roughly from the 20 minute mark of the video. I have not transcribed parts of the talk that are not about RCTs vs other evidence, and I have removed references to slides in the talk, and  added hyperlinks to the relevant studies.

" I want to introduce you to two very different tribes of scientists. The first tribe is called evidence-based medicine and its totem is the hierarchy of evidence and the second tribe is one I called pragmatic public health and its totem is the multifaceted real world case study. Now the evidence-based medicine tribe adheres to a hierarchy of evidence - they're actually a hierarchy of methods with randomized controlled trials at the top. Good science is defined by the use of correct methods. Some methods are accepted as better than others, indeed a poor example of a higher up method may be seen as better than a good example of a lower down method. 

It is a deeply held myth among most members of this tribe that if participants are randomized in an experiment that's good science and if they're not it's less good science. Now here's an example of science produced by the evidence-based medicine tribe. Jefferson and Hennigan identified randomized trials of masks for preventing respiratory infections. They used a checklist called a risk of bias tool. Each research study they looked at got a score according to how biased they judged it to be now using quality standards that were culturally agreed among their tribe. Jefferson and Hennigan concluded that there was no good evidence for the efficacy of masks. They placed non-randomized trial evidence in their hierarchy of evidence in a metaphorical trash can because this tribe ranks by method they didn't even have to look at any other kinds of research and thus was born the fact that there was no evidence that masks work.

You will note that the evidence-based medicine tribe rarely enters into dialogue with other tribes.They largely consider them irrelevant but they spend a lot of time arguing about and refining their own hierarchy of evidence and their risk of bias. Now let's look at a different tribe. It too has totems and rituals and since i myself identify with this tribe i'm going to find it more difficult to be critical of it. This tribe holds passionately to the belief that there is no universally applicable hierarchy of evidence though some methods may be more or less fit for purpose.Good science is defined by this tribe as the use of multiple methods adaptively and pragmatically and also ethically and democratically to build a nuanced narrative of what has happened in a particular real-world case study and why.Theory is assumed to be at least as important as method- as Ken Judge put it: “strong theory, flexible methods”. The narratives, say the pragmatic public health specialists, needs to make sense and be plausible to the natives.

Now in pragmatic public health a lot of additional evidence gets brought to the table. Studies that are ignored by the evidence-based medicine tribe become salient. For example: sneeze videos in which the unmasked person is shown to emit huge turbulent clouds of respiratory droplets and airborne particles, or choir stories in which most people attending a choir practice developed Covid 19 even when they didn't get within six feet of the index case nor touch any common surface. Now these pieces of evidence taken in isolation are not proof that masks work but they demand a scientific explanation and they add to the overall picture. The same goes for natural experiments around the world. Christian Leffler’s study of Covid mortality country by country in the days after the first documented case in each country showed that the countries which introduced mandated or widespread voluntary masking by 30 days had orders of magnitude fewer deaths than countries which delayed introducing masks beyond the 100 days- again not in itself proof that masks work but pretty good evidence that they don't kill you. 

Now let's go back to social media. In 280 characters the evidence-based medicine narrative is summed up by professor Paul Glasziou in his interpretation of the facts based on the sparse literature from randomized controlled trials the evidence that masks protect anyone is weak and he is minded to speculate about multiple possible harms including the question of risk compensation. For example, if you wear a mask you'll think you're protected so you won't bother washing your hands. The pragmatic public health narrative is summed up by professor KK Cheng who brings a much wider menu of facts to the table and concludes that there's strong evidence of benefit and even stronger evidence of no serious harm. Incidentally professor Glasziou’s comments about harms of masks were not borne out- a review by Theresa Marteau's team showed that risk compensation doesn't occur. In fact, wearing masks is associated with increased compliance with other preventive measures. Furthermore video analysis of thousands of people walking past subway cameras shows that the ones wearing masks touch their faces less than the ones not wearing masks. 

okay so back in march i got together with some other members of the pragmatic public health tribe and we wrote this article for the British Medical Journal. We argued as pragmatists do for the precautionary principle we said we don't have 100 proof that masks work yet but let's act pragmatically on the basis of the numerous facts which point in the direction of a positive benefit harm balance. Now our paper didn't initially have much impact- it was another three months before Public Health England introduced a recommendation to wear a face covering in crowded public places. 

You may have heard of the Great Barrington Declaration led by three professors from Harvard, Stanford and Oxford which was signed in about September it states if I may summarize that Covid 19 isn't as bad as claimed, especially for the healthy under 60s, that the evidence base for interfering with people's lives is weak and that the economy should be prioritized over further lockdown. One argument made by this group centers on the lack of rigorous randomized controlled trial evidence. There's another declaration out there the John Snow memorandum to which i'm a signatory which brings in mechanistic and case study evidence and also evidence from ethics and political philosophy to argue the opposite- that Covid19 is serious and sometimes deadly, that all citizens count and that the best way to save the economy is to address public health and that we all need to make compromises for the sake of society.  "