For future searching, where/how did you come across that paper?
Good find - thanks for sharing that paper which I hadn't included. If I update the post I'll add that.
I haven't thought much about this so can't add anything useful at the moment. If I think of / come across anything I'll reply again.
Good point. This is similar to what I was trying to get at when talking about lack of willingness to engage in probabilistic reasoning.
Thanks a lot for the comment. I was a bit nervous to put my first post up so some positive feedback is very much appreciated.
Thanks a lot for the comment. I do think that what your gesturing at makes sense: if I understand correctly you are saying that certain physical interventions can have more predictable effects that ‘biological’ ones because we have a decent idea of exactly how they work. In some cases this is definitely true: as an extreme example, we don’t need RCTs of aeroplane safety as we have a very good understanding of the physical processes and are able to model them well. If we have an airborne pathogen, it’s hardly necessary to run an RCT to see whether or not there is an effect of a stay at home order: there will be one.
In many of the example questions I gave though, I think the fact that there is a large behavioural component pushes us closer to the situation we have with drugs than to the aeroplane. For example, although it could be demonstrated in a laboratory which of mask or shield is actually more effective at blocking exhaled particles, it would be harder to capture the different effects that each has on how often you touch your face, how often it is removed, or other aspects of compliance. These will differ a lot between people, so you’d need to test it on a large group, and the social setting might influence behaviour. I don’t think that we can decompose the often important behavioural component of these interventions in the same way that we can the physical components.
That said, the air filtration question I posed might not have been well chosen. As you point out, it seems reasonable that we can get a good understanding of whether that is likely to be helpful by applying what we know about the filters and viral transmission. Of the questions I posed, RCTs are likely to be the least useful there and may not be useful at all.
However, I do have some thoughts on why an RCT could still be worthwhile. I’m not saying these because I disagree with your points; I’m just providing some possible counterargument.
Overall, I think the areas where trials would be most useful are those where we can expect relatively modest effects and where there is a larger behavioural component. The combination of modest effects, if better understood, might be quite important.
Some quick thoughts (there is certainly already research on these but they seem important, and I don't know about reliability of existing research):
I like this perspective. I've never really understood why people find the repugnant conclusion repugnant!
Not really answering your question, but there is some recent work attempting to forecast clinical trial results that may be relevant: Can Oncologists Predict the Efficacy of Treatments in Randomized Trials? Kimmelman (the senior author) is doing other work on the topic too (e.g. here). I'm not aware of much published work in this space in a biomedical context.
My guess is that key decision makers in medicine (e.g. funders of trials), would not be very open to paying attention to forecasts (even if shown to be accurate to some degree), as there is a very strong culture of relying on data and in particular on RCTs.
This may be less meta than you are hoping for, but may contain some useful advice/references: The dos and don’ts of influencing policy: a systematic review of advice to academics. Influencing policy is at least one way that academic ideas can travel to the wider world.
I expect another is producing accessible content on the topic in question (e.g. writing popular blog posts, books, documentaries). It seems like these can sometimes be a catalyst for ideas becoming more widely known in the public. Examples of books that might have had or could have a broad impact are Animal liberation (Peter Singer), Silent Spring (Rachel Carson), Doing Good Better (Will Macaskill) or Human Compatible (Stuart Russel).