This is the clearest thinking I've ever encountered on this topic. From an international development perspective, but could apply to RCTs for any social intervention, really. Knowing stuff is hard!
There is this sort of belief in magic, that RCTs are attributed with properties that they do not possess.
...
Causality can change locally too. Even if you’ve uncovered a causal effect that doesn’t mean that causality will work that way somewhere else. It’s not just the size of the effect.
...
For instance if you go back to the 70s and 80s and you read what was written then, people thought quite hard about how you take the result from one experiment and how it would apply somewhere else. I see much too little of that in the development literature today.
...
MDRC wrestled with that problem of finding mechanisms from the very beginning but they never resolved it. They thought that by going into the details they could find mechanisms that would generalize or transport and they never managed to do that. You can’t do that with RCTs. You’ve got to combine them with theory and observational data so you’re right back where you were.
...
Each study has to be considered on its own. RCTs are fine, but they are just one of the techniques in the armory that one would use to try to discover things. Gold standard thinking is magical thinking.
The appropriate response- already adopted by the best social sciences where possible- is triangulation of research methods. (https://en.wikipedia.org/wiki/Triangulation_(social_science)) Other than that, blanket fetishisation or blanket dismissal of RCTs are both vices.
I found some of what Deaton said here very odd. Especially what sounds like a false equivalence of comparing the inference of causation from correlational data with experiments, which are "just like any other method of estimation."
Obviously, there can be external validity problems with RCTs, but this point seems to be overstated especially when it comes to global health interventions.
It is very difficult, expensive, and time consuming to unbundle the basket of causes that the average RCT is actually checking and test each one separately to see if it is truly critical. Two things that can help: 1. more money for RCT's so the most important ones can be investigated with follow ups, 2. Better RCT design via sharing of expertise among the developmental econ field and drawing in methodology expertise from outside the field.