Julian Jamison is an economics professor at the University of Exeter and a Senior Research Affiliate at the Global Priorities Institute.
Thanks for this! A small point: as one of the coauthors of the Blattman et al Liberia CBT study, we didn't use trained counselors either (they basically don't exist in Liberia) so I think this helps somewhat with your legitimate concern about scaling. I wasn't sure what you meant by job training in that context?We are just finishing a ten-year follow-up of the same population, with encouraging results, which should be public soon.
Great question and great answers so far. I always liked Alexander Fleming (discoverer of penicillin, which has certainly saved millions of lives) but I suspect someone else would have found it relatively quickly if he hadn't - although I've never looked into the details.What about George Washington? He obviously wasn't the intellectual leader of the early USA and its form of government (and should only get a fraction of the credit / blame for everything that followed), but my sense (?) is that he played a key political (leadership) and military role in the revolution. Perhaps more importantly, he voluntarily stepped down after two terms as president, when he was still very popular. This seems rare in history (before or after his time) and likely led to more stable governance since then not only in the USA but also around the world. Arguably the US is the world's oldest continuous government and oldest democracy; the UK is tough to pin down, but I'd go with the Reform Act of 1832 if I had to pick a year when the House of Commons truly escaped from aristocratic influence.
Another point of agreement: the economics profession currently focuses too much on empirical work. Meanwhile my own personal view is that people like Esther and Chris B are slightly 'too far' in the pro-RCT camp, and that people like Lant (and you) are 'too far' in the anti-RCT camp. But I don't see anyone in this discussion as being extreme (except possibly Lant...); healthy disagreement is to be expected and encouraged. Note that Esther and Abhijit's most recent book tackles macro issues like migration, trade, climate change, and yes growth - using RCTs when possible / relevant but also plenty of other results (including lots of theory! Abhijit started life as a theorist, like I did). Meanwhile Chris has a forthcoming book on war and peace (macro level! no easy RCTs) for which he uses other approaches like machine learning. You can find all sorts of quotes, but the proof is in the pudding. Final point on this is that one can easily combine RCTs with admin data, ML, etc, and researchers (including me) are doing more and more of that, which imo is great - it's not always one or the other.
As you say, the efficacy of deworming seems to be a point of disagreement between us. Again pulling back somewhat, you link to Eva's paper as supporting your claim that RCTs have minimal external validity, but her paper is about all forms of impact evaluation (and she notes in the conclusion that the subset of RCTs aren't special). So this would be extremely damning for economics if true, but her results don't support your claim. For instance she notes that bednets and conditional cash transfers seem to do very well on this front. More relevantly, her point (as I read it) is to see how much of the nominal variation in effect sizes can be explained by other contextual variables, and she finds that typically a nontrivial amount of it can be. This is good news for external validity, since it means we can often explain / predict the differences even when they do arise.
I think I haven't been very clear about 'apples to oranges' - I agree that these can & should absolutely be compared. I just felt like the way you were doing it glossed over an important difference. I can write a check to AMF and feel very confident that something will change in the world; we can then debate the expected magnitude of the impact of that change. But I can't write a check to "growth reform in the developing world", so even before we debate the relative benefits of changing immigration policy vs distributing bednets we have to calculate the probability that the desired policy will get implemented. I realize you're fully aware of this, but that's the part I keep coming back to because that's the part where I'm pessimistic (partly having worked for the US government, although for a counterargument I liked this recent forum post) and suspect that our intuitions disagree, and mostly you keep talking about the benefits of more migration and of GDP growth (which are great!) and not so much about how we sit down and estimate the likelihoods of bringing those about. I'll admit that the "pessimistic" estimate of 1% on ICRIER in the original post with Hauke really made me distrust everything afterward, since the pessimistic estimate in that case is a negative number and a plausible median estimate seems to be about 1 in a million.
On China I suppose my main point is still that I think it's simply very very hard to quantitatively estimate most of this. Just because you (or I, or anyone) thinks that something is extremely conservative (when you admit you haven't put in as much time on all this as you'd like, and indeed it's not your job to do so) doesn't make it so. In this specific case, if you forced me to take a stand, my best guess is to agree with you that economists have helped push policy in a better direction and that that made a big difference to global welfare. Even if I felt more confident about that, what is the counterfactual you are comparing to? Did some NGO or the WB cause that to happen on the margin, or would economists have tried to learn about the world and influence policy anyway? Are there similar opportunities going forward? The Taliban says they want economics expertise, so perhaps. But I don't think we know the answers to these questions (yet), even within orders of magnitude, and whether or not this type of approach will beat RCT-type approaches depends entirely on those particular probabilities.
I'm going to try to step back first and speculate where we actually disagree, in hopes of getting at what you actually think should be happening differently, if anything. You seem to be arguing to some extent against things that do not exist, and in particular that neither I nor others are saying. I think we agree that (i) 1-5% of work in economics should be RCTs; (ii) RCTs are not the right approach for many, indeed most, questions in social science; (iii) there exists lots of policy-relevant and actionable information from non-RCT sources; (iv) intuitions can be a useful input, as long as one is transparent that that's what they are; (v) the EA community should be spending resources studying policy interventions, including around growth but also imo health (e.g. lead paint, tobacco); and (vi) economists do more good than harm in the world.
Where I think we disagree is that your intuition is that with another three person-years of effort, the EA community will find growth policy funding opportunities (not based on RCTs...) that are far more effective than the current top GW charities, and my intuition is that we won't (but that I still think we should look, as I've said many times, because the uncertainty is high and we might find them!). Neither of us knows for sure, as this hasn't been done yet. Is it more than that?
And thanks for your reply! I hope that you are now satisfied, since the issue is being discussed :) More seriously I really am glad you brought it to the fore again, because it deserves it, and I'm being [sincerely] critical only because I take it seriously and respect it.
Re the FP report: so did it find anything promising or not? My reading (could be wrong, obviously, so let me know) was that they/you didn't in fact find anything to point to; that they/you believe such a thing does exist; but that it would take a lot of time and effort to find it. This is not especially encouraging, given that between them and the experts and you and Lant, at least one person-year has been spent on this already. If all you want is more discussion / research, as I said upfront I agree 100%. If you want to convince me that it is likely to succeed, you need to point to something more than intuition (in part because even the numbers you have thrown around are somewhat suspect).
There still seems to be an apples-to-oranges comparison here. You can criticize specific RCTs, sometimes validly and sometimes not (I'm happy to share the 10+ page report I wrote for GiveWell which on the whole validated the deworming results, and David Roodman's extensive research came to a similar conclusion). But it's not helpful to compare a single intervention (note that GiveWell has multiple charities with similarly high estimated ROIs) to "whatever it is that caused [those countries] to grow" which neither you nor the best macroeconomists seem to uncontroversially agree on even in retrospect. My claim is that if deworming had been added to their policy mix (and other aspects had adjusted however they adjusted), that would have been a good thing.
Sure you can run an RCT on allowing foreign investment: subsidize it in some regions and not others. Same for immigration: encourage it in some places (or times) and not others. Same for road pricing. These won't pick up anything that is truly nationally systemic (and for that reason I agree full-scale tax reform is more difficult, although many taxes can be varied locally), but it doesn't have to be perfect for us to learn a lot. It also doesn't have to directly involve a "policy that you might enforce in the real world" - it just has to have useful implications for such policies. Some of these might be hard and take time and involve theory; so is figuring out what causes growth and how to make it happen.
The ICRIER case that you and Lant and FP point to is simply wrong. You suggested in the comments below that >30% of the economics profession is working in the RCT space; this is wrong for development econ (where it is less than 10%) and even further from the truth in other sub-fields. You say in your response to me that "EAs do not support a randomista approach to the rich world" - obviously I can't speak for EAs in general, but I have worked on RCTs in the US (here and here and forthcoming here), RCTs in Europe (here and here), and for what it's worth non-RCTs in the US (here and here and here) as well. Eva Vivalt, another EA-affiliated economist, is also working on randomized cash transfers in the US. David Reinstein is working on RCTs of charitable giving.
Purely anecdotally, I have found 'randomistas' to be highly practical, perhaps in part because they work more often in the field. I think most of them are happy to apply the technique anywhere it can help. I also think most of them admit that it isn't always the best approach, whereas I rarely see Lant or Angus Deaton saying the same about RCTs (perhaps because they incorrectly believe the method can only be applied to second- or third-order domains? then again they are very smart people so I have some trouble accepting that explanation).
And finally back to the main question of this particular post. I'm personally not convinced that US policy work will necessarily get close to the GiveWell bar. Whether it does or not, I do think it's nontrivially possible that developing-world policy wouldn't beat it by an order of magnitude or more. As has been noted by others here I think, policy work often requires local contextual knowledge and connections, which for many poor countries would first have to be built up by the EA community. Most such work will be country specific, and most such countries (with some important exceptions, like Nigeria and India) have an order of magnitude lower population than the US. Most such countries have less stable governments, so any policy changes won't last as long in expectation. Lots of factors come into play; it's not so simple as saying 100x and that you personally don't see how it couldn't be true. Don't get me wrong: it might be true! In fact I do suspect that policy work, like other interventions, will in general have a higher return in the global south. But it's not a priori obvious.
I've been meaning to respond to the original post but never got around to it, so thank you for bringing it back up and encouraging more discussion. I'm not a 'randomista' per se, but I have published RCTs as well as non-RCTs, and I have worked in the US as well as many developing countries. FWIW, like Lant I have a PhD in econ from MIT (where I was one year ahead of Esther and TA'd for Abhijit), have taught at Harvard (and elsewhere), and used to work at the World Bank. We're also fairly evenly matched at tennis, judging from an interesting doubles game many years ago. I was one of the 'experts' for the Founders Pledge work on this topic, which as you know didn't suggest anything specific despite thinking hard about it for at least a little while.
TL/DR: I 100% agree that we should be doing more research on effective ways to leverage broad policy initiatives, including for growth (but also e.g. health). We know a lot about basically good macro policies (and imo very little beyond that; then again I'm a microeconomist) and it would be great to get countries to do more of those. This is hard, and no one seems to have particularly convincing concrete ideas about what to do differently (create another World Bank to compete with the first one??). The potential returns are high, but in expectation not necessarily any higher than for 'RCT-led' policies.
I liked the post (I have worked a lot on RCTs, and a little on NPIs, but not together alas!). Here's another paper that I didn't see mentioned (although maybe I missed it) which I think roughly falls into the category you're considering?https://www.nber.org/system/files/working_papers/w27496/revisions/w27496.rev0.pdf
I feel like there are also some behavioral econ papers looking at e.g. social distancing in queues, but off the top of my head I'm not certain if there are actual randomized field experiments in that space...
Thanks for this great post. I'm closer to left-libertarian or classical liberal myself, but I have many friends and family (mostly in the US) who are more traditional progressives and much more sympathetic to typical social justice concerns than to EA. I agree with many of the issues identified here (including in the comments); my own experience has been that it is largely that they want to be able to "walk and chew gum at the same time". As an economist, I'm imbued with notions like opportunity cost and only being able to optimize one goal at a time (potentially itself an aggregation of course), but this is very foreign and off-putting to them. Either they don't understand the size of the actual disparities between issues, or... well actually I'm not sure, it's hard for me to wrap my head around.
However I particularly wanted to mention an illuminating recent post by Matt Yglesias (who came up elsewhere in the comments) on his substack: https://www.slowboring.com/p/slate-star-codex
The main topic is distinct, but from "The radicalism of effective altruism" onward it is very relevant and informative. On the one hand Yglesias is criticizing the journalist's progressive critique of EA, SSC, Silicon Valley, etc. On the other hand Yglesias (who is definitely on the left, and who likes evidence and reason a lot) doesn't end up very sympathetic to EA himself. He thinks of it as purely consequentialist, extreme, etc. Even if it's hard to attract some full-on progressives, someone like him should be exactly the type of person who supports EA. Something has gone wrong with the messaging if that isn't the case, and we are missing out.
I agree with several of the previous responses, but just to add something I haven't seen mentioned: it is a new / different experience 'outside the convex hull' of anything else. I selfishly enjoy that, because I like to experience new things (travel, changing research focus, ultra-running, etc), but I also believe that all of this gives me a more flexible and broader view of the world and how it can work and what it can contain, in a way that improves my perspective and my thinking. Imagination only goes so far, even for the most creative amongst us.