Effective Altruism Forum
EA Forum

Comment Permalink
rootpi38
0
0

I've been meaning to respond to the original post but never got around to it, so thank you for bringing it back up and encouraging more discussion. I'm not a 'randomista' per se, but I have published RCTs as well as non-RCTs, and I have worked in the US as well as many developing countries. FWIW, like Lant I have a PhD in econ from MIT (where I was one year ahead of Esther and TA'd for Abhijit), have taught at Harvard (and elsewhere), and used to work at the World Bank. We're also fairly evenly matched at tennis, judging from an interesting doubles game many years ago. I was one of the 'experts' for the Founders Pledge work on this topic, which as you know didn't suggest anything specific despite thinking hard about it for at least a little while.

TL/DR: I 100% agree that we should be doing more research on effective ways to leverage broad policy initiatives, including for growth (but also e.g. health). We know a lot about basically good macro policies (and imo very little beyond that; then again I'm a microeconomist) and it would be great to get countries to do more of those. This is hard, and no one seems to have particularly convincing concrete ideas about what to do differently (create another World Bank to compete with the first one??). The potential returns are high, but in expectation not necessarily any higher than for 'RCT-led' policies.

Detailed comments:

  1. In a response here you say that no one would encourage Rwanda to do RCTs as growth policy - of course not: the RCT is not the policy, rather the evaluation. Research is not policy. In the original post you talk about NGO assistance (which you identify with small RCTs) vs govt policy (big impacts), but the point of many RCTs is to influence governments (and on the flipside the EA community is most likely to influence macro policy via NGO intermediaries as well).
  2. More broadly I think you need to be careful to compare apples and apples. In the OP you say that you don't think e.g. deworming is in the top 100 ways to increase growth. [As it happens I think deworming or bednets would have been a fantastic idea for most countries in say 1930; the fact that Denmark didn't do it and still grew does not imply that it wouldn't have been a better way for them to spend resources than most other policies they pursued.]  Are you comparing to "institute macroeconomic stability" or to "fund a start-up that tries to convince Liberian officials to pay more attention to exchange rate fluctuations" ? It's not well-defined as stated.
  3. Also in the OP you say that no one argues that the huge welfare gains in e.g. China were due to interventions of "this [RCT] type" - well I do!  There were huge gains in health, agriculture, urban migration, etc, all of which are regularly studied by RCTs. Also contra your claim in the comments here that none of immigration, housing reform, road pricing, ag subsidies, tax reform, etc can be evaluated with RCTs - of course they can be. NB I'm not claiming that RCTs led China to make those changes (which hadn't happened yet).
  4. You cited Weil (2014) as saying that health doesn't have much effect on growth, but I noticed that he doesn't reference either Hoddinott et al (Lancet 2008) or Baird et al (QJE 2016; working paper was public years earlier), both of which find that adult incomes are 10-15% higher after very low-cost childhood health interventions (a very high ROI!). You can call the latter work controversial if you want, but when I wrote a background piece on deworming for GiveWell there actually seemed to be a lot of evidence that ~10% income gains was pretty reasonable to expect.
  5. Okay let's talk about ICRIER, the example that Lant and you point to. He talks about an initial gift from Ford of $36mil (chosen to fit nicely with India's additional $3.6trillion of GDP, but which Lant "strongly suspects" is the right order of magnitude). In fact I went back and read ICRIER's entire 20th anniversary report, which thanked Ford for $250k. Their earliest annual report that I could find online (from 2002) shows an endowment of just over $2mil. Hence $36mil is wrong by a couple of orders of magnitude. If anything this would make your argument stronger of course: somehow Ford predicted ex ante (else this is not a useful example for us now) that donating a few hundred thousand had a nontrivial chance of increasing Indian GDP by trillions of dollars. Wow :) 
  6. On the other hand, you'd think that if ICRIER believed that their actions (quoting Lant) "increased by 10 percent the odds India adopted growth accelerating policies (my read of the situation is that it was higher)" they would have mentioned this somewhere in their celebratory 20th anniversary document? They don't. Lant mentions the crisis of 1991; they don't. Lant mentions the growth acceleration of 1993; they don't. Closest seems to be the claim that a book by I.J. Ahluwalia (at one point an ICRIER affiliate), published by Oxford in 1985 after many years of work, may have influenced policy, although her wikipedia page doesn't say anything along those lines and I couldn't find any other evidence for it.
  7. I love Lant and Michael Clemens' work on immigration reform, and I think it's hugely important. But to compare that or ICRIER to RCTs you need to make ad hoc back-of-the-envelope calculations about actually influencing policy. I can come up with similarly large numbers for RCTs (except based on much more evidence): GiveWell says we can save a [probabilistic] life for less than $5k, and the typical VSL is $5mil, so that's an ROI of 1000 right there...
Reply
Showing 3 of 8 replies (Click to show all)
2
rootpi
I'm going to try to step back first and speculate where we actually disagree, in hopes of getting at what you actually think should be happening differently, if anything. You seem to be arguing to some extent against things that do not exist, and in particular that neither I nor others are saying. I think we agree that (i) 1-5% of work in economics should be RCTs; (ii) RCTs are not the right approach for many, indeed most, questions in social science; (iii) there exists lots of policy-relevant and actionable information from non-RCT sources; (iv) intuitions can be a useful input, as long as one is transparent that that's what they are; (v) the EA community should be spending resources studying policy interventions, including around growth but also imo health (e.g. lead paint, tobacco); and (vi) economists do more good than harm in the world. Where I think we disagree is that your intuition is that with another three person-years of effort, the EA community will find growth policy funding opportunities (not based on RCTs...) that are far more effective than the current top GW charities, and my intuition is that we won't (but that I still think we should look, as I've said many times, because the uncertainty is high and we might find them!). Neither of us knows for sure, as this hasn't been done yet. Is it more than that? 1. You're right that finding leads but not making recommendations could cause one to update upward, downward, or not at all (depending on one's prior) so I shouldn't have suggested that it was necessarily a bad signal. However I didn't see anything quantitative (I did look at the appendices, but I believe the only quantitatively worked-out example was the incorrect and retrospective  ICRIER one in the main text). This makes it very hard to meaningfully compare against current top interventions. I would love if you would be willing to go on the record, with as many caveats as you want, and pick one or two of the ones Hauke listed and put some back-
9[anonymous]
Yes, that seems like the main thing we disagree about. It also seems like we disagree about the likely impact of deworming.  1. I would also like to do that but I don't have time to do that properly unfortunately - my time is now focused on longterm relevant stuff.  2. I think the point here is that the growth decelerations don't seem like they would be big enough to make the back of the envelope calcs on the world bank, IMF and all economists in mine and Hauke's post lower than RCT stuff.  3. Thanks for clarifying. First, (this is not a criticism of you) this is different to the view taken by Open Phil which does try to compare its rich world policy to AMF and is what prompted the main post. So, at least Open Phil can't think it is an apples to orange comparison. Second, I don't really get how it is an apples to oranges comparison. You seem to be saying that the judgement of the probability is too high. but that is different to the metric used to compare the options being incomparable. I don't understand why you think the probability estimate is too high. The back of the envelope on the world bank and IMF seems like a pretty good guide. You really have to believe some implausible things to think that the probability of success is low enough. On the 'all economists' one for example, the estimate is extremely conservative because it only counts the economic gains in China and assumes that is the only thing that all economists achieved.  4. The reason I cited the Cochrane review was that GiveWell also cited it in their review of deworming. My main aim is to dispute GiveWell's reasoning as that is what is driving a lot of EA money, which I think could do much more good elsewhere. On external validity, I'm going off the Vivalt paper. 5. It's a natural way to interpret GiveWell's views because the only charities they recommend, or (I think) have ever recommended, have been tested by RCTs. All of the charities on their standout list have all been tested by RCTs. Thi
rootpi4
0
0

Another point of agreement: the economics profession currently focuses too much on empirical work. Meanwhile my own personal view is that people like Esther and Chris B are slightly 'too far' in the pro-RCT camp, and that people like Lant (and you) are 'too far' in the anti-RCT camp. But I don't see anyone in this discussion as being extreme (except possibly Lant...); healthy disagreement is to be expected and encouraged. Note that Esther and Abhijit's most recent book tackles macro issues like migration, trade, climate change, and yes growth - using RCTs ... (read more)

Reply
See in context
by [anonymous]
2 min read 26

108

Global health & developmentPolicyOpen Philanthropy
Frontpage

When funding policy advocacy in the rich world, Open Philanthropy Project aims to only fund projects that at least meet the '100x bar', which means that the things they fund need to increase incomes for average Americans by $100 for every $1 spent to get as much benefit as giving $1 to GiveDirectly recipients in Africa. The reason for this is that (1) there is roughly a 100:1 ratio between the consumption of Americans to GiveDirectly cash transfer recipients, and (2) the returns of money to welfare are logarithmic. A logarithmic utility function implies that $1 for someone with 100x less consumption is worth 100x as much. Since GiveWell's top charities are 10x better than GiveDirectly, the standard set by GiveWell's top charities is a '1,000x bar'. 

Since 2015, Open Phil has made roughly 300 grants totalling almost $200 million in their near-termist, human-centric focus areas of criminal justice reform, immigration policy, land use reform, macroeconomic stabilisation policy, and scientific research. In 'GiveWell’s Top Charities Are (Increasingly) Hard to Beat', Alex Berger argues that much of Open Phil's US policy work probably passes the 100x bar, but relatively little passes the 1,000x bar. 

The reason that Open Phil's policy work is able to meet the 100x bar is that it is leveraged. Although trying to change planning law in California has a low chance of success, the economic payoffs are so large that the expected value of these grants is high. So, even though it is a lot harder to increase welfare in the US, because the policy work has so much leverage, the expected benefits are high enough to 100x the $ benefits. 

This raises the question: if all of this true, wouldn't advocating for improved economic policy in poor countries be much better than GiveWell's top charities? If policy in the US has high expected benefits because it is leveraged, then policy in Kenya must also have high expected benefits because it is leveraged. We should expect many projects improving economic policy in Kenya to produce 100x the welfare benefits of GiveDirectly, and we should expect a handful to produce 1,000x the welfare benefits of GiveDirectly. 

This is an argument for funding work to improve economic policy in the world's poorest countries. Lant Pritchett has been arguing for this position for at least 7 years without any published response from the EA community. Hauke Hillebrandt and I summarise his arguments here. My former colleagues from Founders Pledge, Stephen Clare and Aidan Goth, discuss the arguments in more depth here.

Updated addendum: At present, according to GiveWell, the best way to improve the economic outcomes of very poor people is to deworm them. This is on the basis of one very controversial RCT conducted in 2004. I don't think this is a tenable position. 

108

0
0

Reactions

0
0
Mentioned in
Comments26
Sorted by
New & upvoted
Click to highlight new comments since:
rootpi38
0
0

I've been meaning to respond to the original post but never got around to it, so thank you for bringing it back up and encouraging more discussion. I'm not a 'randomista' per se, but I have published RCTs as well as non-RCTs, and I have worked in the US as well as many developing countries. FWIW, like Lant I have a PhD in econ from MIT (where I was one year ahead of Esther and TA'd for Abhijit), have taught at Harvard (and elsewhere), and used to work at the World Bank. We're also fairly evenly matched at tennis, judging from an interesting doubles game many years ago. I was one of the 'experts' for the Founders Pledge work on this topic, which as you know didn't suggest anything specific despite thinking hard about it for at least a little while.

TL/DR: I 100% agree that we should be doing more research on effective ways to leverage broad policy initiatives, including for growth (but also e.g. health). We know a lot about basically good macro policies (and imo very little beyond that; then again I'm a microeconomist) and it would be great to get countries to do more of those. This is hard, and no one seems to have particularly convincing concrete ideas about what to do differently (create another World Bank to compete with the first one??). The potential returns are high, but in expectation not necessarily any higher than for 'RCT-led' policies.

Detailed comments:

  1. In a response here you say that no one would encourage Rwanda to do RCTs as growth policy - of course not: the RCT is not the policy, rather the evaluation. Research is not policy. In the original post you talk about NGO assistance (which you identify with small RCTs) vs govt policy (big impacts), but the point of many RCTs is to influence governments (and on the flipside the EA community is most likely to influence macro policy via NGO intermediaries as well).
  2. More broadly I think you need to be careful to compare apples and apples. In the OP you say that you don't think e.g. deworming is in the top 100 ways to increase growth. [As it happens I think deworming or bednets would have been a fantastic idea for most countries in say 1930; the fact that Denmark didn't do it and still grew does not imply that it wouldn't have been a better way for them to spend resources than most other policies they pursued.]  Are you comparing to "institute macroeconomic stability" or to "fund a start-up that tries to convince Liberian officials to pay more attention to exchange rate fluctuations" ? It's not well-defined as stated.
  3. Also in the OP you say that no one argues that the huge welfare gains in e.g. China were due to interventions of "this [RCT] type" - well I do!  There were huge gains in health, agriculture, urban migration, etc, all of which are regularly studied by RCTs. Also contra your claim in the comments here that none of immigration, housing reform, road pricing, ag subsidies, tax reform, etc can be evaluated with RCTs - of course they can be. NB I'm not claiming that RCTs led China to make those changes (which hadn't happened yet).
  4. You cited Weil (2014) as saying that health doesn't have much effect on growth, but I noticed that he doesn't reference either Hoddinott et al (Lancet 2008) or Baird et al (QJE 2016; working paper was public years earlier), both of which find that adult incomes are 10-15% higher after very low-cost childhood health interventions (a very high ROI!). You can call the latter work controversial if you want, but when I wrote a background piece on deworming for GiveWell there actually seemed to be a lot of evidence that ~10% income gains was pretty reasonable to expect.
  5. Okay let's talk about ICRIER, the example that Lant and you point to. He talks about an initial gift from Ford of $36mil (chosen to fit nicely with India's additional $3.6trillion of GDP, but which Lant "strongly suspects" is the right order of magnitude). In fact I went back and read ICRIER's entire 20th anniversary report, which thanked Ford for $250k. Their earliest annual report that I could find online (from 2002) shows an endowment of just over $2mil. Hence $36mil is wrong by a couple of orders of magnitude. If anything this would make your argument stronger of course: somehow Ford predicted ex ante (else this is not a useful example for us now) that donating a few hundred thousand had a nontrivial chance of increasing Indian GDP by trillions of dollars. Wow :) 
  6. On the other hand, you'd think that if ICRIER believed that their actions (quoting Lant) "increased by 10 percent the odds India adopted growth accelerating policies (my read of the situation is that it was higher)" they would have mentioned this somewhere in their celebratory 20th anniversary document? They don't. Lant mentions the crisis of 1991; they don't. Lant mentions the growth acceleration of 1993; they don't. Closest seems to be the claim that a book by I.J. Ahluwalia (at one point an ICRIER affiliate), published by Oxford in 1985 after many years of work, may have influenced policy, although her wikipedia page doesn't say anything along those lines and I couldn't find any other evidence for it.
  7. I love Lant and Michael Clemens' work on immigration reform, and I think it's hugely important. But to compare that or ICRIER to RCTs you need to make ad hoc back-of-the-envelope calculations about actually influencing policy. I can come up with similarly large numbers for RCTs (except based on much more evidence): GiveWell says we can save a [probabilistic] life for less than $5k, and the typical VSL is $5mil, so that's an ROI of 1000 right there...
Reply
[anonymous]8
1
0

Hello, thanks for this and thanks for assisting with our report on that!

To clarify, it's not the case that the FP report didn't have any recommendations because we didn't find anything good. We stopped because we couldn't guarantee that we could move enough money to the area to make it worth our own time and that of the organisations we evaluated. We didn't discover anything that made me think we wouldn't find something better than deworming if we were to put a couple of person-years into it. 

On your tldr - the point of my post is that I don't see how it could be true that the returns are not higher than direct RCT-tested projects. Either  US policy doesn't actually get close to the GiveWell bar, or poor world policy beats it. 

  1. Yes, what I meant there was RCT-evaluated things rather than run RCTs. Though I think the randomistas would indeed have wanted the Rwandan government to run RCTs?
  2. I have to say I disagree on deworming. If I had philanthropic resources to spend, I think it would have been a clear grave error to go to East Asia after 1950 and try to deworm people rather than getting them to do whatever it is that caused them to grow. This is basically for the reasons outlined by Lant's sniff test. It is true that "the fact that almost no rich or quickly growing  countries didn't do mass school deworming does not logically guarantee that mass deworming is not the most cost-effective economic intervention in the world". The point is that it makes it very unlikely. It is especially unlikely given that the case for deworming only rests on 1 RCT in the early 2000s, which I think has a lot of problems. eg they didn't find any improvements in health that might explain the economic benefits deworming is meant to have produced; the external validity of RCTs in general has been shown to be poor but, while some external validity factors are considered by GiveWell, this broader evidence (as far as I can tell) is not considered. 
  3. I agree that some things in the realms you mention  (agriculture, urban migration and so on) can in principle be tested by RCTs. But almost all of what almost everyone seems to think caused huge economic growth in China  is not testable by RCTs. You can't run an RCT into allowing foreign investment. Realistically, you can't run an RCT into allowing state-owned enterprises to charge a market price for their goods. I don't see how you could run an RCT on an immigration policy that you might  enforce in the real world - how would that work? Could you actually do a useful RCT on road pricing? I don't see how you could do a practicable and useful RCT on tax reform. All of these examples also illustrate that you don't need to run RCTs to know whether these things are good. You could run an RCT on decollectivising agriculture but given the economic history of the 20th century, I don't understand why you would want to do that. 
  4. I haven't looked into the ICIER case, so thanks for doing that. This is exactly the kind of analysis I would like to see happen, especially with organisations doing work now. Note that there are many other cases of economists influencing policy in the world. 
  5. I actually don't think some of the GiveWell numbers are based on more evidence. I don't  think there is more evidence that deworming produces the claimed income benefits than that agricultural decollectivisation is good, for example. The evidence appealed to is one RCT from Kenya, whereas there are countless examples of agricultural decollectivisation producing beneficial effects. The point of my post is that if ad hoc back of the envelope calculations are ok for US policy, then they are ok for poor world policy. Relatedly, I don't understand why EAs do not support a randomista approach to the rich world. 
Reply
rootpi2
0
0

And thanks for your reply!  I hope that you are now satisfied, since the issue is being discussed :)  More seriously I really am glad you brought it to the fore again, because it deserves it, and I'm being [sincerely] critical only because I take it seriously and respect it.

Re the FP report: so did it find anything promising or not? My reading (could be wrong, obviously, so let me know) was that they/you didn't in fact find anything to point to; that they/you believe such a thing does exist; but that it would take a lot of time and effort to find it. This is not especially encouraging, given that between them and the experts and you and Lant, at least one person-year has been spent on this already. If all you want is more discussion / research, as I said upfront I agree 100%. If you want to convince me that it is likely to succeed, you need to point to something more than intuition (in part because even the numbers you have thrown around are somewhat suspect).

There still seems to be an apples-to-oranges comparison here. You can criticize specific RCTs, sometimes validly and sometimes not (I'm happy to share the 10+ page report I wrote for GiveWell which on the whole validated the deworming results, and David Roodman's extensive research came to a similar conclusion). But it's not helpful to compare a single intervention (note that GiveWell has multiple charities with similarly high estimated ROIs) to "whatever it is that caused [those countries] to grow" which neither you nor the best macroeconomists seem to uncontroversially agree on even in retrospect. My claim is that if deworming had been added to their policy mix (and other aspects had adjusted however they adjusted), that would have been a good thing. 

Sure you can run an RCT on allowing foreign investment: subsidize it in some regions and not others. Same for immigration: encourage it in some places (or times) and not others. Same for road pricing. These won't pick up anything that is truly nationally systemic (and for that reason I agree full-scale tax reform is more difficult, although many taxes can be varied locally), but it doesn't have to be perfect for us to learn a lot. It also doesn't have to directly involve a "policy that you might  enforce in the real world" - it just has to have useful implications for such policies. Some of these might be hard and take time and involve theory; so is figuring out what causes growth and how to make it happen.

The ICRIER case that you and Lant and FP point to is simply wrong. You suggested in the comments below that >30% of the economics profession is working in the RCT space; this is wrong for development econ (where it is less than 10%) and even further from the truth in other sub-fields. You say in your response to me that "EAs do not support a randomista approach to the rich world" - obviously I can't speak for EAs in general, but I have worked on RCTs in the US (here and here and forthcoming here), RCTs in Europe (here and here), and for what it's worth non-RCTs in the US (here and here and here) as well. Eva Vivalt, another EA-affiliated economist, is also working on randomized cash transfers in the US. David Reinstein is working on RCTs of charitable giving. 

Purely anecdotally, I have found 'randomistas' to be highly practical, perhaps in part because they work more often in the field. I think most of them are happy to apply the technique anywhere it can help. I also think most of them admit that it isn't always the best approach, whereas I rarely see Lant or Angus Deaton saying the same about RCTs (perhaps because they incorrectly believe the method can only be applied to second- or third-order domains? then again they are very smart people so I have some trouble accepting that explanation). 

And finally back to the main question of this particular post. I'm personally not convinced that US policy work will necessarily get close to the GiveWell bar. Whether it does or not, I do think it's nontrivially possible that developing-world policy wouldn't beat it by an order of magnitude or more. As has been noted by others here I think, policy work often requires local contextual knowledge and connections, which for many poor countries would first have to be built up by the EA community. Most such work will be country specific, and most such countries (with some important exceptions, like Nigeria and India) have an order of magnitude lower population than the US. Most such countries have less stable governments, so any policy changes won't last as long in expectation. Lots of factors come into play; it's not so simple as saying 100x and that you personally don't see how it couldn't be true. Don't get me wrong: it might be true! In fact I do suspect that policy work, like other interventions, will in general have a higher return in the global south. But it's not a priori obvious.

Reply
[anonymous]12
0
0

Hello,

  1. My view is that I  think it is very likely (>95% chance) that one would find something better than  GiveWell's top charities if one were to put in 4 person-years of time into the project, which was the claim in my original post. My team put in about 6 months of person time on the project. Hauke and I put in a few months writing up our post, which wasn't about trying to find donation opportunities but just to make the case that there is reason to think we would find something if we tried. I don't know how much time Lant has put in to actually trying to find a donation opportunities, but that doesn't seem to be what he spends his time doing just looking at his research output. A substantial fraction of the time that Stephen and Aidan spent on the project was on establishing what happened if you analysed things in terms of life satisfaction rather than $s, as this is what Alex Berger has asked for in the response to mine and Hauke's post. I wouldn't expect to find a good donation opportunity after about 3-4 person months of trying from two (very smart) people without much prior experience in the area. I think that would take ~4 years, which is what I suggested in the original post. No-one has yet put that in, so I don't think it is surprising that no-one has found anything. 
  2. My claim that we would find something better than deworming is not just based on intuition. There are numerous independent lines of argument that suggest that working on economic policy must be better than directly funding deworming: (a) economic growth explains a substantial fraction of the variance in human welfare in the world today, variations in RCT-evaluable things explain little; (b) the era of focusing on national development rather than programmatic RCT-evaluable things produced greater improvements in human welfare than all prior human history combined, so let's keep doing that; (c) if you conservatively add up the impact of all development economists and make conservative assumptions about the effect they had on China, it looks better than directly funding RCT-type stuff; (d) similar thoughts apply to the World Bank and IMF.  As I say here,  I appreciate that the ICIER numbers may not be right (I haven't checked), but the value here is illustrative and does not affect the basic arguments. Even if you think there is only a slim chance of success of any feasible campaign succeeding, the size of the rewards are so large that the expected benefits are very large, and must be larger than deworming. 
  3. I don't think it is an apples to oranges comparison. I am asking what I would do as a philanthropist with money in eg East Asia in the 1950s, 60s and 70s, or in any developing country at any point. I have the option of something that could in principle be tested by an RCT and something that might increase growth, like privatising agriculture, legalising supermarkets or reducing tariffs. I don't think it would ever be the correct response to fund the RCT thing. I don't think the appeal to what is uncontroversial among economists works in this debate. It is also highly controversial whether deworming produces the posited income benefits. I don't understand why it is not fair to compare a single intervention to the suite of things that might increase growth rates. That seems like a fair comparison to me and the benefits are measured in $s, which is not an apples to oranges comparison. I use the deworming example because the benefits are provided in monetary terms, which makes the comparison easier. I also think the same applies for GW's other top charities. Indeed, I think the health benefits of their other charities is overrated because GiveWell overrates the life satisfaction benefits of saving lives. 
  4. I expect you and I to differ on this, but I do think there should be more criticism of the recommendation of deworming. EA is generally quite concerned about replicability and the various issues with science. But the deworming recommendation is only based on 1 RCT. Two meta-analyses have found that "deworming did not have an impact on outcomes such as cognition, height, and school performance", so it is very unclear how deworming is meant to have produced these economic benefits. There is evidence of data mining and multiple comparisons in the paper. There is strong evidence that the external validity of all RCTs is very low - this is not accounted for systematically and formally in GW's research. This is  a classic case where we would expect the study finding to not be real. 
  5. I don't think those would be useful RCTs because they have low value of information. With the road pricing one for example, i find it hard to see how you would set a control group in any useful sense. I know it is (very) unfashionable in economics to say this, but I think it is  true that most of the case for road pricing can be known from the armchair (i'm not saying this is true of all topics!)  Economists now tend to form their views based on the median published empirical study on a topic, which we have every reason to believe will be wrong. The approach of complete agnosticism on a topic before an RCT is produced on it just seems to me to be similarly wrong 
  6. I'm not sure your 'less than <10%' is right. Analysing the same top 5 journals for the same year (2015) Duflo suggests that a third of development economics papers are RCTs, which is what I based my claim on. 
Reply
rootpi11
0
0
  1. Fair enough that not all of that time (much less all of Lant's career) was spent trying to come up with a good example here. I still think that if after a nontrivial amount of cognitive resource spent on this by very smart people (e.g. Lant is in an excellent position to have run across something and at least mentioned it), no one has even come up with a single plausible example that's worthy of further study, that's discouraging in a bayesian sense. I fully agree it's worth more research; I simply doubt that anything fantastic will appear after four years of work (or quite possibly ever).
  2. I disagree with all of your (a)-(d), not least because economists and the WB/IMF have caused lots of bad things to happen in addition to lots of good things. However even if we accept those general lines of argument, the conclusion that despite there being only a small chance of success "the size of the rewards are so large that the expected benefits are very large, and must be larger than deworming" is entirely your intuition. I don't share it, because I think the probability of success is very very small indeed. I could be wrong, and if you have a quantitative argument regarding the probability of any given high-quality campaign being successful a priori (not the actual ex post GDP change over decades) which isn't based on your intuition, then we can discuss that.
  3. This is indeed apples to oranges. A philanthropist can cause more deworming or bednets to be implemented. A philanthropist cannot privatize agriculture or legalize supermarkets or reduce tariffs. That's the whole point. China in the 1950s/60s/70s was led by Mao, so how exactly would you (an external EA philanthropist) have convinced him to alter national policy?
  4. External validity is certainly under-studied and -appreciated (at least by most economists, in part because of publication incentives), and I hope that GW takes it into account. The Poland paper I linked to is interesting in part because it reaches very different conclusions than earlier RCTs in the same domain, and I have published specifically on this problem. Criticism is also great, and the worm wars are evidence that Miguel & Kremer has received a lot of criticism! However none of this implies that we "expect the study finding to not be real"; if you have specific criticisms in this case, what are they? The Cochrane report you refer to is highly problematic, in part precisely because it only considers RCTs. For the interested reader, here are some non-RCT papers that all point in the same direction: one two three four
  5. First you said RCTs were not possible in those cases (which I disputed and you seem to have walked back), now you say they are not necessarily useful. Neither I nor anyone else has ever claimed that an RCT is the right way to answer every question in economics, nor that we should be agnostic before any RCTs, so this is a strawman. I'm a theorist at heart (my PhD was on repeated games), and I fully agree with you that modern econ has gone on average too far toward "let the data speak for itself" (whether RCTs or otherwise), and that progress can sometimes be made from armchairs. I just don't think theory is going to tell us whether in the real world funding an NGO to lobby for change to Tanzania's macro policy is in expectation a better bet than funding deworming or bednets.
  6. Your phrase was "a substantial chunk of the economics profession (>30%?)", which I disputed (and linked to a source showing that it is less than 10% even for development). If your claim is instead that a nontrivial minority of the top development papers are RCTs, then I agree. And that's an important indicator, although very different than the profession as a whole. What do you think the 'right' proportion would be in each case? I would also repeat myself that all of those authors value non-RCTs (and many of them also work on non-RCTs), and that both JPAL and GiveWell have become increasingly interested in broader policy work (as they ought to be, imo). 
Reply
[anonymous]3
1
0
  1. You seem here to be eliding the claim that we didn't find any leads with the claim that we didn't recommend anything. We found many  leads. FP found 7 different organisations that we thought it might be worth investigating if we had control of money. In Hauke's appendices to our post,  he lists several organisations and interventions that would be worth investigating. So, there are many plausible examples of things that are worthy of further study. If I put 3 months into a climate change report, I wouldn't argue that the failure to recommend a climate charity proves that aren't any good climate charities.
  2. It is true that economists have caused some bad things, but on balance, as Hauke and I discuss in section 3.2, the net gains of growth episodes have been extremely large since 1950. If economists affected the growth accelerations and decelerations in equal measure, then the effect would be strongly positive. For what you say to be true, the economists would have to have massively disproportionately affected the growth decelerations, which I don't think happened. Further to your criticism of intuition, your argument seems to be either that (i) all subjective probability assessments must be rejected because they are based on intuition or (ii) our subjective probability assessments are too high. Firstly, these two arguments are inconsistent. You cannot both say that you reject our argument because it is based on intuition and then say that according to your intuition the true probability is a lot lower. Secondly, argument (i) applies to all policy work, including the policy work that GiveWell are now going to investigate and that Open Phil already funds; I don't see why economic policy would be more subjective than air quality work. Thirdly, the case for deworming is also based on subjective probability judgements. For example, the various staff replicability adjustments that GiveWell staff members used to provide varied by a factor of 10. If you read the documentation of the way the replicability discount is now done, a lot of it is based on informal and qualitative reasoning. James Snowden describes GiveWell's mission as 'estimating the unknowable'. I don't think there is a sharp distinction in terms of non-subjectiveness between the things that GW recommends and policy stuff.
  3. To avoid misunderstanding here, am I right in interpreting you as saying the comparison as apples to oranges because philanthropists can never cause beneficial policy change? Do you also think comparing San Francisco planning reform and AMF is an apples to oranges comparison? I do in fact think there were ways to influence Chinese economic policy, namely by funding growth research, which is a global public good. As Hauke and I discuss in section 4 of our post, China under Deng sought extensive advice from foreign economists before making its reforms.
  4. I laid out several specific criticisms of the deworming estimate earlier in the thread. There are many many obviously important things that could affect the external validity of the study, and not just the worm burden which is what GW adjusts for. eg the intervention will now take place 20 years later, in a different country, with a different economy, different culture etc etc. The external validity problems are quite clear - there's a study showing that deworming some schoolchildren in Kenya increases income (one of many outcomes they were measuring) and we then need to make the assumption that this result will apply in all other contexts even though we have evidence from lots of other domains that the external validity of all RCTs is pretty minimal.  The 'only considers RCT' criticism of the Cochrane review is one that it is difficult for GW to make given its epistemic approach, which is the one I am criticising. 
  5. It's a bit tricky to carve out a difference between 'not practically possible' and 'not doable in such a way that you could get value of information'. Like you could sort of do an RCT where you allowed foreign investment in some companies but not in others, but it just seems like it would be a total mess to do that. The view that we should be agnostic before RCTs is I think an important guiding implicit assumption that a lot of people make. For example, it seems like by far the most obvious way to explain the approach GiveWell has taken since its inception. Lant discusses an example of a paper in the AEJ which found that building schools closer to children increases enrollment and says "In this paper, we evaluation a simple intervention entirely focused on access to primary schools. The empirical challenge is the potential endogenous relationship between the school availability and household characteristics. [Footnote 1].   Governments, for example, may place schooling either in areas of high demand for education or in areas with low demand for education, in the hopes of encouraging higher participation levels. Either will bias simple cross-sectional estimates of the relationship between access and enrollment." But the finding that proximity to school increases enrollment is something economists have known for ages on the basis of theory and non-RCT evidence .  Not all possible confounders are actual confounders. I didn't say or imply that "theory is going to tell us whether in the real world funding an NGO to lobby for change to Tanzania's macro policy is in expectation a better bet than funding deworming or bednets", it is so implausible that I have never given the impression of having believed it, and explicitly said in my last comment that I wasn't making such a claim- "(i'm not saying this is true of all topics!)"
  6. Yes you're right I'm happy to retract that. I had based my estimate on the Duflo numbers not on the wider field number. In answer to your question, I would have less than 5% of the field work on RCTs, maybe less than 1%. I do also think it is important that top 5 journals disproportionately publish RCTs as this does suggest that the top talent is disproportionately working on RCTs. 
Reply
rootpi2
0
0

I'm going to try to step back first and speculate where we actually disagree, in hopes of getting at what you actually think should be happening differently, if anything. You seem to be arguing to some extent against things that do not exist, and in particular that neither I nor others are saying. I think we agree that (i) 1-5% of work in economics should be RCTs; (ii) RCTs are not the right approach for many, indeed most, questions in social science; (iii) there exists lots of policy-relevant and actionable information from non-RCT sources; (iv) intuitions can be a useful input, as long as one is transparent that that's what they are; (v) the EA community should be spending resources studying policy interventions, including around growth but also imo health (e.g. lead paint, tobacco); and (vi) economists do more good than harm in the world.

Where I think we disagree is that your intuition is that with another three person-years of effort, the EA community will find growth policy funding opportunities (not based on RCTs...) that are far more effective than the current top GW charities, and my intuition is that we won't (but that I still think we should look, as I've said many times, because the uncertainty is high and we might find them!). Neither of us knows for sure, as this hasn't been done yet. Is it more than that?

  1. You're right that finding leads but not making recommendations could cause one to update upward, downward, or not at all (depending on one's prior) so I shouldn't have suggested that it was necessarily a bad signal. However I didn't see anything quantitative (I did look at the appendices, but I believe the only quantitatively worked-out example was the incorrect and retrospective  ICRIER one in the main text). This makes it very hard to meaningfully compare against current top interventions. I would love if you would be willing to go on the record, with as many caveats as you want, and pick one or two of the ones Hauke listed and put some back-of-the-envelope numbers behind them just to see how it plays out (you said that you "actually don't think some of the GiveWell numbers are based on more evidence" - so where is your evidence?). Then we could have a substantive discussion. Or maybe I would be convinced once I saw the concrete reasoning.
  2. To say that the net impact of historic policy economists has been positive (which I agree with) does not begin to tell us that it is "better than directly funding RCT-type stuff": we need (prospective) numbers, or else we're basing it on your intuition. I don't reject your argument because it is based on intuition; I reject the claim that it is based on evidence, if it is based on intuition. No inconsistency from me, and certainly no claim that subjective assessments should all be rejected; indeed I have published those and am devoting a large portion of my current / upcoming research agenda to surveying people about their values in order to use those estimates in policy work. No RCTs whatsoever, except some light randomization around framing!
  3. Thanks for checking - no you are not right. If I thought that philanthropists can never cause beneficial policy change, why would the very first sentence of the TL/DR in my very first reply have been "I 100% agree that we should be doing more research on effective ways to leverage broad policy initiatives"?? The issue is that while  I can donate to AMF, I can't donate to "San Francisco planning reform". There's an extra step involved in lobbying, which may be where our intuitions differ (I think it's possible but low probability; I guess you think it's higher but you never seem to talk about that part as much you do that GDP growth was really big). That's the sense in which every time you compare a specific GW charity to 'whatever it was that caused growth' your claim is difficult to parse and evaluate from the perspective of a philanthropist.
  4. If GW is saying to only consider RCTs, then I very much disagree with them (but see below). NB you're the one who brought up the Cochrane review as support for your position on worms, even though exactly your own arguments about RCTs not being the holy grail - which I fully agree with - show that it is deeply flawed. I'm not sure who you're arguing with here. Btw do you have any citations for the claim that "the external validity of all RCTs is pretty minimal"?
  5. GW paid me some years ago specifically to do an informally bayesian analysis of the deworming results (i.e. what should our prior have been before Baird et al?), putting it in the context of everything we know about not just deworming (from RCTs and otherwise) but also the impact of early childhood interventions / shocks on adult outcomes, most of which was not from RCTs. So I honestly don't know where you get the idea that there's a group of people out there who only believe in RCTs (except that Lant and Angus D sometimes like to give that impression, so that they can argue against it). The fact that Lant can find one imperfect paper in one good-but-not-fantastic journal (if he had found something in a top-5 journal he would have used it instead, so clearly he didn't) doesn't prove much. Can you point to any randomista or organization saying that we should be agnostic before doing RCTs? I would be very surprised if so, because literally no one I know believes that. As far as Tanzanian macro policy, sorry if I gave the impression that you believed that particular example to be true. The difficulty is that you appear to be unwilling or unable to give any specific example whatsoever, so I had to make something up just to instantiate the point that we (as EAs) don't implement policy; we fund organizations who indirectly influence policy (whereas we directly implement 'RCT-type' interventions). But I really didn't mean to give the impression that you might have claimed that it was somehow always going to be a great approach, so apologies.
  6. Thanks, and as I already wrote above I fully agree that prevalence in top-5 journals is [also] an important indicator - for the reasons you mention. Any new technique is going to be popular at first, both because it's a shiny new toy (academia over-rewards novelty) and also because there will actually be a lot of useful low-hanging fruit that it can be used for at first (if it has any merits whatsoever, which RCTs certainly do). I suspect that over the next 5/10/20 years the proportion in top journals (which is much lower than 30% overall; that's just for development) will go down; the proportion in other journals will first go up and then down; and it will settle at something like 10% for development and well under 5% for econ as a whole. This seems about right to me; maybe you think it's still way too high?
Reply
[anonymous]9
0
0

Yes, that seems like the main thing we disagree about. It also seems like we disagree about the likely impact of deworming. 

  1. I would also like to do that but I don't have time to do that properly unfortunately - my time is now focused on longterm relevant stuff. 
  2. I think the point here is that the growth decelerations don't seem like they would be big enough to make the back of the envelope calcs on the world bank, IMF and all economists in mine and Hauke's post lower than RCT stuff. 
  3. Thanks for clarifying. First, (this is not a criticism of you) this is different to the view taken by Open Phil which does try to compare its rich world policy to AMF and is what prompted the main post. So, at least Open Phil can't think it is an apples to orange comparison. Second, I don't really get how it is an apples to oranges comparison. You seem to be saying that the judgement of the probability is too high. but that is different to the metric used to compare the options being incomparable. I don't understand why you think the probability estimate is too high. The back of the envelope on the world bank and IMF seems like a pretty good guide. You really have to believe some implausible things to think that the probability of success is low enough. On the 'all economists' one for example, the estimate is extremely conservative because it only counts the economic gains in China and assumes that is the only thing that all economists achieved. 
  4. The reason I cited the Cochrane review was that GiveWell also cited it in their review of deworming. My main aim is to dispute GiveWell's reasoning as that is what is driving a lot of EA money, which I think could do much more good elsewhere. On external validity, I'm going off the Vivalt paper.
  5. It's a natural way to interpret GiveWell's views because the only charities they recommend, or (I think) have ever recommended, have been tested by RCTs. All of the charities on their standout list have all been tested by RCTs. This suggests at least that they put extremely high weight on RCT evidence. Even if they are not at the extreme of complete agnosticism sans RCT, they are very close to it. To use an example from Lant, Chris Blattman said that the best investment to fight world poverty would be to run an RCT comparing giving people cash or giving people chickens.  I find this stance extremely hard to understand without the implicit premise of agnosticism in the absence of RCTs. I don't think it would be difficult to find similar claims by duflo and banerjee with a bit of extra time. I do think that this goes beyond RCTs and extends to an over-reliance on empirical studies, as I argued here.
  6. I'm not sure which direction RCTs will go in in the future. The publication incentives  are to do that type of work.  
Reply
rootpi4
0
0

Another point of agreement: the economics profession currently focuses too much on empirical work. Meanwhile my own personal view is that people like Esther and Chris B are slightly 'too far' in the pro-RCT camp, and that people like Lant (and you) are 'too far' in the anti-RCT camp. But I don't see anyone in this discussion as being extreme (except possibly Lant...); healthy disagreement is to be expected and encouraged. Note that Esther and Abhijit's most recent book tackles macro issues like migration, trade, climate change, and yes growth - using RCTs when possible / relevant but also plenty of other results (including lots of theory! Abhijit started life as a theorist, like I did). Meanwhile Chris has a forthcoming book on war and peace (macro level! no easy RCTs) for which he uses other approaches like machine learning. You can find all sorts of quotes, but the proof is in the pudding. Final point on this is that one can easily combine RCTs with admin data, ML, etc, and researchers (including me) are doing more and more of that, which imo is great - it's not always one or the other.

As you say, the efficacy of deworming seems to be a point of disagreement between us. Again pulling back somewhat, you link to Eva's paper as supporting your claim that RCTs have minimal external validity, but her paper is about all forms of impact evaluation (and she notes in the conclusion that the subset of RCTs aren't special). So this would be extremely damning for economics if true, but her results don't support your claim. For instance she notes that bednets and conditional cash transfers seem to do very well on this front. More relevantly, her point (as I read it) is to see how much of the nominal variation in effect sizes can be explained by other contextual variables, and she finds that typically a nontrivial amount of it can be. This is good news for external validity, since it means we can often explain / predict the differences even when they do arise.

I think I haven't been very clear about 'apples to oranges' - I agree that these can & should absolutely be compared. I just felt like the way you were doing it glossed over an important difference. I can write a check to AMF and feel very confident that something will change in the world; we can then debate the expected magnitude of the impact of that change. But I can't write a check to "growth reform in the developing world", so even before we debate the relative benefits of changing immigration policy vs distributing bednets we have to calculate the probability that the desired policy will get implemented. I realize you're fully aware of this, but that's the part I keep coming back to because that's the part where I'm pessimistic (partly having worked for the US government, although for a counterargument I liked this recent forum post) and suspect that our intuitions disagree, and mostly you keep talking about the benefits of more migration and of GDP growth (which are great!) and not so much about how we sit down and estimate the likelihoods of bringing those about. I'll admit that the "pessimistic" estimate of 1% on ICRIER in the original post with Hauke really made me distrust everything afterward, since the pessimistic estimate in that case is a negative number and a plausible median estimate seems to be about 1 in a million.

On China I suppose my main point is still that I think it's simply very very hard to quantitatively estimate most of this. Just because you (or I, or anyone) thinks that something is extremely conservative (when you admit you haven't put in as much time on all this as you'd like, and indeed it's not your job to do so) doesn't make it so. In this specific case, if you forced me to take a stand, my best guess is to agree with you that economists have helped push policy in a better direction and that that made a big difference to global welfare. Even if I felt more confident about that, what is the counterfactual you are comparing to? Did some NGO or the WB cause that to happen on the margin, or would economists have tried to learn about the world and influence policy anyway? Are there similar opportunities going forward? The Taliban says they want economics expertise, so perhaps. But I don't think we know the answers to these questions (yet), even within orders of magnitude, and whether or not this type of approach will beat RCT-type approaches depends entirely on those particular probabilities.

Reply
Ramiro21
0
0

I've been following this debate for a while now... Though I tend to agree that deworming is certainly not the most impactful, and probably not the optimal option for EA investments, I'm still puzzled here: What are you proposing, especifically? I mean, what determinate project, with measurable goals, would you propose that someone invest on - which is not already being well funded by international organizations, banks, governments or think tanks? 
My point is that I don't think anyone in the EA community disagrees about the importance of economic development - particularly so-called randomistas such as Duflo and Banerjee. I certainly don't - but I have been baffled with how little the field has gained in certainty, despite having way more data, since Nurkse v. Hirschman debates on the "balanced growth theory." I think the problem is that people are not convinced about where is the low-hanging fruit here - or that there's a clear path to take that will actually yield more overall marginal utility.

Notice that the "smell test" does not answer those questions; and though I'm quite hopeful there might be an optimal answer for development economics, I'm also skeptical Pritchett's test actually makes the right question - which, for me, sounds a bit like criticizing a training program for sedentary persons, based on an athlete's routine. And when I think about it, the one "X thing" that developed countries clearly seem to have more is that they are geographically similar and close to each other - except for small nations with outlier trajectories in very peculiarly strategical positions, such as Taiwan, Brunei or Singapore.

Reply
[anonymous]41
0
0

Hi there

On your first question - what am I proposing? 

The main thing I am proposing is quite weak - I am proposing that there be some public discussion of the arguments. Hauke and I published our piece summarising Pritchett's argument two years ago. It is the second most upvoted post in EA Forum history, which suggests that lots of people in the community found the post persuasive. On the face of it, Lant is worth taking seriously: he has a PhD in economics from MIT, has been a professor of development at Harvard and Oxford, and worked at the World Bank for 15 years. He has been defending economic growth and criticising the RCT-led approach to economics for at least 7 years. Some of the most prominent economists in the world (Rodrik, Acemoglu, Deaton, Cowen, Caplan) have endorsed similar conclusions. In spite of this, no-one in the EA community has ever published a response to these arguments. 

The initial thing Hauke and I proposed was that "A ~4 person-year research effort will find donation opportunities working on economic growth in LMICs which are substantially better than GiveWell’s top charities from a current generation human welfare-focused point of view." I would have tried to  put this to the test  when I was at Founders Pledge if I knew we would move sufficient funds to that area to make it worthwhile. I got two members of the team to explore the question of whether growth increased life satisfaction (here), which was one main piece of pushback we got to our post in the comments. 

Turning to the object-level, I take it that you are making two arguments here - (1) scepticism about growth economics, (2) neglectedness. I have several responses. 

  1. It is not possible to sustain the position that we know how to make policy-led improvements to economic welfare in the US, but not in Kenya. If we know enough about the effects of macroeconomic stabilisation and housing reform in the US, then we can also know enough about their effects in Kenya. Indeed, both of these reforms would, in my view, bring large economic benefits in many low and middle-income countries. With housing reform, there are credible estimates that this would give an enormous boost to GDP in rich countries. It would be amazing if there were one of only two things we know about how to boost GDP anywhere in the world, and they both happen to be in the US.
  2. If you read about the actual policies in place in many poor countries, it is just difficult to be sceptical that changes would bring increases in GDP. For example, Indian farmers are forced to sell their products to state middlemen  at a guaranteed minimum price. According to Wikipedia, foreign supermarkets are banned in many Indian states. The time taken to start a business ranges from 1 day in New Zealand to 230 in Venezuela. The list of such obviously bad policies is literally almost endless.
  3. The experience of the world since 1950 makes it extremely unlikely that we don't know anything about growth. Economists and states were prominently trying to increase development from 1950, and development did in fact increase. Indeed, there was more progress on all subjective and objective measures of wellbeing than all prior human history combined. It might be true that this was an enormous coincidence, but at the very least someone actually needs to sit down and explain why it was.
  4. If we truly don't know anything about growth economics, the value of information from further research must be enormous. For this not to be the case, one would have to hold that we don't know anything about growth and also that, in expectation, we can't know anything it. This seems to me to equate to the view that we can't know anything about social science - surely anything you might want to know about social science would have an effect on GDP?
  5. Suppose you were a philanthropist with the opportunity to influence policy in Rwanda in 1994. Would you focus on encouraging them to do some RCTs of social programmes, or would you try as hard as possible to get them to  do whatever it is that they did that led to sustained economic growth in Rwanda of 9% since 1995? We are in an analogous case with respect to all developing countries. I don't think it would ever be the right move to take the RCT option. 
  6. Suppose you were trying to increase economic welfare in your own country. What would you do? If you browse the manifesto of any political party, literally none of the policies will be testable by RCTs. If I was looking at something to work for to improve economic welfare in my own country, I would look to advocate for things like immigration, housing reform, road pricing, ending agricultural subsidies, advocacy for tax credits, land value taxation etc. None of these can be tested by RCTs. I don't understand why the RCT-led approach is the right one in developing countries but not in rich countries. 
  7. If our knowledge of growth is incomplete, I don't think it helps that a substantial chunk of the economics profession (>30%?) have started to work on RCT-evaluable things, which do not attempt to make a non-trivial dent on GDP per capita.
  8. A side-point on Banerjee and Duflo. In their book they do seem to come out in favour of degrowth as a solution to climate change. “Mitigation through better technologies may not do the trick; people’s consumption will need to fall. We may have to be content not only with cleaner cars but also with smaller cars, or no cars at all.” Abhijit V. Banerjee and Esther Duflo, Good Economics for Hard Times: Better Answers to Our Biggest Problems, 01 edition (Penguin, 2019), chap. 6

Neglectedness

  1. US policy is also not neglected. In fact, I would guess that US policy (land use reform, macroeconomic policy) is much more crowded than Kenyan economic policy. If the neglectedness argument works for Kenya, it works for the US as well
  2. My former colleagues Stephen and Aidan discuss the neglectedness argument here. No-one has yet engaged with these arguments.


 

Reply
Ramiro25
0
0

Thanks for the reply. I'll have to answer it... it was supposed to be short, I really didn't have the time, but then I started enjoying it. But I have a TL;DR.
TL;DR: I guess we’re not understanding each other very well, as you seem to be responding to other people (unfortunately, because I’m a big fan). I don’t see why you categorize me as a skeptic. I think we actually agree (i) RCT shouldn’t be the main path for dev-eco researchers, and (ii) there should be more research focused on developing countries. But: (iii) development economics is more complex than you make it look like, (iv) I keep the discussion on dev-eco research apart from the discussion on which cause areas to choose. Also, please, take a charitable look at my answer to your point (5)... but I don’t see how you can talk about Rwanda’s growth after 94 without talking about genocide and war.

  • It is the second most upvoted post in EA Forum history, which suggests that lots of people in the community found the post persuasive

Congrats 😉 on the other hand, like many people who commented on the subject, I upvoted your post back then, but I don’t find your case against randomistas so persuasive; I don’t upvote only posts I agree with – but if this is sending the wrong signals, perhaps I should start down-voting more.

  • he has a PhD in economics from MIT, has been a professor of development at Harvard and Oxford, and worked at the World Bank for 15 years

Actually, I think your constant emphasis on Pritchett (as if the argument for dev-eco depended on him) is one of the weakest points of the post.

  • Some of the most prominent economists in the world (Rodrik, Acemoglu, Deaton, Cowen, Caplan) have endorsed similar conclusions.

Also, five economists who disagree a lot about how development can be achieved. For instance, Acemoglu emphasizes the role of culture and institutions (but I think Melissa Dell research makes an even better case for this), but then Hausmann, Pritchett and Rodrik (2004) say, in the same article, that “Political-regime changes” and “economic reform” are “significant predictors of growth acceleration,” even though “the vast majority of growth accelerations are unrelated to standard determinants such as political change and economic reform, and most instances of economic reform do not produce growth accelerations.”

Also, Rodrik (my favorite one among them) claimed globalization (with free capital flows), national sovereignty and democracy are incompatible, defended an industrial policy approach to growth for a while, but admitted, in an interview with Tyler Cowen in 2015 (I recommend it), that the window for policies based on East Asia growth is likely gone; also in the same interview, he predicted Brazil would return to growing faster than India again – which didn’t happen (I mean, it’s sad for me, a Brazilian, to here it today).

In spite of this, no-one in the EA community has ever published a response to these arguments

I’m not sure what’s the problem here and what sort of response you’re expecting. But as mentioned before, you also received a lot of comments, many of them discussing some of your main points; and you cite Founder’s Pledge analysis of the area, which concludes (very persuasively, in my opinion) that “We’re not continuing this project because we think identifying funding opportunities would be too costly for us and for the organisations under consideration.”

Perhaps you’re victims of your own success, since searching for “randomista” in the EA Forum mostly leads to references to your post – but it shows people are actually taking it into account; I think it might provide some sort of middle ground between short-term and longterm causes). But, yeah, the only thing that seems as deep as your original post is K. Sarek’s long summary of Ogden’s “RCTs in Development Economics.”

However, I must add that I think EA is more often identified with randomista economics by outsiders – precisely because RCTs in economics provide a useful rhetorical advice to distinguish EA from other approaches to philanthropy.

 

Turning to the object-level, I take it that you are making two arguments here - (1) scepticism about growth economics, (2) neglectedness. I have several responses.

I’m sorry, but calling my position scepticism is getting too close to a strawman – an unfortunately common move among us philosophers, as if it’s impossible to engage with someone without putting a tag on them. For instance, you don’t call someone who says “people disagree about cancer” a skeptic. I explicitly said economic development is very important, and I don’t deny we know a lot about it, nor that people should research it (quite the opposite); I’m just claiming that arriving to specific recommendations on growth, at the required scale, is way harder than you make it look like (particularly if we restrict the discussion to a small sample of economists, in a limited period of time). If I'm skeptical about anything, it is that we will figure out something like a feasible way to, e.g., make Haiti grow like Botswana did, or even to ensure the latter keeps its current trajectory for decades.

Also, I suspect there’s a widespread Dunning-Kruger effect in discussions about economics, where experts usually express themselves with way less confidence than the average amateur. So, I’m not saying we’re in the dark here – actually, we are more likely being obfuscated by too much noisy light. Thus, sorry, but I think your “replies to skepticism” are answering the wrong man (unfortunately, I’d like it to be me, as I’m a big fan of your work), and I’m afraid we’re not quite understanding each other.

On the other hand, I feel compelled to answer in full, as you do provide me with some examples I can target:

  1. It is not possible to sustain the position that we know how to make policy-led improvements to economic welfare in the US, but not in Kenya

Actually, I don’t see why this is impossible.

  1. We may know how to improve A’s health (because she’s kinda healthy, or has a well-known condition), but not B’s – because he has a very peculiar condition, or because you never studied a non-WASP person like B, or because he has diabetes + allergies + cancer + covid-19 etc. and anything we do has a significant risk of causing harm (because of, e.g., noise). Nothing here implies that we only know “one of two things about how to boost” health – actually, precisely because we know a lot about health we can recognize we’re uncertain.
  2. Actually, development economists often consider it quite obvious that the advice you would give to poor countries (such as “improve capital formation”) is quite different from what you would focus on with rich countries (“get back to investing on innovation, lest others will leave you behind”).
  3. Again, if most of our research is about developed countries and mostly focus on their trajectories (“see what developed /OECD countries have done more”), we’re very likely neglecting risks of selection bias and path dependencies. I can’t stress this enough: instead of thinking about Kenya in terms of what I know from US, I’d start studying why Cape Verde and Mauritius (hmmm… bad example: small insular nations), or Botswana and Gabon are better (in terms of HDI) than Kenya.

However, I must remark: I think we actually agree economic researchers should focus more on developing countries, right?

2) For example, Indian farmers are forced to sell their products to state middlemen at a guaranteed minimum price. According to Wikipedia, foreign supermarkets are banned in many Indian states. The time taken to start a business ranges from 1 day in New Zealand to 230 in Venezuela.

I agree these examples show problems that could be fixed with simple recommendations from economics. Do you think there’s any initiative which is likely to lead to these intended outcomes? What do you think its cost-effectiveness could be? I'd gladly know more about it.

But notice they're very different issues, and they're more like symptoms than causes of the underlying conditions affecting these nations. I don’t think the problem here is anything like a lack of economic literacy, or advocacy for the right economic policies – it’s more things like inadequate equilibria.

3) The experience of the world since 1950 makes it extremely unlikely that we don't know anything about growth. Economists and states were prominently trying to increase development from 1950, and development did in fact increase. Indeed, there was more progress on all subjective and objective measures of wellbeing than all prior human history combined. It might be true that this was an enormous coincidence, but at the very least someone actually needs to sit down and explain why it was.

I think this remark is surprising, because it totally ignores my point on the Nurkse v. Hirschman debate, which is my very example of how these discussions are still unsettled. And even now, we still don’t really know, e.g, what kickstarted the Industrial Revolution in England, but we do know economists back then were wrong about many things – even though economies have increased exponentially since then.

4) I don't think there's any point in answering (4), as it seems premised on the accusation of broad skepticism. I agree that finding out, e.g., how to make everyone grow like Botswana would be huge. Except that perhaps you're framing the problem in a way that underestimates the costs of obtaining new information here. Again, development economics has been a hot topic for a long time.

5) Suppose you were a philanthropist with the opportunity to influence policy in Rwanda in 1994. Would you focus on encouraging them to do some RCTs of social programmes, or would you try as hard as possible to get them to do whatever it is that they did that led to sustained economic growth in Rwanda of 9% since 1995?

I’d first try to avoid another Rwandan genocide and the following Congo Wars, which are among the worst things that have happened in the world in the last 30 years. Seriously.

My point is not the rhetorical effect of “how could you forget that?”, but that any explanation of Rwanda’s growth after 1994 that fails to take it into account is likely flawed. WorldBank’s overview of the country refers it explicitly, for instance.

6) I will answer this separately. It’ll take longer, and, unfortunately, I’m not paid to do this – quite the opposite, perhaps.

7) If our knowledge of growth is incomplete, I don't think it helps that a substantial chunk of the economics profession (>30%?) have started to work on RCT-evaluable things, which do not attempt to make a non-trivial dent on GDP per capita.

 Well, this is the debate I actually want to have – and I guess you do, too. Dufflo and Banerjee response to this would be (in my interpretation from Good Economics for Hard Times: Better Answers to Our Biggest Problems) that we know more about how to make progress in this area (RCT), and that researchers can learn useful stuff with that (which might provide positive feedback loops); besides, we already had, and will keep having, public policies that will need to be oriented by research.

They have a point, but I agree with you (as I said before) this is likely not the best approach to development – as it is more about redistributing resources than creating them (i.e., growth), and it still leaves a huge blank in our theory of dev-eco. But this is about the roads economic researchers are trailing, not about EA recommendations.

8) This scared me a bit, too. But you have to read into the whole context. They're emphasizing the need to change consumption patterns (particularly in developed countries), because energy-efficiency programmes failed to deliver results, and so recognizing there's a trade-off here (and pending towards lowering emissions).  They are not against growth. But I got scared: what if Cowen's "Crusonia plants" are getting harder to find?

Neglectedness

I’d like to see someone investigate it more thoroughly. I agree US policy is not neglected. But I guess neglectedness is more about opportunity costs: how much do you expect the counterfactual effect of more input on this area to be? What will it add? Will it crowd-in or crowd-out additional investment?

(You know, this is analogous to a very controversial point on the role of the State in dev-eco since at least Nurkse: will a state intervention here crowd-out or crowd-in private investments?)

Finally, summarising my point: some of the best minds in economics are already taking dev-eco seriously, funded by governments and wealthy organizations, and things are still quite uncertain when it comes to feasible particular recommendations – as my future answer concerning my own country (I’ll see what I can say, despite a small but positive risk of suffering future persecution - sorry, but paranoia is starting to spread around) might exemplify. But I agree with you that sending 30% of economic researchers to do RCTs on things like transfers is likely a wrong move.

Reply
[anonymous]15
1
0

Thanks for engaging with me so deeply on this. To avoid misunderstandings, the comments about my desire for a debate were not meant as a criticism of you. I suppose I am a bit disappointed that no-one from GiveWell or Open Philanthropy has responded to Lant's arguments. When I mentioned the upvotes mine and Hauke's post got, I wasn't trying to blow my own horn (much as I like doing that), I was just trying to say that there is at least a case to answer. But two years on, no-one has engaged with the post. A lot is at stake here - I think we're leaving an awful lot of value on the table.

It seems I misunderstood your stance from your first comment. I'm not sure I've got time to respond to the new comment in full but will try if I find time. 

Reply
Stephen Clare7
1
0

Hey, I really appreciate this discussion! I wanted to jump in on one point. You note that the Founders Pledge follow-up to the original growth post (which I co-wrote) concluded that it would be too costly to continue the research to identify funding opportunities. I just wanted to note that taht was the case that because of how FP's funding model works. FP staff don't directly control the pledged funds - the members make the final decision over where to donate, and can take or leave the recommendations. 

Since policy orgs are difficult to evaluate, I was quite worried that we would take a lot of time to conduct these evaluations and then our members wouldn't end up donating to the recommendations. This would not be a concern for evaluators that have direct control over some funds. They can guarantee funding to organizations they're evaluating that reach some bar, making it worth it (in expectation) for the organizations to spend some time engaging.

Reply
GMcGowan3
0
0

Off topic, but I didn't realise you'd left Founders Pledge. May I ask what you're up to now?

Reply
JamesÖz 🔸13
1
0

I'm surprised (as it seems you are) that it has been two years since yours and Hauke's original post and there hasn't been any movement on this issue. Do you have any ideas on why this might be the case?

Seems like a potentially worrying pitfall of the EA Forum (or EA community?) if new-ish ideas are posted, solid cases are made that people agree with but nothing changes.

Reply
aggg13
0
0

While I can't find any EA work on economic policy in poor countries, two Charity Entrepreneurship incubated charities are working on health policy: 

I think the most obvious reason that this work isn't happening is just that the EA community is overwhelmingly concentrated in richer countries, and it's really hard to work on policy change without having local understanding and context. LEEP has had  success but it's likely because they are focused on such a niche and unpolarized issue, with almost no significant stakeholders that would be against this policy. Economic policies like the ones you mention are probably going to be incredibly difficult for even experienced policy people from the countries they are working in, let alone for EAs with no experience in those countries. It would also be difficult for funders like Open Phil to evaluate which grants are more or less likely to succeed, since their staff doesn't have experience in these countries.

Of course, none of these barriers are an excuse for not focusing on this topic, if the expected impact is very high. The longer-term solution here is probably to encourage EAs studying and/or working in poorer countries to consider policy careers, and for EA funders like Open Phil to focus on developing capacity/expertise in these countries (looks like they're doing this with the new South Asian air quality program), or to partner with other organizations that have more knowledge in the area. 

Final note: J-PAL and IPA have obviously been working on this for a while, though they might be pursuing smaller-scale economic policy changes than you're suggesting.

Reply
Fernando Moreno 🔸10
1
0

1.

"Open Phil has made roughly 300 grants totalling almost $200 million in their near-termist, human-centric focus areas of criminal justice reform, *immigration policy*"

I think immigration policy may be an exception for the 1,000x bar reasoning since you are pretty much helping people who now live in poor countries (although not necessarily the poorest and also probably with a middles-class selection bias - the ones who can actually afford to leave).

Huh...It made me think if we should have some kind of GiveDirectly for Immigration (FlyDirectly?) where the poorest from the most remote villages would get selected to a visa or something.

 

2.

The view from Brazil: 

We have a high influential academic establishment that cultivates marxists and outdated nationalist-industrialist approaches to economics. They pass down those "traditions" (religions?) to the younger students generation. The Washington Consensus liberalizing reforms are usually called by the derogatory neoliberal epithet and serious investigation of its impact or even actually what was originally proposed is never transmitted to students.

Maybe our academics and students should hear more about Effective Altruism. Or maybe they should also hear more about our friends at the Neoliberal Project. I'm an activist in both movements and trying my best to change some of that but good luck for me on that! =)

(In any case, here is our website for the Portuguese-speaking members out there: neoliberais.com, and if you are from Brazil and want to help with Neoliberal Project or EA drop me a line)

Economic policies is a highly political subject and since "politics is the mind-killer" this is probably why few EAs want to explicitly interfere with this. Maybe there should be some indirect support and this kind of “labor division”. For example, EAs could be stimulated to join their local Neoliberal Project chapters but keep it as two separate groups, with different meeting days, etc.

Reply
CarlShulman8
0
0

Alexander Berger discusses this at length in a recent 80,000 Hours podcast interview with Rob Wiblin.

Reply
[anonymous]18
0
0

I do think it is a key pillar of EA that there is open public discussion of arguments for and against different positions. I haven't seen much engagement with the case for focusing on economic growth. 

Reply
RyanCarey3
0
0

One excerpt worth quoting (emphasis added):

Rob Wiblin: ...if you were able to get even 10x leverage using science and policy by trying to help Americans, by like, improving U.S. economic policy, or doing scientific research that would help Americans, shouldn’t you then be able to blow Against Malaria Foundation out of the water by applying those same methods, like science and policy, in the developing world, to also help the world’s poorest people?

Alexander Berger: Let me give two reactions. One is I take that to be the conclusion of that post. I think the argument at the end of the post was like, “We’re hiring. We think we should be able to find better causes. Come help us.” And we did, in fact, hire a few people. And they have been doing a bunch of work on this over the last few years to try to find better causes...

The most relevant comments in the transcript seem to be in the section "GiveWell’s top charities are (increasingly) hard to beat".

Reply
Benjamin_Todd6
0
0

We put this question to Alexander Berger in our recent podcast. Best to engage with his response directly, but the very short version of his response was that they do expect to be able to find some opportunities that are even more leveraged than AMF within policy (AMF is 20x leveraged on cash transfers, but maybe 100x or more is possible), and that's why they're currently hiring for someone to work on each of SE Asian air quality advocacy and advocacy for more effective international aid, and also people to look for other areas like this. Though, my impression is that Alexander doesn't expect to be able to find $1bn per year of opportunities like this, so they also need to fund more scalable things like AMF.

Reply
[anonymous]5
0
0

Hello, yes this was in part a response to the arguments there where he suggested that policy is in the same ballpark as GiveWell top charities, which I don't think can be true given other things he says. 

"Yeah, I think that’s totally right. And I think, again, if you had more of a dominance argument, where it’s like, look, the returns to the policy are just always going to outweigh the returns to evidence-based aid, then I think you would end up with more back and forth and debate between them. But when you see the arguments for cost effectiveness actually ending up in the same ballpark, the same universe, it’s just like, cool, we can all get along. People are going to sort into buckets that appeal to them, or styles that work for them, or interventions that they’re more personally excited about. I think that’s totally healthy. And then when some of them think, actually, the expected value-type argument seems to lead you to think one of these is going to just totally destroy the other, that’s where I think you get a little bit more friction and tension and debate sometimes."

Because there is little public discussion about what happens in the near-termist area,  it is difficult to know why certain decisions are taken. I think it would be better for decisions that affect millions of dollars to be made with more public discussion and scrutiny. 

Reply
Marcel D4
0
0

I think I get the idea of what you’re saying, but I could be missing something in which case feel free to clarify in response to these two thoughts/comments:

  1. It seems that some economic policy reforms rely a lot on percentage improvement gains, e.g., a 5% improvement in purchasing power/income, rather than solely absolute/fixed increases (e.g., +$100 per person)—although I imagine most are varying mixtures of fixed, percentage/linear, log, and/or exponential benefits relative to starting income. However, if a given policy results primarily in a constant percentage increase in income across two countries—one with high income and one with low income—does it matter which one gets the economic improvement under the logarithmic utility function? Doesn’t a 5% increase in income for a country with $100 of average income produce the same amount of utility as a 5% increase in income for a country with $1,000 average income (for a strictly log income => utility function)?

  2. It’d be interesting to get a sense of how difficult it is to support/cause economic policy reform in rich countries vs. developing countries: that tractability consideration could shift things.

Overall though I think it’s good to be thinking about this alternative! I don’t know enough about the debate to really take a side, and like I said, I may missed some of your reasoning that dealt with this.

Reply
[anonymous]3
0
0

Hello! I will attempt to clarify, let me know whether this helps

  1. Yes you're right that it is the percentage increase that matters for welfare - roughly each doubling of income  produces the same amount of welfare, according to GiveWell. Let me put it another way. You can actually donate to Americans through GiveDirectly. Open Philanthropy currently doesn't do this, but it does donate to US policy work. This must be because of leverage. Exactly the same arguments apply in Kenya.
  2. Immigration reform and housing reform in SF seem pretty intractable to me! The former is the most controversial policy issue in contemporary US politics.
Reply
Curated and popular this week
Relevant opportunities
View more