All of Rory Fenton's Comments + Replies

Love the clarity of the post but I agree with Geoffrey that the $ impact/household seems extremely low and I also don't follow how you get to $1k+/HH (which would be like doubling household income).

Back calculating to estimate benefits/household:

  • $1.5m national savings over 5 years = $300k/year
  • Number of adopters:
    • 50m people in Uganda
    • 5 people/household means 10m households
    • 1/3 of households use charcoal: 10m/3 = ~3m households use charcoal
    • 1% adopt: 3m * 1% = 30k adopting households
  • Benefits/household: $300k/year over 30k adopting households = $10/household/ yea
... (read more)
2
NickLaing
8mo
Nice one Rory and George I agree with most of your points and appreciate the engagement a lot! Its super true that change around cooking practise is super hard - perhaps I was being ambitious at a 20% chance of 1% conversion to soaking with a $300,000, but its very hard to know. Sorry just to clarify, the apparent $1000 a year was in response to this -. "To put that into a scale my first-world brain can understand, 0.006% over 100,000 USD is 60 USD. It’s definitely something but also feels low return for the habit change. And at that price, could easily see someone reverting back to cooking beans w/o soaking for the convenience."  I was just correcting Geoffry's 60 x 3 (for the 1/3 of population) and then x 5-10 (family size people are cooking for) saying that on his "American scale", the apparent savings to the person buying the charcoal might be more like 1000 dollars a year. Probably shouldn't have waded into that because of the cnfusion. I think when we look at dollar benefits, it is important to look both on an individual and population level. I completely agree that those small individual monetary benefits will make it hard to convince people to change - but one advantage is the benefits (however tiny) are visible on a day to day basis.  But if 1% of the population could be convinced, then the aggregate benefit of money saved would be big - even if many families/individuals barely noticed the difference. As a side note, I was pretty conservative on some inputs (charcoal saved, charcoal cost) so the benefits might be higher than stated here, even if only by a maximum of maybe 3x at the upper end.  

Thanks Chris, that's a cool idea. I will give it a go (in a few days, I have an EAG to recover from...)

One thing I should note is that other comments on this post are suggesting this is well known and applied, which doesn't knock the idea but would reduce the value of doing more promotion. Conversely, my super quick, low-N look into cash RCTs (in my reply below to David Reinstein) suggests it is not so common. Since the approach you suggest would partly involve listing a bunch of RCTs and their treatment/control sizes (so we can see whether they are cost-o... (read more)

As a quick data point I just checked the 6 RCTs GiveDirectly list on their website. I figure cash is pretty expensive so it's the kind of intervention where this makes sense. 

It looks like most cash studies, certainly with just 1 treatment arm, aren't optimising for cost: 

StudyControlTreatment
The short-term impact of unconditional cash transfers to the poor: experimental evidence from Kenya432503
BENCHMARKING A CHILD NUTRITION PROGRAM
AGAINST CASH: EVIDENCE FROM RWANDA
74 villages74 villages (nutrition program)
100 (cash)
Cash crop: evaluating large c
... (read more)
2
david_reinstein
1y
The “problems caused by unbalanced samples” doesn’t seem coherent to me; I’m not sure what they are talking about. If the underlying variance is different between the treatment and the control group: * That might justify a larger sample for the group with larger variance * But I would expect the expected variance to tend to be larger for the treatment group in many/most relevant cases * Overall, there will still tend to be some efficiency advantage of having more of the less-costly group, generally the control group

Hi Christian-- agreed but my argument here is really for fewer  treatment participants, not smaller treatment doses

Ah, that's helpful data. My experience in RCTs mostly comes from One Acre Fund, where we ran lots of RCTs internally on experimental programs, or just A/B tests, but that might not be very typical!

1
roboton
1y
Would be super interested to see the results of some of these RCTs / AB tests. Were any of them published apart from the Lime SMS study? We're looking for great examples of learning orgs that do this and some studies from 1AF would be a great motivator/example.

Hey Aidan-- that's a good point. I think it will probably apply to different extents for different cases, but probably not to all cases. Some scenarios I can imagine:

  1. A charity uses its own funds to run an RCT of a program it already runs at scale
    • In this case, you are right that treatment is happening "anyway" and in a sense the $ saved in having a smaller treatment group will just end up being spent on more "treatment",  just not in the RCT. 
    • Even in this case I think the charity would prefer to fund its intervention in a non-RCT context: p
... (read more)
2
NickLaing
1y
Thanks Rory  - I think your general idea is good, and in some cases could be a good option! I could be wrong, but from my experience working in the development world these 4 scenarios aren't really how RCTs generally happen. Usually there will be a partnership with a RCT running NGO  (like IPA) or a university department (J-PAL at MIT) where the partner organisation pays for and organise everything. Sometimes scenario 4 could happen as part of a grant application

Hi Nick-- thanks for the thoughtful post! 

I think cash arms make a lot of intuitive sense, my main pushback would be a practical one: cash and intervention X will likely have different impact timelines (e.g. psychotherapy takes a few months to work but delivers sustained benefits, perhaps cash has massive welfare benefits immediately but they diminish quickly over time). This makes the timing of your endline study super important, to the point that when you run the endline is really what determines which intervention comes out on top, rather than the ... (read more)

1
NickLaing
1y
Thanks so much Rory and for the links to your earlier post and the USAID stuff! I think your criticism is a good criticism of RCTs in general, but it seems to me more a criticism comment about RCT design then being a clear argument against comparing with cash transfers. RCTs on development NEED longer term outcome measurement, and surely need at a minimum 2 data points at 2 different times after the study. And of course the most important data point is after many months or even many years as you talked about in your article.    I'm not at all sure about the ethical side either . Medical RCTs  compare a new trial treatment against the most up-to-date treatment - not so much because we worry about "tricking" a patient like you say (there are still plenty of RCTs with sugar placebo pills which is deemed ethically OK), we are  still OK with a kind of 'deception'. What we AREN'T OK with is doing a trial where we give the control arm nothing at all, when we know there is a better option than nothing for the medical condition. And I'd argue that cash is usually a better option than nothing for many development conditions.  That's a great and sobering point about the counterfactual potentially being inaction if cash transfers won the day. Why should the counterfactual be inaction though? I would hope as development people we are good enough that if Cash was equivalent or better than intervention X, this wouldn't lead us not to inaction but instead to give more cash instead. Maybe I'm naive and idealistic though, and maybe you're right that there is actually a practical advantage in seeing a positive impact of intervention X, even if it is worse than a cash transfer. I don't think that should be the case though. That's the whole question really - should we spend our millions on RCTs asking "What is the impact of X", or "Is X better than cash". What we really want to know, the practical question which underlies the research question is is "Should we be implementing this i

Hey!

LEEP is indeed working on this -- I mentioned them in my original comment but I have no connection to them. I was thinking of a campaign on the $100M/year scale, comparable to Bloomberg's work on tobacco. That could definitely be  LEEP,  my sense (from quick Googling and based purely on the small size of their reported team) is they would have to grow a lot to take on that kind of funding, so there could also be a place for a large existing advocacy org pivoting to lead elimination. I have not at all thought through the implementation side of things here. 

Hi! We at LEEP would also be excited about a campaign at something like $100 million/year - great to see you submitted the idea Rory. We recently wrote this  proposal aimed at the Biden administration with some of our ideas: https://www.dayoneproject.org/post/eliminating-childhood-lead-poisoning-worldwide

And yes, we’re currently a small team (3 FTE), but hoping to expand significantly later this year!

How does the time and monetary cost of buying these products compare to the time and monetary cost of giving cash?

The total value of the bundle ($120) includes all staffing (modelled at scale with 100k recipients), including procurement staff, shipping, etc. This trial was a part of a very large nonprofit, which has very accurate costs for those kinds of things.

But obviously the researchers didn't know beforehand that the programs would fail. So this isn't an argument against cash benchmarking.

That's true, I don't think I made my point well/clearly with th... (read more)

I really like the idea of asking people what assets they would like. We did do a version of this to determine what products to offer, using qualitative interviews where people ranked ~30 products in order of preference. This caused us to add more chickens and only offer maize inputs to people who already grew maize. But participants had to choose from a narrow list of products (those with RCT evidence that we could procure), I'd love have given them freedom to suggest anything. 

We did also consider letting households determine which products they rece... (read more)

Thanks for the interesting reflections. 

I agree that longer term data collection can help here in principle, if the initial differences in impact timing wash out over the years. One reason we didn't do that was statistical power: we expected our impact to decrease over time, so longer term surveys would require a larger sample to detect this smaller impact. I think we were powered to measure something like a $12/month difference in household consumption. I think I'd still call a program that cost $120 and increased consumption by, say, $3/month 10 yea... (read more)

1
Mo Putera
2y
Life satisfaction score is a snapshot, while WELLBY is its integral over time, so it's WELLBY you want.  The World Happiness Report's article on this is a good primer.

Thanks for sharing! 

 

My initial sense is that China's method is focused on controlling rainfall, which might mitigate some of the effects of climate change (e.g. reduce drought in some areas, reduce hurricane strength) but not actually prevent it. The ideas I had in mind were more emergency approaches to actually stopping climate change either by rapidly removing carbon (e.g. algae in oceans) or reducing solar radiation absorbs on the Earth's surface (making clouds/oceans more reflective, space mirrors). 

Will all funding applications be made public? If so, is it possible for ask for specific application not to be public?  No problem if actual funding will be publicized, I'm just wondering about the applications themselves. Thanks!

3
Nick_Beckstead
2y
No, funding applications will not be made public.

Eliminate all mosquito-borne viruses by permanently immunizing mosquitoes 

Biorisk and Recovery from Catastrophe

Billions of people are at risk from mosquito-borne viruses, including the threat of new viruses emerging. Over a century of large-scale attempts to eradicate mosquitoes as virus vectors has changed little: there could be significant value in demonstrating large-scale, permanent vector control for both general deployment and rapid response to novel viruses. Recent research has shown that infecting mosquitoes with Wolbachia, a bacterium, out-co... (read more)

Campaign to eliminate lead globally

Economic Growth

Lead exposure limits IQ, takes over 1M lives every year and costs Africa alone $130B annually, 4% of GDP: an extraordinary limit on human potential. Most lead exposure is through paint in buildings and toys. The US banned lead paint in 1978 but 60% of countries still permit it. We would like to see ideas for a global policy campaign, perhaps similar to Bloomberg’s $1B tobacco advocacy campaign (estimated to have saved ~30M lives), to push for regulations and industry monitoring.

Epistemic status: The “prize”... (read more)

Pilot emergency geoengineering solutions for catastrophic climate change

Research That Can Help Us Improve

Toby Ord puts the risk of runaway climate change causing the extinction of humanity by 2100 at 1/1000, a staggering expected loss. Emergency solutions, such as seeding oceans with carbon-absorbing algae or creating more reflective clouds, may be our last chance to prevent catastrophic warming but are extraordinarily operationally complex and may have unforeseen negative side-effects. Governments are highly unlikely to invest in massive geoengineering so... (read more)

3
Kirsten
2y
I thought China has already done some low-key geoengineering? https://80000hours.org/podcast/episodes/kelly-wanser-climate-interventions/