Holden wrote a 2011 blog post related to this question; I would summarize by saying that you have to make a Bayesian update of your priors, and if the evidence has a high point but low reliability then you should consider improving the reliability of your evidence.
Not to pile on but this seems not quite right to me:
Moreover, 3 years of waiting to scale and finding out that the results are same, would create a huge amount of deaths not averted, happy to provide more precise estimates at a later stage.
This argument seems not to consider the downside of the alternative, that we spend charitable dollars on something ineffective, that we didn't know about because we didn't do an RCT. This situation would also "create a huge amount of deaths not averted".
I would be happy to join a call too, but I should be clear that I personally would be very unlikely to fund this — I delegate all my GHD grantmaking to GiveWell.
Great comment, Nick, and largely agreed. I want to push back a bit on the idea that blinding doesn't matter though. We should expect to see a placebo effect from an intervention like this, even if the monitors only produced noise. Any change to the workflow, new equipment, training, researcher monitoring, etc. can disrupt routines in a way that could be positive or negative (and which may not be sustained).
A related example from a completely different context is that almost any traffic safety intervention shows beneficial effects in the first 6 months or so but then usually crash statistics return to baseline after that — one explanation is that changes in traffic patterns push drivers out of System 1 and into System 2, making them more cautious and aware in the process. Eventually drivers get used to the new design and we may see there was no permanent improvement.
It would (probably?) be unethical to use fake monitors, but some other control ideas could include:
Set up the system, including all the staff training and making the staff connect the monitor leads, but then don't display anything and don't produce any alerts.
Provide the control hospital with some other medical equipment of comparable cost (e.g. pulse oximeters, blood pressure cuffs, sterilizers, etc.), and a similar quantity of training on some other area of medicine (for example, airway management).
Provide the control hospital with a cash grant comparable to the amount of installing and operating the IMPALA program.
If we can't run RCTs for whatever reason, another option would be (as in the traffic example above) long-term monitoring to see if the effects fade out over time. One could use a staged deployment design where hospitals are randomized to the time at which they receive the intervention, then 5 years later you could compare the hospitals that have had it for a long time to the ones that received it only recently.
It seems to me that the Chinese are mainly fast-following, for example via distillation. I doubt they could take the lead in capabilities, with or without a U.S. slowdown. Price is another matter of course.
Given the headline, I was a bit surprised that the article did not discuss kidnapping or cross-border human smuggling. Frankly, the Afghan practices described here seem tame compared to what's been documented in the cocoa or shrimp industries.
As someone privy to only a small fraction of it, I suspect there is a large quantity of private information that leads those in the know to conclude that a lot of philanthropic dollars are coming, including to global health and development.
Would be interested in seeing a version of this that looks at WELLBYs or otherwise uses moral weights to holistically judge all of the portfolio's effects. But, this piece does seem like a convincing example of the idea the best charities are 100–1000× more effective than the average charity.
You may be interested to know that GiveWell didn’t start out as a GHD evaluator. The initial idea was to evaluate many different cause areas, including economic opportunity and education. But I think what happened is that once they had compiled the research, donors saw that the GHD stuff was like 10,000× more cost effective, so the other cause areas didn’t attract much money moved and in 2012 they shut the other cause areas down. 2009 was the year GiveWell discovered AMF, and other than that, their 2009 top charities list is pretty unrecognizable.
It sounds like you are struggling with whether it is better to have true belief or comforting ones. Maybe you can take comfort in knowing this is not a new dilemma! As early utilitarian John Stuart Mill wrote, “better to be Socrates dissatisfied than a fool satisfied.”
I think partly this is a reaction to dishonest charity fundraising communications that say things like “Donate today and your gift will be matched 8×”, when in reality either the match limit is virtually guaranteed to met or else the matching donor would offset any unused matching funds with a separate donation.
So consider your last sentence — maybe if I do the thing you will give an extra $100 at the end of the year or if I don’t do the thing you will give an extra $150 at the end of the year. Technically I only gave the $50 gift if you did the thing, but in reality there was no counterfactual difference in total giving.
Holden wrote a 2011 blog post related to this question; I would summarize by saying that you have to make a Bayesian update of your priors, and if the evidence has a high point but low reliability then you should consider improving the reliability of your evidence.
Not to pile on but this seems not quite right to me:
This argument seems not to consider the downside of the alternative, that we spend charitable dollars on something ineffective, that we didn't know about because we didn't do an RCT. This situation would also "create a huge amount of deaths not averted".
I would be happy to join a call too, but I should be clear that I personally would be very unlikely to fund this — I delegate all my GHD grantmaking to GiveWell.
Great comment, Nick, and largely agreed. I want to push back a bit on the idea that blinding doesn't matter though. We should expect to see a placebo effect from an intervention like this, even if the monitors only produced noise. Any change to the workflow, new equipment, training, researcher monitoring, etc. can disrupt routines in a way that could be positive or negative (and which may not be sustained).
A related example from a completely different context is that almost any traffic safety intervention shows beneficial effects in the first 6 months or so but then usually crash statistics return to baseline after that — one explanation is that changes in traffic patterns push drivers out of System 1 and into System 2, making them more cautious and aware in the process. Eventually drivers get used to the new design and we may see there was no permanent improvement.
It would (probably?) be unethical to use fake monitors, but some other control ideas could include:
If we can't run RCTs for whatever reason, another option would be (as in the traffic example above) long-term monitoring to see if the effects fade out over time. One could use a staged deployment design where hospitals are randomized to the time at which they receive the intervention, then 5 years later you could compare the hospitals that have had it for a long time to the ones that received it only recently.
It seems to me that the Chinese are mainly fast-following, for example via distillation. I doubt they could take the lead in capabilities, with or without a U.S. slowdown. Price is another matter of course.
Thank you for this interesting writeup.
Given the headline, I was a bit surprised that the article did not discuss kidnapping or cross-border human smuggling. Frankly, the Afghan practices described here seem tame compared to what's been documented in the cocoa or shrimp industries.
As someone privy to only a small fraction of it, I suspect there is a large quantity of private information that leads those in the know to conclude that a lot of philanthropic dollars are coming, including to global health and development.
Would be interested in seeing a version of this that looks at WELLBYs or otherwise uses moral weights to holistically judge all of the portfolio's effects. But, this piece does seem like a convincing example of the idea the best charities are 100–1000× more effective than the average charity.
You may be interested to know that GiveWell didn’t start out as a GHD evaluator. The initial idea was to evaluate many different cause areas, including economic opportunity and education. But I think what happened is that once they had compiled the research, donors saw that the GHD stuff was like 10,000× more cost effective, so the other cause areas didn’t attract much money moved and in 2012 they shut the other cause areas down. 2009 was the year GiveWell discovered AMF, and other than that, their 2009 top charities list is pretty unrecognizable.
It sounds like you are struggling with whether it is better to have true belief or comforting ones. Maybe you can take comfort in knowing this is not a new dilemma! As early utilitarian John Stuart Mill wrote, “better to be Socrates dissatisfied than a fool satisfied.”
I think partly this is a reaction to dishonest charity fundraising communications that say things like “Donate today and your gift will be matched 8×”, when in reality either the match limit is virtually guaranteed to met or else the matching donor would offset any unused matching funds with a separate donation.
So consider your last sentence — maybe if I do the thing you will give an extra $100 at the end of the year or if I don’t do the thing you will give an extra $150 at the end of the year. Technically I only gave the $50 gift if you did the thing, but in reality there was no counterfactual difference in total giving.