Teacher for 7 years; now working as a Researcher at CEARCH: https://exploratory-altruism.org/
I construct cost-effectiveness analyses of various cause areas, identifying the most promising opportunities for impactful work.
This is excellent. It will definitely inform the way I work with RCTs from now on.
I would just like to quibble your use of the word "discount". In most of the post you use it synonymously with "multiplier" (ie. a 60% discount to account for publication bias means you would multiply the experimental result by 0.6). However in your final worked example under External Validity you apply "discounts" of 20%, 20% and 5% for necessary conditions, special care effects and general equilibrium effects respectively, with the calculation 100/(100+20+20+5)*100% = 74%. This alternative definition took me some time to get my head around.
I'd also expect to see the 20%, 20%, 5% combined as multipliers, since these effects act independently: 1/(1.2*1.2*1.05) *100% = 66% (although I realise the final result isn't too different in this case).
Some quick observations:
Nice question!
Firstly I would challenge your assumptions in using "lives saved" as currency in your very brief final estimate. Depending on your moral basis, the lives of people in need of a kidney transplant are probably less valuable than those typically saved by malaria interventions:
Consider using DALYs instead.
I think you are right to think about considering government funding. It seems plausible that transplants would save govt money compared to dialysis. But transplant recipients also live longer (that's the point of this) and will incur healthcare costs for a longer time.
If you do come up with a figure for "costs saved" you could try to convert govt spending into DALYs or similar. Some countries' healthcare systems evaluate treatments based on the cost per QALY (in the UK publicly-funded drugs are supposed to cost less than £30,000 per QALY). If you assume that any money the system saves is invested in DALYs at the threshold rate, converting is simple. I don't think this applies in the US, where costs are split between private and public sources in a confusing way.
All of this brings me back to broad agreement with your instinct to count the health gains as "free", because it's quite complicated to do otherwise. But I would caution about complexity. Objections to the buying/selling of organs are partly based on notions of sanctity that we might see as stupid (much like the fear of GM crops that is blocking Golden Rice) but also partly on valid concerns about the hard-to-predict secondary effects of an organ trade (exploitation, decreased concern for the poor who "can always sell a kidney if they are truly struggling").
As far as I can tell, ex-ante cost-effectiveness is not the most important figure for someone considering whether to fund a future project. I think the expected benefit per unit cost is more important.
For reference, you give this definition:
We define the ex-ante cost-effectiveness of an R&D project as the expected value of its ex-post cost-effectiveness, given only the information that is available before the project is conducted.
I think I understand what this means, but I am going to attempt to show why it's not that useful using a simple example.
Scenario 1: Suppose we know that a project will have benefit and that the projected cost has distribution
Then the ex-post cost-effectiveness of the project will have distribution
and thus has expected value
Why is this not useful? It does not reflect the expected return-on-investment, and is not sensitive to high-cost scenarios. Consider Scenario 2, a similar project with known benefit and cost with distribution
Scenario 2 is clearly much less cost-effective than Scenario 1. But the ex-ante cost-effectiveness is , very close to .
What a decision-maker really wants to know is the amount of benefit they can expect from each unit of investment. This can be given by.
Scenario 1:
Scenario 2:
We can see that this does appropriately reflect the difference in cost-effectiveness between the two scenarios. What I'm not so sure about is how we might give the expected benefit per unit cost as a distribution, rather than just a point-estimate.
It seems likely that I'm missing something.
I think Ghandi's point nods to the British Empire's policy of heavily taxing salt as a way of extracting wealth from the Indian population. For a time this meant that salt became very expensive for poor people and many probably died early deaths linked to lack of salt.
However, I don't think anyone would suggest taxing salt at that level again! Like any food tax, the health benefits of a salt tax would have to be weighed against the costs of making food more expensive. You certainly wouldn't want it so high that poor people don't get enough of it.
Thanks again!
I think I have been trying to portray the point-estimate/interval-estimate trade-off as a difficult decision, but probably interval estimates are the obvious choice in most cases.
So I've re-done the "Should we always use interval estimates?" section to be less about pros/cons and more about exploring the importance of communicating uncertainty in your results. I have used the Ord example you mentioned.
Thanks for your feedback, Vasco. It's led me to make extensive changes to the post:
Overall I agree that interval estimation is better suited to the Drake equation than to GiveWell CEAs. But I'd summarise my reasons as follows:
Thanks again, and do let me know what you think!
Nice question.
First it's worth noting that "size" is only one contributing factor in cost-effectiveness. But I acknowledge that there are some issues (rare genetic diseases, for example) that are so small that we could never justify spending lots of resources on them.
You ask how EA values diversity of issues. Clearly EA works on a number of issues, and I'd argue this is because:
Yet clearly EA does not work on everything. Some things are just not plausibly among the most cost-effective things we can do. This does mean admitting that we are not going to "solve" type 1 diabetes in the foreseeable future.
Responding to your diabetes example, it really is sometimes better to turn down the opportunity to "solve" a problem in order to have impact elsewhere. Doctors in the 1960's already had a way of saving ~100% of cholera sufferers in hospital using intravenous saline solution. But most people couldn't make it to the hospital. Scientists developed an oral hydration that was less effective, but could be administered at home. This oral solution was, and has been, much more important and revolutionary than the hospital treatment. As one of the scientists put it, “It’s better to reach 80 percent of people with something that’s 80 percent effective than five percent of people with something that’s 100 percent effective.”
Ultimately, EA is about doing the most possible good with the resources available. The result is that if "your" niche or issue is something very ineffective (like training guide dogs, which is far less cost-effective than some ways of preventing or reversing blindness in poor countries) then you should change your niche.