Stan Pinsent
Bio
Participation2
Teacher for 7 years; now working as a Researcher at CEARCH: https://exploratory-altruism.org/
I construct cost-effectiveness analyses of various cause areas, identifying the most promising opportunities for impactful work.
Posts 9
Comments42
Topic Contributions1
This is excellent. It will definitely inform the way I work with RCTs from now on.
I would just like to quibble your use of the word "discount". In most of the post you use it synonymously with "multiplier" (ie. a 60% discount to account for publication bias means you would multiply the experimental result by 0.6). However in your final worked example under External Validity you apply "discounts" of 20%, 20% and 5% for necessary conditions, special care effects and general equilibrium effects respectively, with the calculation 100/(100+20+20+5)*100% = 74%. This alternative definition took me some time to get my head around.
I'd also expect to see the 20%, 20%, 5% combined as multipliers, since these effects act independently: 1/(1.2*1.2*1.05) *100% = 66% (although I realise the final result isn't too different in this case).
Some quick observations:
- Eastern Europe comes out well when economy, workforce and law are given equal weighting. They are helped by the scaling on the GBP and income indicators - in the “rankings” version, Western & Northern European start to compete
- India and Philippines look better as the emphasis on economy increases
- Weighting only on economy and law (cheap, stable countries) gives a top 5 of India, Rwanda, Georgia, Indonesia, Uzbekistan
- With a double weighting on workforce, the top 5 are New Zealand, Australia, US, UK, Georgia
- Georgia comes out looking surprisingly good. It appears to have flukey scores on education and crime.
- I did some top 10 rankings with different ratings
Nice question!
Firstly I would challenge your assumptions in using "lives saved" as currency in your very brief final estimate. Depending on your moral basis, the lives of people in need of a kidney transplant are probably less valuable than those typically saved by malaria interventions:
- they probably gain less extra lifetime
- they are probably more likely to have health problems that diminish their quality of life
- they probably have less productivity and more healthcare costs ahead of them for the previous two reasons
Consider using DALYs instead.
I think you are right to think about considering government funding. It seems plausible that transplants would save govt money compared to dialysis. But transplant recipients also live longer (that's the point of this) and will incur healthcare costs for a longer time.
If you do come up with a figure for "costs saved" you could try to convert govt spending into DALYs or similar. Some countries' healthcare systems evaluate treatments based on the cost per QALY (in the UK publicly-funded drugs are supposed to cost less than £30,000 per QALY). If you assume that any money the system saves is invested in DALYs at the threshold rate, converting is simple. I don't think this applies in the US, where costs are split between private and public sources in a confusing way.
All of this brings me back to broad agreement with your instinct to count the health gains as "free", because it's quite complicated to do otherwise. But I would caution about complexity. Objections to the buying/selling of organs are partly based on notions of sanctity that we might see as stupid (much like the fear of GM crops that is blocking Golden Rice) but also partly on valid concerns about the hard-to-predict secondary effects of an organ trade (exploitation, decreased concern for the poor who "can always sell a kidney if they are truly struggling").
As far as I can tell, ex-ante cost-effectiveness is not the most important figure for someone considering whether to fund a future project. I think the expected benefit per unit cost is more important.
For reference, you give this definition:
We define the ex-ante cost-effectiveness of an R&D project as the expected value of its ex-post cost-effectiveness, given only the information that is available before the project is conducted.
I think I understand what this means, but I am going to attempt to show why it's not that useful using a simple example.
Scenario 1: Suppose we know that a project will have benefit and that the projected cost has distribution
Then the ex-post cost-effectiveness of the project will have distribution
and thus has expected value
Why is this not useful? It does not reflect the expected return-on-investment, and is not sensitive to high-cost scenarios. Consider Scenario 2, a similar project with known benefit and cost with distribution
Scenario 2 is clearly much less cost-effective than Scenario 1. But the ex-ante cost-effectiveness is , very close to .
What a decision-maker really wants to know is the amount of benefit they can expect from each unit of investment. This can be given by.
Scenario 1:
Scenario 2:
We can see that this does appropriately reflect the difference in cost-effectiveness between the two scenarios. What I'm not so sure about is how we might give the expected benefit per unit cost as a distribution, rather than just a point-estimate.
It seems likely that I'm missing something.
- What is your rationale for focusing on expected value of ex-post cost-effectiveness ?
- Could you use an adapted method to make an ex-ante prediction of the benefit per unit cost of Baumsteiger’s R&D project?
I think Ghandi's point nods to the British Empire's policy of heavily taxing salt as a way of extracting wealth from the Indian population. For a time this meant that salt became very expensive for poor people and many probably died early deaths linked to lack of salt.
However, I don't think anyone would suggest taxing salt at that level again! Like any food tax, the health benefits of a salt tax would have to be weighed against the costs of making food more expensive. You certainly wouldn't want it so high that poor people don't get enough of it.
Thanks again!
I think I have been trying to portray the point-estimate/interval-estimate trade-off as a difficult decision, but probably interval estimates are the obvious choice in most cases.
So I've re-done the "Should we always use interval estimates?" section to be less about pros/cons and more about exploring the importance of communicating uncertainty in your results. I have used the Ord example you mentioned.
Thanks for your feedback, Vasco. It's led me to make extensive changes to the post:
- More analysis on the pros/cons of modelling with distributions. I argue that sometimes it's good that the crudeness of point-estimate work reflects the crudeness of the evidence available. Interval-estimate work is more honest about uncertainty, but runs the risk of encouraging overconfidence in the final distribution.
- I include the lognormal mean in my analysis of means. You have convinced me that the sensitivity of lognormal means to heavy right tails is a strength, not a weakness! But the lognormal mean appears to be sensitive to the size of the confidence interval you use to calculate it - which means subjective methods are required to pick the size, introducing bias.
Overall I agree that interval estimation is better suited to the Drake equation than to GiveWell CEAs. But I'd summarise my reasons as follows:
- The Drake Equation really seeks to ask "how likely is it that we have intelligent alien neighbours?", but point-estimate methods answer the question "what is the expected number of intelligent alien neighbours?". With such high variability the expected number is virtually useless, but the distribution of this number allows us to estimate the number of alien neighbours. GiveWell CEAs probably have much less variation and hence a point-estimate answer is relatively more useful
- Reliable research on the numbers that go into the Drake equation often doesn't exist, so it's not too bad to "make up" interval estimates to go into it. We know much more about the charities GiveWell studies, so made-up distributions (even those informed by reliable point-estimates) are much less permissible.
Thanks again, and do let me know what you think!
Nice question.
First it's worth noting that "size" is only one contributing factor in cost-effectiveness. But I acknowledge that there are some issues (rare genetic diseases, for example) that are so small that we could never justify spending lots of resources on them.
You ask how EA values diversity of issues. Clearly EA works on a number of issues, and I'd argue this is because:
Yet clearly EA does not work on everything. Some things are just not plausibly among the most cost-effective things we can do. This does mean admitting that we are not going to "solve" type 1 diabetes in the foreseeable future.
Responding to your diabetes example, it really is sometimes better to turn down the opportunity to "solve" a problem in order to have impact elsewhere. Doctors in the 1960's already had a way of saving ~100% of cholera sufferers in hospital using intravenous saline solution. But most people couldn't make it to the hospital. Scientists developed an oral hydration that was less effective, but could be administered at home. This oral solution was, and has been, much more important and revolutionary than the hospital treatment. As one of the scientists put it, “It’s better to reach 80 percent of people with something that’s 80 percent effective than five percent of people with something that’s 100 percent effective.”
Ultimately, EA is about doing the most possible good with the resources available. The result is that if "your" niche or issue is something very ineffective (like training guide dogs, which is far less cost-effective than some ways of preventing or reversing blindness in poor countries) then you should change your niche.