Click the link above to see the full article and charts. Here is a summary I wrote for the latest edition to the 80,000 Hours newsletter, or see the Twitter version.
Is it really true that some ways of solving social problems achieve hundreds of times more, given the same amount of effort?
Back in 2013, Toby Ord1 pointed out some striking data about global health. He found that the best interventions were:
- 10,000x better at creating years of healthy life than the worst interventions.
- 50x better than the median intervention.
He argued this could have radical implications for people who want to do good, namely that a focus on cost-effectiveness is vital.
For instance, it could suggest that by focusing on the best interventions, you might be able to have 50 times more impact than a typical person in the field.
This argument was one of the original inspirations for our work and effective altruism in general.
Now, ten years later, we decided to check how well the pattern in the data holds up and see whether it still applies – especially when extended beyond global health.
We gathered all the datasets we could find to test the hypothesis. We found data covering health in rich and poor countries, education, US social interventions, and climate policy.
If you want to get the full picture on the data and its implications, read the full article (with lots of charts!):The bottom line is that the pattern Toby found holds up surprisingly well.
This huge variation suggests that once you’ve built some career capital and chosen some problem areas, it’s valuable to think hard about which solutions to any problem you’re working on are most effective and to focus your efforts on those.
The difficult question, however, is to say how important this is. I think people interested in effective altruism have sometimes been too quick to conclude that it’s possible to have, say, 1,000 times the impact by using data to compare the best solutions.
First, I think a fairer point of comparison isn’t between best and worst but rather between the best measurable intervention and picking randomly. And if you pick randomly, you expect to get the mean effectiveness (rather than the worst or the median).
Our data only shows the best interventions are about 10 times better than the mean, rather than 100 or 1,000 times better.
Second, these studies will typically overstate the differences between the best and average measurable interventions due to regression to the mean: if you think a solution seems unusually good, that might be because it is actually good, or because you made an error in its favour.
The better something seems, the greater the chance of error. So typically the solutions that seem best are actually closer to the mean. This effect can be large.
Another important downside of a data-driven approach is that it excludes many non-measurable interventions. The history of philanthropy suggests the most effective solutions historically have been things like R&D and advocacy, which can’t be measured ahead of time in randomised trials. This means that restricting yourself to measurable solutions could mean excluding the very best ones.
And since our data shows the very best solutions are far more effective than average, it’s very bad for your impact to exclude them.
In practice, I’m most keen on the “hits-based approach” to choosing solutions. I think it’s possible to find rules of thumb that make a solution more likely to be among the very most effective, such as “does this solution have the chance of solving a lot of the problem?”, “does it offer leverage?”, “does it work at all?”, and “is it neglected?”
Hypothetically, if we could restrict ourselves to solutions that are among the top half and then pick randomly from what remains, we can expect a cost-effectiveness that’s about twice the mean. And I think it’s probably possible to do better than that. Read more in our article on choosing solutions.
So, suppose you use a hits-based approach to carefully pick solutions within an area. How much more impact can you have?
My overall take is something like 10 times more. I feel pretty uncertain, though, so my range is perhaps 3-100 times.
A 10-times increase in impact given the same amount of effort is a big deal. It’s probably underrated by the world at large, though it may be overrated by fans of effective altruism.
A final thought: I think you can increase your impact by significantly more than 10 times by carefully choosing which problem area to focus on in the first place. This is a big reason why we emphasise problem selection in career choice so much at 80,000 Hours. Overall, we’d say to focus on exploring and building career capital first, then start to target some problem areas, and only later focus on choosing solutions.
Hey Ben, thanks for the replies -- adding some more to get closer to the same page 🙂
Re your 1), my criticism here is more one of emphasis and of the top-line messaging, as you indeed mention these cases of advocacy and research.
I just think that these cases are rather fundamental and affecting the conclusions very significantly -- because we are almost never in the situation that all we can choose from are direct interventions so the solution space (and with it, the likely variance) will almost always look quite different than what is discussed as primary evidence in the article (that does not mean we will never choose direct interventions, to be sure, just that the variance of solutions will mostly be one that emerges from the conjunction of impact differentials).
Re your 2), I think this is mostly a misunderstanding -- my comment was also very quickly written, apologies.
I am not saying we should always choose the most leveraged thing ever, but rather that the solution space will essentially always be structured by conjunction of multipliers. There are reasons to not only choose the most leveraged solution, as you point out, but I don’t think this is enough to argue that the most effective actions will not usually be conjunctive ones.
I agree that the data in the article is useful for specifying the shape of a particular impact differential, I am mostly arguing that it understates the variance of the solution space.
(I worry that we are mixing expected and realized value here, I am mostly talking about conjunctive strategies affecting how the variance of the solution space looks like on expected value, this does not preclude the realized value sometimes being zero (and that risk aversion or other considerations can drive us to prefer less leveraged actions.)).
Re your 3) & 4) I agree -- my understanding was that these are the factors that lead you to only 10x and my comment was merely that I think direct intervention space variance is not that informative with regards to solution selection in most decision contexts.
Aside: I agree with you that I don’t think that advocacy by itself is a 100x multiplier in expectation.