EA's efforts to maximise impact uses important concepts / thinking tools, including:
- Maximisation
- Expected value
- The scientific method
- Cost-efficiency
- Hierarchies of evidence
- The ITN Framework
- Counterfactual reasoning
- Use of probabilities derived from belief and expert opinion
- Forecasting and modelling
- Comparing point estimates with wide confidence intervals
- Measuring and quantifying everything, even if there's very high uncertainty involved
- Cost-effectiveness bars
While these tools are used formally in write-ups and cost-effectiveness models, I think EAs also use these tools informally to make a variety of decisions.
As far as I'm aware, the only concept a where a weakness or criticism is widely known and widely pointed out, is expected value, where Pascal's mugging comes up often.
Given that expected value has this weakness, I think
- EA should seek out weaknesses of other thinking tools we use in EA, and maybe pay philosophers, statisticians and economists to make these weakness clearer to EAs.
- More EAs should become aware of the weaknesses of other thinking tools often used in EA.
For example,
Givewell has written about how expected value also fails to consider evidence quality
There's this Wikipedia section of criticisms of various hierarchies of evidence
I don't understand what the value of criticizing the Scientific Method in the context of EA would be.
There are criticisms of the idea that a scientific approach is sufficient to gain a more complete understanding of morality but my impression is that that criticism mostly doesn't apply to EA. EA seems more focused on applying science to the "how to do" of the question of "how to do the most good," as opposed to the "good" part.
I've met EAs who have the conviction a scientific approach to morality is sufficient but they're an unrepresentative minority. For moral philosophers in EA trying to advance EA as a theory of moral progress, while their work may be informed by science or a scientific mindset, I've only ever known most of them to not take a particularly scientistic approach.
As to applying scientific methods to develop practices in EA, there are also criticisms that the scientific method is not sufficient to identify what the best interventions are. I'm aware of almost no EA-aligned organizations that pursue their goals based on that conviction. There may be something to criticize about how EA relies too heavily on the scientific method as a still-imperfect tool. As a matter of kind, though, it seems like there aren't any potential flaws of applying the scientific method in EA that haven't already been internalized in the movement.
This was my one outlier in reading the list too.
I like the idea but I've got a question. What is a cost-effectiveness "bar?"
Do you mean a pre-existing set of one or more standard metrics that interventions being newly evaluated are measured against?
Or a preconceived threshold for cost-effectiveness an intervention has to clear before it will be seriously considered as plausibly competitive with existing top interventions? Or something else?
The second one, a theshold!