From some previous work on technology timelines I know that NASA used Technology readiness levels to gauge the maturity of technologies for the purposes of space exploration. The scale then later was criticized as becoming much less useful when it became applied more widely and more holistically, particularly by the European Union. Here is a picture of NASA’s technology readiness levels:
The general idea is an awareness that technology has a funnel, from theoretical proposal to actually being used in space, and that the drop-out rate —the rate at which technologies don't make it to the next stage— is pretty brutal. For Effective Altruism cause candidates, the funnel might look something like this (more on the meaning of each stage later):
So if one is trying to pick technologies for the next generation of rockets, as in the case of NASA, one would want to focus on ideas with a high level of maturity. Similarly, if one is an EA incubator who wants to create a charity which has a chance of becoming a, say, GiveWell top charity, or a valuable long-termist charity, one might also want to choose an idea to implement which has a very high level of maturity.
But one would also want to pay attention to the health of the funnel. That is, one would want to make sure that each stage has an adequate amount of resources, that each stage has enough survivors left for the next stage, and that there isn’t any particularly grievous bottleneck at any stage. For example, it could be the case that many ideas are suggested, but that they are mostly not fleshed out or followed up on, which means that the set of ideas for which there is a cost effectiveness analysis might be very small (and thus its best ideas not as promising as if they had been selected from a larger pool).
This brings me to the question of what the actual stages are. I’m currently thinking in terms of the following stages:
- Ideation stage: A new idea which hasn’t been considered before, or which hasn’t been salient recently, is proposed.
- Deconfusion stage: The proposed idea is fleshed out, and concrete interventions are proposed or explored.
- There is a theory of change from intervention to impact, an understanding of how progress can be measured, etc.
- Evaluation: How does the evidence base for this kind of intervention look?
- What unit of benefit would an intervention produce and how many units would it produce per unit of resources? What does a cost-effectiveness analysis of proposed interventions reveal?
- Overall, is this cause competitive with GiveWell's/ACE's/That Long-Termist Charity Evaluator Organization Which Doesn't Exist But Probably Should's top charities?
- If the evidence base is unclear (for example, because the cause area under consideration is speculative), what specific evidence is there to think that the idea is worth it? Are there robust heuristics that allow us to conclude that working on the cause under consideration is very much likely to be worth it?
- Implementation stage: It has been determined that it's most likely a good idea to start object-level work on a specific cause/intervention (e.g., starting a charity, starting a new programme in an existing org or government, doing independent research and advising key decision-makers). Now the hard work begins of actually doing that
To clarify, I’m not saying that all EA charities or projects need to be started after a quantitative cost-effectiveness analysis. But I’m guessing it helps, particularly for charities in global health and development, and particularly if someone wants to convince EAs to go along with a new cause. Other strong evaluation methods are of course also possible.
I’m holding this model somewhat lightly. For example, I’ve noticed that the “deconfusion” and the “evaluation” stages can sometimes go in either order. For instance, we can guess that an intervention which positively affects vast amounts of people in the long-term future of humanity is probably very cost-effective, even if we’re confused about how such an intervention looks like in practice. It can also be the case that some area seems particularly valuable to become less confused about, and so an organization could be created for that purpose. Further, ongoing interventions still undergo deconfusion and evaluations on later variations and specificities.
Despite holding that model lightly, and understanding that it doesn’t fully capture reality, I’ll be referring to it in subsequent posts, because I think it’s useful to be able to talk about the “readiness level” of a cause candidate. Thanks to Ozzie Gooen and Edo Arad for comments and discussion.
Thanks for this. I have two initial thoughts which I'll separate into different comments (one long, one short - guess which one this is).
OK so firstly, I think in your evaluation phase things get really tricky, and more tricky than you've indicated. Basically, comparing a shorttermist cause area to a longtermist cause area in terms of scale seems to me to be insanely hard, and I don't think heuristics or CEAs are going to help much, if at all. I think it really depends on which side you fall on with regards to some tough, and often contentious, foundational questions that organisations such as GPI are trying to tackle. To give just a few examples:
Basically my point is, depending on answers to questions such as the above, you may think a longtermist cause area is WAY better than a shorttermist cause area, or vice versa, and we haven't even gone near a CEA (which I'm not sure would help matters). I can't emphasise that 'WAY' enough.
To some significant extent, I just think choice of cause area is quite personal. Some people are longtermists, some aren't. Some people think it's good to reduce x-risk, some don't etc. The question for you, if you're trying to apply a funnel to all cause areas, is how do you deal with this issue?
Most research organisations deal with this issue by not trying to apply a funnel to all cause areas in the first place. Instead they focus on a particular type of cause area and prioritise within that e.g. ACE focuses on near-term animal suffering, and GiveWell focuses on disease. Therefore, for example, GiveWell can make certain assumptions about those who are interested in their work - that they aren't worried by complex cluelessness, that they probably aren't (strong) longtermists etc. They can then proceed on this basis. A notable exception may be 80,000 Hours that has funnelled from all cause areas, landing on just longtermist ones and
So part of me thinks your project may be doomed from the start unless you're very clear about where you stand on these key foundational questions. Even in that case there's a difficulty, in that anyone who disagrees with your stance on these foundational questions would then have the right to throw out all of your funnelling work and just do their own. (EDIT: I no longer really endorse this paragraph, see comments below).
I would be interested to hear your thoughts on all of this.
This cube approach is interesting, but my instinctive response is to agree with MichaelA, if someone doesn’t think influencing the long-run future is tractable then they will probably just want to entirely filter out longtermist cause areas from the very start and focus on shorttermist areas. I’m not sure comparing areas/volumes between shorttermist and longtermist areas will be something they will be that interested in doing. My feeling is the cube approach may be over complicating things.
If I were doing this myself, or starting an ’exploratory altruism’ ... (read more)