How should we assess very uncertain and non-testable stuff?


There is a good and widely accepted approach to assessing testable projects - roughly what GiveWell does.  It is much less clear how EA research organisations should assess projects, interventions and organisations with very uncertain non-testable impact, such as policy work or academic research. There are some disparate materials on this question on blogs, Open Phil's website, on 80k's website, in the academic/grey literature etc. However, this information is not centralised; it's not clear what the points of agreement and disagreement are; lots of the organisations who will have thought about this question will have insights that have not been shared with the community (e.g. maybe CSER, FHI?); and the mechanisms for sharing relevant information in the future are unclear. 

Ultimately, it would be good to collate and curate all the best material on this, so that EA researchers at separate EA orgs would have easy access to it and would not have to approach this question on their own. As a first step, we invite people who have thought about this question to discuss their insights in the comments to this post. Topics could include:

  • How far should we use quantified models?
    • e.g. The Oxford Prioritisation Project used quantified models to assess really uncertain things like 80k and MIRI. 
    • Open Phil doesn't appear to do this (they don't mention that often in their public facing docs.)
  • What role should the Importance/Neglected/Tractable framework play?
    • Should it be used to choose between interventions and/or causes?
    • Should quantitative models be instead of ITN?
    • How quantified should the ITN framework be? As quantified as 80k's? More intuitive?
  • What are the key takeaways from the history of philanthropy, and the history of scientific research?
  • What's the best way to assess historical impact?
    • Process tracing or something like it?
    • What are the main biases at play in assessing historical impact?
    • Who do you ask ?
  • Is hits-based giving the right approach and what follows from it?
    • How relevant is track record, on this approach? Sometimes Open Phil takes account of track record, other times not. 
    • Should we favour choosing a cause area and then making lots of bets, or should we be more discerning?
  • What are the most important considerations for assessing charities doing uncertain-return stuff?
    • Strength of team
    • Current strategy
    • Potential to crowd in funding. 
  • What are the best theories of how to bring about political change?
  • How much weight should we put on short to medium-term tractability?
    • Given the nonlinear nature of e.g. political change, current tractability may not be the best guide. 
  • Are there any disciplines we could learn from?
    • Intelligence analysis.
    • Insurance (especially catastrophe insurance). 


Thanks, John and Marinella @ Founders Pledge.