I could go on a rant about this... :) Very briefly, my own impression is that:

• ~All organizations should do explicit strategic planning, including research organizations.
• Usually these plans should be public.
• A theory of change diagram would be a good example of what I'd like to see when doing strategic planning, though not exhaustive / sufficient in itself. More broadly, I'm not too fuzzed about the specific methodology used, or way in which plans are communicated.
• Perhaps surprisingly given these views, I mostly agree with the adage that "plans are useless, planning is everything". I expect that usually the main value of strategic planning will be due to improving the thinking of organizational leadership rather than for example due to enabling external feedback. (This can be different for some organizations, e.g. when buy-in by many external stakeholders is unusually important.) So for example the main reason I think plans should be public is that it incentivizes leadership to expend more effort and rigour when doing strategic planning. (I expect pushback along the lines of "but making plans public incentives plans that sound defensible rather than being actually useful". I don't have time to go into more detail but my response is: yes, these effects are real, but they can be mitigated to a large extent, and overall I think being public is still worth it on net).

But I've also found that I tend to disagree with staff at EA (research) organizations on this, including the people running the organizations or responsible for strategy. Sometimes the disagreement seemed fairly large, to the extent that it was a significant and sometimes decisive factor in my decisions between different jobs.

So after updating on the revealed and stated preferences of people with roughly similar amounts of or more experience in organizational leadership, my best guess is that the optimal policy is a bit closer to their impressions than mine.

(Of course, these disagreements are gradual rather than binary. It's not that people say "we should never think about our strategy". It's more that e.g. they think they have thought about strategy organically anyway, and that there's not sufficient value added by explicitly engaging in something labelled 'strategic planning' to justify the time cost.)

At HLI, we've found creating a Theory of Change (TOC) very useful. It was (at least for me) quite a painful process of making explicit various assumptions and uncertainties and then talking through them. I think if we hadn't done it explicitly we would (a) have made a less thoughtful plan and (b) different members of the team would be carrying around their own plans in their heads.

Going through a ToC process has also helped us to focus on meeting the needs of our target audiences. After developing our TOC, we sent out surveys to some of our key stakeholders to identify their concerns about subjective well-being measures and what new information would make them more likely to use them. Their responses provided the basis for our research agenda and the questions we have chosen to investigate this year.

We have a slightly more detailed version of our ToC diagram on our blog. Thanks for pointing out that it’s hard to find; we’ll think about putting it on a main page.

Sadly don’t have time to go into much depth on this, but we strongly recommend it to all charities that run through our CE program (including all the research orgs) and create a ToC for each idea we research.

Here are my own current thoughts on those questions. These thoughts are of course tentative, and my main aim here is to learn more on this matter (otherwise I would’ve made a regular post rather than a question post):

On Q1: It seems like it’s rare for research orgs to make (publicly accessible) ToC diagrams? As I said, I only know of two examples of research orgs that have made such diagrams. But I haven’t done a systematic or extensive search.

If it’s indeed uncommon for research orgs to make ToC diagrams, that might be due to the “potential downsides” I list below.

On Q2: I have basically no evidence regarding whether many orgs make non-public ToC diagrams. I’ve never seen an example of that, but that’d likely be the case even if there are many such non-public diagrams, since I’ve only worked at one research org thus far (Convergence), and their ToC diagram was public anyway.

On Q3: My independent impression - i.e., what I’d believe if I wasn’t updating my beliefs based on what others seem to believe - is that, indeed, more research orgs should make ToC diagrams. But if so few research orgs do that, and most research orgs are run by smart people, maybe I should interpret that as evidence that there’s some good reason not to do this which I’m missing?

But I’m not sure how strong that update should be, because it might be more a matter of norms, inertia, and/or not having thought of doing this.

Some potential benefits of making ToC diagrams (even if they aren’t publicly accessible) that came to my mind are that doing so could help the org:

• Make decisions about which research questions/directions to pursue
• E.g., maybe you’d realise that a line of research that seems fascinating to your researchers actually doesn’t have any clear path to influencing your intended outcomes. This would likely push in favour of deprioritising that line of research. (But this consideration could be outweighed. See also Can we intentionally improve the world? Planners vs. Hayekians)
• Make decisions about which audiences to target
• E.g., you might realise that, in order to achieve your ultimate intended outcomes, you mainly need to influence decisions by [policymakers / AI researchers / EAs interested in global health / company leaders / whatever]. You can then have that in mind as you choose research questions, writing styles, types of output (e.g., EA Forum posts vs academic articles), etc.
• Make decisions about what non-research activities to do, how to do them, and how much time to allocate to them
• E.g., you might realise that most of the outcomes you want to bring about require specific people to read your work, such that it’s worth dedicating a decent amount of time to building relationships with those people, attending relevant conferences, etc., rather than just doing research and assuming it’ll have an impact somehow.
• Work out how they can assess their progress and their impact
• E.g., my personal opinion is that, once Convergence had made a ToC diagram, this helped us come up with metrics for assessing our progress and impact. It’s practically impossible to “directly” assess whether one has improved the expected value of the future (which was our ultimate aim), but easier to assess how well we’re achieving outcomes on the causal path from our actions to that outcome.
• Note I said “easier” rather than “easy”, and that I’m comparing this to something that’s “practically impossible” - I definitely still find assessing the impact of longtermist research very difficult!

(Note that each of those benefits could likely be captured to at least some extent by an org just describing their theory of change fairly explicitly, without actually making a diagram about it. The same point applies to the remainder of this “answer”.)

And some potential downsides of making ToC diagrams that came to my mind are that doing so could:

• Take up a lot of time
• Not actually provide much clarity anyway
• Lead to focusing too much on a particular set of paths to impact that one explicitly maps out, at the expense of doing things that seem worthwhile based on intuitions and heuristics
• See again Can we intentionally improve the world? Planners vs. Hayekians, and perhaps also Goodhart’s law.
• Of course, it’s hard to say when it’d be better to operate based on intuitions and heuristics rather than a ToC
• It also seems possible that the ideal would be to make a ToC, and sometimes/often refer to it, but not always refer to it or always require that a project “fits” within that ToC.

On Q4: I’d guess that, if a research org has a ToC diagram, it’d typically make sense to make it publicly accessible. This could mean actively sharing it (e.g., in a post like this one or this one), making it prominently visible on the org’s website, or burying it in some obscure corner of the org’s website.

Some potential benefits of making ToC diagrams publicly accessible (rather than just making them in the first place) that came to my mind are that doing so could:

• Help potential donors, potential employees, etc. understand why you’re doing what you’re doing, and thus make informed decisions about whether to donate to you, work for you, etc.
• E.g., perhaps what you’re doing seems inefficient or unlikely to connect to important real-world outcomes unless people understand your ToC
• Or perhaps what you’re doing is misguided, and it’d be good for the world for people to have more evidence of that if that’s true, so that they can then support something better instead
• Help other orgs learn from and adopt clever aspects of your strategy and thinking
• Help stakeholders coordinate with you or support you
• I’m less sure what I could mean by this point
• One vague and speculative example: If many orgs’ ToC diagrams suggest that a particular type of conference would be useful in order for those orgs to make their research impactful, and those ToC diagrams are publicly accessible, this could lead to someone setting up such a conference.

(It might be possible to capture some of these benefits by just sharing ToC diagrams with specific people, such as potential donors or employees, rather than making those diagrams totally publicly accessible.)

And some potential downsides of making ToC diagrams publicly accessible that came to my mind are that doing that could:

• Lead to misperceptions about an org’s strategy
• E.g., it could lead to stakeholders focusing too much on the set of pathways to impact highlighted in the ToC diagram (rather than other, vaguer reasons to believe impact might occur), or perceiving the org’s thinking as quite simplistic
• Allow other actors to harm your org or harm things you value, as a result of them learning from your strategy and thinking
• E.g., another research org might improve itself by adopting some of your ideas, and then get some funding you’d have otherwise gotten
• But maybe there’s often little harm in that, as that other org could then do useful things as well
• More speculatively, if there’s some actor that actively wants to prevent you accomplishing your mission, perhaps seeing your ToC would make that easier for them

But again, these are just my current, tentative thoughts, and I’m primarily interested in hearing what others have to say!

My thanks to Courtney Henry and David Kristoffersson for discussions that informed this answer.

A forum resource on ToC in research which I found insightful: Are you working on a research agenda? A guide to increasing the impact of your research by involving decision-makers

Should they

Yes, but ToC don't improve impact in isolation (you can imagine a perfectly good ToC for an intervention which doesn't do much). Also, if you draw a nice diagram, but it doesn't actually inform any of your decisions or change your behavior in any way, then it hasn't really done anything. A ToC is ideally combined with cost-benefit analyses, the comparing of multiple avenues of action, etc and it should pay you back in the form of generating some concrete, informative actions e.g. consulting stakeholders to check your research questions, generally creating checkpoints at which you are trying to get measurements and indicators and opinions from relevant people.

For more foundational and theoretical questions where the direct impact isn't obvious, there may be a higher risk of drawing a diagram which doesn't do anything. I think there's ways to avoid this - understand the relevance of your research to other (ideally more practical) researchers who you've spoken to about it such as a peer review process, make a conceptual map of where your work fits in to other ideas which then lead to impact, try to get as close to the practical level as you realistically can. If it's really hard to tie it to the practical level it is sometimes a sign that you might need to re-evaluate the activity.

Do they

Back in academia, I didn't even know what a "theory of change" was, so I think not. But, one is frequently asked to state the practical and the theoretical value of your research, and the peer review and grant writing process implicitly incorporates elements of stakeholder relevance. However, as an academic, if you fail to make your own analyses, separately from this larger infrastructure, you may end up following institutional priorities (of grant makers, of academic journals, etc) which differ from "doing the most good" as you conceptualize it.

Nuno Sempere of QURI just published a short post on Pathways to impact for forecasting and evaluation, which has two big diagrams and opens with:

As part of the Quantified Uncertainty Research Institute's (QURI) strategy efforts, I thought it would be a good idea to write down what I think the pathways to impact are for forecasting and evaluations. Comments are welcome, and may change what QURI focuses on in the upcoming year. 

He adds in a section on "Reflections":

Most of the benefit of these kinds of diagrams seems to me to come from the increased clarity they allow for when thinking about their content. Otherwise, I imagine that they might make QURI's and my own work more legible to outsiders by making our assumptions or steps more explicit, which itself might allow for people to point out criticism. Note that my guess about the main pathways are highlighted in bold, so one could disagree about that without disagreeing about the rest of the diagram.

I also imagine that the forecasting and evaluations pathways could be useful to organizations other than QURI (Metaculus, other forecasting platforms, people thinking of commissioning evaluations, etc.) 

It seems to me that producing these kinds of diagrams is easier over an extended period of time, rather than in one sitting because one can then come back to aspects that seem missing.

Brilliant to raise this topic, and I like what you wrote but both diagrams are weak. For me a good diagram shows very specifically how a single change will be achieved, and shows if there is too long a chain for success to be likely.

Regardless of diagrams, we all have conscious or unconscious theories of change, and many (especially in climate change) have been useless.

The classic unconscious theory of change is:

brainy guy does research > publishes > civil servants write a policy > wise politicians decide > funds are allocated > policy is implemented well

The main weakness here is that it's a very long chain, with many obstacles in each link.

Compare to coal industry's ToC, which they learned from Big Tobacco:

"create confusion about climate science" + "capture Congress" > block all carbon tax proposals nationally and internationally

Good ToC for EAs involves:

• selecting good and astute targets of change (whether in real world, movement or metta)
• smart routes to achieving the change
• updating appropriately (at a Goldilocks frequently, not too rarely to stay current, not too often to frustrate the teams doing the work)

For "natural conservatives", this may sometimes involve finding ways of opposing harmful change, and proving that some policies are a bad idea, or need fine tuning.

In Owen Cotton-Barratt's recent 80,000 Hours interview, there's some discussion that felt to me interesting and somewhat relevant here (though it wasn't explicitly about theory of change diagrams). The most relevant part starts around 1:52:00, where he says:

I think that there are also pretty interesting questions about when to try and be strategic about what can I achieve here? And like, “Okay, I would like to create this change in the world. Now I’ll come up with a plan for that.” Versus trying to be helpful and say, “Look, it just seems like it would be good if someone was providing this kind of information in this context, and then I don’t know how people will use it”, but trusting that other people or information processing systems, which might be bigger than people, it might be communities will make use of the things which one is feeding into that.

(That's just the start of the relevant part - he then goes on to say more interesting stuff on that.)

Here's a quote from a comment Owen Cotton-Barratt's made elsewhere (focusing on explicit theories of change in general, rather than specifically ToC diagrams):

Some general thoughts:

• Advantages of having an explicit theory of change:
  • Makes it easier to sync up about direction/priorities/reasons for doing things
  • Makes it easier for people to engage critically, or otherwise to notice mistakes and course-correct
• Disadvantages of having an explicit theory of change:
  • Easy to have the case where your best expression of something is dumber than your real internal sense of it
    • In this case it may be preferable to be guided by the internal sense rather than the explicit version
    • (this is at least some distant relative of Goodhart's law)
  • To the extent that you're going to be guided by your internal sense rather than an explicit version, sharing something as an explicit theory of change can be misleading
• In general I think it's good to encourage lots of explicit discussion about theories of change
  • Ideally without committing to reaching an "answer", but having that as a goal may be helpful for prompting the discussion

I think that I find the disadvantages quite emotionally resonant, which may pull me to err too far in the direction of not being explicit. I have appreciated some cases where people have pushed me towards "let's have a discussion where we're pretty explicit about best guesses".

I've just finished a project working with a large American foundation (not sure I'm okay to say which, but it's in the top 10 largest). They use logic models / ToC diagrams internally as their lingua franca: everything is expressed as a diagram. I feel a little ambivalent.

On the one hand, they are a clear and expeditious way of expressing information that might otherwise be crammed into a memo no one will read. They also very clearly express causal flow in a way that other media might not, which can facilitate understanding. At the foundation I worked with, they seem to be used primarily as a way of rapidly communicating mechanisms of action (e.g. in proposed foundation grants or investments) to and between program officers, who seem extremely pressed for time.

On the other hand, I also saw reports and presentations crammed full of incredibly detailed logic models. I'm talking about pages and pages on which small-type boxes and arrows completely fill each page. I really don't think this is useful. These incredibly detailed models are not easy to understand at a glance, and they seem to sit in an unhappy middle ground: by being complicated, they challenge comprehension, but by being simplifications, they occlude important details relevant to the mechanism being described.

I got the impression that because the order had come down from on high to put everything in a logic model, it was being done even in contexts where these models made no sense. I worried that the focus on logic models encourages only a logic model - level understanding of the world, while simultaneously eating up huge amounts of foundation time creating diagrams that few will look at or understand.

However, I am still a convert. I think theory of change / logic models do have a lot of value, but I think they need to be used sparingly and kept small. I'd make some kind of a rule: no more than twenty boxes in a model, or something like that.