Comments
One of the things that surprised me when trying to develop potential interventions for Shrimp Welfare Project, was trying to understand a clear pathway from A (wanting to help shrimps cost-effectively) to Z (having a demonstrably impactful intervention scaling-up).
As a result of this, I spent a lot of time reading up on and learning about Programme Development Methodologies - trying to Frankenstein something together that I thought would make sense for us. I want to share this work in case it’s useful to other NGOs (in particular for NGOs in contexts similar to ours). I hope it’s useful to you 🙂
A quick note - Though this Roadmap is written in such a way as to outline the steps from A to B to C, etc. in reality, some of these steps may run in parallel (or at least, overlap in some ways). For example, one member of your team may be conducting an Evidence Review, while others are preparing to do some on-the-ground Needs Assessment work. In particular, throughout the Roadmap, you’ll likely continue to revisit and reflect on your Theory of Change, as well as updating and refining your Impact Monitoring.
I also wrote some "Appendix posts" to complement this one that some people may find useful:
1. Key Decisions - The most important decisions we’ll make, and how to make them
There are some useful Decision-Making and Creative-Thinking that it makes sense to learn early on, as they can be used throughout the roadmap. Not least, to help you with some of the most important decisions you’ll make on the path to impact, such as defining your values and choosing your career area based on personal fit. If starting a nonprofit is the best fit for you, then we can move on to deciding on a charity idea, finding a co-founder and selecting your (initial) implementation country. We can now begin to more concretely understand our path to impact through an (initial) Theory of Change.
2. Theory of Change - How and why the program will work
Now that you’ve made your key decisions, you can move on to defining your potential program. You’ll do this by first outlining some Project Objectives, before creating your first (likely somewhat broad at this point) Theory of Change diagram, sketching out the Activities you will undertake which lead to your projects Outputs, resulting in the overall positive Outcomes of your project, which ultimately achieves Impact. This will provide some clarity around the impact you intend to make, as well as the pathway to that impact. It’s likely though that there are some gaps in your knowledge and some uncertainties and assumptions in your Theory of Change, which brings us to an Evidence Review.
3. Evidence Review - A critical review of secondary sources related to our program
We now have an idea of how we intend to make an impact. It’s time to review any evidence that could help us more concretely answer some of our uncertainties and clarify our assumptions. To do this, we’ll undertake a literature review to help us answer these questions (i.e. What interventions have been tested or studied in relation to this program? Under what conditions were they implemented? Were they effective? Why or why not?). Still, however, we will need to get a better understanding of our chosen context, so we’ll go on to undertake a Needs Assessment.
4. Needs Assessment - Seeking out perspectives from the populations we aim to help
We now have a good overview of the relevant evidence in our field, but it’s time to get a deeper understanding of the context we aim to work in. This will involve gathering inputs and perspectives from the community, and more specifically, helping us (again) answer any uncertainties and clarify any assumptions within our Theory of Change. This will involve detailing what we want to learn, and how to gather data to support this. After your Needs Assessment is complete, you’ll be moving on to developing your intervention.
5. Intervention Development - Developing our ideas for theoretically impactful interventions
We now know about as much as we could hope to before actually trying stuff out in the field. So it’s time to start developing our actual intervention ideas. We’ll begin by brainstorming all the theoretically impactful interventions there could be (based on our knowledge so far). This can be broken down into four stages - 1) defining and understanding the problem and its causes, 2) clarifying which factors are malleable and have the greatest scope for change, 3) Identifying how to bring about change: the change mechanism, and 4) identifying how to deliver the change mechanism(s). After this, we should have a bunch of (ideally 10+) ideas that could theoretically be impactful. We’re going to shortlist them to our top 3-5 ideas and test them in the field.
6. Viability & Impact Testing - Testing the efficacy and effectiveness of our most promising theoretical interventions
It’s now time to develop our Implementation Plan(s), outlining how we intend to do small-scale tests (and refinements) on our shortlist of theoretically impactful interventions, in order to ensure they are actually viable in the context we’re working in. Depending on our interventions, we may have a number of potential deliverables that accompany the plan. This will also likely be the point where you start working on your Impact Monitoring systems in earnest. When we’re ready to go, we’ll start doing our small-scale tests on the most promising interventions, by conducting our Viability Testing (these are small scale, and designed to be adjusted as they go). After this, we’ll be doing our Impact Testing in order to test which of our most promising (and viable!) intervention(s) are the most impactful.
7. Growth - Investing in rigorous evaluation or implementation of the intervention
Congratulations! You’ve completed your “Explore” phase. You now have an impactful intervention to either invest in a rigorous evaluation of your intervention (i.e. an RCT or Quasi-Experimental Design) or to grow your organisation around. You’ll likely be spending this time building up your organisational capacity and relying on your Impact Monitoring systems to continuously improve your internal learning (and external accountability).
8. Scale - Expanding our proven intervention to scale
Now that you’ve had the time to rigorously evaluate (or demonstrate the effectiveness of) your intervention, you might reach the point where it’s time to scale. In which case you’ll need to detail your scale strategy, then develop your scale model, before finally, implementing your scaled program.
The most important decisions we’ll make, and how to make them.
At the outset of your path to impact, you will have a number of key decisions to make, such as what your broad charity idea might be, or how to pick an intervention country. This stage is intended to present some guidance around how you might end up answering some of these key decisions.
Before we start though, I wrote an Appendix post with decision-making tools that should help throughout the roadmap. It’ll likely be useful to refer to throughout the roadmap (which is why it’s a separate post), but I’d recommend reading through it first if you can.
[Links: I'll try and share useful further reading at the end of each sub-section. For this first section, most of these will link to resources by Charity Entrepreneurship. In particular, these Key Decisions (and Decision-Making Tools) are discussed in depth in the CE Handbook - How to Launch a High-Impact Nonprofit. This also just seems like an opportune place to recommend applying to the Incubation Program if you're interested in this kind of stuff :)]
You may already have a good sense of this, but I found the process of working through this and trying to concretely articulate my values to be a worthwhile process. Being able to articulate your values will help with the rest of the decisions in this section.
[Links: Good resources on this can be found within the Know Your Values article on the Good Enough website.]
The first of your big key decisions now is going to be your career path. Is founding a charity the best way to go? Or does your personal fit and values suggest a different career path is optimal? Even if you’re pretty set on the idea of creating an idea and seeing it through to fruition (as is the intention of this Roadmap), Non-profit Entrepreneurship might not be the best fit for you (or your idea), and instead For-Profit Entrepreneurship, or Social Entrepreneurship (a double-bottom line) might be a better choice. Additionally there are alternatives, such as working at an existing Effective Altruism organisation, or in Research, or Communications, or Policy, or Earning-to-Give, or developing Career Capital.
[Links: Your best bet here is probably going to be Career Planning Tools, such as those by 80,000 Hours, Animal Advocacy Careers, and Probably Good. Additionally, a good video on this is this talk and Q&A by Charity Entrepreneurship: “Impactful opportunities around and adjacent to charity entrepreneurship”]
I think this next step would also make sense if your Career Area wasn’t to found a charity (or a new intervention), but I’m not certain of this, and the language of the rest of this roadmap is written with the assumption that you would be starting a charity.
If you don’t have this already, you’ll likely need to go through a process to determine your impactful idea (there are also "pre-vetted" cause area, such as those identified by Charity Entrepreneurship or CEARCH). Peter Wildeford breaks down how this decision can be made for the question of “What Charity to Start?” using the Multi-Factor Decision-Making tool:
[Links: Another good resource I’d recommend that discusses this concept is “How to do Research that Matters” by Karolina Sarek.]
Two co-founders is often the sweet spot. You likely want to find someone who has complementary skills and psychology to you, as well as similar Values. Additionally, you’ll want to find someone who you personally have a strong personal fit with, so that collaboration with this person produces better work than either of you could have produced independently.
The best way to know if you work well with someone is to work with them. So ideally you’ll want to work on some discrete projects with different people, to test your fit with them. Finding a co-founder might be tough, but the Charity Entrepreneurship Incubation Program is designed to help you test your fit with a number of different co-founders, but the basic concepts should apply outside of the context of an Incubation Program like this (I also promise I’ll stop talking about CE and the Incubation Programme so much after this first section, but they really are the experts at this first stage of the Roadmap).
[Links: A good overview of this process can be found in the CE articles “How to Successfully Pick a Co-Founder” and "How to Strengthen Your Co-Founder Relationship".]
We’re now at the final step of our Key Decisions - it’s time to select (or narrow down) your implementation country. This is not set-in-stone, though bear in mind that once a location is selected, it can be difficult to change, and location-specific knowledge and skills that you invest in building may not be directly transferable to other contexts.
The process of determining a country is both practically decision-relevant (it gives you a place to start), as well as helping you think through the factors to consider for selecting an implementation country (such as the scale, neglectedness, and tractability for the problem in that country - what are the proxies you’ll be using to assess that, and how will you answer these questions).
[Links: You'll likely want to use a number of decision-making tools for this stage, in particular, the trusty Weighted Factor Model.]
How and why the program will work.
A Theory of Change (ToC) outlines the logic behind a program and explains how it is expected to bring about impact. It demonstrates how the program's activities and use of resources will lead to changes in behaviour and improvements in the lives of people (or, in our case, shrimps). This theory is an important foundation for any development program.
The Theory of Change should be referred to (and if needed, updated) often. It should function as a living document. Ideally, it should be returned to after every subsequent (major) stage in this roadmap (i.e. Evidence Review, Needs Assessment, Intervention Development etc.) as at every stage you’ll be learning more and more, and those insights can and should be reflected in your Theory of Change.
[Links: New Philanthropy Capital's "Theory of Change in Ten Steps" is a deep dive into this topic.]
Project Objectives detail a project’s intended results, which contribute to improving A, B, C, for X, Y, Z. Developing them allows you to orient yourself (and the team) to ensure you’re on track, motivate your team, demonstrate your work (to external stakeholders, and funders), and aids with your Impact Monitoring down the line. They become the foundation for your project, and ultimately will likely form the “Impact” section of your Theory of Change diagram, which you can then work back from, to get a more grounded understanding of your project’s Activities.
As with the Theory of Change diagram (in the next section), the Project Objectives will likely not remain stable over time, and should be regularly revisited, and updated if required, to quote Social Impact Navigator: "Addressing project objectives is not simply a one-time task limited to the planning phase. Rather, this task should be regularly revisited. After all, the needs of the target group or the project context may change over time...".
[Links: Social Impact Navigator's guide to Project Objectives.]
The Theory of Change diagram links (visually) how we expect to achieve Impact through our Activities (based on theory and evidence, such as: academic evidence, lived or professional experience, and sound reasoning).
There are a number of different ways to do Theories of Change, and they can be as complex or as simple as you need them to be. Often it makes sense to produce a simple, clear Theory of Change, accompanied by a Narrative, to share externally (i.e. with funders), in addition to a more complex, less-polished but detailed Theory of Change, in order to tease out assumptions and get internal synthesis among the team.
Although Theories of Change are often read left to right, it often makes sense to create them right to left. As in “for Y to change, we need X as a precondition”, which is why it often makes sense to start with defining your Project Objectives.
You’re not finished when you’ve drawn out your diagram, there’s still one last step, which is to identify the assumptions that your Theory of Change rests on. It likely makes sense to try and track the links between every box and node in the process, as well as to really articulate them in detail. For example, you might label every step in your Theory with a number, which can then be referenced in a table/spreadsheet which can allow you to write more detail and clarify points about each step, without cluttering up the diagram.
Break down every assumption to be as concise as it can be, such as breaking down ‘Users will pay for this model’, into ‘Users are interested in the model’ and ‘Users will have the means to pay for it’.
[Links: IDInsight's Impact Measurement Guide has a really useful Theory of Change section (and case study). Additionally, Step 10 of New Philanthropy Capital's "Theory of Change in Ten Steps" how to identify assumptions in more detail.]
A critical review of secondary sources related to our program.
An evidence review is a comprehensive examination of existing research on a specific program. It helps to understand what is known about the issues the program aims to address and its design. An evidence review can help answer questions such as: What interventions have been tested or studied in relation to this program? Under what conditions were they implemented? Were they effective? Why or why not? Conducting an evidence review can provide valuable insights into the program and inform its development and implementation.
A literature review is a systematic review of the relevant sources to look for evidence related to your program. The primary benefit of this is to have decision-relevant data for developing your intervention, but in many cases it makes sense to also write up and publish the literature review so that it can be shared with interested parties.
It’s worth noting that in some cases, you may be able to find a high-quality literature review compiled by someone else. If you’re confident in someone else’s work, this step can be skipped.
Write-up - Finally, you can write-up your findings from previous studies (highlighting the relevance of evidence to your program, as well as their rigour) as well as any remaining evidence gaps (particularly links or assumptions in your Theory of Change).
[Links: The Evidence Review section of the ID Insight Guide informed most of the details of this sub-section]
Seeking out perspectives from the populations we aim to help.
To be effective, a program must address the needs of the community it serves. Gathering input and perspectives from the community where the program will be implemented allows for a thorough understanding of the problem the program aims to solve. A needs assessment is a tool that helps identify the challenges facing the target population and the broader economic, political, and social context in which the program will operate. Incorporating a needs assessment into program planning can help ensure that the program is responsive to the needs of the community.
Effectively you’re trying to get answers to some key questions in the context you’re trying to work in:
The Needs Assessment process will then follow these broad steps:
[Links: This step of the process largely follows the Needs Assessment section of the ID Insight Guide.]
Developing our ideas for theoretically impactful interventions.
We now know about as much as we could hope to before actually trying stuff out in the field. So it’s time to start developing our actual intervention ideas. We’ll begin by brainstorming all the theoretically impactful interventions there could be (based on our knowledge so far). This can be broken down into four stages - 1) defining and understanding the problem and its causes, 2) clarifying which factors are malleable and have the greatest scope for change, 3) Identifying how to bring about change, and 4) identifying how to deliver the change mechanism(s).
[Links: This stage broadly follows the first 4 steps outlined in the 6SQuID methodology. Steps 5 and 6 of the 6SQuID methodology are covered in the Viability & Impact Testing section.]
Define and understand the problem and its causes.
Based on the previous stages of the roadmap, we’re confident that we’ve identified there is a problem that requires an intervention. The next step is to clarify the problem with stakeholders and existing research evidence.
[Links: Googling “Causal Pathways” or "Problem Tree" is useful to give some examples, such as this. But the takehome is that drawing out the pathway is really useful at this stage]
Clarify which factors are malleable and have the greatest scope for change.
Most interventions take place within systems and exert their influence by changing relationships, displacing existing activities and redistributing and transforming resources within. Some stakeholders may only change their behaviour if other stakeholders do so too. Interventions that address complex problems through multilevel actions are more likely to maximise synergy and long-term success.
Which of the factors that shape the Problem have the greatest scope to be changed - this could be at any point along the Causal Pathway. It probably makes sense to work through and analyse all of the Causal Factors (i.e. “steps”) identified in the Causal Pathway - what is the importance of the factor? Is there evidence on willingness and ability to change? etc.
Identify how to bring about change: the "change mechanism".
Once the most promising (and modifiable) Causal Factors have been identified, how do we achieve this change? What are the Change Mechanisms we can use (i.e. the process that triggers change for individuals, groups or communities).
[Links: Behaviour Change Strategies - SparkWave’s Conditions for Change tool]
Identify how to deliver the change mechanism(s).
After identifying the potential change mechanisms, we need to figure out how best to deliver them, which is likely to be target-group and context-specific.
Testing the viability, and then the potential impact of our most promising theoretical interventions.
Essentially everything up until this point has been part of a somewhat “theoretical” phase of work (all leading up to generating a list of all possible interventions that could be impactful, and whittling these down to the ones which are the most promising (something like 10+ ideas, ideally more).
Now we can move on to the the “viability” and “impact” phases:
[Links: Viability & Impact Testing is effectively the “Lean Startup” phase of your intervention, so I’d recommend reading the book - the goal is to test your hypothesis, iterate, and achieve validated learning.]
How will we Monitor, Evaluate, and continuously Learn from our interventions?
We can now use our Theory of Change / Programme Theories to identify key questions for our Impact Monitoring:
Essentially, we can use our Theory of Change diagram to understand what data we should collect in order to assess our impact. As a result, there will be two key outputs that we’ll create at this stage, a Monitoring & Evaluation system, and our Cost-Effectiveness Analysis (both of which are discussed in more detail in the Decision-Making Tools Appendix post). You’ll likely begin with a simple version of each, and reflect and iterate on after each subsequent stage of the roadmap.
As a quick aside - your Impact Monitoring systems (much like the Theory of Change) - should be referred to (and if needed, updated) often. It should function as a living document. Ideally, it should be returned to after every subsequent (major) stage in this roadmap. From here on out, your Impact Monitoring system(s) are going to be your trusty compass, and should become more detailed (or streamlined) depending on your intervention(s) needs.
[Links: The books “The Goldilocks Challenge” and "How to Measure Anything" are really useful dives into these concepts]
What is our plan for the interventions we want to test?
Whether you’re testing an intervention as part of Viability Testing, or as part of Impact Testing, you’re likely going to need an implementation plan. The implementation plan clarifies the conditions and resources necessary for successful implementation and the related risks and assumptions.
The key difference to bear in mind when writing the implementation plan for a given intervention is that for Viability Testing the primary aim is the learning value (does this intervention work in this context? Are there any tweaks we can make along the way to improve it?). Whereas the primary aim of Impact Testing is to assess the (cost)-effectiveness of our intervention(s). So the plans might vary somewhat depending on that context. For example, the plan for an intervention that is being tested for Viability will likely be less detailed, and you’ll likely want to build more slack into the plans, to allow for things to go wrong, or to change the way you’re doing things along the way. You’ll also likely spend less on data collection, or prioritise a few proxy data points during this phase, rather than doing a more focused look at evidence of effectiveness.
The following sub-sections are potential deliverables that could be part of your implementation plan. It’s largely based on the implementation plan we created, and yours might look quite different (depending on the intervention(s) you’re testing). But the point is that the implementation plan isn’t the only document you’ll be creating at this stage, but rather, the implementation plan will be your guide in helping you determine the deliverables that will be required during your testing phase.
Tests and refinements on a small scale.
It may turn out that some of our interventions are not viable (in this context) in which case, they should be removed at this stage. As a result, we should prioritise tests that can shed light on multiple major crucial considerations. Additionally, we should aim to anticipate possible unintended effects of the intervention and minimise any that might be harmful.
The point of this step isn’t to unwaveringly stick to the implementation plan, it’s highly likely that small tweaks can be made to your plan along the way, in which case, go ahead and make those tweaks. The point of this step is to learn and iterate - we’re going to be doing a robust Pilot next as part of the Impact Testing phase. This is not your pilot - so don’t be afraid to make changes to your intervention at this stage - in fact, that’s the whole point of this stage!
Collect sufficient evidence of effectiveness to justify rigorous evaluation/implementation.
Depending on the nature of the results of your Viability Testing, this step could look very different. For example, if you were testing interventions that:
This is the phase most people refer to as the Pilot. We now know that the intervention we are testing is viable in the context we’re working in. The goal here is to understand the Impact of the intervention, in particular the (cost)-effectiveness of the intervention. So there should be a stronger focus on a robust plan which likely will not need to be tweaked/changed (much) once the Pilot has begun. Essentially the big difference here is that you’re expecting that the intervention you’re Piloting here is your organisation's intervention (or one of them). So you want to replicate as close to what this intervention will look like as your future Business-As-Usual as possible (of course, you can, and should, still update your intervention based on new evidence during and even after the Pilot, but the intention is to have already unearthed all the learnings you can, so you shouldn’t have to). You’re likely testing your intervention on a much bigger scale, and the Implementation Plan you create should reflect that. You’re also likely trying to calculate an explicit cost-effectiveness of your intervention that captures nuance, so your implementation plan should also reflect that.
It’s quite possible you’ll just be doing a single Pilot here, Impact Testing an intervention you’re pretty confident in. But it’s also possible you have a few interventions that seem promising, in which case you’ll be doing 2-3 pilots (potentially at the same time). In fact, that’s even potentially preferred, as the goal at this point is to figure out what the most impactful intervention is, in order to begin implementation in earnest. Depending on the results of your Pilot(s), you may either decide to move on to implementing the intervention (if successful), moving back to the Viability Testing phase (if there is still more learning/tweaking to be done), or even shutting down the project altogether. If you’re testing multiple interventions, you may simply find one is the most cost-effective and then scale that one up (in which case, you’re hopefully following a “Scaling Down SOP” for the interventions that will no longer continue).
Investing in rigorous evaluation or implementation of the intervention.
You’ve determined which intervention is the most impactful, it’s time to roll it out. In the beginning of your Exploit phase (I’m not a huge fan of this terminology, but Explore/Exploit does succinctly capture a useful idea), you’ll likely be in a stage of slow, sustainable growth, finding beneficiaries and rolling out your intervention, increasing your organisational capacity (i.e. staff / equipment / fundraising) along with your growth.
Alternatively, you may determine that the next step is to invest in a Randomised Control Trial (RCT), or a Quasi-Experimental Design (QED). I’m hoping to add more information on these in a future update of the Roadmap.
At this point, your key tools are likely your Theory of Change and Impact Monitoring systems. We essentially want to update our Theory of Change to reflect the new understanding of our intervention after all that Testing. From there, we want to make sure that our Impact Monitoring systems are in place and robust and accurately reflect our detailed Theory of Change. From here, we let the ship take its course, learning and improving as we go.
And for some projects, this might be enough, a slow and steady growth until we reach a sort of “impact equilibrium”, likely depending on your cause area being constrained in some way (i.e. talent or funding constrained), or even just because you work on an issue that is neglected and tractable, but not particularly scaleable. However, for those organisations working on a potentially scalable solution (for example, by scaling up operations in one country, or attempting to explore replication in other countries), it’s time to think about our scale strategy.
[Links: “Managing to Change the World', along with the resources and templates they provide on their website. Especially if a big part of your Growth phase is building organisational capacity and resilience. Additionally, the “Improving Social Impact” section of Social Impact Navigator is useful here, in particular the checklist for becoming a Learning Organisation.]
Expanding our proven intervention to scale.
Now that you’ve had the time to rigorously evaluate (or demonstrate the effectiveness of) your intervention, you might reach the point where it’s time to scale. Spring Impact’s “Scaling Impact Toolkit” outlines three key phases for scale:
They have released a bunch of tools for Strategy, including the Scale Readiness Diagnostic, Problem Definition, Intended Impact, Design Your Core, Identify Risky Assumptions, Test Risky Assumptions, Sweet Spot for Scale, Measure what Matters, and Doer and Payer at Scale. Their tools for the Model and Implementation phases are still in development.
As these tools are still in development, I'm going to leave this section fairly brief for now, but hope to add to it in a future update of the Roadmap.
[Links: The Scaling sections here follow the path laid out in Spring Impact’s open-source “Scaling Impact Toolkit”.]
