Written by Sophie Gulliver and Thomas Billington
TL;DR - EAs want to do the most good with the resources we have. But how do we know if we are actually doing the most good? How do we know if our projects are running efficiently and effectively? How do we know we are achieving impact? How do we check we are not causing harm? Monitoring and evaluation (M&E) theories and tools can help EA organisations answer these questions—but are currently not being applied to a sufficient degree. These theories and practices will help us achieve more impact and value in the short-term and long-term.
Here, we outline a few ways to build M&E knowledge and skills within EA, including:
- Helpful resources
- The EA M&E Slack community group
- Signing up for our pro bono M&E support
- M&E as a career choice
EA is “a project that aims to find the best ways to help others, and put them into practice”. Specifically, EA distinguishes itself from general do-gooding through its commitment to doing the most good possible with its resources.
To these ends, the EA community is only as good at achieving its goal of “finding the best way to help others” as it is at knowing what the best way to help others is. But how can we know what the best ways to help others are? And how will we know if what we are doing is actually helping others in a cost-effective and impactful way?
Monitoring and evaluation (M&E) experts, tools and concepts are some of the best ways to realise effective altruism’s philosophy of maximising impact and doing good better.
M&E are two interrelated concepts that help us track and assess a project or organisation’s progress, impact, and value. In essence, if you really care about whether a project is making the world better, then you should also care about M&E.
However, in our experience, the integration of M&E tools and expertise within the EA community has been variable and mostly restricted to the global health and development sector.
This post argues that the broader EA movement should also be engaging with M&E more to ensure we are doing the most good possible.
Note: This post aims at being a broad overview of M&E. Our objective is to increase knowledge and awareness of M&E in the EA space, as well as give some first steps to learn more. Don't worry if you feel like you can't apply it immediately. Future posts will delve into details, and see our section below for what you can do right now.
What is M&E?
Monitoring and evaluation are two distinct functions that work synergistically. They can be defined as:
Monitoring: the systematic and routine collection of information to track progress and identify areas for improvement. Monitoring asks questions like:
- Are we on track?
- Are we reaching the right people?
- Are we using our money and time efficiently?
- What can we improve?
For example, if you were running a project to reduce deaths through improved water quality, you might regularly monitor chlorine availability at water points, or run quarterly surveys asking community members how often they are chlorinating their water.
Evaluation: the rigorous assessment of the value of a project or programme to inform decision-making. This assessment usually measures the performance of the project against criteria that define what a ‘valuable’ project looks like. These could be criteria like ‘impactful’ or ‘cost-effective’ or ‘sustainable’. Unlike monitoring, evaluations answer bigger questions to inform important decisions about a project’s future:
- How well is this project doing overall?
- How valuable is it?
- Is it worth it?
- Should we adapt, scale up or scale down?
Given the deeper analysis required, evaluations are typically undertaken at important decision points within a project’s life.
For example, for our above water quality project our evaluation might assess cholera and mortality rates in our program area and compare these with similar non-treatment areas. This might help us decide if our program achieves our planned outcomes and ultimate impact.
We operationalise M&E throughout the entire life of a project—from the conception of an idea, through piloting and implementation, all the way to scaling up (or down). The diagram below gives a basic illustration of the different stages of M&E and what we aim to do at each stage.
The amount of M&E and how you might go about it changes from organisation to organisation depending on the size, budget, and resources. However, a basic M&E system often has the following elements:
A Theory of Change (ToC) which maps out the step-by-step process of how you think your project will address a problem and create change. The theory starts at your inputs (e.g. chlorine and chlorine dispensers) and leads through your outputs (e.g. amount of chlorine dispensed) to intermediate outcomes (e.g. reduced cases of cholera) all the way to your eventual impact (e.g. reduced mortality). This helps you understand how your project works and helps you think about what to track and evaluate.
A Monitoring and Evaluation Framework. Now you know the steps for how you think your project will achieve change, you need to write down how you will know if these changes are happening. What data can you gather, from whom, how often, and using what methods to track progress? This includes both quantitative and qualitative data.
Evaluation questions. What questions would you want answered to know if your project was good or worth it? For example: How cost-effective is my project? How well are we reaching those in most need? How impactful is our project in terms of lives saved? You can answer these questions through different quantitative, qualitative, and mixed evaluation methods.
In smaller organisations it can be hard to find the time, money, and expertise to do all this. If you are interested in doing this, our linked resources can be a great place to start! You can also contact us for additional pro bono advice and join the Slack M&E community to learn from others. For larger organisations it is beneficial to invest in M&E and hire M&E experts internally or externally to support you with this work.
What are some common struggles with M&E?
EA is a broad movement, and so there are lots of ways organisations are approaching M&E depending on their size, maturity, cause area, and other factors. Here are some common issues we have seen some organisations face that can hold them back from achieving a rigorous M&E system:
- No explicit theory of how their program works. It is hard to monitor and evaluate our work when we don’t have a clear theory of change that shows how certain activities should lead to outputs, outcomes, and eventual impacts. This means we don’t know what to look for to see if our program is on its way to achieving impact.
- A theory of change exists but lacks detail like the level of evidence/ certainty, the risks, and the assumptions we have for each causal ‘linkage’. For example: How certain are we based on existing evidence that an activity like training leads to an outcome like behaviour change? What are we assuming? What could risk this relationship?. Without sufficient detail, M&E can’t help us focus on assessing the most important linkages and thus cannot determine if our program is actually working.
- Not fully diagnosing the problem they are trying to solve. A needs assessment helps us systematically think through the root causes and effects of the problem we are trying to solve with our program and weigh the pros and cons of all possible solutions. This helps us design a robust project that has considered all available evidence; it also helps us identify alternative pathways we can take if our project is not working as intended.
- Conducting only monitoring, often focused on quality assurance. That is, are we delivering what we said we would deliver?. While quality assurance monitoring is necessary and often the easiest place to start, it is important in time to move beyond looking at inputs and outputs and start monitoring early outcomes and unintended consequences. It is also important not to miss the forest for the trees— a phenomenon that occurs when you are so preoccupied with monitoring specific indicators that you aren’t asking or answering bigger evaluation questions like, how valuable is this project to our communities? or is this project worth it?
- No evaluation plan. Often, organisations want to jump straight to evaluation without having done the planning required. First, we need to know what criteria we are evaluating the program against. Within EA, we often consider impact and cost-effectiveness as valuable criteria. However, people within our teams or the communities we work with might have different ideas about what makes a program valuable (e.g. sustainability, equity, cultural acceptability). M&E can surface important, but sometimes difficult, conversations about what matters most, but we need to decide the criteria against which we will evaluate a program before we start evaluating. This is why it is important to frame some evaluation questions that indicate what is most important to answer to determine the value of your program. Even if you are only just starting, it is important to start thinking about the type of evaluative evidence you want so you can start to prepare. Many types of evaluation, like Randomised Control Trials or quasi-experimental designs, require early planning, so you have time to collect baseline data. It is also easier if your monitoring data harmonises with your evaluation questions!
- No good examples of M&E in their field. Animal welfare and longtermist causes can struggle as there are not as many resources available in these areas on how to do M&E. This can make it harder to formulate indicators as there are no common standards to help think through different evaluation designs; for instance, those in longtermist causes can’t conduct an RCT or quasi-experimental design.
- No clear M&E responsibilities. Smaller organisations especially struggle with allocating M&E responsibilities because they have lots to do and not enough people. These organisations also often have lots of “jacks of all trades” but not necessarily someone with expertise in M&E. Without clear M&E responsibilities and in-house expertise, it can be hard to develop a rigorous M&E system beyond basic monitoring.
- No resources dedicated to M&E. M&E can be expensive given the required staff/external time and tools. And every dollar that goes towards M&E is a dollar not going towards direct programming. M&E, like projects, must be cost-effective. It requires some investment, matched to the needs of the organisation.
Why should I care about doing M&E better?
Every programme that aims to change the world contains assumptions about how the world works and a theory (either explicit or implicit) about how these programmes will make change (a “theory of change”). As leaders/supporters of these projects, it is incumbent on us to carefully monitor them and evaluate whether those assumptions and theories hold true.
M&E allows you to apply the philosophical concepts of EA to your project practically. It is beneficial for you as a practitioner, for your organisation, and for the EA movement as a whole.
For your project
Rigorous monitoring and evaluation can…
Help you make informed decisions
We all want to do the most good with the least resources. That means we need ways to rigorously assess if we are doing valuable and cost-effective work. M&E can provide robust evidence to help us decide whether a project should continue, adapt, scale up or scale down.
Keep you accountable
It is important to be accountable to ourselves, our donors, our partners and the communities we serve and show we are doing what we committed to do, generating the results we intended and not causing harm. M&E can help us verify that a project is rolling out as planned, that the activities are reaching the intended participants, that funding is being spent appropriately and that there are no adverse consequences.
Improve your project design
If You Fail to Plan, You Are Planning to Fail. M&E practices like needs assessment/problem diagnosis and theory of change help you to think through the problem you intend to solve and the theory for how you think you will achieve change. Bringing your theory of change into the light and inspecting each aspect of it thoroughly helps you ensure your proposed project actually addresses the problem on the ground and is evidence based.
Identify risks and areas for improvement so you can learn and adapt
Even with a robust theory of change, things can change when you start implementing in the messy real world. By systematically monitoring your project’s progress and evaluating your performance, it’s possible to see where a project is or could be going wrong and proactively address this. Learning and adapting your project as you go makes it more robust and improves its quality. This can also lead to important innovations.
Motivate you, your partners and your project participants
You, your team and the people you work with need motivation to keep going. Otherwise, especially with complex problems, it can feel like nothing is changing. A powerful way to keep people motivated is to share M&E findings with them to show if and where change is happening.
Ultimately, these things make your project more effective and impactful.
For the whole movement
Practise our values.
EA is built on the foundation of doing good better. To truly live this value, we need to invest in monitoring and evaluating systems across the EA ecosystem to know if we are indeed doing good better.
Aggregate EA organisations' impact.
The more monitoring data and rigorous evaluations we conduct, the better placed we are to conduct systematic reviews and evaluations of the impact of the EA movement (and sub-communities) as a whole. This is important evidence to prove the value of the EA movement to members and to the world.
Build our knowledge of what works.
If every organisation in the EA movement undertook regular monitoring and evaluation and shared the key learnings from these efforts, we could help other EA, and non-EA organisations avoid similar mistakes and learn from innovations, resulting in better programming for those we most want to benefit.
Help decision-makers make better allocative decisions for funding within the EA movement so we are doing the most good possible with the resources available.
What can I do right now to improve M&E within EA?
M&E is fundamental to achieving the goals of EA to ensure we do the most good possible. To improve M&E within EA, we need to build M&E knowledge and skills within the movement and build linkages and partnerships with M&E organisations and experts. To start this, we propose the following actions:
- Individuals and organisations
- Join the EA M&E community of practice slack group
- A place to share resources, advice, and learning with others working on or interested in M&E within the EA space.
- Conduct IDinsight’s M&E health check for your project.
- Apply for a pro-bono M&E consultation for your project
- You can contact us by filling out this form
- Reach out for paid M&E support
- For example, organisations like IDInsight, Innovations for Poverty Action Rightfit evidence, JPAL, or ITAD
- Consider M&E as a career option
- You can read a summary on this from Probably Good here.
- Learn more about M&E. More detailed resources will also be shared in the online Slack group
- Better Evaluation resource hub
- Intrac resource hub
- JPAL research resources
- Evalforward resources
- IDInsight Impact Measurement Guide
- UNICEF Impact evaluation series
- Join the EA M&E community of practice slack group
- Consider adding M&E consultancy and resources onto the list of projects you want to see more of.
- Incentivise groups to use M&E in their funding applications and reporting and factor this into their appraisal.
- Forge linkages with M&E experts and organisations to create robust M&E systems.
EAs want to do the most good possible with the resources we have. If we want to live this value, then we need ways to assess that we are actually doing good.
M&E theories and tools can help EA organisations, and the movement as a whole, monitor and evaluate what we are doing so we can create better designed projects, adapt and improve our work, hold ourselves accountable for the commitments we make and, ultimately, make better decisions about how to do the most good.
Currently, we are not using M&E skills and knowledge in EA as much as we could. We hope that this post helps to get more EA projects interested in M&E (see our pro bono support) and begin to build a community for those interested in M&E within EA (see our Slack group).
This post will be the first in a series around M&E in the EA space. If you have topics you would like us to cover in future posts, please leave a comment below.
Some shameless self-promotion as this might be relevant to some readers: I work at cFactual, a new EA strategy consultancy, where one of our three initial services is to optimize ToC's and KPI's together with organizations. Illustrative project experience includes the evaluation of the ToC and design of a KPI for GovAI’s fellowship program, building a quantitative impact and cost-effectiveness model for a global health NGO, internally benchmarking the impact potential of two competing programs of an EA meta organization with each other, doing coaching with a co-founder of a successful longtermist org around Fermi-estimates and prioritization of activities as well as redteaming the impact evaluation of a program of a large EA organization.
I wholeheartedly agree with the key statement and goals of this post, thank you for writing this up!
One thing I think could be really useful to see (perhaps for this post, or perhaps this is coming in a future post?) would be even just a simple sample M&E system with a Theory of Change, a Framework, and Evaluation questions for a particular project. It could make it easier to understand at a glance what the type of thing is that it would be great for a project or org to have and a sample template could easily be copied and adapted by readers in 80/20 fashion.
I'm looking forward to the next post(s?).
I've found this resource from IDinsight suuuuper helpful, and I think it's approximately what you're looking for?
Wow, that looks great!
Are you using it for EA Netherlands?
Yup! Basically structuring all of our M&E work around it. So far have mostly used it to help with our ToC and a recent needs assessment of our subgroup organisers. Lynn from EA UK showed me it, so all credit to her :)
I scrolled the comments to write exactly the same comment as Judith :) Basically wanted to ask authors to take e.g. plant-based campaign (e.g. replacing meat meals with vegan meals in canteens or something more difficult even) and show how one can go about it. So sort of step-by-step M&E guide. That would be super valuable I think.
I am happy that this exists.
I think a good way to approach how to build out a good M&E culture in the community will be a guide with a checklist augmented by case studies of how other orgs/project teams implemented M&E at different stages. I have some confidence most will be happy to pick this up and reach out for specific advice. I also particularly encourage that all our EA project leads / orgs "self-require" some baseline level of an M&E system before asking donors for funds so we can be more accountable for our own requests for grants, etc.
Happy to collaborate on the resource building part of this.
Yes totally agree and we have been talking about putting together another post about how this might be applied in practice so great to hear you are interested.
Great post. It's somewhat ironic how spotty the internal M&E is for most EA orgs, given how much we demand from the orgs we recommend.
An addition: you mention reaching out to IDInsight and IPA for paid M&E services, but they operate at a scale and budget that is way beyond most organisations. Individual, freelance M&E consultants are much more likely to be able to design a framework, help you collect data and then evaluate the findings, for a cost appropriate to your org size.
This is what my wife does for a living and she could try to recommend colleagues if orgs were looking for some freelance help (for clarity, she doesn't do this for EA orgs and isn't touting for business).
Yes agree! I think the broader international development world where I have worked has a lot more freelance consultants who provide these services and it would be great to see more consultants come into the EA space. We also know though that cost can sometimes be a barrier so we are also trying to offer some kind of probono support and the MEL community to help people get started.
Very pleased to see this! I'd love to see more focus from EA orgs (and others of course) on the fundamentals of being an effective nonprofit (e.g. having a strong, well-evidenced theory of change, and using M&E to test the weakest links in that theory of change and measure impact).
In particular, on theory of change, I'd like to add the following impassioned rant:
A non-profit’s theory of change is analogous to a business model in the for-profit world. Just as you wouldn’t found a company without a clear business model (and nobody would fund you), one shouldn’t found or fund a charity without a strong theory of change.
In fact, theory of change is more important for charities than business models are for businesses, because businesses have better feedback loops. If a for-profit is based on a bad idea, or has bad execution, it will see poor revenue and profits and it will soon go out of business. If a charity is ineffective on the other hand, it may limp along for years without having any impact, squandering limited funding and talent in the process.
@Lizka I want to suggest this post for curation and definitely putting it back on the Front Page :) I think someone by mistake tagged it as "Community" while it is extremely valuable content that should be at the Front Page.
If people are interested in getting better at M&E, I highly recommend The Goldilocks Challenge by Dean Karlan (founder of IPA).
How to Measure Anything is also a classic for a reason.
The animal movement would tremendously benefit from M&E consultations or even a separate organization that will help create M&E plans and give people the know-how to do it in-house.
Has there been a theory of what size a project should be before an M&E stack is justified, in generality? I'm thinking of the "value of moral information" chapter in Moral Uncertainty, that talked about the formula for calculating what percentage of an endowment should be spent on deciding how to spent the rest of the endowment.
Great question! In partial answer to your question, there is a commonly thrown-around figure that 5-10% of an organisation's budget should be spent on evaluation (e.g. see INTRAC). But this is a pretty basic heuristic (indeed in the linked article they talk about a study that found a range of 0-25%). I think at a minimum all organisations should have some kind of theory of change (even if it is very basic) and should be collecting basic monitoring data to enable project management. But I think we do need more thinking and guidance around when and how you decide to do more M&E (especially the E!) and decide how to do this within your means. In other sectors this is also often driven by demand from funders too.