Interesting discussion. I agree incentives can be tricky and I have seen my fair share of bad evaluations and evaluation organisations with questionable practices. Some thoughts from me as an evaluator who has worked in a few different country contexts:
I think M&E is not just internal or completely external. A lot of times M&E orgs are hired to provide expert support and work alongside an organisation to develop an evaluation. M&E can be complex and it can really help orgs to have experts guide them through this process and clarify their thinking. And as you say when we have internal buy in we are more likely to see the findings taken up and actioned. When we only see M&E as an outside judgement commissioned by a funder with no input from the org being evaluated we make M&E out as antagonistic or adversarial which can be an unhelpful dynamic. I have seen orgs who have been unhappy with an external evaluation because they feel the evaluators made judgements when they didn’t fully understand the operating context (and how can they with often only a fly by visit to project locations) or did not properly take into account the values of the organisation or the community but rather only listened to the funder. This can be very disempowering and may not lead to positive changes.
I think many organisations do want to learn and improve but fear harsh judgement which is quite a natural, human response. I think bringing partners/orgs on board early, and establishing a pre-evaluation plan (see here) highlighting what your standards for evidence are and what actions you will take in response to certain findings before the evaluation is helpful. This also gives the organisation ownership over the evaluation results. I think it is important to frame your evaluation so that feedback can be taken on board in a culturally appropriate way. The last thing you want is for an organisation to feel harshly judged with absolutely no input or right of reply.
We speak like M&E is clear cut but M&E assessments often don’t come out fully positive or negative. A lot of evaluations occupy a messy middle. There is often some good things, some not so good things, some thing which we think are good or bad but we don’t have conclusive evidence. Sensemaking can be subjective as it often comes down to how you weigh different values or the standards you set for what good or bad look like. This can be different between the funder, the org, the community and the evaluators. For example if you find an education project is cost-effectively increasing test scores, but only for female students and not struggling male students what do you say? Is it good? Is it bad? Is this difference practically significant? Should the program be changed? What if changing this makes it less cost-effective? This comes down to how you weigh different values and standards of performance.
I agree with Dan and think integrity is a very important internal driver. While I agree with Nick that acting this way can be more difficult for local staff given the connections and relationships both professional and personal that they have to navigate, I don’t think integrity as an incentive is hard for them to compute, it is just harder for them to action. I don’t think the response should be that all evaluations should be done by non-local/international firms. This is highly disempowering, would drain local capacity, and again puts decision-making back in the hands of people often from high-income and supposedly ‘more objective’ contexts rather than building a strong local ecosystem of accountability, re-hashing problematic colonial power dynamics.
These kinds of dilemmas exist everywhere. Evaluation is always a tricky tightrope walk where you are trying to balance the rigour of evidence, the weight of different values, and the broader political and relational ecosystem so that what you say is actually used and put into practice to improve programs and impact.
Amazing work and well done on trying to wrangle such a complex topic to help inform prioritisation.
I have a few thoughts, especially on the cost side of the equation which may be useful.
On costs, firstly I am not sure that some of your organisational costs are representative.
For example, the costs for Resolve to Save Lives. In 2021 (the year used in your estimation) RSL was split from its parent org Vital Strategies Initiative however this transition was not complete until mid-2022. As such only a very small fraction of activities are covered by this financial report and these activities under the $600,000 grant were for a COVID-19 project (contracted back to Vital Strategies) not for anything to do with salt reduction (this likely was still covered by Vital strategies). There were also no employees on the books. I don't think this $600K figure is a reliable estimate for costs for salt reduction activities. You may wish to look at financial reports for other years and if possible find ones that directly relate to their salt work.
On the Action on Salt initiative, their costs are also very low but I think it is also important to recognise that this initiative is supported by the Queen Mary University London (e.g. their website is hosted by them, many of the volunteer experts are staff). So I don't think this expenditure figure accurately reflects the costs it takes to run this project like volunteer time and other costs borne by others like QMUL.
I think this all means you may be underestimating.
Finally, I think there is also a bigger picture question on costs versus expenditures. Above you are using org expenditures but these are not costs. Expenditures are the dollar outlays by a specific group. For example, the cost of going to university is not just the tuition price paid by students. The cost includes all resources to provide university including public subsidies, charitable support etc. In cost analysis we are interested in all resources no matter who pays.
In your CEA you are assuming a magnitude of positive effects related to the successful reduction in hypertension DALYs from activities of salt taxation, food reformulation, school meals, mass public education and package labelling. While the costs for a small policy advocacy charity to nudge these interventions into being may be small that is not the true cost of these interventions. The costs borne by others to actually put them into practice (e.g. govt, industry, schools etc) would be significant and this is not accounted for even though these costs are instrumental to realising the effects of these interventions that you are claiming.
Because you don't cost in the full cost of these interventions a cross-comparison to a Givewell charity is not necessarily fair. A charity like AMF are responsible for all the costs of running their mosquito net distribution program which support their claimed effects. In your example, you have only costed the relatively small costs of an advocacy charity and left out many other (substantial) costs necessary to realise the effects of these salt reduction activities like paying for mass media campaigns, or changing formulations in factories, or changing food packaging. But you are still claiming all the effects. So it seems to me that the true cost is vastly underestimated and is resulting in a skewed comparison.
Great question! In partial answer to your question, there is a commonly thrown-around figure that 5-10% of an organisation's budget should be spent on evaluation (e.g. see INTRAC). But this is a pretty basic heuristic (indeed in the linked article they talk about a study that found a range of 0-25%). I think at a minimum all organisations should have some kind of theory of change (even if it is very basic) and should be collecting basic monitoring data to enable project management. But I think we do need more thinking and guidance around when and how you decide to do more M&E (especially the E!) and decide how to do this within your means. In other sectors this is also often driven by demand from funders too.
Yes agree! I think the broader international development world where I have worked has a lot more freelance consultants who provide these services and it would be great to see more consultants come into the EA space. We also know though that cost can sometimes be a barrier so we are also trying to offer some kind of probono support and the MEL community to help people get started.
Yes totally agree and we have been talking about putting together another post about how this might be applied in practice so great to hear you are interested.
Interesting discussion. I agree incentives can be tricky and I have seen my fair share of bad evaluations and evaluation organisations with questionable practices. Some thoughts from me as an evaluator who has worked in a few different country contexts:
Hi Joel,
Amazing work and well done on trying to wrangle such a complex topic to help inform prioritisation.
I have a few thoughts, especially on the cost side of the equation which may be useful.
On costs, firstly I am not sure that some of your organisational costs are representative.
For example, the costs for Resolve to Save Lives. In 2021 (the year used in your estimation) RSL was split from its parent org Vital Strategies Initiative however this transition was not complete until mid-2022. As such only a very small fraction of activities are covered by this financial report and these activities under the $600,000 grant were for a COVID-19 project (contracted back to Vital Strategies) not for anything to do with salt reduction (this likely was still covered by Vital strategies). There were also no employees on the books. I don't think this $600K figure is a reliable estimate for costs for salt reduction activities. You may wish to look at financial reports for other years and if possible find ones that directly relate to their salt work.
On the Action on Salt initiative, their costs are also very low but I think it is also important to recognise that this initiative is supported by the Queen Mary University London (e.g. their website is hosted by them, many of the volunteer experts are staff). So I don't think this expenditure figure accurately reflects the costs it takes to run this project like volunteer time and other costs borne by others like QMUL.
I think this all means you may be underestimating.
Finally, I think there is also a bigger picture question on costs versus expenditures. Above you are using org expenditures but these are not costs. Expenditures are the dollar outlays by a specific group. For example, the cost of going to university is not just the tuition price paid by students. The cost includes all resources to provide university including public subsidies, charitable support etc. In cost analysis we are interested in all resources no matter who pays.
In your CEA you are assuming a magnitude of positive effects related to the successful reduction in hypertension DALYs from activities of salt taxation, food reformulation, school meals, mass public education and package labelling. While the costs for a small policy advocacy charity to nudge these interventions into being may be small that is not the true cost of these interventions. The costs borne by others to actually put them into practice (e.g. govt, industry, schools etc) would be significant and this is not accounted for even though these costs are instrumental to realising the effects of these interventions that you are claiming.
Because you don't cost in the full cost of these interventions a cross-comparison to a Givewell charity is not necessarily fair. A charity like AMF are responsible for all the costs of running their mosquito net distribution program which support their claimed effects. In your example, you have only costed the relatively small costs of an advocacy charity and left out many other (substantial) costs necessary to realise the effects of these salt reduction activities like paying for mass media campaigns, or changing formulations in factories, or changing food packaging. But you are still claiming all the effects. So it seems to me that the true cost is vastly underestimated and is resulting in a skewed comparison.
Great question! In partial answer to your question, there is a commonly thrown-around figure that 5-10% of an organisation's budget should be spent on evaluation (e.g. see INTRAC). But this is a pretty basic heuristic (indeed in the linked article they talk about a study that found a range of 0-25%). I think at a minimum all organisations should have some kind of theory of change (even if it is very basic) and should be collecting basic monitoring data to enable project management. But I think we do need more thinking and guidance around when and how you decide to do more M&E (especially the E!) and decide how to do this within your means. In other sectors this is also often driven by demand from funders too.
Yes agree! I think the broader international development world where I have worked has a lot more freelance consultants who provide these services and it would be great to see more consultants come into the EA space. We also know though that cost can sometimes be a barrier so we are also trying to offer some kind of probono support and the MEL community to help people get started.
Yes totally agree and we have been talking about putting together another post about how this might be applied in practice so great to hear you are interested.