As the effective altruism (EA) movement continues to grow, so does our emphasis on rigorous impact measurement. While this focus on evidence-based giving is commendable, it's crucial to recognize the significant challenges that social sector organizations face in measuring their impact.
As someone involved in social sector impact measurement and evaluation for a full career, I would point to some difficulties with the current direction of travel and suggest that the EA community's expectations sometimes need adjustment.
The Real-World Challenges of Impact Measurement
1. Resource Constraints
Many nonprofits operate on shoestring budgets with limited staff. Conducting thorough impact evaluations can be expensive and time-consuming, potentially diverting resources from direct program work. For a small organization providing urgent services, spending a significant portion of their budget on measurement might mean serving fewer people in need.
2. Complexity of Social Issues
Unlike controlled laboratory experiments, social interventions often deal with complex, interconnected problems. Isolating the impact of a single program can be extremely difficult. For instance, an education program's success might depend on factors like family income, nutrition, and community support - all of which are hard to control for.
3. Long Time Horizons
Some interventions, particularly those addressing systemic issues, may take years or even decades to show measurable results. This timeframe often exceeds typical funding and reporting cycles. Climate change mitigation efforts, for example, might not show clear impacts for generations.
4. Diverse Stakeholder Expectations
Different funders, beneficiaries, and regulatory bodies may have varying (and sometimes conflicting) ideas about what constitutes meaningful impact and how it should be measured. An organization might find itself trying to satisfy multiple, incompatible reporting requirements.
5. Lack of Standardization
Unlike financial accounting, there are no universally accepted standards for social impact measurement, making comparisons across organizations challenging. This lack of standardization can make it difficult for donors to make informed decisions.
6. Data Collection Challenges
Gathering reliable data in resource-poor or crisis-affected areas can be logistically difficult and sometimes dangerous. Imagine trying to conduct a detailed survey in a conflict zone or a remote rural area with poor infrastructure.
Why EA Expectations Can Be Unrealistic
Given these challenges, some common expectations in the EA community can be out of step with the realities of the social sector:
1. Overemphasis on Quantitative Metrics
While numbers are important, they don't tell the whole story. The EA community sometimes undervalues qualitative data or anecdotal evidence that can provide crucial context.
2. Preference for RCTs
Randomized controlled trials (RCTs) are powerful, but they're not always feasible or appropriate. Expecting RCT-level evidence for every intervention is often unrealistic and can bias us towards easily measurable interventions.
3. Focus on Short-Term Outcomes
Our desire for clear, attributable impact can lead to a focus on short-term, easily measurable outcomes at the expense of important long-term or systemic changes.
4. Underestimating Complexity
Social change is messy and non-linear. Our quest for clear cause-and-effect relationships can sometimes oversimplify complex realities.
5. Neglecting Local Knowledge
In our pursuit of "objective" data, we may undervalue the insights of local communities and practitioners who have deep, contextual understanding of the issues.
6. Problematic Cross-Domain Comparisons
While it's tempting to directly compare the cost-effectiveness of diverse interventions (e.g., global health vs. climate change mitigation), such comparisons often rely on questionable assumptions and may not be truly meaningful.
7. Insufficient Appreciation of Capacity Building
Efforts to improve an organization's ability to deliver and measure impact in the long term may not show immediate results but are crucial for sustained change.
Bridging the Gap
To address these challenges and set more realistic expectations, the EA community could:
1. Invest in Capacity Building
Instead of simply demanding better data, we should help organizations develop their impact measurement capabilities. This could involve funding measurement-focused staff positions or providing training and resources.
2. Embrace Diverse Evidence
We need to recognize the value of multiple types of evidence, including qualitative and mixed-methods approaches. A compelling case study or ethnographic research can provide insights that numbers alone cannot.
3. Fund Long-Term Evaluations
We should support studies that track impact over many years, acknowledging that significant change takes time. This might mean committing to longer funding cycles or creating pooled funds for longitudinal research.
4. Collaborate with Practitioners
Working closely with those on the ground can help us develop practical, context-appropriate measurement approaches. We should view social sector workers as partners, not just as subjects of evaluation.
5. Advocate for Shared Standards
Supporting efforts to develop flexible, widely-applicable frameworks for impact measurement in the social sector can help make evaluation more accessible and comparable.
Towards More Effective Altruism
By recognizing these challenges and adjusting our expectations, we can foster a more constructive dialogue between the EA community and the broader social sector. This doesn't mean abandoning our commitment to evidence and effectiveness. Rather, it means expanding our understanding of what constitutes valuable evidence and meaningful impact.
As we refine our approach to impact measurement, we'll be better equipped to support truly effective interventions - even when their impacts are complex, long-term, or hard to measure. This more nuanced perspective will ultimately lead to more effective and sustainable positive impact, bringing us closer to the goals that drive the EA movement.
Remember, the aim isn't just to measure impact effectively, but to create real, lasting positive change in the world. By bridging the gap between ideals and realities, we can do both.
