Comments
We appreciate that you read our previous posts on systemic change and EA.
To start this fourth and - for now - last section, we would like to critically discuss obstacles we encountered, open questions and uncertainties that remain, and ideas we would love to see more work and reflection about. We are then especially looking forward to discussing all this with you in the comment section below.
That said… let’s hop right in!
When this project started, none of us expected it to take so long - close to three years all in all!
It turns out that trying to be rigorous about this topic made us run head-first into obstacles that we hadn’t foreseen. For a group of interested people meeting every few weeks, just arriving at a shared definition of systemic change took months of going back and forth.
The questions
The process of coming up with our questionnaire, trialling it, and improving it incrementally took even longer. This left us with the insight that phrasing and understanding the questions is very hard. Minor changes in the questionnaire can lead to large differences in the results. So our questionnaire ended up being a chaotic system by definition. This beast ended up being hard to draw conclusions from at the low response rates we received (3-5 reviewers per organisation).
Information on the internet
We decided that raters should use information on the evaluated organisations’ websites in order to have a reliable baseline for inter-rater comparisons. This seemed sensible compared to asking people to rate organisations based on their knowledge or hours-long deep-dives into every single available publication. However, this means that our data only covers whatever might be presented on the organisations’ own websites, which has its own limitations.
These websites are created for a few specific target groups. They thus present the organisation with these target groups in mind. Organisations might need to attract donors, or speak to a particular political context, which influences how they choose to talk about their goals, values, and principles. Organisations also need to consider how well-understood and well-accepted their core mission already is in mainstream discussions. For well-established problem areas, such as global poverty and ill-health, there may be comparatively little value in discussing root causes and system dynamics because website visitors can be expected to know about these already. For less conventional problem areas, such as global catastrophic risks, the situation may be different. Hence, the specific language used may differ between organisations not because they are fundamentally different in outlook, but rather because they speak to different audiences.
Additionally, such websites are often too large to quickly skim and find the relevant information for a reviewer. If raters don’t reliably find a relevant piece of information, the ratings are not reliable.
Insider knowledge
To add more meaningful information and compare our external reviews with the perspective of people working inside given organisations, we decided to ask people working for the respective organisations for internal ratings based on our questionnaire. It seemed helpful to let them answer a largely unedited copy of the questionnaire to receive data that is comparable. Even so, we remain unsure if it is possible or meaningful to compare the aggregate scores from self-ratings to the external raters’ answers. Here lies an opportunity for the future: If we receive more ratings from external and internal raters, we might look in more detail at these answers. We could compare internal and external raters’ answers for particular, single sections and try to draw conclusions from these.
Overall, this sequence of posts relies on the results of our questionnaire less than we had originally hoped. We are left wondering what other approaches might be helpful to draw firmer conclusions about systemic change in society at large and in EA in particular for the future.
It wasn’t just our particular approach that ended up proving quite difficult to execute. Throughout this project, we came to the understanding that systemic change overall is quite a hard concept to grasp. This is especially true if you are trying to look at it from a perspective that makes sense to people with different worldviews and starting assumptions.
Take our definition as introduced in the second post of this sequence:
“A transformation of an entire system, rather than just an incremental improvement of its constituent parts. Change happens by addressing the root cause of an issue through changing the system or levers that cause it. Building on this definition, an organisation working on systemic issues should pay attention to three dimensions of systemic change: 1) power dynamics; 2) systems thinking; 3) root causes.”
We use this definition as a basis for the project. We don’t suggest that the definition we chose is the only sensible one. In fact, one of the goals of this sequence is to encourage readers to scrutinise our definition. We want you to consider whether it captures the kind of “systemic change” that we should be talking about, and propose and argue over alternatives that may or may not be more in line with what EA should be focusing on.
The motivation for this search for a useful definition is not just idle curiosity or an abstract wish for rigour. When the use of a concept differs from person to person, any kind of meaningful exchange about it becomes difficult. This leads to unproductive disagreements. People state what they feel are obvious, core truths about the world, while others disagree simply because they understand the concepts at the heart of the argument differently. What we want to arrive at is a shared understanding that lets us find out where we actually do end up agreeing or disagreeing with each other on the underlying matter, and not just nomenclature.
It is because of this desire for more meaningful, coherent, and productive discussions that we wrote the previous three posts of this sequence. What we want to do in this last post is to prompt you to take these topics and open questions to your next meet-up, coffee break, or conference 1-1s.
In the first post of our series we made a case for the impact potential of systemic change applying the ITN framework: We argued for systemic change to be highly important (high confidence), moderately tractable (medium confidence), and probably neglected (low confidence: anecdotal but directionally supported by our survey). Beyond ITN, systemic change also offers durability that one-off interventions often lack, though this cuts both ways since negative systemic shifts can be equally hard to reverse. Such arguments suggest there is room in the EA ecosystem for more research and entrepreneurial work on radical systems change. In the following paragraphs, we dive a bit deeper to provide concrete and plausible directions for work on systemic change:
As we laid out in post 2 of the sequence, organisations might benefit from reflecting on their systemic impact from time to time. Our second post suggests a list of key intervention/strategy design criteria that are derived from our definition of systemic change. Naturally, if one does not agree with our definition for whatever reason, then the derived checklist might not be useful as a strict guideline. Instead of adopting these design criteria as they are, organisations that aim for systemic impact could use these points to start an internal discussion, for example in a workshop setting. This discussion could focus on their work with regard to the three dimensions of systemic change we outline: Power dynamics, systems thinking, and root causes. The checklist itself will not immediately lead to more systemic change. It can be a starting point for reflection in an organisation, and ultimately inspire more systemic thinking and change. This reflection could be done on a regular basis, e.g. in the context of strategy workshops. Learnings should ideally not only influence the inner workings of the organisations. If communicated publicly, they could also influence other organisations to think more critically about systemic change. We would love to see more recognition of the topic in strategic considerations of organisations in the EA ecosystem.
Please build on our work:
As the third post in this sequence showed, undertaking such a project as a group of volunteers is not easy. Organisations such as the Social Change Lab that evaluate the effectiveness of social movements give some idea of what an extension of our project could look like. Analysing specific interventions externally and from inside the organisations which implement the intervention could lead to a better understanding of present and potential systemic change. As another option, adapting the type of work the System Change Lab is doing on climate change to other cause areas could be a basis to orient a whole movement towards work on the most important levers on a systems level. This could apply to other cause areas too, including animal welfare, existential risks, extreme poverty alleviation and global inequality reduction among others.
Please build on our work:
Our effort with this sequence of posts is only a start. We’ve emphasised over and over again that our definition of systemic change is provisional, our list of criteria for systemic work is subject to trial and debate, and our analysis of the existing EA ecosystem has limitations. Many questions remain open and in need of serious discussion: which next steps should we all collectively as a community take to gain more clarity and insight about all this? Would it be helpful to gather more meaningful data on the current state of systemic approaches within EA – or on whether there even is a need for more such approaches? Should we formulate visions of how more systemic interventions might look like in the future?
Please do not restrict yourself to discussing these ideas right here, with us. Instead, carry these topics with you into conversations you have with others, both people involved in EA but also anyone else in your life. We believe it will be incredibly valuable to gain insights from as many perspectives as possible – and we cannot wait to hear what these insights will be!
Please build on our work:
We are incredibly excited about what might come from all this – will we, collectively, be able to tackle even larger problems? Make the world better on whole new levels? Are there low-hanging fruits of systemic approaches that have eluded us for far too long – or will we realize that we are already doing the best that can be done?
For us, this journey has just begun. After roughly three years of discussing, surveying and drafting in a small group, we are approaching our own little singularity by finally publishing this sequence; what (if anything) actually emerges from it is hard to predict – but all the more exciting for it!
