Context
This post is a project plan for a Cause-specific Effectiveness Prize, intended to be launched locally in Israel, as a part of EA Israel’s activities.
The project is currently in the stage of market research, and we wanted to post this plan in order to receive your feedback before we continue.
Definition
This project’s aim is to establish a competition/prize for cost-effectiveness, between charities that work on the same cause.
Motivation
Our assumptions for potential value in this project:
- Encourage a conversation of impact effectiveness within local charity culture.
- Drive innovation inside specific causes. Both in terms of creating/researching new interventions, and in terms of publicizing those interventions. For that reason, this project can actively put a spotlight on interventions rather than on charities.
- Spark interest in EA within charities that aren’t naturally involved in EA. As of today, when a charity that doesn’t involve with EA’s top causes encounters EA, the message that charity receives is mainly “you better switch a cause or change everything you’re doing”; a message which doesn’t take into account personal attachments to that charity’s cause, and the emotional difficulty of changing assumptions. Instead, this charity could receive a message of (and potentially even guidance for) “here is how you can apply evidence and reason in order to improve your impact”.
In general, we want to test the potential of this route as a lead for new EA causes, and as a community-building approach (either by reaching out to new potential members, or encouraging individuals to move into EA-related causes from non-EA-related causes). - An important consideration regarding the cost-effectiveness of this project, is the potential of this project to provide resources we otherwise would have to work much harder for. Many donors have strong preferences to donate locally and are reluctant to donate to our top charities for that reason, but could find this project attractive. New volunteers had also shown much interest in this project.
- Regarding the previous argument, this project can also spark the interest in EA by donors that aren’t naturally interested.
If successful, this project could be launched:
- Locally, in other countries.
- Globally, calling out for charities all over the world that are involved in a specific cause area.
- Could evolve (after it is somewhat known within local charities) into more complex forms such as Impact Prizes, which seems very complicated to implement or pitch well without having an existing strong status.
Method
The general outline of this project consists of three main steps; Gathering resources for the project (especially funding), publishing the contest, and evaluating results. To simplify the evaluation process, we will pick a couple of different cause areas for the prize.
Below is a scale for the actions that can be undertaken by prize candidates. These actions are ordered by the amount of effort required by the charity.
- Level 1: Figure out what to measure, then employ measuring techniques (a required action for applying).
- Level 2: Make modifications in current interventions. Such modifications could be minor, for instance, increasing social influence by text modifications (example. Video about such modifications).
- Level 3: Duplicate an intervention that is carried out somewhere else. We hypothesize that a majority of charities in Israel haven’t done that global-level research.
- Level 4: Innovate an intervention based on subject-matter experience, theory or adjacent causes.
The prize requirements should be adjusted so charities can undertake only level 1 actions, and still have a chance of winning (in order to minimize the effort required by charities for competing). Nevertheless, we should provide guidance (at least through online material) for all levels.
Open Questions
- Do we provide counseling to charities? Personal counseling, or online comprehensive guides?
- RCTs are the golden standard of measuring impact, but are expensive and complicated. What easier alternatives do we allow? What’s the level of evidence and reliability we accept?
- Evaluation process: Are charities required to file documents, do we pay for third-party analysts? Should we have a committee? How transparent can/should we be about the results?
- How much money do we need?
- There might be extremely important distinctions between the definition of the project: Is this a grant, prize, contest?
Possible Risks
- Not enough charities will care or want to put in the effort we require.
- Such contests might eventually discourage knowledge sharing between charities.
- Legal risk - let’s not get sued.
- Is EA Israel ready for such exposure, or for handling this amount of money? Could this create a misleading image of EA as mainly focused on cause-specificity (given that it’s our only project with this approach).
- Miscalculating how complex our criteria for evaluation are, or leaving out opportunities for disruption of results.
Plan
- Consulting
- With other prize-focused projects in EA community.
- With prize-focused organizations in Israel, such as Midot.
- Market research (with charities)
- Amount of money that will create interest.
- Intent of participating. What would make them apply?
- What information do they have today, or they can make available, for analysis?
- Evaluate resources needed for this project: What personal do we need, evaluate costs.
Additional Steps
- Funding - starting with our current funding leads.
- Writing down precise definitions and requirements for the prize.
- Definitions (and resolving the open questions).
- Legal consulting.
- Creating online materials.
- Advertisement and reaching out to charities.
- Follow-up on competing charities.
- Evaluations of charity filings.
- Optional: Final event for announcing the prize winner. Only if we can be certain enough about holding a big and fancy event. The main value of this is EA Israel’s status within Israeli charities, for other projects and future partnerships.
- Retrospective analysis of the project (conclusions for another round of this project in Israel, publicizing results in EA community, organizing project’s resources for future replication).
Thanks fo the post, I really enjoyed this idea; please, let us know about its progress. I'd like to see something analogous for academic research and published papers.
As for your open questions:
I guess comprehensive guidelines are better, but they’re not exclusive alternatives, right?
Maybe you should let your contestants and committee have this discussion; of course you can't RCT everything - e.g., no one would do it for parachutes. Actually, it might even be an easier problem than finding a common metric to compare different interventions - e.g., take a look at GiveWell's blog.
Yes, yes, a lot. Suggestion: evaluate contestants in a two, perhaps three-level process:
a) approval voting in an online platform: you can either have an open website where people can vote for free (after identifying themselves), or you can charge voters a fee (two pros: voters will have some skin in the game, and partially fund your costs);
b) individual reviewers rate / select the n most voted contestants;
c) your committee decides the winners.
Risks: I guess reputational risk might be your real issue. Some sort of worst-case scenario: you lose your money and resources to this awful winner... but the real problem is that it goes viral, and everyone starts associating "Effective Altruism" with something like "Canadian Satanists chanting religious poetry in rich schools". Even my illustrative example sucks. Is there any way of letting EA get the benefits of the exposure, but not its risks?
BTW, do you plan to do it just once, or do you intend people to expect it to become a periodic contest? Signalling it could be repeated could influence future projects (in case you end up being successful)