Hello Katie, thanks for writing this up. I'm very concerned, however, about the evaluation criteria:
The primary metric used to assess grants at the end of the first year is the number of group members who apply for internships or graduate programs in priority areas and reach at least the interview stage [...]
The 80k's priority paths are basically X-risk, work at an EA org (does 'cause prioritisation' happen anywhere else?) and earn to give. As such, on your stated primary metric, it seems anyone who switched their career from doing nothing to working on any of animals, global poverty, mental health or climate change would not count as a valuable shift. In other words, CEA only counts a career change as worthwhile if it focuses on the far future.
Three points on this. First, if this is the case - and please tell me if I'm mistaken - I consider it to be deeply regrettable. It moves the EA community from being a morally inclusive one (I've written on this previously), which brings together individuals who do - according to their own lights - the most good they can, and seek to help each other, to a morally exclusive one, where there is a right answer, we need to fight over what it is, and CEA will financially reward you if you accept their answer. It seems the sort of thing that would lead to the split of the EA movement (i.e "if EA is just effective futurism, those of us who aren't effective futurists should go and do our own thing").
Second, I note that, according to the latest EA survey, 54% of those who identify as EA consider those four areas - animals welfare/rights, global poverty, mental health and climate change - to be the 'top cause'. Hence the evaluation metric is markedly not representative of the EA community (in the sense of matching the community's priorities). You might think the EA has just got it wrong, but there is still an oddness here, because you've stated
We hope to avoid these negative effects by providing after-the-fact assignment of credit for outcomes outside the scope of the primary success criteria, and by emphasising the criterion of ‘being a good representative of EA’
I submit that your evaluation criteria - using 80k's list - fails your own representativeness test, because it judges impact by a standard that would not be widely accepted among self-identifying effective altruists.
Third, given the evaluation criteria is quite sharply divergent from what people would expect, I would have appreciated if this had been flagged more clearly.
It seems like the nature of your concern originates from this paragraph:
“We used the 80,000 Hours list of priority paths as the basis for our list of accredited roles, but expanded it to be somewhat broader. The areas and roles that we intend to accredit are still being decided upon, and we expect the number of accredited roles and areas to increase in the future. We’ve chosen a relatively restricted set of criteria for the time being, as we think the costs to later restricting the criteria will be significantly higher than the costs of expanding them.”
I think some additional color on how we expanded the criteria to be broader than 80K priority paths and how we expect to expand it to be broader in the future will mitigate some (but not all) of your concerns.
80K priority paths are the initial basis of our list of accredited roles because we have high credence that these are high-impact roles and because 80K can provide infrastructure to assist in helping members of local groups connect with some of these roles. We don’t expect that these are the only roles that are important and it is not our intent to accredit only these roles.
Interestingly, our initial list of accredited roles included any Open Phil or GiveWell grantee in order to make the list more inclusive. We now feel that this expansion might have been too inclusive because of the significant expansion in the organizations Open Phil has provided with grants. (For example, we probably wouldn't be happy to accredit someone working at VasoRx without further information.)
Our current best guess for how to proceed is to accredit anyone working at an organization on the 80K Job Board (which includes global poverty and animal welfare organization) but continue to review individual positive outcomes on a case-by-case basis and accredit some career-related outcomes that are neither on the 80K job board nor Open Phil/GiveWell grantees.
Unfortunately, any list of accredited outcomes that we distribute to grantees will be imperfect. As we mentioned in the original post, our goals in designing an evaluation process for EA community grants were to:
Give groups clear guidance on what would cause us to evaluate their activities favourably.
Make it easy for the EA community and potential funders to understand and evaluate the success of EA Community Grants.
Provide sufficient evidence of the value produced to enable CEA to make well-informed decisions on whether to renew funding for given groups, and whether to scale the EA Community Grants process.
Avoid incentivising groups to optimise for our metrics rather than for what is actually highest-impact.
Minimise the time cost to CEA and to groups in evaluating their results.
Our current best guess for how to do this is to use the cause prioritization research done by others to create an initial list of clear, easy-to-understand positive outcomes that we can communicate to CEA’s donors and grantees and then to accredit additional positive outcomes on a case-by-case basis. We think this is superior to accrediting everything on a case-by-case basis because it reduces uncertainty for grantees and makes the project easier for funders to evaluate. We also think that 80K represents the best research on what careers have an impact, so it would be surprising if we don’t take advantage of that resource.
If you think we should accredit entirely different outcomes, think this is the wrong approach for accrediting career-related outcomes, or think that we should use a different list, we’d be very open to suggestions. We plan to make the next iterations to our accredited criteria before the next application round in January so feedback now would be particularly timely.
The primary metric used to assess grants at the end of the first year is the number of group members who apply for internships or graduate programs in priority areas and reach at least the interview stage. For subsequent years, the primary metric used to assess the grants is the number of group members who go on to work in priority areas.
This seems to assume most group members will be quite young. Is this expected? Desired? Depending on how much this is emphasized, it could be a turn-off to older members with established careers they’re unlikely to change.
Out of curiosity, was this metric selected by CEA, the groups, LEAN, or some combination?
Hi John, thanks for your comment (and sorry for the delay getting back to you)! These metrics were selected by CEA, but we gave current recipients the opportunity to provide feedback on the proposed evaluation procedure.
We think that one of the main route to value of EA Groups is to cause people to change their career trajectory such that they end up contributing to solving some of the world’s most important problems. It does seem that people who are willing to make such changes are more likely to be younger, however there are a number of senior management positions (e.g. Chief of Staff, Chief Operating Officer etc) available within organisations that we would accredit that one could imagine would be more suitable for people more advanced in their careers.
As mentioned in the post, we don’t think career changes are the only route to value for EA Groups, and that there are likely ways in which people who are unlikely to change careers can contribute. Though given that we think career change is a major route to value, we’ve chosen to assess this specifically. We also hope to find out the extent of the value of contributions from individuals who don’t change their career by using a case-by-case, after-the-fact assessment.
FYI, I had applied to do this in Berkeley, but was previously turned down due to lack of data/metrics. I re-applied in the fall, and have now received a grant to help fund renting the space for REACH. I plan to do more active outreach now that I have a more stable source of funding.
In October 2018, BERI (Berkeley Existential Risk Initiative) awarded a grant of $24k toward REACH operations for 2018-2019. So far, $10k of that has been dispersed as salary for the REACH Manager, Sarah “Stardust” Spikes (me).
In November 2018, the EA Meta Fund awarded a $5k grant to for work on the REACH Panel.
In December 2018, CEA (Centre for Effective Altruism) awarded a $14k community building grant toward rent for REACH, which has been dispersed in full to the REACH Bank account.
CEA doesn't run any regular events, community spaces, or fund people to do active community building in the Bay that I know of, which seemed odd given the density of EAs in the area and thus the marginal benefit of increased coordination there.
See my reply to casebash: my comment was not about whether it would be in general good to have a good community builder working on stuff not covered by existing orgs, but on whether it is "really quite weird" that there isn't anyone working on it payed by this particular program.
For example, in some places, community builders may organize events like MIRIx or AI safety reading group. In Bay, people can just get in touch with MIRI or CHAI. Some local organizations organize EAGx conferences, but there is EAG in SF organized by CEA. In some places, community organizers organize shared housing, but there is a lot of rationality houses already. Etc.
In other words, for efforts which in some places take some fraction of time of one community builder, there is often a whole dedicated organization in the Bay Area.
Enough possibility space still exists that it seems worth at least experimenting to see if a professional organiser could value add, even if it were only by better advertising and promoting the opportunities that already exist. Especially now that CEA is increasing its focus on one-on-one meetups.
My model for it is the whole space of opportunities in the Bay Area is large, on the other hand, many parts are already exploited by existing organizations. Whether the remaining part warrants someone should be working on it full-time, supported particularly by community building grant, is unclear to me, but quite likely yes.
Also it may be the case if someone who the grant-makers would be excited about had applied, they would had given them support, but there weren't such applicants. (Note that Bay Area biosec got the the grant)
Overall my comment was not about whether it would be good to have a good community builder working on stuff not covered by existing orgs, but on whether it is "really quite weird" that there isn't anyone working on it payed by this program. IMO it is not really weird, and not really surprising. (Btw if CEA wants to hire someone todo this it, it is plausible it is better to have such person employed directly by CEA, and sitting in the office.)
Also it may be the case if someone who the grant-makers would be excited about had applied, they would had given them support, but there weren't such applicants. (Note that Bay Area biosec got the the grant)
When I spoke to ~3 people about it in the Bay, none of them knew the grant existed or that there was an option for them to work on community building in the bay full time.
Effective Altruism Community Building Grants is a project run by the Centre for Effective Altruism (CEA), providing grants of between $5,000 and $100,000 to individuals and groups doing local effective altruism community building for the academic year of 2018-19. We believe this could be a high impact project for the EA community in the long term, because it is a way in which we can increase the time and effort that goes into EA community building.
This post is an update on how the project is going so far, including an overview of the application process, the grants we’ve made, our evaluation criteria, and plans for the future.
Background
Effective altruism groups are an integral part of the EA community and have played a central role in helping the community achieve many of its aims. They help direct funds to effective causes, influence people’s career trajectories (including placing people in high-impact organisations), contribute to the intellectual progress of the community, and help shape EA culture.
The goal of Community Building Grants is to scale the impact of high-potential effective altruism groups by increasing their organisational capacity. We expect this to have a number of beneficial effects, but we see a key benefit being the empowerment of group members to work on the world’s most important problems. We also expect Community Building Grants to enable group organisers to better coordinate and support each other, to pursue valuable projects outside the scope of EA community building, and to more more effectively disseminate a high fidelity understanding of effective altruism.
This is a new project for CEA and can be seen as a way of testing the hypothesised route to impact of providing significant funding to effective altruism groups. If we judge EA Community Grants to have been successful, we intend to expand the project and the amount of funding allocated to it. Because the project is new, we’re also in the process of resolving any kinks and expect to continue to tweak the process as we learn more about how to execute it effectively.
To read more about EA Community Building Grants and the core rationale for the project, see this launch post.
Application Process
The EA Community Grants were advertised via the EA Forum in March, in the grants announcement post, with a link to the application form. Applicants could apply as individuals or with a joint application. Applicants who passed the first interview stage were then interviewed by Harri Besceli, with Kerry Vaughan monitoring the interview process during April and May. After the interviews, final decisions were made by June. Applicants also nominated referees who were asked to provide a reference in borderline cases.
Evaluation Criteria
The three main criteria used to assess applications were:
Ability and skill set of applicant
Applicant understanding of and engagement with effective altruism and effective altruism community building
Future potential of the effective altruism group (where relevant)
Applicants were scored on three criteria, and the scores were combined (using equal weighting) to create an overall rating of the application. This acted as a key input to the final decisions but did not solely determine the final decisions.
Grant Offers
Since launching in March, 22 grant offers were made, for a total of $623,000, with the significant majority of grants offers being to enable group organisers to work full- or part-time on organising their effective altruism group.
Of the 22 offers made, 18 are to fund individuals directly working on their effective altruism group, and 4 were for general expenses for groups or specific projects. The table below lists the grants made, the number of people who will be working part- or full-time as a result of the grants, and the length of the grant period. The majority of the grant periods started at the beginning of this month. A small number of applications to EA Community Grants were referred to EA Grants for evaluation, and aren’t included below.
We ultimately granted significantly more funding than we expected to. This was largely due to the quality of applicants exceeding our expectations, but also because we concluded that the potential for harmful indirect effects was smaller than we previously thought.
Support
Recipients of EA community grants will largely work independently from and autonomously to CEA. The main support offered is regular calls with CEA staff, a private Slack channel, and a retreat which took place in September (hosted and co-organised by the Czech Association for Effective Altruism). The structure and extent of support for grantees is something that we intend to experiment with significantly over the duration of the grant period.
Evaluation
Evaluating the impact of groups is challenging, because by they provide a complex mix of direct and indirect benefits to both individuals and the wider EA community. Our goals in designing an evaluation process for EA community grants were to:
Give groups clear guidance on what would cause us to evaluate their activities favourably.
Make it easy for the EA community and potential funders to understand and evaluate the success of EA Community Grants.
Provide sufficient evidence of the value produced to enable CEA to make well-informed decisions on whether to renew funding for given groups, and whether to scale the EA Community Grants process.
Avoid incentivising groups to optimise for our metrics rather than for what is actually highest-impact.
Minimise the time cost to CEA and to groups in evaluating their results.
On the basis of these criteria, we’ve chosen to evaluate the success of grants primarily by assessing outcomes pertaining to influencing the career trajectories of their members, but also by taking into account how groups are good representatives of the EA community and by evaluating other valuable outcomes on a case-by-case basis.
The primary metric used to assess grants at the end of the first year is the number of group members who apply for internships or graduate programs in priority areas and reach at least the interview stage. For subsequent years, the primary metric used to assess the grants is the number of group members who go on to work in priority areas.
We have chosen to prioritise career-related outcomes because we believe that this will track a key source of value of the community grants and because we feel like we have a reasonable understanding of the value of these kinds of outcomes. Though career-related outcomes will be a useful proxy for success, it’s worth emphasising that we don’t expect them to capture all of the value that the groups produce. However, we feel it is important to have some concrete metrics for our projects, especially ones that require a lot of people’s time and money.
We used the 80,000 Hours list of priority paths as the basis for our list of accredited roles, but expanded it to be somewhat broader. The areas and roles that we intend to accredit are still being decided upon, and we expect the number of accredited roles and areas to increase in the future. We’ve chosen a relatively restricted set of criteria for the time being, as we think the costs to later restricting the criteria will be significantly higher than the costs of expanding them.
A general expectation of all grantees and a necessary condition for receiving further funding is that they abide by CEA’s Guiding Principles of Effective Altruism, and are good representatives of the EA community. By this, we mean that we expect the indirect effects of grantees to the community to be positive, both by affecting the public perception of effective altruism, and the cultural norms of the effective altruism community.
Beyond career-related outcomes, we will encourage grantees to submit other outcomes that they’ve produced, which we can evaluate on a case-by-case basis. Such outcomes could include career-related outcomes not previously specified, or non-career related outcomes, such as supporting other EA Groups, influencing donations, etc. This is in part intended to minimise the potential of creating adverse incentives, and to help us better understand the value created by grantees for future evaluation rounds.
We are aware of how focusing on career-related outcomes could have negative effects. One way this could happen is by disincentivising valuable activities because they don’t count towards the success criteria. This could include:
Having people enter the priority area where they will have the highest expected impact (rather than the one where they have the highest chance of ‘success’);
Helping promote good epistemic norms and considerate culture so that people entering these areas will have larger positive externalities than otherwise;
Helping out other EA groups, with information and shared best practices.
Another way this could happen is incentivising suboptimal or harmful activities because they do count towards the success criteria. For example:
People pushing group members to apply for the priority positions even when it isn’t right for them;
The use of potentially harmful strategies such as large-scale outreach in order to achieve the success conditions;
Distorting thinking of the grantees by giving them incentive to believe this is just the correct thing to do and not question it too closely.
We hope to avoid these negative effects by providing after-the-fact assignment of credit for outcomes outside the scope of the primary success criteria, and by emphasising the criterion of ‘being a good representative of EA’.
We recognise that these measures may be imperfect, and may seek to make improvements to future evaluation criteria based on our mid-term review of current grantees’ reported outcomes.
Future Plans
CEA will run another, smaller, EA Community Grants round in January 2019, and in the summer of 2019, when the success of the first community grants will also be evaluated.
Prior to the summer 2019 funding round, we’ll evaluate the grants provided so far. If we judge specific EA Community Grants to be successful, we’ll renew the funding provided to those groups in the funding round, and for groups we judge to have been particularly successful, we’ll provide grants for time periods longer than a year. Additionally, if we judge the process as a whole to have been successful, we’ll expand the number of groups we fund.
Given that EA Community Grants is in its first iteration, it is difficult to make strong predictions, as its success currently remains to be seen. However, one vision of the EA community building landscape for within a few years’ time could involve:
The equivalent of 50 full-time people doing local EA community building
Full-time EA community builders in the majority of (but not restricted to) ~15 major universities and cities in Europe and North America
5 national or regional EA organisations, employing a handful of people working on both EA community building and ‘direct’ projects
5 people working on supporting and coordinating EA Community Grantees, providing a combination of retreats, conferences, training, mentorship, online infrastructure etc.
100+ people going into high priority positions each year from EA groups with a full-time coordinator
If you’d like to support EA Community Building Grants, you can do so here. Additional funding would likely increase our willingness to fund more groups in the January funding round, although this depends on the applicant pool.
If you have any questions relating to EA Community Grants, please contact [email protected].
This is a crosspost from the new Animal Welfare Alignment Newsletter by Anima International. You can subscribe on Substack if you are interested in following these efforts. Audio reading also available on Substack.
The goals of this post are to:
1. Raise a question I see as crucially important to the goal of aligning AI to animal welfare...
Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job.
My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI.
...
“How long have you been v*g*n?”
This is one of the most common icebreakers at animal protection events. It’s a baseline assumption, and it mostly holds true: if you’re out advocating for animals not to be tortured or abused, realistically these days you are v**n, or close. And it makes for good conversation. It seems fairly safe to assume when you meet strangers.
But this assumption is hurting the movement in a way which we don’t always notice: someone new comes into the sp...
Hello Katie, thanks for writing this up. I'm very concerned, however, about the evaluation criteria:
The 80k's priority paths are basically X-risk, work at an EA org (does 'cause prioritisation' happen anywhere else?) and earn to give. As such, on your stated primary metric, it seems anyone who switched their career from doing nothing to working on any of animals, global poverty, mental health or climate change would not count as a valuable shift. In other words, CEA only counts a career change as worthwhile if it focuses on the far future.
Three points on this. First, if this is the case - and please tell me if I'm mistaken - I consider it to be deeply regrettable. It moves the EA community from being a morally inclusive one (I've written on this previously), which brings together individuals who do - according to their own lights - the most good they can, and seek to help each other, to a morally exclusive one, where there is a right answer, we need to fight over what it is, and CEA will financially reward you if you accept their answer. It seems the sort of thing that would lead to the split of the EA movement (i.e "if EA is just effective futurism, those of us who aren't effective futurists should go and do our own thing").
Second, I note that, according to the latest EA survey, 54% of those who identify as EA consider those four areas - animals welfare/rights, global poverty, mental health and climate change - to be the 'top cause'. Hence the evaluation metric is markedly not representative of the EA community (in the sense of matching the community's priorities). You might think the EA has just got it wrong, but there is still an oddness here, because you've stated
I submit that your evaluation criteria - using 80k's list - fails your own representativeness test, because it judges impact by a standard that would not be widely accepted among self-identifying effective altruists.
Third, given the evaluation criteria is quite sharply divergent from what people would expect, I would have appreciated if this had been flagged more clearly.
Hi Michael,
It seems like the nature of your concern originates from this paragraph:
“We used the 80,000 Hours list of priority paths as the basis for our list of accredited roles, but expanded it to be somewhat broader. The areas and roles that we intend to accredit are still being decided upon, and we expect the number of accredited roles and areas to increase in the future. We’ve chosen a relatively restricted set of criteria for the time being, as we think the costs to later restricting the criteria will be significantly higher than the costs of expanding them.”
I think some additional color on how we expanded the criteria to be broader than 80K priority paths and how we expect to expand it to be broader in the future will mitigate some (but not all) of your concerns.
80K priority paths are the initial basis of our list of accredited roles because we have high credence that these are high-impact roles and because 80K can provide infrastructure to assist in helping members of local groups connect with some of these roles. We don’t expect that these are the only roles that are important and it is not our intent to accredit only these roles.
Interestingly, our initial list of accredited roles included any Open Phil or GiveWell grantee in order to make the list more inclusive. We now feel that this expansion might have been too inclusive because of the significant expansion in the organizations Open Phil has provided with grants. (For example, we probably wouldn't be happy to accredit someone working at VasoRx without further information.)
Our current best guess for how to proceed is to accredit anyone working at an organization on the 80K Job Board (which includes global poverty and animal welfare organization) but continue to review individual positive outcomes on a case-by-case basis and accredit some career-related outcomes that are neither on the 80K job board nor Open Phil/GiveWell grantees.
Unfortunately, any list of accredited outcomes that we distribute to grantees will be imperfect. As we mentioned in the original post, our goals in designing an evaluation process for EA community grants were to:
Our current best guess for how to do this is to use the cause prioritization research done by others to create an initial list of clear, easy-to-understand positive outcomes that we can communicate to CEA’s donors and grantees and then to accredit additional positive outcomes on a case-by-case basis. We think this is superior to accrediting everything on a case-by-case basis because it reduces uncertainty for grantees and makes the project easier for funders to evaluate. We also think that 80K represents the best research on what careers have an impact, so it would be surprising if we don’t take advantage of that resource.
If you think we should accredit entirely different outcomes, think this is the wrong approach for accrediting career-related outcomes, or think that we should use a different list, we’d be very open to suggestions. We plan to make the next iterations to our accredited criteria before the next application round in January so feedback now would be particularly timely.