Estimates of highly engaged EA retention

by Gabby_O6 min read13th Aug 20212 comments

28

Value driftData (EA Community)Centre for Effective AltruismCommunity
Frontpage

​Summary

  • Note: I expect this post is only interesting to a small number of people who are highly engaged with EA, so I haven't spent a lot of time cleaning it up. Please feel free to comment or reach out to me or Ben West with any questions you might have.
  • One of CEA’s goals is to help individuals who are highly engaged with effective altruism to stay highly engaged in a meaningful way.
  • In order to provide useful support and to evaluate our work, we need to learn what retention looks like today.
  • This analysis builds on the data analysis Ben West presents here, evaluating the retention rates of highly engaged EAs by tracking 3 representative cohorts from 2019-2021.
  • The highest estimate I found was a 95.05% yearly retention rate for highly engaged EAs.

 

Background

One of CEA’s goals is to help individuals who are highly engaged with effective altruism to stay highly engaged in a meaningful way. In order to provide useful support and to evaluate our own work, we need to learn what retention looks like at the present. As part of this work, I have explored ways of evaluating whether someone is still highly engaged that can allow us to evaluate retention from year to year, even quarter to quarter. 

This was difficult for a number of reasons. One major challenge was the variation in email addresses used at different times and in different places; this made it difficult to trace individuals’ engagement with CEA’s programs and effective altruism in general. 


As such, we decided to track 3 cohorts of people on whom we had reliable data: 

  • Employees at certain organizations connected to effective altruism
  • Recipients of Community Building Grants
  • Attendees of EA Global
     

I conducted this work as a CEA contractor.
 

Previous Work on Retention

For those interested in this topic, there have been efforts to estimate retention of EAs and highly engaged EAs before. In the Discussion section, I will discuss how my results compare to these estimates: 


I also recommend Marisa Jurczyk’s qualitative analysis of value drift discusses factors that influence retention.


 

Method

Definition of a highly engaged EA

CEA defines an engaged EA as someone who takes significant actions motivated by EA principles (we sometimes also use the term “impartially altruistic, truth-seeking principles). In practice, this can look like selecting a job or degree program, donating a substantial portion of one’s income, working on EA-related projects, and so on.


Data sources & method

We identified markers of retention that we could reliably analyze from year to year for the three cohorts. 

We first checked if individuals were still supported by a Community Building Grant or working at an EA-related organization. If they were, we marked them as still being highly engaged. If they were not, we checked what they were currently doing and whether they still fulfilled the definition of being highly engaged. (These checks either involved looking at data from CEA’s various programs, LinkedIn, talking to people who knew about the individual’s activities, or the individual directly.)

EA-related organizations: We generated a list of organizations with strong connections to EA, and reached out to ask for data on how many of their employees (among those who worked there at any time from 2019-2021) were (a) still employed by the organization, or (b) otherwise still highly engaged. We received data from 11 organizations in total.

Community Building Grantees: We checked the recipients of all grants made from 2019-2021, and whether they were still highly engaged. Grantees varied in how long they were supported by a grant and the fraction of their time that went toward supported activities. Some worked at 50% of a 40-hour week (“full-time equivalent”, or FTE) for 3 months, while others worked at 100% of an FTE for 3 years. As such, there’s a lot of variance in how much time people in this cohort spent engaging with EA.

EA Global: We used attendance at EA Global in multiple years as a proxy for continued high engagement, analyzing how many attendees of 2019 conferences also attended a conference in 2020. We looked at data from the following conferences:

  • EAG SF 2019, EAG London 2019
  • EAG SF 2020, EA Student Summit 2020

 

We expect retention rates for highly engaged EAs to be indicative of an upper bound for retention rates in the broader community, given our past data on dropout rates among people with different levels of engagement. We think that highly engaged people drop out less often in part because they have stronger social and professional connections to the EA community, and thus gain more value by staying involved.


 We wished to analyze whether individuals in each cohort performed any of the following actions each year: donated through EA Funds, viewed or commented on the EA Forum, attended an EA Global conference. But we found that the overlap of email addresses was too low for this data to provide an accurate understanding of retention rates. Ben West has produced an estimate of retention based on this (50-70% per year), “using a naïve method of matching people (mostly looking at email addresses)”. Now that CEA has moved over to a single registration system for their digital services (event applications, the EA Forum, and EA Funds/GWWC), we expect to see more people stick with one email address, which would make our data analysis more accurate.

 

Results

Retention rates:

2019-2020

2020-2021

2019-2021

Projected retention 
over 2 years

Projected retention 
over 5 years

Projected retention
over lifetime

EA-relevant org Employees

96.0%

94.3%

91.3%

91.3%

80%

16%

Community Building Grantees

95.6%

94.3%

93.5%

93.5%

84%

26%

EAG Reattendees

29.8%

  

9%

0.23%

0.00%

Average of all cohorts

73.8%

94.3%

92.4%

64.5%

55%

14%


 * This is not a projection, but the retention rate observed 2019-2021. 

 

Retention rates among most highly engaged:

Retention rates:

2019-2020

2020-2021

2019-2021

Projected retention 
over 2 years

Projected retention 
over 5 years

Projected retention
over lifetime

EA-relevant org Employees

96.0%

94.3%

91.3%

91.3%

80%

16%

Community Building Grantees

95.6%

94.3%

93.5%

93.5%

84%

26%

Average of most engaged 

95.8%

94.3%

92.4%

92.4%

82.1%

21.1%



 

Discussion

The highest estimate from my data set (95.05% year-over-year, averaged over 2019-2020 and 2020-2021 among employees at EA-related organizations and Community Building Grantees) is very similar to those that Ben West (95.3%) and Ben Todd (95.6%) found. 

EA Global attendance data provides a significantly lower estimate of retention (29.8%), and is also lower than the lower end (50%) of Ben West’s estimate of retention between 2020 and 2021 (the fraction of people who engaged with CEA’s services in 2021 after also doing so in 2020). However, EA Global attendance is less accurate as attendees use different emails to sign up year to year. It also seems likely that the unusual circumstances of 2020 (conferences moved to virtual spaces) may have driven the EA Global estimate to an especially low point. 29.8% is much closer to the annual retention estimate produced by Peter Wildeford based on the 2018 EA Survey. However, the EA Survey is filled out by many people who aren’t highly engaged with EA, so we’d expect higher retention in highly engaged cohorts. 

 

We hope to continue measuring this estimate of retention in coming years and expand this analysis beyond these cohorts to gain more representative retention data of the community. 


 

28

2 comments, sorted by Highlighting new comments since Today at 4:40 PM
New Comment

Thanks for the post. It looks like useful data!

Definition of a highly engaged EA

CEA defines an engaged EA as someone who takes significant actions motivated by EA principles (we sometimes also use the term “impartially altruistic, truth-seeking principles). In practice, this can look like selecting a job or degree program, donating a substantial portion of one’s income, working on EA-related projects, and so on. [italics added]

Just to clarify my understanding, are you defining/taking yourself to be looking at "highly engaged" EAs or just "engaged" EAs? For reference, the criteria for the CEA definition of "engaged EA" above are met by over half the EA Survey sample (more like 2/3rds depending on how you defined a "substantial" portion of one's income). In contrast,  EA org employees/CBG recipients is a much higher bar for engagement (~10% of EA Survey are current EA org employees), while attending EAG is ~30% of EA Survey respondents, so I'd expect very different retention rates for these different populations.

We first checked if individuals were still supported by a Community Building Grant or working at an EA-related organization. If they were, we marked them as still being highly engaged. If they were not, we checked what they were currently doing and whether they still fulfilled the definition of being highly engaged. (These checks either involved looking at data from CEA’s various programs, LinkedIn, talking to people who knew about the individual’s activities, or the individual directly.)

Were you able to get definitive answers about all the individuals on your lists using these methods? If not, I'm curious what numbers the percentages were drawn from (e.g. just the people you could get a definitive answer about or were unclear cases assumed to have dropped out/not dropped out etc.?).

29.8% is much closer to the annual retention estimate produced by Peter Wildeford based on the 2018 EA Survey. 

Note that in the comments on Ben's earlier post, Peter suggests that the other method that we used in that post would be more accurate (which gives an estimated ~60% retention after 4-5 years). We should be able to improve on that estimate quite a bit now that we have more cross-year data.

Hi Gaby. Thanks for the post!

In case it's helpful, I wanted to let you know that I embedded a series of public comments using the Hypothes.is tool.

If you get an account and a browser plugin at https://hypothes.is/ (it's quick) you will see my comments on the page itself, and you can respond to them. Let me know if you have problems or want me to post these in another way.