Yikes; this is pretty concerning data. Great find!
I'd be curious to hear from anyone at GWWC how this updates them, and in particular how it bears on their "realistic calculation" of their cost effectiveness, which assumes 5% annualized attrition. (That's not an apples to apples comparison, so their estimate isn't necessarily off by literally 10x, but it seems like it must be off by quite a lot, unless the survey data is somehow biased.)
We're definitely aware that Giving What We Can's 2015 analysis comes away with a more optimistic conclusion than other more recent data sources like the EA Survey indicate (and I believe the Slate Star Codex survey, though I haven't seen a careful analysis of that one as it bears on Giving What We Can). We've just made some improvements to the donation recording platform, and once a few last things are ironed out we'll be sending out reminders for members to record their donations that may not have been recorded. Once people have had time to respond to those reminders, we plan to do an update on our 2015 estimates of members' follow-through.
Quick note on the 'bunching' hypothesis. While that particular post and suggestion is mostly an artefact of the US tax code and would lead to years that look like 20%/0%/20%/0%/etc., there's a similar-looking thing that can happen for non-US GWWC members, namely that their tax year often won't align with the calendar year (e.g. UK is 6th April - 5th April, Australia is 1st July - 30th June I believe).
In these cases I would expect compliant pledge takers to focus on hitting 10% in their local tax year, and when the EA survey asks about calendar years the effect will be that the average for that group is around 10% but the actual percentage given will range anywhere from 0 - 20% (if ~10% is being given), but often look like 13% one calendar year, 8% the next, 11% the year after that, etc. In other words, they will appear to be meeting the pledge around 50% of the time in your data. Yet the pledge is being kept by all such members continuously through that period. Eyeballing your 2017 graph of the actual distributions of percentages given, there are a lot of people in the 8-10% range, who are the main candidates for this.
Since both most US members and most non-US members have good reasons to not hit 10% in every calendar year, the number I find most compelling is the one in the bunching section that averages 2015 and 2016 donations (and finds 69% compliance when doing so). But that number suffers from not knowing if those people were actually GWWC members in 2015. It just knows they were members when they took the survey in 2017. GWWC had large growth around that time, so that's a thorny issue. Then the 2018 survey solves the 'when did they join' problem but can't handle any level of donations not exactly aligning with the 2017 calendar year.
My best guess thinking over all this would be that 73% of the GWWC members in this EA survey sample are compliant with the pledge, with extremely wide error bars (90% confidence interval 45% - 88%). I like Jeff's suggestion below as a way to start to reduce those error bars.
One complexity here may be in how members are treating taxation and tax deductibility in relation to their donations. "income" in your sample is self-reported and it is presumably pre-tax income. However, per GWWC: "While we have defined income as pre-tax in the past, after speaking with members in a variety of situations we believe there should be some flexibility here.
Thus people may be keeping to the terms of their pledge but donating less than 10% of their pre-tax income if they are not getting a tax deduction
I checked and ~22% of GWWC members* did not donate more than 5% of their income in 2017, so even assuming taxes accounted for a large portion of the issue, there are still a lot of people who are not reporting data consistent with keeping the GWWC pledge.
*this analysis was limited to people who (a) took the 2018 EA Survey, (b) reported having taken the GWWC pledge, (c) reported income and donation data, (d) are non-students, (e) have income >$10K, and (f) reported joining GWWC prior to 2017. N=253.
Great post, very thought-provoking! I understand the rationale for excluding student pledge-takers from this analysis, but I think looking at the share of this cohort that goes on to give >=10% after graduating could shed some light on the broader attrition question. Also seems highly relevant to the strategy for campus chapters of both GWWC and One for the World.
My hunch is that the graduation stage would be the “leakiest part of the funnel”, because a) it’s a lot more palatable to donate 1% of a small amount than 10% of a larger amount; b) after graduation people’s social circles are likely to be less EA-centric.
I just logged on to my GWWC pledge dashboard and noticed I was under 10%, even though I've been giving 10-20% the past 4 years. It seems that my reported incomes were each included twice, possibly leftover from the site migration.
I'm wondering if this happened to other people - can you check if there multiple or duplicate entries for the same date ranges?
I'm not a good person to ask about that... I'd reach out to [email protected] and [email protected].
It's worth noting that this post here uses data from the Effective Altruism Survey, which is different from the GWWC pledge dashboard. I don't think GWWC pledge dashboard has been used yet to calculate any retention statistics.
I don't think that "going silent" or failing to report donations is indication that people are not meeting the pledge. Nowadays I don't pay GWWC as an organisation much / any attention, but I'm still donating 10% a year (and then some).
To be honest I haven't read closely enough to understand where you do and don't account for "quiet pledge-keepers" in your analysis, but I at least think stuff like this is just plain wrong:
total number of people ceasing reporting donations (and very likely ceasing keeping the pledge)
In this analysis, I'm looking specifically at people who do report donations that are distinctly and clearly inconsistent with pledge keeping. "Quiet pledge-keepers" who do not report any data would not be included in this analysis because they would not be reporting data to the EA Survey. So the phenomena I report here cannot be mere instances of quiet pledge-keeping.
As for the point "total number of people ceasing reporting donations (and very likely ceasing keeping the pledge)" which refers to GWWC analysis that may refer to quiet pledge-keeping, it is impossible to know to what degree people do or do not keep the pledge but fail to report it. My intuition, shared by others, is that people who don't return GWWC's emails asking for whether people keep the pledge are likely not keeping the pledge, but I agree there can be exceptions to this (such as maybe you) and that perhaps there are actually a lot of quiet pledge-keepers.
Some updates on this in the latest donations EA survey post
especially here.
As I've discussed with Peter, the results mainly agree with this post, but are a bit more pessimistic.
@Peter_Hurford
To look at this in depth, we turn to data from the 2017 and 2018 EA Surveys.
Link doesn't work.
The Giving What We Can (GWWC) Pledge asks people to donate 10% of their income to the organizations they believe can most effectively use it to improve the lives of others. It was once used as a benchmark for the EA movement in determining growth and allocating grant money, however it has been de-emphasized over the past few years. The pledge is notably not intended for everyone, and taking the pledge can result in a loss of flexibility[1]. It does, however, seem that taking the pledge, or convincing someone else to take the pledge, is of at least some value, since donations are generally a good thing and the pledge plausibly convinces people to donate more than they otherwise would. But there’s a question of how much value pledging has and how long people stick with it.
To look at this in depth, we turn to data from the 2017 and 2018 EA Surveys. Looking at our sample, we find 476 people self-identifying as Giving What We Can members[2] in the 2017 data and 843 people self-identifying as Giving What We Can members in the 2018 data. However, most of this analysis will focus on the ~400 GWWC members that are non-students and have self-reported income and donation information.
Our analysis finds that:
GWWC members donate more than non-GWWC members, both absolutely and as a percentage of income.
~40% of self-reported GWWC members are not reporting donation data that is consistent with keeping their pledge -- far more pledgers than GWWC originally reported based on data ending in 2014.
This lack of keeping the pledge cannot be fully or adequately explained by appealing to people bunching donations in a particular year, discrepancies in when people took the survey versus joined GWWC, or people being non-earners (e.g., students, retired people, or homemakers).
No relationship is found between when someone takes the pledge, joins EA, or someone’s age and whether or not they keep their GWWC pledge, frustrating arguments that this analysis is biased by timing issues. However, a significant relationship between pledge-keeping and income is found. Predictably, for example, people earning more than $100K are far more likely to keep the pledge than people earning under $30K.
This trend is most likely the result of attrition over time, but could include methodological shortcomings, such as sampling bias or inaccurate self-reporting (either to us, GWWC, or both).
Looking at EA Survey data from 2018, self-identified GWWC members in our sample reported donating an average of $14,847.77 and a median of $2,890.56 during 2017, whereas non-members reported donating an average of $9,162.14 and a median of $520.12. Looking at non-students only[3], self-identified GWWC members in our sample report donating an average of $20,327.76 and a median of $4,199 in 2017, whereas non-members report donating an average of $12,287.59 and a median of $901.04 (Details are in Table 1).
GWWC’s influence is also clear when looking at the percentage of income donated. When looking at GWWC members, excluding students, we see they donated a mean of 13.8% and a median of 10% in 2017, compared to a mean of 7.64% and a median of 2.2% among non-members. Even after the high variability in percentage of income donation is considered, the difference is still considered statistically significant.
Another possibility for impact that is harder to analyze is the influence GWWC has on the moral norms of EA. For example, there may be EAs who choose not to take the pledge itself for concerns of flexibility, but still donate more than they otherwise would because of the influence of GWWC.
Indeed, in the distribution chart above we do see a peak at 10% even for non-GWWC members, suggesting that this target has some pull. However, the 10% target has cultural and historical precedent before GWWC existed, so it’s not easy to tell what part of this effect is attributable to GWWC or not.
While the GWWC pledge is meant to be taken seriously as a lifelong commitment, not everyone who takes the pledge ends up following through. As of Giving What We Can’s 2014 Impact Analysis, they had noted that 1.7% of members leave each year and an additional 4.7% of members “go silent” each year, meaning that GWWC has not been able to hear a response from them after two years (doing roughly annual check-ins), with a total number of people ceasing reporting donations (and very likely ceasing keeping the pledge) at 5.8%.
It was encouraging that Table 1 found the median donation of GWWC members to be 10%, showing that the median GWWC member is on track. However, when we bin our EA survey respondents into donating 10% or more or not doing so and exclude students and people earning $10K or under, we find that only 62% of GWWC members reached their pledge in 2015 (see Table 4), only 54% did so in 2016 (see Table 5), and only 52% did so in 2017 (see Table 3).
How do we explain that many GWWC members are reporting EA Survey data inconsistent with GWWC pledge keeping? One potential explanation is that perhaps GWWC members are “bunching” their donations, grouping two years of donations into a single year. Luckily, while the 2018 EA Survey only has us look at 2017 donation data, the 2017 EA Survey captured both 2016 and 2015 donation data for the same individuals, letting us look at a snapshot of two years of donation data for the same individual, allowing us to look to see if people meet the pledge over a two-year period instead of one[4].
We try to capture this in Table 6 and find that when you average individuals’ donations over both 2015 and 2016, 69% of GWWC members (and 17.6% of non-members) are on track to keep the pledge. There could be additional bunching not in the 2015/2016 time horizon of this survey (such as 2014-2015 bunching or 2016-2017 bunching) that might create false negatives. However, looking at data in 2018 did not find any large population of people donating in the region of 20%, which makes bunching uncaptured by Table 6 fairly unlikely.
In the 2018 EA Survey, we captured data directly about when people joined GWWC (whereas in the 2017 survey we only had data about when people joined EA generally, which was also recorded in the 2018 survey). Table 7 shows that 13.64% of those who took the pledge in 2018 were already donating 10% or more in 2017. Going beyond that we can see that around half of those who took the pledge in a year before 2017 were donating 10% or more in 2017, though there are fewer people in the earlier year groups. We can also see a discernible trend of people being more likely to keep their pledge the longer they’ve been involved in GWWC. The results of two Cochran-Armitage trend tests (approximated, and simulations based) are close and lead us to conclude that there is a linear trend in the proportions[5].
It’s harder to see how this trend holds up in our prior data. For previous years when we did not ask specifically when people joined GWWC, we can try to make up for this by filtering on when people claim they first started self identifying as an EA. Notably, this is not the same as the year they joined GWWC[6], and they may report being a GWWC member as of the time of the survey, and being an EA before the time of their donations, despite not being a GWWC member at the time of their donations. Thus not donating 10% in that year would not be breaking the pledge, because they had not taken the pledge yet.
Another factor would be that while we filter out people who report being students as of the time of them taking the survey, it’s possible (and, given the age of EAs, quite likely in many cases) that they were a student at the time of the donations. Thus they would only need to meet a lower pledge of 1% of spending money, rather than 10% of income.
Table 8 shows us that, predictably, those joining EA in 2018 have very low pledge adherence as they were overwhelmingly likely not to be GWWC members at the time of their reported donations. This is why we have excluded them from our analyses elsewhere. Beyond that, we see a similar trend as before, and again trend tests lead us to conclude, with a very low risk of being wrong (lower than 1.44%), that there is a linear trend in the proportions.
This trend is likely the case due to a variety of potential factors, including but not limited to:
First we can’t rule out mere survivorship bias - people less likely to keep their pledge are also less likely to keep showing up in our survey data.
People recruited into EA and GWWC earlier are less likely to be the result of mass media pushes and thus are more likely to have discovered GWWC organically and be more committed. (However, GWWC media pushes have now died down again, which could complicate this trend.) Additionally, early GWWC had a higher barrier to entry (having to mail in a physical form as opposed to join online) which screened for commitment.
People who have been in EA longer have also generally have higher earning potential, thus making it easier to keep a 10% pledge.
Looking specifically at that third factor (confounding by age), we look to Table 10 and Table 11, which seems to show a discernible trend of pledge keeping increasing as age increases, and trend tests lead us to conclude that there is a linear trend in the proportions[7]. However, while statistically significant, it does not hold up to a stricter multiple comparisons check[8].
One question is whether people who are not reporting keeping the pledge are just mistaken when they self-report being a GWWC member - maybe they took the smaller “Try Giving” pledge which does not have a 10% commitment or maybe they didn’t pledge at all. To check this, I cross-referenced the list of names I have of GWWC members in the EA Survey with GWWC’s member roster. In the 2018 EA Survey data, I found 190 self-reported GWWC members who joined before 2017 and gave their name[9] plus enough information to determine if they kept the pledge in 2017. Of these people, 69% of them were keeping the pledge. Of the 59 identifiable non-pledge keepers, 48 (81%) of them were identifiable in the roster. I also selected a random subset of 50 identifiable pledge keepers and found 46 (92%) of them in the roster. Based on this, people being mistaken about whether they are GWWC members is a possible explanation for a small fraction of people missing the pledge but still leaves a substantial fraction unexplained.
When looking at factors that best predict whether one will keep the GWWC pledge, the most important factor that emerges is income. The EA Survey looked at both individual and household income (e.g., a couple filing taxes jointly). Here, we chose to look at individual income. As seen in Table 12 and Table 13, there was a strong relationship (logistic regression average marginal effect of 11% p<0.0001) between income brackets and pledge adherence. This is also the case when looking at continuous income data where individual income is correlated with giving 10% or more (Point-Biserial r = 0.1480, p<0.03).
While certainly correlated with income, one’s area of employment could also be associated with pledge keeping. As seen in Table 14 and 15, those in Earning-to-Give careers are more likely to keep their pledge than those in direct charity, non-profit work, or research. Additionally, those working in academia and for-profit industries were a lot more likely to keep their pledge than those who were unemployed or worked for the government[10].
However, this may be the result of some confounding where people in some job categories are more likely to be engaged EAs. Additionally, one would think that those self-identifying as taking an “Earning to Give” career are basically defining themselves by donating money, so they would be more likely to keep the pledge.
It is important to note that members who are not meeting their pledge are still, on average, donating more than their non-member counterparts. We find that of the non-students who donate less than 10%, those reporting GWWC membership donate 4.5% on average, compared to a 2.4% average of non-GWWC members who are not students and donate less than 10%. This difference is statistically significant (t-test, p < 0.0001).
Based on data across multiple EA Surveys, it does appear that a proportion of GWWC members are not reporting donation data that is consistent with keeping their pledge, and that this proportion is much larger than would’ve been expected. GWWC originally reported in 2014 an annual membership attrition rate of ~4.8%, however we record ~40% of the sampled GWWC membership population as not keeping their pledge in any particular year (not to mention EAs who drop out of the EA Survey altogether[11]). A variety of attempts to explain away this trend do not appear adequate enough, though we cannot fully rule out some sampling or other methodological issue in calculating this conclusion.
Despite this, GWWC members do donate more than non-GWWC members (though this difference is only statistically significant in 2017 and not in 2015 or 2016 because there is very high variability in donation amounts) and GWWC members donate more as a percentage of income than non-GWWC members (this is statistically significant). Together with provisional data on EA retention and EA growth rate, this could be taken to be a springboard for further discussion and analysis of how people join and leave the effective altruism movement. We hope to continue to explore this in the forthcoming 2019 EA Survey and we’d appreciate your feedback.
This post is part of the supplementary posts for the EA Survey 2018 Series. The annual EA Survey is a project of Rethink Charity with analysis and commentary from researchers at Rethink Priorities.
This post was written by Peter Hurford and Neil Dullaghan, with graphs by Neil Dullaghan, and analysis by Peter Hurford, Neil Dullaghan, David Moss, and Tee Barnett. Thanks to Julia Wise and Greg Lewis for providing very helpful feedback.
Previous articles in the EA Survey 2018 Series include:
I - Community Demographics & Characteristics
II - Distribution & Analysis Methodology
III - How do people get involved in EA?
IV - Subscribers and Identifiers
VIII- Where People First Hear About EA and Influences on Involvement
IX- Geographic Differences in EA
X- Welcomingness- How Welcoming is EA?
XI - How Long Do EAs Stay in EA?
Prior EA Surveys include:
The 2017 Survey of Effective Altruists
The 2015 Survey of Effective Altruists: Results and Analysis
The 2014 Survey of Effective Altruists: Results and Analysis
If you like work from Rethink Priorities, please consider subscribing to our newsletter. You can see all our work to date here.
See also “Considerations against pledging donations for the rest of your life”. ↩︎
These are people who answered “Yes” when we asked "Have you have taken the Giving What We Can pledge?" ↩︎
When comparing these figures, it is more instructive to exclude students, since students have both a significantly lower income and are bound by a lower “1% of spending money” pledge, rather than a 10% pledge. ↩︎
We can compare entries across different EA Surveys for those who (a) give an email address and (b) give the same email address for each survey. This is done by comparing a hashed version of the email address for anonymity. However, few people fulfill these two criteria and thus the population for our longitudinal sample is frequently not large enough to be useful. See the provisional data on EA retention for yet more information. ↩︎
The Cochran–Armitage test for trend shows a very low risk of being wrong (lower than 4.65%). ↩︎
There is a mean lag of 1.07 years between when someone joins EA and when they take the GWWC Pledge, for those people who have actually taken the GWWC Pledge so far. Notably, this lag calculation excludes people who have so far never taken the GWWC pledge but might in the future. ↩︎
Cochran-Armitage trend test (Asymptotic p-value) / (Monte Carlo method - Number of simulations = 5000)/Two-tailed test: p<0.038 p<0.035 (however, this fails to meet a stricter multiple comparisons threshold). Test shows a very low risk of being wrong (lower than 3.83%). ↩︎
Given the number of hypothesis tests we do in this piece, we may run into problems of multiple comparisons, so it is also helpful to consider Bonferroni correction - that is count the number of hypothesis tests in the analysis and note whether any p-values significant at the p<0.05 level remain significant at the p<0.05/(N of tested hypotheses) level. We ran ~12 such tests, making our adjusted threshold p<0.00455. We admit that this correction method may now err on the side of being too strict and that more balanced correction methods are available in the literature, but we decided not to look into this more as it did not change our endline conclusions. ↩︎
Names are available in confidential, non-shared, non-public EA Survey data for those who gave it. Consistent with our privacy policy, we do not provide any individual-level data unless explicit permission is given. This analysis was aggregated and the identities and pledge-keeping status of individuals was kept strictly confidential. ↩︎
Also note that while a huge majority of those who reported being homemakers are donating more than 10%, this comes from only 5 individuals who span the individual income spectrum from below $10K to above $1M. ↩︎
The data on EA retention shows a ~40% rate of people leaving EA over a five year time span (though this number is very preliminary and can be dramatically different based on the chosen methodology used). Note, however, this number cannot be fairly compared to the rate of people not keeping their pledge, since that is on the basis of looking at one particular year, not a five year window. ↩︎
One group I'm especially interested in is people who were active in EA, took the GWWC pledge, and then drifted away (eg). This is a group that likely mostly didn't take the EA Survey. I would expect that after accounting for this the actual fraction of people current on their pledges would be *much* lower.
Since we don't know the fraction of people keeping their pledge to even the nearest 10%, the survey I would find most useful would be a smallish random sample. Pick 25 GWWC members at random, and follow up with them. Write personalized handwritten letters, place a phone call, or get a friend to contact them. This should give very low non-response bias, and also good qualitative data.