That makes sense. That said, while it might not be possible to quantify the extent of selectin bias at play, I do think the combination of a) favoring simpler explanations and b) the pattern I observed in the data makes a pretty compelling case that dissatisfied people being less likely to take the survey is probably much more of an issue than dissatisfied people being more likely to take the survey to voice their dissatisfaction.
As I mentioned in point 3 of this comment:
Looking at the distribution of satisfaction scores in late 2022 vs late 2023, we see the reduction in satisfaction over that period coming from fewer people giving high ratings of 9 or 10, and more people giving low ratings of 3-5. But for the lowest ratings, we see basically no change in the number of people giving a rating of 2, and fewer people (almost nobody) now giving a rating of 1.
This suggests we could crudely estimate the selection effects of people dropping out of the community and therefore not answering...
Great post! Another excellent example of the invaluable work Rethink Priorities does.
My observations and takeaways from this latest survey are:
Many thanks!
I think it would be very valuable to closely examine the cohort of people who report having changed their behavior...
All behaviour changes were correlated with each other (except for stopping referring to EA, while still promoting it, which was associated with temporarily stopping promoting EA, but somewhat negatively associated with other changes).
All behaviour changes were associated with lower satisfaction, with most behavioural changes common only among people with satisfaction below the midpoint, and quite rare with satisfaction above the ...
Hi David! Any update on what (if anything) is going on with this survey and sharing its results? Was this part of the survey that was conducted in late December?
FWIW I’m pretty surprised and disappointed that my question about whether EV will continue to exist after the spinoffs never got a response. It’s a key piece of information about one of the most important EA orgs, and it’s clear that others are curious about it from the karma of my question, the comments replying to it, and the same question being raised in another thread. Even if the answer is “we don’t know yet”, that would be valuable to communicate. And if the answer is known, giving clarity to the rest of the community feels like extraordinarily low h...
Thanks for this update! Two questions…
1. It's unclear what the legal status of EV will be at the end of the process. If it does exist, I expect it would be in a minimalist fashion and I wouldn't expect it to resemble what it has historically looked like (e.g. I don't expect it to be a fiscal sponsor for multiple projects).
2. No specific timeline. It's in a queue with other pieces of public communications I expect to do after I transition into a new role at CEA, and I'm not planning on it being the first piece.
Thanks Ben, I appreciate this detailed response!
Re: 2, very helpful to know CEA’s top of funnel target. To the best of my knowledge, this hasn’t been shared before. Are there also targets for middle and bottom of funnel growth, and if so, would you mind sharing those?
Re: 3, I agree that both of your points suggest raising the target might make sense. But in the other direction, all else equal we should expect growth rates to slow over time (30% annual growth obviously isn’t sustainable in perpetuity).
Re: 4, I would VERY much like to see EA develop gr...
Thanks for clarifying this! I agree with Jason’s point that the percentage of CEA’s funding coming from OP’s GCRCB team is high enough that the exact figure doesn’t really change the incentives much. But I’m very glad that CEA has made the effort to try and diversify OP’s funding.
I do want to flag that if your understanding is correct, it seems quite bad/weird that the incentives/priorities/strategies of the most important EA community building organization are significantly (mainly?) influenced by differences in “willingness to pay for talent” between OP’s GCRCB and GHWCR teams.
Has CEA ever solicited funding from Open Phil's Global Health and Wellbeing team and/or does CEA have plans to do so? If not, why not?
IMO a world where CEA got e.g. 50% of its funding from OpenPhil's GCR team and 35% of its funding from OpenPhil's GHW team would be vastly preferable (due to incentives being better aligned with the broader community) to CEA getting 80% of its funding from OpenPhil GCR.
Yep! We have solicited and received funding from Open Philanthropy’s Effective Altruism (Global Health and Wellbeing) program. We most recently worked together with them on an Effective Giving Summit earlier this year (funded by them and organized by our Partner Events team).
We have discussed whether they or other OP teams (other than GCRCB) would be interested in funding CEA. While I don’t want to speak for them, my (briefly summarized) understanding of why we have had more success with GCRCB is because they have a higher willingness to pay for talent.
I’ve looked over Angelina’s (excellent) report on EA growth rates, and it raised some questions about how you see top of funnel growth rates for the community.
In July’s mid-year update on CEA, you wrote: “our priorities haven’t changed since we wrote about our work in [December] 2022: helping people who have heard about EA to deeply understand the ideas, and to find opportunities for making an impact in important fields. We continue to think that top-of-funnel growth is likely already at or above healthy levels, so rather than aiming to inc...
Thank you Angelina for this excellent post! I really hope that this analysis, or something quite similar, gets conducted on a regular basis and shared with the entire community.
A few thoughts on this:
Sure! Here’s a comment I wrote in response to a similar question in March. I don't think it is exhaustive, but should be fairly illustrative.
...Some specific examples of EA leaders putting SBF on a pedestal that I found with a bit of brief digging:
- At the time FTX blew up, SBF was featured on 80k’s homepage. Also, if you clicked “start here” on that homepage (the first link aside from a subscription form) you were brought to an article that featured SBF as one of three individual profiles.
- Both of these mentions linked to a more in depth pro
IMO the big mistakes before the collapse were 1) making SBF an EA poster child and 2) not giving enough weight to the possibility of a massive drawdown or complete collapse in FTX’s valuation.*
In each case, I think the magnitude of the mistake was dramatically increased by the fact that some parts of EA leadership apparently reasons to think SBF was/could be shady well prior to the FTX collapse. (Though to be clear, it does not seem reasonable to think that those leaders should have foreseen the extent of the fraud that appears to have taken place). I also...
Thanks Angelina! Your post is terrific, and I hope that you or someone else will update it in the future. It seems like a great starting point for the sort of regular analysis of community metrics I’ve called for. And I appreciate your clarification on some of the data points in my analysis.
Luke Freeman from GWWC has shared some excellentobservations around reduced willingness for community members to advocate for GWWC post-FTX (partially but not solely due to FTX). I think it would be quite valuable if the follow-up survey examined the dynamics he describes, as I doubt they are unique to GWWC.
Thanks for clarifying. FWIW, given community desires for more communication from leadership and greater transparency, I think it would have been good for the board to share this decision publicly.
While I agree in terms of the ethos, I also understand the difficulty and a reasonable amount of the complexity of the situation and why this would have been difficult (and possibly bad, all things considered) to do at the time. I do not envy the position of those involved in this level of decision making/crisis response, and recognise the need to do a lot of satisficing at the time.
we were instructed to pause all our giving season campaigns around the time of the crisis
This line surprised me, as my impression was that GWWC is essentially autonomous. Who gave this instruction? The EV board? I think the decision was reasonable, I’m just trying to understand how it was made.
Thanks Luke, for this extremely thoughtful and informative comment!
The dashboard is a fantastic resource, very impressive. I poked around a bit and noticed that the “monthly active effective givers” series look a lot less spikey than the pledge data (which makes sense). It seems like the various permutations of the active effective givers metrics (pledged, unpledged, reported, facilitated, one-off, recurring) all follow a pretty similar patterns: many years of growth (with some periods growing faster than others and some rare short flattish periods for som...
Thanks 😀 Glad it was helpful!
Is that a fair characterization?
Yep!
have you seen any signs of this dynamic easing?
No strong signs yet. This upcoming giving season will be a true test of that though.
I’m curious whether you think these issues are still as impactful as they were right after they occurred.
My impression is that it's less, but far from back to normal. Also bearing in mind that "normal" is hard to define (especially with GWWC) as a lot has changed over the last decade!
...Also, do you think the other issues would have had as much of an impact if FTX h
Thanks for clarifying that Caleb, that does seem substantially less problematic than granting to Nonlinear themselves.
Thanks for flagging the disclaimer ("Please note that this page is in beta testing and grant data may not be accurate"), I'd missed that.
Catherine, you and Nicole are both CH team members who advise the EAIF and the LTFF. Given that CH “heard about many of the concerns mentioned in [Ben’s] post” in mid-2022, did either of you share those concerns with the EAIF team prior to that fund granting $73k to Nonlinear in 4Q22?
You’ve previously written that “meta work like incubating new charities, advising inexperienced charity entrepreneurs, and influencing funding decisions should be done by people with particularly good judgement about how to run strong organisations, in addition to having...
I wholeheartedly endorse a more rigorous analysis of the data, I just didn't have the capacity or capability to undertake that. The EA Funds dashboard allows users to export the raw data. The CEA dashboard doesn't, though I assume CEA could provide the raw data without too much hassle (and hopefully they'll add an export functionality).
Thanks for sharing your thinking and predictions Ben. I look forward to the publication of the internal report and RP's updated survey.
Yeah I think we’re in agreement that for people outside of EA, FTX didn’t really have an impact because the vast majority of those people have never even heard of EA.
What I guess I’m more interested in is your current thinking (and whether/why it has changed) about how FTX has impacted people already in EA and the implications for EA’s future trajectory. I assume the internal report and my analysis are more relevant to the EA community rather than the general public.
Thanks so much for sharing this data Bella! I agree 80k’s metrics look a lot better than everything else I looked at. I’ll add an edit to the OP to that effect, and will link to your comment. To the extent 80k is doing something different from other orgs that could explain the better performance, it would be great to identify that (I wonder if your third point about advertising might be a differentiating factor).
...To make a more general point, I think a number of the metrics you cite in the post show returning to levels from early 2022, or 2021, which (at le
Also, if you had asked for growth predictions in September 2022 -- and told the forecasters that 2023 would be a year in which public awareness of / interest in the nearness and risks of powerful AI greatly increased -- one would likely factor this information into the prediction in addition to historical trends.
By rough analogy, if the stock market is unexpectedly up 25 percent over last year, your sector is up 35 percent, but your company's stock is flat . . . the market seems to be saying something negative about your company. The flat stock value may well mean that positive macroeconomic trends and sector trends have been cancelled out by negative outlook on your company.
Thanks for replying & editing the OP — appreciate it! :D
I wonder if your third point about advertising might be a differentiating factor
Yeah — to my knowledge, we have the biggest team focused on outreach of any (single) EA organisation (3 FTE). I think this is probably a big part of it.
...we’re not seeing a “normal level of growth”. While growth had generally been the norm pre-FTX, we’re now either seeing zero growth (EA forum metrics) or outright contraction (EA Funds donations/donors, EA.org intro page, EA newsletter).
I think that's fair...
Thanks Ben! I look forward to reading CEA’s report on this topic. While we wait for publication, are you able to share the big picture takeaway? Having seen the data in CEA’s internal report and the information I’ve presented here, do you still hold your June belief that: “it seems that the FTX crash hasn’t, overall, impacted sentiments toward EA very much”?
I also look forward to seeing the results of the updated RP survey, though it sounds like it hasn’t been conducted yet and then will take some time to analyze the results. Do you know roughly when...
Having seen the data in CEA’s internal report and the information I’ve presented here, do you still hold your June belief that: “it seems that the FTX crash hasn’t, overall, impacted sentiments toward EA very much”?
Yes, my best understanding is still that people mostly don't know what EA is, the small fraction that do mostly have a mildly positive opinion, and that neither of these points were affected much by FTX.[1] This understanding was not affected by your post or the internal document I referred to very much, but I suspect we are talking past ea...
Yeah, the OP is all about what has happened and punts on the question of why it has happened. Reduced outreach could definitely be an explanation for how some of these metrics have changed (IMO the other likeliest explanations include reduced funding, disillusionment with EA principles, disillusionment with EA orgs, and disillusionment with EA leaders, though there could certainly be other factors at play). Personally I think reduced outreach makes much more sense for some metrics (e.g. newsletter subscriptions) than others (e.g. forum engagement).
I ...
FYI I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.
I included your comment about 80k's metrics being largely unaffected, but if there's some updated data on if/how 80k's metrics have changed post-FTX that would be very interesting to see.
This is a very good point, and something I probably should have addressed in the OP.
I agree it’s totally possible for non-FTX factors to be causing the deterioration in metrics I describe. However, I think the evidence points to FTX being the dominant factor for two main reasons:
Note that many EA organizations (including CEA) paused/lowered outreach post-FTX. So decreased engagement might be because of FTX in some sense, but maybe not for the reason one would initially think.
FYI, I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.
FYI, I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.
I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.
To manage expectations: While I hope this post generates some productive discussion, I have very limited capacity to engage with a lot of back and forth. I’ll likely offer some high level thoughts but will not contribute extensive comments.
They also suggested some steps that we didn’t see as feasible for us.
Can you disclose the specifics of some or all of these steps and the reasons why you didn't think they were feasible?
I agree with all of this, and hope the CH team responds. I'd also add that the video of Kat's talk has a prominent spot on the EAG 2023 playlist on CEA's official youtube channel. That video has nearly 600 views.
If initial due diligence conducted by an independent third party didn’t uncover obvious evidence about which side is correct, IMO that’s very helpful info for the broader community and it really seems like there should be a way of expressing that in a way that doesn’t introduce legal liability.
I would like to see an updated post from the team on what the community should and should not expect from them, with the caveat that they may be somewhat limited in what they can say legally about their scope.
Agree this would be helpful. In addition to clarifying what community expectations should be, I’d like to know whether the Nonlinear affair will be included either or both of the internal and external reviews that are (were?) being conducted. And if so, would that inclusion have taken place if Ben hadn’t published his post?
Whether or not CEA/EV had in-house counsel, I’d like to think they had an ability to access legal advice. If not, that seems like a poorly thought out setup.
I agree it makes sense for EV to have a lower risk tolerance in light of the Charity Commission investigation. However, I’m making the following assumptions (it would be great if a lawyer could opine on whether they are accurate):
First, CEA definitely have access to legal counsel.
Second, I don’t think these issues are that relevant, after reading Ben’s posts.
Regardless of legal risk, the reasons for not making claims public are clear -
(A) It took Ben hundreds of hours to feel confident and clear enough to make a useful public statement while also balancing the potential harms to Alice and Chloe. This is not uncommon in such situations and I think people should not expect CH to be able to do this in most cases.
(B) CEA is not in charge of Nonlinear or most other EA orgs. Just like Be...
I find it hard to believe CH would be liable if they’d issued a public statement along the lines of “Alice and Chloe report XYZ about Nonlinear; Nonlinear disputes these claims. CH is not publicly picking a side but wants to make people aware of the dispute.”
In that example, what would CH be adding relative to Alice and Chloe making the public statement themselves? For example, if the idea is that people will give the report greater weight because it comes via CH and people know that CH wouldn't host a report like this if they didn't give it some creden...
Thank you for all your efforts in this endeavor Ben, you’ve performed a very valuable service to the community.
Your comments about CEA’s Community Health team in this post seem particularly important to me. If CEA’s CH team had in depth knowledge of how Alice and Chloe described their experiences, had found no reason to doubt those accounts, and still declined to make any kind of public statement, that’s incredibly damning. I'm open to hearing CH's take on things, but if that’s actually the case I agree with your view that “the world would probably be bett...
While I think I disagree pretty strongly with the idea CEA CH should be disbanded, I would like to see an updated post from the team on what the community should and should not expect from them, with the caveat that they may be somewhat limited in what they can say legally about their scope.
Correct me if I'm wrong but I believe CEA was operating without in-house legal counsel until about a year ago. This was while engaging in many situations that could have easily led to a defamation suit should they have investigated someone sufficiently resourced and lit...
Thanks for running this analysis Ollie! Interesting findings!
it does suggest the results are sensitive to my scoring system. I'm not sure where this leaves me; that isn't ideal and I'd like something more robust but, on the other hand, I think these high-scoring results (people securing jobs, teams being formed) are exactly the kind of things we want to happen at our events so I think it's reasonable to put significant weight on them.
Agree that this exercise doesn’t yield an obvious conclusion. Given that you’ve found the results to be sensitive to t...
Hey Anon,
I ran this analysis, beheading (?) the data to remove all responses which I scored over 5 and changed them to 5 (18 responses).
I realised that all of these responses were from EAGx attendees - that's not that surprising since 90% of survey responses were from EAGx attendees but that highlights a limitation of my analysis. Plausibly, if I had data from CEP retreats, we might find more high-scoring impact stories.
Previously:
...This means the cost per counterfactual impact point at EAGx events is ~$1,219, while the cost per counterfactual impact point a
Thanks for providing that background Jessica, very helpful. It'd be great to see metrics for the OSP included in the dashboard at some point.
It might also make sense to have the dashboard provide links to additional information about the different programs (e.g. the blog posts you link to) so that users can contextualize the dashboard data.
Same goal as your analysis really- to find the most cost-effective event models for producing valuable outcomes. With a very fat-tailed scoring rubric, I’m concerned that legitimate differences between the event types might be overshadowed by the particulars of the rubric. As some of the other comments indicate, it’s not obvious how to value different outcomes on a relative basis.
Even if you don’t want to use an equal-weighted scoring system, you could see if the results change materially if you use a much less fat-tailed rubric (e.g. have scores ranging f...
Thanks so much for this additional data and analysis! Really interesting stuff here. To me, the most interesting things are:
- The midpoint of the satisfaction scale being a pretty good threshold for the point at which we see behavior changes
- The relatively high frequency of people with high satisfaction temporarily stopping promoting EA (and the general flatness of this curve)
- I was surprised that for the cohort that changed their behavior, “scandal” was just one of many reasons for dissatisfaction and didn’t really stand out. The data you provide looks
... (read more)