All of AnonymousEAForumAccount's Comments + Replies

Thanks so much for this additional data and analysis! Really interesting stuff here. To me, the most interesting things are:

  1. The midpoint of the satisfaction scale being a pretty good threshold for the point at which we see behavior changes 
  2. The relatively high frequency of people with high satisfaction temporarily stopping promoting EA (and the general flatness of this curve)
  3. I was surprised that for the cohort that changed their behavior, “scandal” was just one of many reasons for dissatisfaction and didn’t really stand out. The data you provide looks
... (read more)

That makes sense. That said, while it might not be possible to quantify the extent of selectin bias at play, I do think the combination of a) favoring simpler explanations and b) the pattern I observed in the data makes a pretty compelling case that dissatisfied people being less likely to take the survey is probably much more of an issue than dissatisfied people being more likely to take the survey to voice their dissatisfaction.

As I mentioned in point 3 of this comment:

Looking at the distribution of satisfaction scores in late 2022 vs late 2023, we see the reduction in satisfaction over that period coming from fewer people giving high ratings of 9 or 10, and more people giving low ratings of 3-5. But for the lowest ratings, we see basically no change in the number of people giving a rating of 2, and fewer people (almost nobody) now giving a rating of 1.

This suggests we could crudely estimate the selection effects of people dropping out of the community and therefore not answering... (read more)

8
David_Moss
17h
This is a neat idea, but I think that's probably putting more weight on the (absence) of small differences at particular levels of the response scale than the smaller sample size of the Extra EA Survey will support. If we look at the CIs for any individual response level, they are relatively wide for the EEAS, and the numbers selecting the lowest response levels were very low anyway. 

Great post! Another excellent example of the invaluable work Rethink Priorities does.

My observations and takeaways from this latest survey are:

  1. Overall satisfaction is low. On a 10 point scale, a 6.77 rating isn’t very good. Only ~12% give a rating of 9 or 10, and <40% give a rating of 8 or higher. (I’m aware that a 6.8 rating on a ten point scale still suggests more satisfaction than dissatisfaction, but given that we're talking about people rating their satisfaction with a community they've chosen to participate in I think we should interpret this resu
... (read more)

Many thanks!

I think it would be very valuable to closely examine the cohort of people who report having changed their behavior...

All behaviour changes were correlated with each other (except for stopping referring to EA, while still promoting it, which was associated with temporarily stopping promoting EA, but somewhat negatively associated with other changes).

All behaviour changes were associated with lower satisfaction, with most behavioural changes common only among people with satisfaction below the midpoint, and quite rare with satisfaction above the ... (read more)

Hi David! Any update on what (if anything) is going on with this survey and sharing its results? Was this part of the survey that was conducted in late December?

FWIW I’m pretty surprised and disappointed that my question about whether EV will continue to exist after the spinoffs never got a response. It’s a key piece of information about one of the most important EA orgs, and it’s clear that others are curious about it from the karma of my question, the comments replying to it, and the same question being raised in another thread. Even if the answer is “we don’t know yet”, that would be valuable to communicate. And if the answer is known, giving clarity to the rest of the community feels like extraordinarily low h... (read more)

Thanks for this update! Two questions…

  1. When all the sponsored projects have been spun out, will EV continue to exist? If so, what will it do?
  2. “I plan to share other non-privileged information on lessons learned in the aftermath of FTX and encourage others to share their reflections as well.” Do you have an estimated timeline for this? 

1. It's unclear what the legal status of EV will be at the end of the process. If it does exist, I expect it would be in a minimalist fashion and I wouldn't expect it to resemble what it has historically looked like (e.g. I don't expect it to be a fiscal sponsor for multiple projects).

2. No specific timeline. It's in a queue with other pieces of public communications I expect to do after I transition into a new role at CEA, and I'm not planning on it being the first piece.

2
AnonymousEAForumAccount
2mo
FWIW I’m pretty surprised and disappointed that my question about whether EV will continue to exist after the spinoffs never got a response. It’s a key piece of information about one of the most important EA orgs, and it’s clear that others are curious about it from the karma of my question, the comments replying to it, and the same question being raised in another thread. Even if the answer is “we don’t know yet”, that would be valuable to communicate. And if the answer is known, giving clarity to the rest of the community feels like extraordinarily low hanging fruit.
6
RyanCarey
3mo
Related to (1) is the question: which sponsored projects are definitely being spun out?

Thanks Ben, I appreciate this detailed response!

Re: 2, very helpful to know CEA’s top of funnel target. To the best of my knowledge, this hasn’t been shared before. Are there also targets for middle and bottom of funnel growth, and if so, would you mind sharing those?

Re: 3, I agree that both of your points suggest raising the target might make sense. But in the other direction, all else equal we should expect growth rates to slow over time (30% annual growth obviously isn’t sustainable in perpetuity). 

Re: 4, I would VERY much like to see EA develop gr... (read more)

Thanks for clarifying this! I agree with Jason’s point that the percentage of CEA’s funding coming from OP’s GCRCB team is high enough that the exact figure doesn’t really change the incentives much. But I’m very glad that CEA has made the effort to try and diversify OP’s funding.

I do want to flag that if your understanding is correct, it seems quite bad/weird that the incentives/priorities/strategies of the most important EA community building organization are significantly (mainly?) influenced by differences in “willingness to pay for talent” between OP’s GCRCB and GHWCR teams.

Has CEA ever solicited funding from Open Phil's Global Health and Wellbeing team and/or does CEA have plans to do so? If not, why not? 

IMO a world where CEA got e.g. 50% of its funding from OpenPhil's GCR team and 35% of its funding from OpenPhil's GHW team would be vastly preferable (due to incentives being better aligned with the broader community) to CEA getting 80% of its funding from OpenPhil GCR.  

Yep! We have solicited and received funding from Open Philanthropy’s Effective Altruism (Global Health and Wellbeing) program. We most recently worked together with them on an Effective Giving Summit earlier this year (funded by them and organized by our Partner Events team). 

We have discussed whether they or other OP teams (other than GCRCB) would be interested in funding CEA. While I don’t want to speak for them, my (briefly summarized) understanding of why we have had more success with GCRCB is because they have a higher willingness to pay for talent.

I’ve looked over Angelina’s (excellent) report on EA growth rates, and it raised some questions about how you see top of funnel growth rates for the community. 

In July’s mid-year update on CEA, you wrote: “our priorities haven’t changed since we wrote about our work in [December] 2022: helping people who have heard about EA to deeply understand the ideas, and to find opportunities for making an impact in important fields. We continue to think that top-of-funnel growth is likely already at or above healthy levels, so rather than aiming to inc... (read more)

8
Ben_West
4mo
This is a great and well researched question, thanks for asking! First: I think my personal opinion here might matter less than you would imagine: I am intentionally trying not to push for any major strategic changes to CEA while I am in the interim role. That being said: 1. I agree that FTX causes the drop to be overstated. (And more importantly, Angelina, who collected this information, also agrees.) 2. Historically CEA has targeted a top of funnel growth of 30%, which, if you take these metrics at face value, means we are perfectly on target. 3. My personal low confidence guess is that our target growth rate should be a bit higher than it was historically, because: 1. CEA now has two full-time communications people, an increase of infinity percent over the zero people we had last year 2. EA (and particularly AI safety) have become mainstream/professionalized (more people know what the causes are, why they are important, stable funding is available, etc.) which allows us to attract a broader demographic, specifically older/more senior people 4. Some details that seem worth considering: 1. Will is planning to "distance myself from the idea that I’m “the face of” or “the spokesperson for” EA". I have not heard a compelling suggestions for an alternative person to replace him. (E.g. there are a few people writing books or doing interviews, but my sense is that none of them are likely to be cited as a major place people first heard about EA in the next EA survey.) 2. Peter Singer is doing a tour. I'm interested in seeing if we can rope him into doing some stuff, he has historically been very successful at top of funnel outreach. 5. Some cruxes that would change my mind: 1. More employers (or things further down in the funnel) saying stuff like "the difference between my top candidates is really small; getting more people into the hiring pool doesn't seem very useful." I hear this occasionally, but not enough to make me currently worry about

Thank you Angelina for this excellent post! I really hope that this analysis, or something quite similar, gets conducted on a regular basis and shared with the entire community.

A few thoughts on this:

  • There are a lot of complications, judgement calls, assumptions that need to go into something like this. I think you did a very good job of articulating these and the associated limitations, and also made reasonable judgement calls along the way.
  • You mention that some metrics measure “levels” while others measure “changes”. I haven’t fully thought through this,
... (read more)
4
Angelina Li
4mo
Thanks for your thoughtful feedback on this! I really appreciate it. I might not have time to respond in depth here, but will come back to this if/when we pick this project back up again.

Sure! Here’s a comment I wrote in response to a similar question in March. I don't think it is exhaustive, but should be fairly illustrative.

Some specific examples of EA leaders putting SBF on a pedestal that I found with a bit of brief digging:

  • At the time FTX blew up, SBF was featured on 80k’s homepage. Also, if you clicked “start here” on that homepage (the first link aside from a subscription form) you were brought to an article that featured SBF as one of three individual profiles.
... (read more)
2
Ben_West
5mo
Awesome, thanks!

IMO the big mistakes before the collapse were 1) making SBF an EA poster child and 2) not giving enough weight to the possibility of a massive drawdown or complete collapse in FTX’s valuation.*

In each case, I think the magnitude of the mistake was dramatically increased by the fact that some parts of EA leadership apparently reasons to think SBF was/could be shady well prior to the FTX collapse. (Though to be clear, it does not seem reasonable to think that those leaders should have foreseen the extent of the fraud that appears to have taken place). I also... (read more)

6
Ben_West
5mo
Could you clarify what was done to "make SBF an EA poster child"/ "putting SBF on a pedestal"? I think I know some things you're referring to (e.g. he was on an 80 K podcast) but I'm not sure I can generate the entire list of things you are pointing to, and I would find it helpful to have more examples.

Thanks Angelina! Your post is terrific, and I hope that you or someone else will update it in the future. It seems like a great starting point for the sort of regular analysis of community metrics I’ve called for. And I appreciate your clarification on some of the data points in my analysis. 

Luke Freeman from GWWC has shared some excellentobservations around reduced willingness for community members to advocate for GWWC post-FTX (partially but not solely due to FTX). I think it would be quite valuable if the follow-up survey examined the dynamics he describes, as I doubt they are unique to GWWC.

Thanks for clarifying. FWIW, given community desires for more communication from leadership and greater transparency, I think it would have been good for the board to share this decision publicly.

While I agree in terms of the ethos, I also understand the difficulty and a reasonable amount of the complexity of the situation and why this would have been difficult (and possibly bad, all things considered) to do at the time. I do not envy the position of those involved in this level of decision making/crisis response, and recognise the need to do a lot of satisficing at the time. 

Thanks for taking the time to write this! Very helpful to get your perspective!

we were instructed to pause all our giving season campaigns around the time of the crisis

This line surprised me, as my impression was that GWWC is essentially autonomous. Who gave this instruction? The EV board? I think the decision was reasonable, I’m just trying to understand how it was made.

8
Luke Freeman
5mo
It was a board level decision. While that is largely true for day to day operations, EV UK and EV US are still essentially responsible for the actions of their projects.

Thanks Luke, for this extremely thoughtful and informative comment!

The dashboard is a fantastic resource, very impressive. I poked around a bit and noticed that the “monthly active effective givers” series look a lot less spikey than the pledge data (which makes sense). It seems like the various permutations of the active effective givers metrics (pledged, unpledged, reported, facilitated, one-off, recurring) all follow a pretty similar patterns: many years of growth (with some periods growing faster than others and some rare short flattish periods for som... (read more)

Thanks 😀 Glad it was helpful!

Is that a fair characterization?

Yep!

have you seen any signs of this dynamic easing?

No strong signs yet. This upcoming giving season will be a true test of that though.

I’m curious whether you think these issues are still as impactful as they were right after they occurred.

My impression is that it's less, but far from back to normal. Also bearing in mind that "normal" is hard to define (especially with GWWC) as a lot has changed over the last decade!

Also, do you think the other issues would have had as much of an impact if FTX h

... (read more)

Thanks for clarifying that Caleb, that does seem substantially less problematic than granting to Nonlinear themselves.

Thanks for flagging the disclaimer ("Please note that this page is in beta testing and grant data may not be accurate"), I'd missed that.

Catherine, you and Nicole are both CH team members who advise the EAIF and the LTFF. Given that CH “heard about many of the concerns mentioned in [Ben’s] post” in mid-2022, did either of you share those concerns with the EAIF team prior to that fund granting $73k to Nonlinear in 4Q22

You’ve previously written that “meta work like incubating new charities, advising inexperienced charity entrepreneurs, and influencing funding decisions should be done by people with particularly good judgement about how to run strong organisations, in addition to having... (read more)

4[anonymous]5mo
I don't know if the grant information is accurate (there's a disclaimer on the page), but if it is, this is pretty shocking. I would appreciate clarification on this. 

I wholeheartedly endorse a more rigorous analysis of the data, I just didn't have the capacity or capability to undertake that. The EA Funds dashboard allows users to export the raw data. The CEA dashboard doesn't, though I assume CEA could provide the raw data without too much hassle (and hopefully they'll add an export functionality).

4
Angelina Li
5mo
Thanks for the nudge! By the way, we've now created a process for users to request access to our data — more details here.

Thanks for sharing your thinking and predictions Ben. I look forward to the publication of the internal report and RP's updated survey.

Yeah I think we’re in agreement that for people outside of EA, FTX didn’t really have an impact because the vast majority of those people have never even heard of EA.

What I guess I’m more interested in is your current thinking (and whether/why it has changed) about how FTX has impacted people already in EA and the implications for EA’s future trajectory. I assume the internal report and my analysis are more relevant to the EA community rather than the general public.

7
Ben_West
5mo
Cool, in that case: A summary of our internal growth estimates shows more growth than your summary here (though there are a bunch of examples of things with decreased engagement, including stuff you point out here), and my guess is that RP's survey will show mean reversion (i.e. people's opinions about EA will be more similar to pre-FTX levels than they were immediately following the FTX collapse), though to be clear this is just a prediction because RP hasn't done the survey yet.[1] 1. ^ This is a scenario where a prediction market might be action guiding for me, if someone wants to make it. If I thought the answers to RP's survey were going to be the same or lower than they were in December, CEA might change some of its priorities, though I'm not exactly sure how.

Thanks for the update David! These results should be very interesting.

Thanks so much for sharing this data Bella! I agree 80k’s metrics look a lot better than everything else I looked at. I’ll add an edit to the OP to that effect, and will link to your comment. To the extent 80k is doing something different from other orgs that could explain the better performance, it would be great to identify that (I wonder if your third point about advertising might be a differentiating factor).

To make a more general point, I think a number of the metrics you cite in the post show returning to levels from early 2022, or 2021, which (at le

... (read more)
Jason
5mo62
16
1
2
4

Also, if you had asked for growth predictions in September 2022 -- and told the forecasters that 2023 would be a year in which public awareness of / interest in the nearness and risks of powerful AI greatly increased -- one would likely factor this information into the prediction in addition to historical trends.

By rough analogy, if the stock market is unexpectedly up 25 percent over last year, your sector is up 35 percent, but your company's stock is flat . . . the market seems to be saying something negative about your company. The flat stock value may well mean that positive macroeconomic trends and sector trends have been cancelled out by negative outlook on your company.

Thanks for replying & editing the OP — appreciate it! :D

I wonder if your third point about advertising might be a differentiating factor

Yeah — to my knowledge, we have the biggest team focused on outreach of any (single) EA organisation (3 FTE). I think this is probably a big part of it. 
 

...we’re not seeing a “normal level of growth”. While growth had generally been the norm pre-FTX, we’re now either seeing zero growth (EA forum metrics) or outright contraction (EA Funds donations/donors, EA.org intro page, EA newsletter).

I think that's fair... (read more)

Thanks Ben! I look forward to reading CEA’s report on this topic. While we wait for publication, are you able to share the big picture takeaway? Having seen the data in CEA’s internal report and the information I’ve presented here, do you still hold your June belief that: “it seems that the FTX crash hasn’t, overall, impacted sentiments toward EA very much”? 

I also look forward to seeing the results of the updated RP survey, though it sounds like it hasn’t been conducted yet and then will take some time to analyze the results. Do you know roughly when... (read more)

Having seen the data in CEA’s internal report and the information I’ve presented here, do you still hold your June belief that: “it seems that the FTX crash hasn’t, overall, impacted sentiments toward EA very much”?

Yes, my best understanding is still that people mostly don't know what EA is, the small fraction that do mostly have a mildly positive opinion, and that neither of these points were affected much by FTX.[1] This understanding was not affected by your post or the internal document I referred to very much, but I suspect we are talking past ea... (read more)

8
David_Moss
5mo
  I'm hopeful that we can launch the survey sometime between the end of this month and the middle of November (the next 2-4 weeks). Like with Ben's report, we're just waiting on input from a variety of different orgs (the survey is addressing multiple different aims, besides looking at community health/FTX, so there are quite a few different stakeholders). Allowing another 2-4 weeks for the survey to run (taking us up to early-mid December), I would still aim report on the FTX/community health results before the end of the year.

Yeah, the OP is all about what has happened and punts on the question of why it has happened. Reduced outreach could definitely be an explanation for how some of these metrics have changed (IMO the other likeliest explanations include reduced funding, disillusionment with EA principles, disillusionment with EA orgs, and disillusionment with EA leaders, though there could certainly be other factors at play). Personally I think reduced outreach makes much more sense for some metrics (e.g. newsletter subscriptions) than others (e.g. forum engagement). 

I ... (read more)

FYI I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.

I included your comment about 80k's metrics being largely unaffected, but if there's some updated data on if/how 80k's metrics have changed post-FTX that would be very interesting to see.

This is a very good point, and something I probably should have addressed in the OP. 

I agree it’s totally possible for non-FTX factors to be causing the deterioration in metrics I describe. However, I think the evidence points to FTX being the dominant factor for two main reasons:

  1. The timing of the downturn in metrics aligns almost perfectly with FTX’s collapse. That’s not really the case with alternative explanations. Using your examples, Bitcoin (as a proxy for cryptocurrency) lost ~2/3 of its value from November 2021 to October 2022 (prior to FTX co
... (read more)

Note that many EA organizations (including CEA) paused/lowered outreach post-FTX. So decreased engagement might be because of FTX in some sense, but maybe not for the reason one would initially think.

FYI, I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.

FYI, I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.

I’ve just released a post which offers significantly more empirical data on how FTX has impacted EA. FTX’s collapse seems to mark a clear and sizable deterioration across a variety of different EA metrics.

To manage expectations: While I hope this post generates some productive discussion, I have very limited capacity to engage with a lot of back and forth. I’ll likely offer some high level thoughts but will not contribute extensive comments.

They also suggested some steps that we didn’t see as feasible for us. 

Can you disclose the specifics of some or all of these steps and the reasons why you didn't think they were feasible?

I agree with all of this, and hope the CH team responds. I'd also add that the video of Kat's talk has a prominent spot on the EAG 2023 playlist on CEA's official youtube channel. That video has nearly 600 views.

If initial due diligence conducted by an independent third party didn’t uncover obvious evidence about which side is correct, IMO that’s very helpful info for the broader community and it really seems like there should be a way of expressing that in a way that doesn’t introduce legal liability.

I would like to see an updated post from the team on what the community should and should not expect from them, with the caveat that they may be somewhat limited in what they can say legally about their scope.

Agree this would be helpful. In addition to clarifying what community expectations should be, I’d like to know whether the Nonlinear affair will be included either or both of the internal and external reviews that are (were?) being conducted. And if so, would that inclusion have taken place if Ben hadn’t published his post?

Whether or not CEA/EV had in-house counsel, I’d like to think they had an ability to access legal advice. If not, that seems like a poorly thought out setup.

I agree it makes sense for EV to have a lower risk tolerance in light of the Charity Commission investigation. However, I’m making the following assumptions (it would be great if a lawyer could opine on whether they are accurate):

  1. There are simple steps CH could have taken that would carry very little risk of a defamation suit. For instance, I find it hard to believe CH would be liable if they’d issued
... (read more)

First, CEA definitely have access to legal counsel.

Second, I don’t think these issues are that relevant, after reading Ben’s posts.

Regardless of legal risk, the reasons for not making claims public are clear -

(A) It took Ben hundreds of hours to feel confident and clear enough to make a useful public statement while also balancing the potential harms to Alice and Chloe. This is not uncommon in such situations and I think people should not expect CH to be able to do this in most cases.

(B) CEA is not in charge of Nonlinear or most other EA orgs. Just like Be... (read more)

I find it hard to believe CH would be liable if they’d issued a public statement along the lines of “Alice and Chloe report XYZ about Nonlinear; Nonlinear disputes these claims. CH is not publicly picking a side but wants to make people aware of the dispute.”

In that example, what would CH be adding relative to Alice and Chloe making the public statement themselves? For example, if the idea is that people will give the report greater weight because it comes via CH and people know that CH wouldn't host a report like this if they didn't give it some creden... (read more)

Let me know if you'd like me to remove my comment while this gets sorted out.

2
Ben Pace
6mo
Thanks for saying that, but no request from me. (And my guess is it'll be fine and I'll add my bullets back in a day or so.)

Thank you for all your efforts in this endeavor Ben, you’ve performed a very valuable service to the community.

Your comments about CEA’s Community Health team in this post seem particularly important to me. If CEA’s CH team had in depth knowledge of how Alice and Chloe described their experiences, had found no reason to doubt those accounts, and still declined to make any kind of public statement, that’s incredibly damning. I'm open to hearing CH's take on things, but if that’s actually the case I agree with your view that “the world would probably be bett... (read more)

5
Ben Pace
6mo
Appreciate the comment. I sadly decided to edit out a few bullets on that to check in on what's okay to share. That's my fault, I will make sure to leave a new comment when I am able to add them back in, probably in a day or two (but might be longer).

While I think I disagree pretty strongly with the idea CEA CH should be disbanded, I would like to see an updated post from the team on what the community should and should not expect from them, with the caveat that they may be somewhat limited in what they can say legally about their scope.

Correct me if I'm wrong but I believe CEA was operating without in-house legal counsel until about a year ago. This was while engaging in many situations that could have easily led to a defamation suit should they have investigated someone sufficiently resourced and lit... (read more)

Thanks for running this analysis Ollie! Interesting findings!

it does suggest the results are sensitive to my scoring system. I'm not sure where this leaves me; that isn't ideal and I'd like something more robust but, on the other hand, I think these high-scoring results (people securing jobs, teams being formed) are exactly the kind of things we want to happen at our events so I think it's reasonable to put significant weight on them. 

Agree that this exercise doesn’t yield an obvious conclusion. Given that you’ve found the results to be sensitive to t... (read more)

Hey Anon,

I ran this analysis, beheading (?) the data to remove all responses which I scored over 5 and changed them to 5 (18 responses).

I realised that all of these responses were from EAGx attendees - that's not that surprising since 90% of survey responses were from EAGx attendees but that highlights a limitation of my analysis. Plausibly, if I had data from CEP retreats, we might find more high-scoring impact stories.

Previously:

This means the cost per counterfactual impact point at EAGx events is ~$1,219, while the cost per counterfactual impact point a

... (read more)

Thanks for providing that background Jessica, very helpful. It'd be great to see metrics for the OSP included in the dashboard at some point.

It might also make sense to have the dashboard provide links to additional information about the different programs (e.g. the blog posts you link to) so that users can contextualize the dashboard data.

Same goal as your analysis really- to find the most cost-effective event models for producing valuable outcomes. With a very fat-tailed scoring rubric, I’m concerned that legitimate differences between the event types might be overshadowed by the particulars of the rubric. As some of the other comments indicate, it’s not obvious how to value different outcomes on a relative basis.

Even if you don’t want to use an equal-weighted scoring system, you could see if the results change materially if you use a much less fat-tailed rubric (e.g. have scores ranging f... (read more)

4
OllieBase
8mo
Thanks, I like this idea, and it would be easy to implement! There were only a handful of cases with scores above 5 across the dataset and I think they weren't bunched anywhere. I'll report back by the end of next week (probably just in this thread), please DM me if I drop this.
Load more