Hide table of contents

tl;dr: AI Safety became the biggest thing in EA Forum discussion last year, and overall Forum engagement trended downards. What that means for the wider EA movement/philosophy is up for interpretation. If you have your own questions, let me know and I'll dive in (or try to share the data)

1. Introduction

This is a follow-up to my previous post using the EA Forum API to analyse trends in the Forum. Whereas that last post was a zoomed-out look at Forum use for as far back as the API goes, this is a specific look at the Forum aggregations and trends in 2023 alone.

Needless to say, last year was a tumultous year for EA, and the Forum is one of (if not the) primary online hubs for the Community to discuss issues and self-organise. I hoped to see if any of these trends could be spotted in the data available, and also see where the data led me on some more general questions.

I'm sharing this now in a not-quite-perfect state but I'd rather post and see the discussion it promotes than have it languishing in my drafts for much longer, and as noted in section 3.4.2 if you have a query that I can dive into, just ask!

2. Methodology

(For more detail in the general method, see the previous post)

On Monday 6th May I ran two major queries to the EA Forum API:

    1) The first scraped all posts in Forum history. I then subselected these to find only posts that were in the 2023 calender year.

    2) I ran a secondary query for all of these postIds to find all comments on these posts, and again filtered to only count comments made in 2023.

Any discrepancy with ground truth might be because of mistakes of my part during doing the data collection. Furthermore my data is snapshot as of how the 2023 Forum looked at May 6th this year, so any Forum engagement that was deleted (or users who deleted their account) at the point of collection will not be sampled. I'll leave more specific methods to the relevant graphs and tables below.

I used Python entirely for this, and am happy to talk about the method in more coding-detail for those interested. I'm trying to resussicate my moribund GitHub profile this summer, and this code may make its way up there.

3. Results

3.1.1 - Posts and Comments over time

This graph shows the a rolling 21 day mean of total posts and comments made in 2023, indexed to 1.0 at the start,[1] so be aware it is a lagging indicator. 

Both types of engagment show a decline over the course of the year, though the beginning of 2023 was when the Community was still reeling from the FTX scandal, and the Forum seemed to be the primary online place to discuss this. This was causing so much discussion that the Forum team decided to move Community discussions off the front page, so while I've indexed to 1.0 at the beginning for the graph, it's worth noting that January/February 2023 were very unusual times for the Forum.

There is also a different story to be told for the individual engagement types. Posts seem to drop from the beginning of the year, tick up in the spring (due to April Fools'), and then drop away towards the end of the year. Comments, on the other hand, rapidly drop away, presumably as a result of engagement burning out after the FTX-Bostrom-Doing EA Better-FLI-Sexual Harrassment-OCB perfect storm. They then settle to some sort of baseline around May, and then pick up again sometimes in spurts due to highly-engaging posts. I think the September-October one is due to the Nonlinear controversy, the December Spike is the response from Tracing and Nonlinear themselves. There didn't seem to be any clear candidate for the spikes in the Summer though.[2]

This is just an overview, I have more topic results to share in 3.2

(To be specific, this is simply a count of how many times a topic tag appeared on a post in 2023, or how often a comment was made on a post with that tag. The vast majority of posts usually have multiple tags, and a small group actually have none.)

RankingMost PostsMost CommentsTotal Karma (Posts)
1AI safetyCommunity Building effective altruism
2Building effective altruismBuilding effective altruismCommunity 
3Community AI safetyAI safety
4Existential riskCause prioritization Cause prioritization 
5Cause prioritization Criticism of work in effective altruism Existential risk 
6Global health & developmentExistential riskGlobal health & development
7Policy Criticism of effective altruism cultureEffective giving 
8Career choice Criticism of effective altruist organizationsAnimal welfare
9Effective giving Effective givingOrganization updates
10Animal welfareGlobal health & developmentPolicy

Topics which only make it on one list have been highlighted, otherwise it is not surprising that there is a significant amount of overlap between these topics. The biggest difference is the presence of explicit 'Criticism of...' topics in the Most Comments list, but otherwise there's a lot of stability in which topics are most commonly discussed on the Forum.

3.1.3 - Who were the most active users

This is just an overview, I have more user results to share in 3.3

(In this case, for both posts and comments I grouped by user and took the 10 most frequent results. I combined both DataFrames and summed by the karma to get a combined result for the third column)

RankingMost PostsMost CommentsTotal Karma (Posts & Comments)
1EA JapanJason Jason 
2Lizka Vasco GriloJeff Kaufman
3EA Italy Nathan Young Lizka 
4Vasco GriloNickLaing Nathan Young
580000_HoursJeff KaufmanLinch
6christian LinchBen_West
7Center for AI SafetyGeoffrey Miller Habryka
8Nathan YoungMichaelStJules Geoffrey Miller
9Joe_Carlsmith Greg_Colbourn titotal
10Akash titotal Peter Wildeford

This time, I've only highlighted those Users who have made it onto more than 1 list. At a brief glance, it seems users who gained high Karma seem to be those who make the most comments, rather than the most Posts. This was slightly surprising to me given my intuition that Posts have a higher score than Comments, but it seems volume of Comments outweigh this.[3]

Shout out to @Nathan Young for being in the top 10 for posts, comments, and karma in 2023 (which makes his turn away from the Forum in 2024 rather sad, but so it goes). Secondary shoutout to @titotal for being top 10 in Comments and Karma despite taking an explicitly critical stance of many EA causes/arguments.

3.2 - Topics in 2023

The primary reason I set out to do this was to look at the main topics that the Forum discussed over 2023, how that changed over time, and what this can tell us about the state and trajectory of the movement.

3.2.1 - Top 10 Topics of 2023, Rolling Post Frequency

Below is the rolling 21 day mean of proportion of posts that included a given topic, subselected to the 10 most frequent topics over the course of 2023.

I've split out the above chart into 1-5 and 6-10 (open in a new tab for a higher quality image)

We can see some major trends here:

First the 'Building Effective Altruism' and 'Community' topics were dominant at the beginning of February, and most likely continuing a trend from the previous year, and then steadily decreased over the year.

Second - AI Safety takes over as the dominant Forum topic from March onwards (likely due to the release of GPT-4 and the FLI Open Letter) and doesn't relinquish the lead. Note that it is still behind Community and Building EA in terms of Comments and Post Karma. Still, the trend in the Community toward AI Safety continues.

Third - Animal Welfare has slowly risen from around 7.9% of posts at the beginning of the trend to 13.9% at the end. For GH&D the numbers are 9.9% -> 14.7%. Whether this is a return to those topics after Community/AI posts crowded them out at the beginning of the year, or simply a boost driven by Giving Season remains to be seen.[4]

3.2.2 - Which Topics had the highest average karma

Sub-selected to only those topics which were tagged to 20 or more posts

RankingHighest Average KarmaLowest Average Karma
1Good things & impact storiesEffective altruism in Japanese
2Criticism of effective altruist organizationsTranslation
3Criticism of effective altruism cultureThreads
4EA Strategy FortnightLarge Language Models
5Criticism of work in effective altruismClimate change
6FTX collapseRationality
7Nonprofit governanceDecision theory
8Open PhilanthropyNewsletters
9Community experiencesMetaculus
10Data (EA Community)Volunteering

This table, however, more measures 'Engagement' rather than 'Reception'. High Karma posts might be ones that generate a lot more engagement rather than universal acclaim, and Low Karma posts are more likely to have no captured Community interest as much as other topics.

3.2.3 - Risers and Fallers in Topic Popularity

This table was calculated by ranking the Topics by frequency, grouped by January-June and then July-December. Taking the difference between the two lists showed which Topics were highly prominent early in the year and then fell away, along with which ones emerged later on in the year. Top 100 topics only.

RankingMore Active First 6 MonthsMore Active Last 6 Months
1Personal developmentOpinion
2Diversity and inclusionGiving Season (2023)
3Criticism of work in effective altruismAnnouncements and updates
4AI forecastingEffective altruism in Japanese
5Community infrastructureResearch
6Research summaryOpportunities to take action
7Criticism of effective altruismMental health (cause area)
8Less-discussed causesAudio
9

Self-care and wellbeing in 

the effective altruism community

Job listing (open)
10Effective Altruism ForumVideo

For the first column of topics which fell away, I think that these correspond to the Community dramas of early 2023 (see 'Diversity and inclusion' and 'Criticism of effective altruism'). For the second column, I think we see the broadening of content types on the Forum ('Opinion', 'Audio', 'Video').

3.3 - Users in 2023

3.3.1 - Engagement across Users

Forum use is quite unequal between users. In fact, it is so unequal that to make a useable plot below, I had to convert number of Posts and Comments to a log scale:

In all cases, however, there are more users making comments than posts. This 'power-law' pattern in Forum behaviour appeared in mutliple different plots, though I've focused on the User results here. Still, at a first guess, if you had to describe how the EA Forum worked, a power law of engagement is a good first approximation.

3.3.2 - Engagement and Karma

This trend continues when we add Karma into the mix. Most Forum uses engage a small number of times, and accumulate a small amount of Karma. Some, however, post extensively and gain lots of karma as well:

However, the average Karma story is more interesting, as it shows that even as a user keeps engaging, the trend of average Karma decays to sit between between 10 and 20.[5]

Shout out to @Rockwell for an exceedingly high 47.65 average Karma despite 75 combined posts and comments over the year, you can see her dot breaking the trendline just before the '100' mark on the chart above!

3.3.3 - Risers and Fallers in User Engagement

Finally, I wanted to look at who posted more in the first 6 months compared to the last 6 (all within the 100 most-active accounts), similar to the analysis done for Topics in 3.2.3, to see which high volume accounts left the Forum over the year and which have started recently.

RankingMore Active First 6 MonthsMore Active Last 6 Months
1Benjamin_ToddSummaryBot
2Brian_TomasikEA Japan
3Wil PerkinsSiebeRozendal
4DirectedEvolutionJames Herbert 
5Michael_PJcalebp
6MaxRaOmnizoid
7zchaungKat Woods
8peterhartreeHolly_Elmore 
9Ivy MazzolaJackM 
10ChanaMessinger Rebecca 

Additionally, I had some code to look at user behaviour and find 'similar users' for the top candidates, but this had the unintentional side effect of showing which accounts or most similar to Anonymous accounts - so I've decided not to share this for now. (Similar concerns noted by @Jeff Kaufman here).

3.4 - Miscellaneous Findings

3.4.1 - The most 'x'

Given I had the data which gave me the score (proxy for well received), total votes (weak proxy for controversial), and wordcount (very weak proxy for effort), I thought 'why not'. So,[6] here are some posts and comments to note of 2023:

3.4.2 - Your queries

If you've read it this far, I feel like you found this post interesting. In which case, you probably have questions of your own! If you have a quick query I can put to the DataFrames, then put it in a comment below and I will see what comes up.

4. Interpretations and Takeaways

It's been a lot already! So, to focus on what's important, I'll make 2 points on interpretation and 2 on my personal takeaways.

4.1.1 - Be careful with interpretation

  1. The EA Forum is not the movement, and is only one slice of how it organises itself and connects to world. I know of various EAs who are influential, impressive, and doing good who hardly post on the Forum at all (though they might read it). Do not generalise from my spare-time-data-analysis to the state of EA-as-a-whole
  2. Even on the Forum, there's probably a slight 'bikeshedding' effect. It's not easy to give a worked out opinion on what exactly the value of Cohen's d for a recent RCT in nutritional interventions in the third world, for instance. It's very easy to shoot off a scorching hot take about AI from first principles. A Forum naturally trends towards engagement, not impact on the world.

4.1.2 - My takeaways

  1. Is the downward trend in the Forum temporary, or permanent? I noticed this in the last analysis I did, and it seems to have continued across 2023. Of course, the lead up to FTX was a period of rapid growth, and the fallout a period of retrenchment for EA. Has Community engagement stabilised, or is there still evaporative cooling?
  2. AI Safety is the leading current cause, and that makes me feel uncomfortable. I've never bought the simple criticism of "EA 1.0 = Bednets = Good. EA 2.0 = AI = Bad." Nevertheless, I feel like the turn towards AI Safety as overwhelmingly important[7] is a risky one for the future of EA, and I wish that the community's discussions and engagements with the wider world were more varied.

5. Future Work

As always, the planning fallacy hit me with this post. I had the idea but kept procrastinating, and then when I finally ran the initial data scrape I didn't dive into the details, and then when I did each question simply unlock many more to explore! Here are some of the main ideas that could be pursued next:

  • Actually look at the content people post rather than the titles and topics. This will require more in-depth NLP methods to extract signal from that noise.
  • More sophisticated network analysis and/or visualisations of users/topics to see if there really are clearly differentiable EA 'schools of thought' in the data.
  • Look for dates where there are the biggest discontinuities in trends, or changes in directions of trends, and then see what events in the real world seemed to have those effects on the Forum.
  • Track Forum data throughout 2024 to see if it is recovering a lower base, or is continuing its decrease after the frenzy of late 2022/early 2023.

Thanks for reading this post, and I appreciate any feedback or discussion on any of the points I made. If you want to collaborate on any similar future work, then please reach out, I'd love to hear from you :) 

  1. ^

    Hence the x-axis starts at January 21st, and the value for January 21st is indexed at 1.0 for both Posts and Comments

  2. ^

    If you have potential candidates, let me know

  3. ^

    To be precise, the average post has 4.55x more Karma than the average comment. On the other hand, there were 6.94x more Comments than Posts in 2023

  4. ^

    But should be answerable, with 2024 data available

  5. ^

    Note this is counting 2023 only, so (Total Karma Earned in 2023)/(Total Forum Engagements in 2023)

  6. ^

    (at the time of my download and subject to any data-cleaning mistakes I made)

  7. ^

    I think one can make a case for it to be important, but being overwhelmingly the top cause seems a suspect claim, at least to me

Comments8
Sorted by Click to highlight new comments since:

This seems like an interesting data dive, thanks for posting! :)

As an FYI to readers, you can find some high level metrics on the Forum (e.g. hours of engagement, monthly active users) live on the CEA dashboard.

Executive summary: An analysis of EA Forum trends in 2023 shows a decline in engagement, a shift towards AI safety as the dominant topic, and highly unequal participation among users.

Key points:

  1. Posts and comments declined over 2023, with comments dropping rapidly after the FTX scandal and other controversies.
  2. AI safety became the most frequent topic from March onward, while animal welfare and global health & development saw modest increases.
  3. Forum engagement follows a power-law distribution, with a small number of highly active users and many less active ones.
  4. Average karma per user tends to stabilize between 10-20 despite increasing engagement.
  5. The analysis has limitations and should not be overgeneralized to represent the entire EA movement.
  6. The shift towards AI safety as the dominant cause is a potentially risky trend for the future of EA.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

I'm puzzled that in the footnote, you say that there were 7x more comments than posts, but in the graph of 3.1.1, there are more posts than comments on almost every day? And the number of comments is never more than 2x the number of posts in the same day. How does that add up to be 7x more?

I'm assuming the posts and comments were both (separately) indexed to 1 at the start of the graph.

Ahh, that makes sense.

Great stuff. I'm glad you looked into the data, and posted it publicly. Congrats on getting the draft over the line 🙂. I'm glad our public API was able to help, sorry about the lack of documentation. (Oh, you know about our open source repo though?) I want to check on one point that it behaved correctly: did the site redirect you to the bot site?

Oh, random thought: I think there are sometimes effects where the late holiday seasons often sees a dip like you observe. Maybe there's a seasonal adjustment you could do?

(If you did, you would be going beyond what we do internally though, so, like, not a criticism.)

Thanks JP! No worries about the documentation, I think the main features and what they correspond to on the Forum are fairly easy to interpret.

As for the redirection, I actually queried the bot site directly after seeing it you had one for that purpose, so I haven't actually tested the redirection!

Interesting on the holiday seasonality, it'd be interesting to see and I might look into it. My expectation given the top-level data is that extraneous community events are what bump engagement more, but I could be wrong.

Curated and popular this week
Relevant opportunities