I think the total karma across posts would be a better metric than the number of posts, and ones with more than 50 karma. It would combine information about quantity and quality, account for the value of posts with less than 51 karma, and account for varying quality among posts with more than 50 karma. If you believe the value of posts increases more (less) than linearly with karma, you can estimate the sum of karma^alpha across posts for alpha higher (lower) than 1. You can also include the value of comments in the same way, estimating it from the sum of k*karma^beta across comments, where k determines the value of comments compared to that of posts, and beta determine how the value of comments increases with karma (maybe it should simply be equal to alpha).
[...]
Sarah estimates that we raised between $74K and $85K for the charities that took part in our election, via platforming them (i.e. without counting the election money).
[...]
Last year we estimated costs at around $30-35K (most staff time). I’d expect this year was similar, if a little less (there were some process improvements from last year, and we benefited from experimentation on what works).
I estimate you raised 93.7 k$ (= (14.2 + 79.5)*10^3) during the last giving season, 14.2 k$ directly, and 79.5 k$ (= (74 + 85)*10^3) indirectly. For a cost equal to the lower bound of the giving season of 2023 of 30 k$, your multiplier would be 3.12 (= 93.7*10^3/(30*10^3)) assuming a counterfactual of no impact. So the money you influenced had to be more than 32.1 % (= 1/3.12) as cost-effective as the counterfactual for the giving season to have counterfactually increased impact. I guess the counterfactual would mainly be making worse donations, not donating less.
Interesting point r.e. karma. I do think it increases somewhat non-linearly, i.e. a post above 50 karma is one I am very likely to include in the digest, and I think that people do vote based on some sort of fixed deservingness number for karma. I.e. a post can hit 60 karma very quickly from the first day's voters, and then be read by many others who don't vote much at all (because they think the post is worth about 60 objectively). Maybe I'm wrong though, I should check if karma totals anchor around particular numbers or not.
Makes sense. I also think tracking the monthly value of posts and comments along the lines I suggested outside events would be useful. CEA's dashboard has the number of posts with at least 2 upvotes excluding self-votes, but this accounts very little for quality (nitpick; I would account for self-votes, as I guess a post with no votes still has some value).
Executive summary: This reflective memo reviews a year and a half of EA Forum event experiments—including Draft Amnesty Weeks, Debate Weeks, and Giving Seasons—finding that lightweight initiatives like Draft Amnesty are cost-effective for surfacing new content, while higher-effort events like Debate Weeks can successfully stimulate discourse but require careful framing and coordination; future iterations will likely double down on these learnings while adjusting based on engagement and quality tradeoffs.
Key points:
Draft Amnesty Weeks are low-cost and effective at generating Forum posts, encouraging new authors, and increasing engagement modestly; the 2025 event outperformed 2024, and the author plans to experiment with running them twice yearly.
Debate Weeks successfully foster deep discussion, especially when well-framed and accompanied by features like homepage slider polls and symposiums; however, selecting and wording debate topics remains challenging and time-intensive.
Animal Welfare vs Global Health Debate Week was the most successful across metrics—posts, karma, comments, and engagement—highlighting the potential of controversial but well-chosen topics, though some feedback flagged the framing as too combative.
Giving Season 2024 increased content volume significantly over 2023 (111 vs. 63 posts), but had lower total engagement and raised less in direct donations ($15K vs. $30K); however, the new ranked-choice Donation Election and public comment thread did increase discourse.
Marginal Funding Week saw strong growth, with 46 participating organizations in 2024 (up from 19), likely driven by requiring posts/comments for donation election eligibility.
The memo underscores tradeoffs between cost, content volume, and quality, advocating for iterating on successful formats (e.g. Draft Amnesty, Symposiums) while refining goals and expectations for each event type.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
I suggested focussing on karma to measure the impact of events. However, thinking more about it, I would focus more on engagement time. People spending time reading posts and comments, and writing comments suggests these were worth it even if they do not upvote them. So I would roughly assess the impact of events estimating how much additional engagement time they caused. This is less than the engagement time across all the events' posts and comments because these decrease the engagement time across other posts and comments to some extent. You can show which events were happenning on the graph below with the total hours of engagement on the EA Forum, and see if the engagement time usually goes up when there is an even (or for which type of events), as you did for posts related or not to effective giving.
I’ve been running events on the Forum for a year and a half now. How have they gone? What have I learned? What has worked? What should I do next? This post aims to start answering these questions.
Note that this doc is broad to the point of being shallow in some places. It was originally written as a memo for CEA's May team retreat, and it's only been polished up a little in places.
Feel free to ask questions about anything in the retro.
I’ve run two draft amnesty weeks, March 11 – 17 2024, and Feb 24 – March 2 2025.
Draft amnesty week aims to surface good content cheaply, develop relationships with authors, and encourage new authors to post their first pieces.
Before I ran Draft Amnesty Week, Lizka ran a successful Draft Amnesty Day (based on Catherine Low’s suggestion?) in 2022.
Comparing the events
Posts, and what went into them
The first Draft Amnesty Day (DAD) led to as many >50 karma posts as the first Draft Amnesty Week (DAW) (though it was during our biggest year for forum engagement). The second DAW (2025) was more successful in terms of the number of posts, and the amount of posts above 50 karma.
“Prompted posts” refers to the number of posts I remember asking the author to write/ post. I think I’ve missed a few — but it’s definitely interesting that though the majority of work for a DAW event is me talking to people about their posts and encouraging them to post them, the majority of posts didn’t require any particular prompting from me.
Engagement
Engagement on DAW 24 posts was 10.8% of forum engagement that week. On DAW 25, it was 13.2%.
Is this a good thing?
The aim of draft amnesty is to lead to more good posts, both during the week itself, and in the future via encouraging authors. In that sense - engagement is only tangentially related (more posts should equal more engagement- or we aren’t doing sharing right).
But I think there would also be a failure mode if the % of engagement on draft amnesty posts got too high, and the draft amnesty posts were on average lower quality than published posts (which they should be if we are doing enough to encourage people to post the rest of the time).
The cost of draft amnesty
The second draft amnesty week took a lot less time to organise than the first. The work includes:
Writing an announcement post, and a few threads.
Reaching out to authors to ask them if they have any drafts (and pitching people in person)
Running an online writing session (takes a couple hours, wasn’t a big success either time)
The second draft amnesty week (2025) was especially easy to run.
Takeaways
Draft amnesty seems to be a very cheap way to generate a bunch of posts. This year, I’d like to experiment with a second draft amnesty event. If bi-annual (twice-yearly) draft amnestys lead to the same or similar number of posts, then it is a bit of a no-brainer to run them.
Debate Weeks
Overview
The idea of polls had been floating around for a while, but the idea of debate weeks became more concrete on an online team retreat in early 2024.
The aim was to find a way to cause particular crucial discussions to happen on the EA Forum. We especially value long comment chains where people change their mind, or posts which respond to other posts — essentially things which push discourse forward. Debate weeks were hypothesised as a way to do that. Overall, they’ve been very successful.
Debate weeks, as I’ve run them, have a slider poll on the forum homepage banner where users can vote and attach a comment. As of the second debate week (Animal Welfare vs Global Health Debate Week) these comments appear in a discussion thread.
Animal Welfare vs Global Health Debate Week (tag) October 7-13 2024
Existential Choices Debate Week (tag) March 17-23 2025
There was also a debate-week-adjacent event before, AI Pause Debate, which followed a different method (no poll, just posts on a topic, like a lecture with responses). It was held in September 2023, and led to 29 posts, with 17 posts (not counting announcement) over 50 karma.
Purpose and goals
Metrics and comparisons
AIW
AWvsGH
EC
#posts
29
25
14
#posts>50karma
7
17
3
#votes
559
564
370
#comments on tagged posts
221
814
457
#engagement hours total (week and two weeks after)
3860.3
3636.652778
4156.725
#engagement hours on tag (week and two weeks after)
189.4805556
646.7416667
171.4944444
%engagement on event
4.91%
17.78%
4.13%
The most successful debate week on several metrics was Animal Welfare vs Global Health Debate Week. It had the most posts over 50 karma by far, the most votes on the banner, and the highest engagement hours on the discussion. It also led to a massive number of comments — one of the key positive outputs of all debate week events so far.
Unlike draft amnesty weeks, the percentage of engagement on the event is a positive metric for debate weeks, given that the topic was well-chosen.
Engagement hours
The three engagement hours graphs below don’t necessarily tell you much more than the table above (they are partially included because they took me bloody ages to make). One interesting point I’d highlight is that the AWvsGH engagement did lead to a spike in engagement, and is possibly responsible for overall engagement growing in the month afterwards (perhaps because of all the users, new and old, who started commenting (again) because of the event).
Feedback on debate weeks
A large portion of critical feedback during debate weeks is on the wording of the question. It’s very difficult to produce just the right question for the entire audience, with all their different understandings and assumptions, to engage in. I expect this to remain true.
Some other critical feedback:
Some people (anonymous here) were concerned that the Animal Welfare vs Global Health debate was too combative, and that it is unwise to show such a controversial part of effective altruism so publicly.
Personally, I agree that the framing could have been less combative, but I think the effective altruist views on animal welfare and its relative importance are a very important fact to note, whether you agree or disagree, and this was one reason I ran the event.
Existential Choices was a particularly challenging debate question to pin down, and I think this made discussions more difficult.
Debate weeks are more costly (in terms of my time) to run than draft amnesty. I reach out to many people beforehand to encourage them to post, and only a few of those do. Deciding on the question and framing it means consulting experts, reading relevant papers, and so on. This can take a while.
For the Existential Choices Debate Week, I organised a symposium with Will MacAskill — a timeboxed period when a group of experts would engage in a comment debate about the debate question. This was very successful, and led to 145 comments on that post alone. However, it also took a lot of organising and coordination.
Where next with debate weeks?
The core debate week feature is clearly a winner – a slider which encourages you to vote, then prompts you to comment.
The symposium for existential choices week was also a winner – it led to a lot of comments and engagement.
Topics for debate weeks are difficult — I’d like to focus on topics that people could change their mind on, even if it is at the expense of some engagement.
Giving Season & Donation Election
Overview
I’m going to be comparing Giving Season 2023 and 2024 because these are the most direct comparisons (and I was here for both, and project managed 2024).
Heart banner on home page, where people can discuss their donations.
Purpose and goals
About Giving Season 2023, I previously wrote that: “The main benefits of the event were:
More donations
Better donations
Effective giving is more of a thing.”
I think this is a pretty good quick description of the goals of the event, though not necessarily in order. For example, we don’t think of fundraising as the primary goal of giving season — causing discussions that lead to better donations, or more respect and centrality for effective giving, are more important than donations which may or may not be attributable to forum activity.
The biggest change I wanted to make in 2024 was to make events, especially the donation election, more discursive. I.e, I wanted the donation election to cause conversation.
Donation Elections
The first donation election was in 2023, and we also ran one in 2024. Both involved voting for charities to win money from a prize pot. However, we made some changes for 2024:
The voting system was simplified. We didn’t do pre-voting for example.
We included a leaderboard on the banner so that people could argue for changes in voting as it happened.
We made the voting tool much simpler.
We switched to ranked voting (where you rank the candidates you’d like to vote on), rather than assigning points to candidates (as we did last year)
We asked for comments, which we then posted in a discussion thread.
Election engagement
How many voters:
2023: 341
2024: 485
How many comments/ posts/ discussions:
I haven't done a full analysis here, but it looks as though there were more comments overally during 2023's giving season than 2024's. Specifically, there was an average of 86.5 comments or quick takes per day during 2023 giving season, versus 59.5 comments per day in 2024.
However, 2024's giving season did achieve the goal of more conversation about the donation election decisions. Using the same vote -> public comment pipeline we use for debate week events, we ended up with 112 comments on a donation election discussion thread in 2024.
Marginal funding week
Marginal funding week started in 2023, and was very successful. Because of its success, we ran it again in 2024.
The biggest difference which drove more posts in 2024 was that we required a marginal funding comment or post in order for an organisation to be eligible for the donation election.
Other theme weeks, and content
At a first approximation, there was far more effective giving content in the 2024 event than in the 2023.
2024’s Giving Season tag has 111 posts on it, while 2023’s has 63.
Our donation pot for the election raised $30K in ‘23, and only around $15K in ‘24.
Below is the data for 2024 - I don't currently have access to the 2023 data. I know that in 2023 there were a few more large donors, one of which gave at least $5K.
2024
Not bad, but much less than last year.
This shows up even more in the count by day chart (also in the hex), but people were more willing to donate later in the game, i.e. when they already know where the money is going.
Most donors were relatively small donors.
Raising money indirectly
Sarah estimates that we raised between $74K and $85K for the charities that took part in our election, via platforming them (i.e. without counting the election money). However, we are missing some key data on some of the top charities in the election, including the EA AWF and Rethink Priorities, so we expect this is a significant underestimate.
It’s hard to compare this to 2023’s, because the model my colleague wrote for last year includes even more uncertainty (between $23,500 and $1.24M raised).
Engagement
Total engagement was much lower at the end of 2024 than at the end of 2023, so it's tricky to do apples to apples here. One key difference is that giving season seemed to cause an overall engagement boost during 2023 in a way it didn’t in 2024, despite the fact that there was far more content in 2024.
The cost of giving season
Last year we estimated costs at around $30-35K (most staff time). I’d expect this year was similar, if a little less (there were some process improvements from last year, and we benefited from experimentation on what works).
Summary –
2024 got more votes in the donation election.
However it had lower engagement, and raised less in the donation election.
Appendix/ extras
Glossary of events:
Below are the events that have happened since I started on the Forum team in November 2023.
This is a crosspost from the new Animal Welfare Alignment Newsletter by Anima International. You can subscribe on Substack if you are interested in following these efforts. Audio reading also available on Substack.
The goals of this post are to:
1. Raise a question I see as crucially important to the goal of aligning AI to animal welfare...
Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job.
My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI.
...
This is a linkpost for Request for Proposals: Research and Applied Work on Digital Minds.
I'm glad to announce a request for proposals for research and applied work on digital minds at Longview Ph...
Thanks for the post, Toby!
I think the total karma across posts would be a better metric than the number of posts, and ones with more than 50 karma. It would combine information about quantity and quality, account for the value of posts with less than 51 karma, and account for varying quality among posts with more than 50 karma. If you believe the value of posts increases more (less) than linearly with karma, you can estimate the sum of karma^alpha across posts for alpha higher (lower) than 1. You can also include the value of comments in the same way, estimating it from the sum of k*karma^beta across comments, where k determines the value of comments compared to that of posts, and beta determine how the value of comments increases with karma (maybe it should simply be equal to alpha).
I estimate you raised 93.7 k$ (= (14.2 + 79.5)*10^3) during the last giving season, 14.2 k$ directly, and 79.5 k$ (= (74 + 85)*10^3) indirectly. For a cost equal to the lower bound of the giving season of 2023 of 30 k$, your multiplier would be 3.12 (= 93.7*10^3/(30*10^3)) assuming a counterfactual of no impact. So the money you influenced had to be more than 32.1 % (= 1/3.12) as cost-effective as the counterfactual for the giving season to have counterfactually increased impact. I guess the counterfactual would mainly be making worse donations, not donating less.
This links to a private doc.
Interesting point r.e. karma. I do think it increases somewhat non-linearly, i.e. a post above 50 karma is one I am very likely to include in the digest, and I think that people do vote based on some sort of fixed deservingness number for karma. I.e. a post can hit 60 karma very quickly from the first day's voters, and then be read by many others who don't vote much at all (because they think the post is worth about 60 objectively). Maybe I'm wrong though, I should check if karma totals anchor around particular numbers or not.
Also, whoops- should have linked to this
Makes sense. I also think tracking the monthly value of posts and comments along the lines I suggested outside events would be useful. CEA's dashboard has the number of posts with at least 2 upvotes excluding self-votes, but this accounts very little for quality (nitpick; I would account for self-votes, as I guess a post with no votes still has some value).