For the past several years, CEA has awarded a regular prize to authors of Forum posts (selected by a group of 4-6 judges, one of whom works for CEA). The frequency has changed somewhat over the years, but has generally been a monthly prize given to 3-5 authors. Our hope was that this prize would encourage the creation of more content of the sort we want to see on the Forum. For more information about the prize, you can see our most recent post.
Perhaps the most plausible way in which this could happen is that the authors of prize-winning posts are incentivized to post more frequently. We therefore examined whether prize-winning authors post more frequently in the six months following their prize than in the six months prior to it, relative to a control group.
We found that there was no increase in frequency. This result, combined with other qualitative information we've heard from Forum users, has caused us to reconsider the Forum Prize process. In September we launched a creative writing contest, separate from the usual prize, and our initial impression is that this has generated much more engagement.
More on that qualitative feedback: While people generally react quite positively to winning the prize, few people have explicitly told us it made them want to write more (even when we asked directly), and our surveys of the Forum's userbase haven’t found many people saying that the chance of winning a prize leads them to invest more time in writing.
Going forward, we no longer plan to run the “standard” Forum Prize process. Instead, we expect to experiment with different ways of incentivizing writing. While nothing is set in stone, examples of things we’ve considered include prizes specifically for new authors and contests for the best posts on particular topics.
(We'll also make this announcement in the upcoming final Forum Prize post, so that it will be more visible.)
Below is more information on our methodology. We expect this to only be of interest to a small group of people, and have therefore left the post unpolished. Please feel free to comment with any questions or clarifications.
Method
Test set
A test set with one record for each Forum Prize awarded
- Get the publication dates of all posts for every user that appears in this set
- Compare each user’s mean post interval in the 6 months before the prize’s announcement date with that user’s mean post interval in the 6 months after the announcement
- Each prize appears as a separate data point (i.e. repeat winners have multiple data points)
Control set
A control set with one row for all posts with >50 karma, excluding authors who won any Forum Prize at any time
- Get the publication dates of all posts for every user that appears in this set
- Pretend that each post won an imaginary prize and allocate a corresponding “announcement date” depending on when the post was published (e.g. a post written in April 2021 has an announcement date of 29th July 2021)
- Compare each user’s mean post interval in the 6 months before the imaginary prize announcement date with the user’s mean post interval in the 6 months after the imaginary announcement
- Each >50 karma post appears as a separate datapoint (i.e. “repeat winners” have multiple data points)
Results
The ratio [mean posting interval after] / [mean posting interval before] is shown. If there was an effect, we should expect the top two box and whisker plots to be to the left of the bottom two plots.
https://chart-studio.plotly.com/~cea_a_kara/1.embed
Here is the same data, but looking at the average change in terms of number of days (i.e. subtracting the after versus before frequency, as opposed to dividing them). Again, an effect would look like the top plots being to the left of the bottom ones.
I'm surprised this seemed the most plausible mechanism. Surely the incentive should have occurred prior to winning the prize? For my own case, I observed the existence of the prize, which encouraged me to put more work into making my post better, and the winning came later, presumably in part due to this extra effort. Is your idea that winning signals that you are high enough quality to be able to win, and hence its worth trying again?
In fact if winners suspected the Judges would be averse to letting them win 'too often' out of some egalitarian sentiment the effect might go in the opposite direction (though I think this would be very small, and I don't think I used this as a judging criteria).
A year ago I suggested the use of randomization to estimate the causal effects of the prize on the parameters of interest. In general, I'm surprised that the EA community (including EA orgs such as CFAR or 80k) doesn't use randomization more. One of Jeff Kaufman's policy proposals is to "randomize everything", and I think this should be an EA policy as much as a government policy.
Thanks Larks! I somewhat regularly encounter people who are hesitant to post on the forum, and can't recall a time when telling them about the existence of the prize made them seem more likely to post. I can, however, think of people who have told me that they were more willing to post after having received the prize or other recognition for their work.
My guess is that something like imposter syndrome is more of a barrier to people posting than money is.
Sad to see this go
Thanks for this and the related work! Interesting to see the relatively small/insignificant effect on post frequency. I don't want to spend much time commenting here so I'll be very concise with my quick thoughts. They are just for the sake of feedback mainly so don't take them too seriously!
I think this was a relatively valuable part of the Prize as well. However, the weekly Forum Digest now gets about 700 opens/week, which is well over the total number of views the average Forum Prize post received. If people want to track good content, the Digest is a good way to do that. (It's much less selective than the Prize, but still seems like a superior form of curation overall, given how many good posts didn't receive prizes.)
However, the Mailchimp archive only goes back 20 issues or so — I've made a note to put up a page displaying all past Digests at some point, to make the curation more complete.
They also give a signal to authors that content is valuable.
Previously I think I responded that it did not motivate me to write more/better, but in retrospect I think this is just false. At least to me, it feels very arrogant to be hopeful that I could win a prize and therefore encourages dishonesty with myself. I expect this to be similar for other people.
For contrast, not the case for me.
I agree with other commenters who have pointed out that using "more posts by previous prize-winning authors" as a proxy for the stated goal of "the creation of more content of the sort we want to see on the Forum" seems like a strange way to evaluate the efficacy of the Forum Prize. In addition to the points already mentioned, I would add two more:
Indeed, I would argue that the prize adjudication process itself offers a useful infrastructure for evaluating the Forum experience. Since you have a record of the scores that posts received each month as well as the qualitative opinions of longtime judges, you have the tools you need to assess in a semi-rigorous way whether the quality of the top posts has increased or decreased over time.
I also wanted to express that if CEA really is ceasing the Forum Prize as such, that seems like a fairly major decision that should get its own top-level post, as the prize announcements themselves do. As it is, it's buried in an article whose title poses what I think most people would consider to be a pretty esoteric research question, so I expect that a lot of people will miss it.
I'd also bet that it's not zero, but my experience interviewing dozens of users and surveying hundreds more suggests that the number is not very high. (I focus my interviews on people we'd especially like to see submit more content, so the Prize being relatively unimportant to them seems quite informative in that light.)
There will be another announcement in the upcoming Forum Prize post, which I expect to be read by more people. This post was ready before the other post, but it seemed a bit odd not to mention the decision to stop the Prize here, given the subject matter. (I'm about to note this in the above post, to clear up potential confusion.)
Given that the number of votes remains basically fixed over time, the "record of the scores" wouldn't reveal any change in quality — we don't use a scale, just approval voting.
The judges are definitely the sorts of people I'd want to hear from w/r/t the perceived quality of top posts over time, but they are also the sorts of people who read the Forum a lot anyway (that's why I asked them to judge), so I expect I'll be able to get their opinion on this kind of thing pretty easily with or without the prize.
***
Overall, I also want to reiterate that the Forum Prize closing doesn't mean we won't be trying to reward quality posts, have judges evaluate posts, etc. — this will just be happening in other forms (using the budget we've freed up by not running the Prize). So I'm hoping that many of the benefits we looked for will be present in greater quantity with our new projects!
Some things I liked about the old Forum Prize:
It doesn't need to have money attached, but I hope a regular way to emphasize good work of all sorts (beyond karma, and more selective than inclusion in the forum newsletter) reappears, for the sake of both authors and readers. As a new writer (this is hypothetical; I haven't yet written any polished posts), winning would be very valuable for me (and while it probably wouldn't really affect my writing much, the existence of the forum prize would make me more excited about writing polished posts). As a reader, a regular prize has helped me find excellent content outside of the stuff I normally read.
See also the discussion under How should we run the EA Forum Prize?
Another vote against this being a wise metric, here. Anecdotally, while writing my last post when (I thought) the prize was still running, I felt both a) incentivised to ensure the quality was as high as I could make it and b) less likely to actually post as a consequence (writing higher quality takes longer).
And that matches what I'd like to see on the forum - better signal to noise ratio, which can be achieved both by increasing the average quality of posts and by decreasing the number of marginal posts.