Hide table of contents

Comment Permalink

I did some quick, far from scientific analytics work on this to get a feeling for the additional room for participation.

First, I wanted to identify a series of posts to examine. I wanted a consecutive series so that there would not be any introduced bias. I wanted the series to be recent (to reflect recent activity) but not within a couple weeks, as I wanted it to reflect the lifecycle of a post. I also wanted the series to have what seemed like a representative distribution of upvotes.
I selected the 5 post series from Kuhn's post about GiveWell's 2015 recommendations to Ben Todd's post on WALYs.
I ran some analytics. I do not know if there is a norm that EA Forum traffic numbers are not released, so instead I'll just report a single statistic that seems most relevant.

In these 5 posts, on average 4.17% of unique visitors to the posts voted (I was sure to include both up and down votes). The range was from 1% to 6% of unique visitors to a post voting.

Of course, this only tells us a bit. If we consider the goal to be to determine what percent of visitors with meaningful opinions voted, we could say this number is inflated, since it only considers those that visited a post, and some people may know with confidence that they want to upvote or downvote based on the title on the front-page alone. This number could be an underestimate, because unique visitors is not a 100% reliable statistic, failing, for example, to adjust for the same visitor reading a post on two different devices.

Besides considering whether it is an over or underestimate, it still lacks context, and I do not believe context is going to be truly available. I did find a reddit post that has a slightly interesting discussion of possible voting behavior on that site: http://bit.ly/1OJq25b

Personally, I am of the opinion that voting behavior, percentage-wise is currently OK compared to other sites, but that this largely accounts for an engaged readership, with the other sites I'm referencing not being highly comparable. I think it is highly likely that our voting should well exceed that of other sites. And I think it is optimal to capture as much feedback as possible, so I would have a leaning toward implementation almost regardless of the current participation data.

See in context

[Meta] Up and down voting should be added next to the circles with the scores (to the left of this title), like on Reddit

by Josh Jacobson

Dec 6 20151 min read 15

5

Frontpage

[Meta] Up and down voting should be added next to the circles with the scores (to the left of this title), like on Reddit

Situation:

Why This is a Problem:

Proposed Solution:

15 comments

Situation:

Posts of drastically different quality often receive nearly identical numbers of upvotes.
The majority of posts receive scores in the single digits.

Why This is a Problem:

Voting exists to serve as a feedback mechanism. When it is only being used to a limited extent, it fails to serve this purpose. Relative vote totals are meant to serve as a filtering mechanism for the reader, and a feedback mechanism for the posters. I hate to see such low vote total for really excellent posts, as those deserve greater recognition of effort / accomplishment.

Proposed Solution:

Allow for up and down voting next to the circles that have post scores, on a trial basis. This involves adding it in two places, the front-page and at the top of each post.
This is likely to be successful at increasing voting activity, simply by making voting more accessible.

Concerns and Determining Success/Fail:

I assume voting was implemented in the way it is currently in order to ensure posts weren't up and downvoted simply based on their headline. This was sound rationale, but I believe it has not succeeded in practice.
This experiment will have succeeded if voting activity increases, and it is generally felt that voting more accurately reflects post quality. If this leads to more clickbaity posts, or quality longer posts not being rewarded in the voting, this will have failed and should be reversed.

5 Reactions

Comments15

Sorted by

New & upvoted

Click to highlight new comments since: Today at 10:10 AM

Peter WildefordDec 6 20156

I assume voting was implemented in the way it is currently in order to ensure posts weren't up and downvoted simply based on their headline. This was sound rationale, but I believe it has not succeeded in practice.

This was the rationale, so I'm curious why you don't believe it has succeeded in practice.

Josh JacobsonDec 7 20158

TL;DR: It is super rare to read to the end of anything online, and many people get value and can make judgments based on partial reads.

I should have added more logic there.

I believe it is often both practical and useful to vote on an article before reading the whole thing. For example, I already strongly believed I'd upvote Sean's announcement upon reading the headline... it's a big deal, and to me, clearly positive. Depending on my time availability, I may have wanted to upvote it then, and not read his announcement at all. This would have had what I believe would have been positive signaling and feedback effects, as it is still an informed decision of support, based on the context I have knowing him, his work, and the needs in the space.

If I had more time, I may have chosen to read some, and as I scanned the article, I may have been satisfied upon seeing that the funding will create 10 new postdoc positions in X-risk. I would have both satisfied my curiosity and been confident in the knowledge that this is awesome and I want to give positive feedback. At this point, it is more practical as a forum reader to scroll up, as that is where the links are that allow me to navigate back to the main page of the forum. To scroll down just to vote requires familiarity, recall, and particular passion of the reader.

I'll add that web analytics support that these ways of reading occur at high frequency. For example, on Slate, 50% of readership exits around the 50% mark: http://slate.me/1XNNRi8

Certainly not all, but a majority of these readers get some meaning from having visited the page, and if the content was not interesting enough for them to continue scrolling and reading, or if they feel positively about the article but get the majority of the value early on, they have feelings that are meaningful and, I believe, should be allowed to be easily expressed.

Josh JacobsonDec 7 20151

Also, I suck at formatting on this forum. Sorry for the huge TL;DR.

[anonymous]Dec 7 201510

It made for a great partial read though, enough for me to upvote the comment

Chris LeongDec 22 20150

You don't have to actually read the entire article. It is perfectly fine it you decide to upvote an article after the first paragraph or two. If it is high quality, I would suspect that enough people will upvote the article that the one person who can't be bothered clicking on it doesn't matter. Furthermore, if it isn't worth your time to click and scroll to upvote, then upvoting that article can't be particularly important.

MichaelDickensDec 6 20154

Indeed. I would expect that the current placement makes the most sense because you can vote as soon as you've finished reading the article rather than having to scroll back to the top.

Josh JacobsonDec 7 20151

I now commented on this as a direct response to Peter.

MichaelDickensDec 7 20150

Thanks for the heads up.

Tom_AshDec 6 20152

Thanks for suggesting a tweak to the forum and doing so in such a clear way. I agree that voting is relatively limited, but it'd be good to first know whether this is an inevitable result of a relatively limited audience who are interested in voting. I'd be curious to hear Ryan Carey's view as to that. If that's the primary explanation, then I don't think we can justify the time arranging for the experiment that you suggest. (It sounds relatively simple technically, but tweaks always take longer than you expect, and we've found the forum code can be surprisingly slow to work with.)

Josh JacobsonDec 7 20156

I did some quick, far from scientific analytics work on this to get a feeling for the additional room for participation.

First, I wanted to identify a series of posts to examine. I wanted a consecutive series so that there would not be any introduced bias. I wanted the series to be recent (to reflect recent activity) but not within a couple weeks, as I wanted it to reflect the lifecycle of a post. I also wanted the series to have what seemed like a representative distribution of upvotes.
I selected the 5 post series from Kuhn's post about GiveWell's 2015 recommendations to Ben Todd's post on WALYs.
I ran some analytics. I do not know if there is a norm that EA Forum traffic numbers are not released, so instead I'll just report a single statistic that seems most relevant.

In these 5 posts, on average 4.17% of unique visitors to the posts voted (I was sure to include both up and down votes). The range was from 1% to 6% of unique visitors to a post voting.

RyanCareyDec 7 20152

Putting up and down arrows on the front page would make sense if the content included lots of hugs and memes that readers are quickly opening and closing. For longer-form pieces, its better for readers to be encouraged to vote on the page in which they read it, from a UX point of view. Especially so since we place a premium on quality. We can see that the quality of the most upvoted posts is currently good. Given that we only have a few levers that influence writing quality, let's not lose this one.

I agree that more votes would be useful, but this should be readily achieved if the overall audience increases. I think that the real challenge is to roughly double readership and participation, and that people's work ought to be put to that goal. If that can be achieved, then the future of the forum would beore stable and the experience of using it will be more fulfilling.

[anonymous]Dec 6 20151

Two other ways to increase voting and help with filtering would be: 1) give users more voting points to spend on each post (e.g. 5 stars), or 2) simply tell users to vote more either by a post like this one (which has already been done!) or by including a text-prompt next to the vote buttons (although that would probably look uncomfortably commercial-like).

DavidNashDec 7 20151

What about users having 1 vote that is next to the score circle, and then 4 more when they reach the bottom of the article?

MatthewDahlhausenDec 7 20150

"I assume voting was implemented in the way it is currently in order to ensure posts weren't up and downvoted simply based on their headline. This was sound rationale, but I believe it has not succeeded in practice."

How about placing the voting at the end of each post, above the comment section? Then people have to at least scroll through a post to vote, and maybe will read something on the way.

RyanCareyDec 18 20152

That's where they are :)