Questions

The current karma system of EA Forum, which is described here^[1], is based on the assumption that karma is positively correlated with impact (of the post/comment). I think this is pretty reasonable, but how strong is the correlation?

I am also curious about the shape of the relationship between karma and impact. For example, if post Y has twice as much karma as post X, is Y roughly 2 times as valuable as X (linear function), less than than (e.g. logarithmic), or more than that (e.g. quadratic)?

Moreover, what is the impact caused by a post with a given amount of karma? Ideally, one would want an answer in terms of the effect on the expected value of the future, but I understand this is hardly feasible. So, in practice, using heuristics may be better. Examples include donations to the Long-Term Future Fund (or other), and the metric quality adjusted research papers (QARPs) used by Nuño Sempere here.

Example answers

I tried to get a very preliminary sense of the answers to the questions above based on Nuño Sempere’s analysis of the first 10 winners of the EA Forum Prize. I estimated the mean impact in QARPs of each of the posts from the mean between the lower and upper limit of the 80 % confidence intervals provided by Nuño. The calculations are in tab “Posts” of this Sheet.

The table below contains the slope and correlation coefficient for the linear regression of the mean impact on various functions of the karma. It also has the p-value for the null hypothesis that there is no correlation. The calculations are in tab “Statistics”.

For the logarithm, square root and square of the karma, and karma alone, the correlation coefficient is 0.4, and the p-value 0.3. The differences are quite small given the sample size of 8^[2]. Interestingly, the above estimates for the correlation coefficient are similar to the 0.47 obtained by Nathan Young here for the relationship between the inflation-adjusted karma and ranking of the posts of the Decade Review.

The table below shows the correspondence between the scale described here by Nuño and the karma predicted by the linear regression of the mean impact on logarithm of karma, which has the best fit among the 4 presented above. The calculations are in tab “Predictions”.

The predictions above are quite poor. For instance, they imply:

• An unreasonably small difference of:
  • 0.367 karma between the impact of “a good blog post, a particularly good comment” and “a thoughtful comment”.
  • 3.67 karma between the impact of “an excellent blog post” and “a good blog post, a particularly good comment”.
• “A particularly valuable paper” is worth 475 karma, whereas I think the right value is of the order of magnitude of 10 kkarma (with huge variation), i.e. 21.1 times as large.

In any case, one should certainly be mindful of Goodhart's Law, and do not start optimising posts just for karma!

1. ^{^}

   For interesting discussions of the system, see this post from Arepo, and this one from Nathan Young.

2. ^{^}

    Nuño evaluated 10 posts, but only scored 8 in terms of QARPs.