# Questions

The current karma system of EA Forum, which is described __here__^{[1]}, is based on the assumption that karma is positively correlated with impact (of the post/comment). I think this is pretty reasonable, but how strong is the correlation?

I am also curious about the shape of the relationship between karma and impact. For example, if post Y has twice as much karma as post X, is Y roughly 2 times as valuable as X (__linear__ function), less than than (e.g. __logarithmic__), or more than that (e.g. __quadratic__)?

Moreover, what is the impact caused by a post with a given amount of karma? Ideally, one would want an answer in terms of the effect on the expected value of the future, but I understand this is hardly feasible. So, in practice, using heuristics may be better. Examples include donations to the __Long-Term Future Fund__ (or other), and the metric quality adjusted research papers (QARPs) used by Nuño Sempere __here__.

# Example answers

I tried to get a very preliminary sense of the answers to the questions above based on Nuño Sempere’s __analysis__ of the first 10 winners of the __EA Forum Prize__. I estimated the mean impact in QARPs of each of the posts from the mean between the lower and upper limit of the 80 % __confidence intervals__ provided by Nuño. The calculations are in tab “Posts” of __this__ Sheet.

The table below contains the slope and __correlation coefficient__ for the linear regression of the mean impact on various functions of the karma. It also has the __p-value__ for the null hypothesis that there is no correlation. The calculations are in tab “Statistics”.

Null intercept linear regression between mean impact (QARP) and... | Slope | Intercept | Correlation coefficient (R) | P-value |
---|---|---|---|---|

Logarithm of the karma | 0.244 | -1.04 | 0.445 | 0.269 |

Square root of the karma | 0.0492 | -0.407 | 0.440 | 0.276 |

Karma | 2.45 m | -0.164 | 0.433 | 0.284 |

Square of the karma | 11.8 µ | -0.0427 | 0.416 | 0.305 |

For the logarithm, square root and square of the karma, and karma alone, the correlation coefficient is 0.4, and the p-value 0.3. The differences are quite small given the sample size of 8^{[2]}. Interestingly, the above estimates for the correlation coefficient are similar to the 0.47 obtained by Nathan Young __here__ for the relationship between the inflation-adjusted karma and ranking of the posts of the __Decade Review__.

The table below shows the correspondence between the scale described __here__ by Nuño and the karma predicted by the linear regression of the mean impact on logarithm of karma, which has the best fit among the 4 presented above. The calculations are in tab “Predictions”.

Mean impact (QARP) | Description | Example | Predicted karma |
---|---|---|---|

0.1 m | “A thoughtful comment” | A thoughtful comment about the details of setting up a charity | 67.0 |

1 m | “A good blog post, a particularly good comment” | What considerations influence whether I have more influence over short or long timelines? | 67.3 |

10 m | “An excellent blog post” | Humans Who Are Not Concentrating Are Not General Intelligences | 71.0 |

100 m | “A fairly valuable paper” | Categorizing Variants of Goodhart's Law | 108 |

1 | “A particularly valuable paper” | The Vulnerable World Hypothesis | 475 |

10 to 100 | “A research agenda” | The Global Priorities Institute's Research Agenda | 4.15 k to 40.9 k |

100 to > 1000 | “A foundational popular book on a valuable topic” | Superintelligence, Thinking Fast and Slow | 40.9 k to > 408 k |

> 1000 | “A foundational research work” | Shannon’s "A Mathematical Theory of Communication" | > 408 k |

The predictions above are quite poor. For instance, they imply:

- An unreasonably small difference of:
- 0.367 karma between the impact of “a good blog post, a particularly good comment” and “a thoughtful comment”.
- 3.67 karma between the impact of “an excellent blog post” and “a good blog post, a particularly good comment”.

- “A particularly valuable paper” is worth 475 karma, whereas I think the right value is of the order of magnitude of 10 kkarma (with huge variation), i.e. 21.1 times as large.

In any case, one should certainly be mindful of __Goodhart's Law__, and do not start optimising posts just for karma!

^{^}^{^}Nuño evaluated 10 posts, but only scored 8 in terms of QARPs.

Thanks for engaging, Evan!

I would say there is a positive correlation between my subjective impression of how impactful the post is, and its worth from an impartial point of view.

Impact has implicitly been operationalised as karma in the EA Forum, to the extent... (read more)