What stands in the way of you doing more forecasting on EA topics? If you started, what made you stop?

New Answer
Ask Related Question
New Comment

5 Answers sorted by

Scoring systems

I like to feel like additional effort makes a difference. The easiest way to get to the top of the metaculus leaderboard is to forecast on everything. Not well, but just to do it. This is pretty dispiriting.

I did Metaculus for a while but I wasn't quite sure how to assess how well I was doing and I lost interest. I know Brier score isn't the greatest metric. Just try to accumulate points?

Not being able to mark predictions as quick and slow.

Sometimes I spend a long time on a forecast (10-20 mins) sometimes I spend about 30s. It frustrates me that all the current sites don't let me track these two quite different types.

Forecasts are rather time-intensive -- I used to make a bunch of metaculus forecasts, but putting serious effort into researching something, just to come up with a number to toss into the community mix, rarely seems worth the effort these days. Instead, I have a backlog of blog posts and other projects that seem higher expected value. (But going forwards, I should try to contribute forecasts when I find myself already researching something EA-adjacent anyways.)

Related to this, unclear impact -- as Nuno judges here, it's hard for forecasts to really hit the sweet spot of being at the right difficulty level and also being decision-relevant. If I had more of an understanding of who finds forecasts useful, I might be more interested in contributing.

Overthinking forecasts, causing writing them down & tracking them diligently to be too much of a mental-overhead for me to bother with.

As you say, there's a tension between effortful vs informal forecasts: Out of humility / modesty / imposter syndrome, sometimes I don't want to mess up a pool of expert forecasts by tossing out an ill-informed hunch.

I think Metaculus could benefit from adopting the more stock-market / prediction-market style mechanism of letting players stake different amounts of their total forecasting "budget" on individual questions... prediction-market style, experienced players or players who wanted to make a very confident prediction could pour lots of points into the system. Then I would be better able to distinguish my confident from my less-confident predictions, and small numbers of experts could bet their beliefs to overturn an ill-judged majority opinion.

Although I recognize that Metaculus's current, more egalitarian style probably helps to grow the userbase and encourage participation.