I'd be interested to see more on how accuracy on Manifold changes with the number of traders and overall trading volume.
In my anecdotal experience, Manifold accuracy improves a lot with more trading, just as you'd expect. Some of the markets in this dataset probably only ever got single-digit number of trades and were obviously mispriced (with nobody seeing them to correct the mispricing). I occasionally see this happen on Metaculus too, but much less often in my experience - I would guess this is in large part because the question set on Metaculus is highly curated and much smaller.
I'd be interested in what the analysis looks like if restricted to questions with at least a certain number of forecasts or forecasters.
The curve for Manifold looks more spiky and less smooth. I expect this to be largely a function of the number of forecasters and the trading volume. To me, the spikes mostly look like noise.
Yeah, the narrow spikes on Manifold are mostly noise due to inexperienced traders not understanding liquidity and price slippage, which are quickly corrected. The platform has changed a lot over the last year, liquidity was generally improved, so I think the amount of those noise spikes has decreased. I'd be curious if you ran the comparison on earlier vs later questions whether we'd see a significant difference in relative performance (although it would be hard to distinguish that from random noise from looking at different questions in a different time period).
it looks like on the set of questions I analyzed Metaculus forecasts tended to update faster.
This is a very interesting observation, and I think it's largely coming from the Metaculus having more predictors update their predictions over time than on Manifold, on this particular question set. Prediction markets that are well-traded should update much faster than Metaculus because there is a large profit incentive for being the first to update the market price with new information, which typically happens within minutes on the most popular markets, whereas Metaculus predictions just reward an update as much as it affects your time-averaged score. Metaculus's recency weighting works fairly well at updating quickly, but we're usually talking about days, not minutes.
Thanks for the post, great analysis!
In my anecdotal experience, Manifold accuracy improves a lot with more trading, just as you'd expect. Some of the markets in this dataset probably only ever got single-digit number of trades and were obviously mispriced (with nobody seeing them to correct the mispricing). I occasionally see this happen on Metaculus too, but much less often in my experience - I would guess this is in large part because the question set on Metaculus is highly curated and much smaller.
I'd be interested in what the analysis looks like if restricted to questions with at least a certain number of forecasts or forecasters.
Yeah, the narrow spikes on Manifold are mostly noise due to inexperienced traders not understanding liquidity and price slippage, which are quickly corrected. The platform has changed a lot over the last year, liquidity was generally improved, so I think the amount of those noise spikes has decreased. I'd be curious if you ran the comparison on earlier vs later questions whether we'd see a significant difference in relative performance (although it would be hard to distinguish that from random noise from looking at different questions in a different time period).
This is a very interesting observation, and I think it's largely coming from the Metaculus having more predictors update their predictions over time than on Manifold, on this particular question set. Prediction markets that are well-traded should update much faster than Metaculus because there is a large profit incentive for being the first to update the market price with new information, which typically happens within minutes on the most popular markets, whereas Metaculus predictions just reward an update as much as it affects your time-averaged score. Metaculus's recency weighting works fairly well at updating quickly, but we're usually talking about days, not minutes.