In this post I describe a hypothetical single site that aggregates a given forecaster’s performance across multiple public platforms into a single place. I detail what I expect the technical hurdles to be in implementing such a site. I justify these costs by arguing that this would help foster an efficient culture of learning within the forecasting community that would accelerate the rate of improvement of forecasters and their techniques. I also detail the current performance metrics displayed on four popular platforms and find them to be vastly different.
Imagine a minimum viable "Linkedin" for forecasting. At a glance, a profile there could answer questions like…
There are currently multiple popular forecasting platforms, and they all surface answers to some subset of the above questions in their own way. A user may not even have the same username across multiple platforms, so I believe the current state of the art is for a prolific forecaster to either link to all of their independent profiles from their own personal site/blog or to summarize their results themselves and manually update them.
A single site could allow a user to link their various accounts and concisely display their performance across platforms such that it can be continually updated, either via API access or web scraping.
I expect there to be some barriers to implementing this that I’m not qualified to fully assess:
Obviously such a site would cost resources to create and maintain. These metrics are already being collected, so why should we care about aggregating them?
I view rigorous judgmental forecasting as a recently developed art. We know, first from academic research and now from these online platforms, that usefully impressive human performance is possible. We even know some predictors of it and some common attributes of accurate forecasts and forecasters.
But how much better can forecasters get? What are the limits of human judgmental forecasting performance and what techniques get us there? Which domains or question types are most amenable to what techniques? These questions can be answered by academia rigorously, but at what feels to me like a glacial pace. Athletes and coaches don’t wait for double blind studies to confirm which strategies or equipment they should use. Instead, they live in a culture of continual experimentation where they’re surrounded by evidence to evaluate and infer causes from. Critically, they can evaluate which teams and players are top performing thanks to clear scoring and statistics.
I believe that lowering the barrier to accessing a forecasters performance metrics (along with other interventions that I will continue to describe on this blog) can help foster a similar culture within the forecasting world, where individuals and teams can better learn from each other. When someone shares a resource on a topic or advice on how to structure a forecast, being able to evaluate their track record at a glance minimizes the friction to weighing the value of that information as you consume it. Making performance more public also increases the incentive to perform well, and explore the limits of current techniques.
With the advent of open, online forecasting platforms, I strongly believe that the most powerful lever on advancing the art of forecasting is to foster communities of open experimentation and collaboration, and making performance as clear and accessible as possible seems fundamental to this. By aggregating these metrics in one place, you also open the door to have a new API that makes surfacing the most salient information in other locations (like Discord servers or on forums) much easier.
All of this depends on what metrics are being surfaced by open forecasting platforms in the first place. In writing this post I was surprised that the current state is massively varied. I expect this to be a subject of continual improvement for these platforms, but I wanted to capture the current state here for posterity. Even now, I believe the kind of aggregation would have strong benefits, and the diversity between the scoring metrics may even make this case stronger.
I only list the metrics/information relevant to forecasting directly, not question writing. I’ve also omitted visual badges/achievements as they represent things already captured in the other scores, but these are typical across the platforms. This list is accurate as of April 1st, 2023.
I like this idea :-)
I think that there are some tricky questions about comparing across different forecasters and their predictions. If you simply take Brier score, this can be Goodharted: people can choose the "easiest" questions and get way better scores than the ones taking on difficult questions.
I can think of some attempts to go at this:
I agree with your concerns on using a pure Brier score with open platforms. I expect that currently it makes the most sense within "tournaments" where participants are answering every question. Technically, I think some sort of objective, proper scoring rule is a prerequisite to a more advanced scoring system that conveys more useful information in open contexts.
I've seen some sort of a "relative Brier score" referenced frequently in associated research (definitely in the good judgement project papers, at a minimum) that scored forecasters based on the difficulty of each question, as determined by the performance of others who forecasted it. This seems promising, and I expect there are a lot of options in that direction.