Effective Altruism Forum
EA Forum

Hide table of contents
by christian
4 min read 0

13

ForecastingAnnouncements and updatesMetaculusOrganization updates
Frontpage
This is a linkpost for https://www.metaculus.com/questions/20025/new-scores-new-leaderboard-new-medals/

leaderboardsgif

We completely overhauled Metaculus’s leaderboards and ranking system, introducing an all new medals framework that better recognizes the myriad ways you can contribute on the platform.

We also built a clearer, more consistent scoring system that better rewards forecasting skill.

Introducing Medals

The Metaculus community helps strengthen human reasoning and coordination with trustworthy forecasts, cogent analysis, careful arguments, and clarifying questions. Our all new leaderboards and medals reward a wider range of contributions to this public-serving epistemic project.

Now you can earn medals for:

  • Writing insightful comments
  • Creating engaging questions
  • Placing well in tournaments
  • Making accurate forecasts

The new leaderboards page features rankings and medals across our competitive categories. (Tournament medals appear on their own respective tournament pages, however.)

staticleaderboard

In each category, within a given time frame:

  • 🥇Gold Medals are awarded to the top 1% of users
  • 🥈Silver Medals to the top 2%
  • 🥉Bronze Medals to the top 5%

Let’s take a look at the different ways you can earn medals:

Comments

The Comments category recognizes insightful commentary as measured by upvotes. It uses an h-index — like the kind that measures scientific research impact — to incentivize quantity and quality: An h-index of 10 means you made at least 10 comments, each with at least 10 upvotes.

Question Writing

The Question Writing category recognizes authorship of engaging questions. A normalized h-index ranks your output according to the number of forecasters your questions attract.

Tournament Ranking

We award medals to the top-performing tournament participants according to the breakdown described above. Every concluded tournament now features medals on its leaderboard!

Clearer, more consistent scores

Integrated with our new leaderboards and medal system is a new, more robust approach to scoring forecasts.

The leaderboards page introduces two Accuracy Leaderboards that rank forecaster performance compared to other forecasters and compared to chance.

To design our new scoring system, we went back to first principles. We wanted a system that is:

  • As intuitive and simple as possible
  • Principled and consistent
  • Able to identify and reward true forecasting skill
  • Mathematically proper so it incentivizes forecasting your true beliefs

We created the new Peer and Baseline scores and leaderboard rankings to achieve these goals.

Peer Accuracy

Your Peer score on a question measures your performance compared to other forecasters. Did you forecast more accurately? You receive a positive Peer score. Forecast less accurately than others and you receive a negative score. The average Peer score for all users on a given question is 0.

The Peer Accuracy leaderboard displays the average of your Peer scores across many questions.

Some forecasters make deeply researched predictions on relatively few questions. The Peer Accuracy leaderboard recognizes their abilities and their contributions to Community Prediction accuracy across the platform.

Click here for more on how Peer Accuracy is calculated.

Baseline Accuracy

Baseline scores measure your performance compared to chance. Your forecast accuracy on a question is compared to an impartial baseline. If your forecast is better than chance, then your Baseline score is positive. Forecast worse than chance and your Baseline score is negative.

Your Baseline Accuracy is the sum of your Baseline scores across many questions.

Some forecasters predict on a large number of questions. The Baseline Accuracy leaderboard recognizes their abilities and their contributions to Community Prediction accuracy across the platform.

Baseline Accuracy is more intuitive than legacy Metaculus Points, and it preserves one of their most appealing properties: the more accurate predictions you share on Metaculus, the higher your score will be.

Learn more about how Baseline scores are calculated here.

Can I still see legacy scores?

Legacy scores — including Metaculus Points and relative log scores — are available in the ‘Legacy scores’ section of the question page. The Metaculus Points leaderboards are accessible here. Current tournaments will continue to use the same scoring system as when they launched, but we plan to launch new tournaments with rankings built on our new scoring.

Where can I see a user’s medals?

trophycase

While building the new leaderboards, we knew we wanted to recognize the many Metaculus users who have shaped the platform over the years. We decided to retroactively award medals for all categories and time frames going all the way back to the site’s founding.

If you’re a longtime user, it’s worth checking your updated Profile page to have a look inside your trophy case!

As you browse the leaderboards, click any username to see their profile. There you’ll discover how many medals they have in each category. You can even click a particular medal to see a full breakdown of how they earned it.

contributions

Do you want to learn more about the calculations behind scoring and medals? You can find those details in the new Scoring FAQ and Medals FAQ.

Medal Time Periods

timeframe

Accuracy medals are awarded for different time periods. Longer-term forecasting questions are combined in longer-duration medals. For example:

  • Selecting Duration ‘5 years’ and Time Period ‘2021 - 2025’ ranks users according to questions that opened after January 1st, 2021, and will close by December 31st, 2025.
  • Selecting Duration ‘2 years’ and Time Period ‘2018 - 2019’ includes questions opened after January 1st, 2018, that closed by December 31st, 2019.

Comments and Question Writing medals are awarded on a simpler annual schedule.

Learn more about how Duration and Time Period work in the FAQ. Learn more about the trade-offs we considered in arriving at this solution here.

Trade-offs and future directions

Before arriving at the new system, we considered a variety of scoring mechanisms, incentives, and trade-offs. We’ve written a more detailed supporting document that shares additional thinking behind our decisions, and also includes future directions for scores and medal categories. Read it here.

We’ve been incredibly excited to share the new medals, leaderboards, and scoring system with you: They represent passionate and dedicated efforts by the entire Metaculus team. These major updates to the platform will help us celebrate a wider range of contributions to the Metaculus community and more rigorously identify top forecasting talent.

13

0
0

Reactions

0
0

More posts like this

Comments


No comments on this post yet.
Be the first to respond.
More from christian
35
Forecast With GiveWell
christian
· · 2m read
6
6
33
Metaculus Is Open Source
christian
· · 1m read
1
1
52
Announcing the AI Forecasting Benchmark Series | July 8, $120k in Prizes
christian
· · 5m read
4
4
View more
Curated and popular this week
Paul Present
 ·  · 28m read
 · 
Note: I am not a malaria expert. This is my best-faith attempt at answering a question that was bothering me, but this field is a large and complex field, and I’ve almost certainly misunderstood something somewhere along the way. Summary While the world made incredible progress in reducing malaria cases from 2000 to 2015, the past 10 years have seen malaria cases stop declining and start rising. I investigated potential reasons behind this increase through reading the existing literature and looking at publicly available data, and I identified three key factors explaining the rise: 1. Population Growth: Africa's population has increased by approximately 75% since 2000. This alone explains most of the increase in absolute case numbers, while cases per capita have remained relatively flat since 2015. 2. Stagnant Funding: After rapid growth starting in 2000, funding for malaria prevention plateaued around 2010. 3. Insecticide Resistance: Mosquitoes have become increasingly resistant to the insecticides used in bednets over the past 20 years. This has made older models of bednets less effective, although they still have some effect. Newer models of bednets developed in response to insecticide resistance are more effective but still not widely deployed.  I very crudely estimate that without any of these factors, there would be 55% fewer malaria cases in the world than what we see today. I think all three of these factors are roughly equally important in explaining the difference.  Alternative explanations like removal of PFAS, climate change, or invasive mosquito species don't appear to be major contributors.  Overall this investigation made me more convinced that bednets are an effective global health intervention.  Introduction In 2015, malaria rates were down, and EAs were celebrating. Giving What We Can posted this incredible gif showing the decrease in malaria cases across Africa since 2000: Giving What We Can said that > The reduction in malaria has be
LintzA
 ·  · 15m read
 · 
Cross-posted to Lesswrong Introduction Several developments over the past few months should cause you to re-evaluate what you are doing. These include: 1. Updates toward short timelines 2. The Trump presidency 3. The o1 (inference-time compute scaling) paradigm 4. Deepseek 5. Stargate/AI datacenter spending 6. Increased internal deployment 7. Absence of AI x-risk/safety considerations in mainstream AI discourse Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on. In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!  Implications of recent developments Updates toward short timelines There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is that 99% of fully-remote jobs will be automatable in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus[1] & Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months). We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achi
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
Recent opportunities in Forecasting
20
Hiring pre-docs
Eva
· · 1m read
0
0
79
HealthLearn: Impact Evaluation and Next Steps
2 authors
· · 12m read
1
1
32
$300 Fermi Model Competition
Ozzie Gooen
· · 2m read
8
8
View more