Hide table of contents

The following is Metaculus's year-in-review CEO letter written by Gaia Dempsey.
 

2022 was a year of progress, growth, and change at Metaculus.

We matured and grew as an organization.

  • Mission Update: In the spring, we expanded the scope of our mission, creating a solid foundation for our work. Specifically, our mission is to build epistemic infrastructure that enables the global community to model, understand, predict, and navigate the world’s most important and complex challenges. We also clearly defined the three primary ways we enact our mission, namely: providing forecasts as a public service, fostering a global forecasting community, and supporting forecasting research.
  • Pro Forecaster Program: In response to a sharp increase in global uncertainty with the outbreak of the war in Ukraine, we designed new operational procedures including an infohazard review process, and recruited over 30 candidates into an exceptionally talented Pro Forecasting team.
  • Core Funding: In the summer, we secured a $5.5M core funding grant from Open Philanthropy to expand our work.
  • Becoming a Public Benefit Corporation: Shortly after, we committed to providing three specific public benefits in our charter, reflecting our mission statement, as part of the process of becoming a Public Benefit Corporation.
  • Growing Our Team: With a mandate to expand our platform, programs, and operations, we set about upgrading our hiring processes thanks to great resource recommendations from Jim Savage (a), and Aaron Hamlin (b), and got laser focused on recruiting. Over the course of the year we doubled our overall headcount, building four strong new teams: leadership, AI forecasting, engineering, and design (some engineering start dates are in early 2023).

We launched over a dozen forecasting tournaments and generated over 1000 aggregate forecasts.

  • Expansion of Forecasting Programs: Over the last year, we expanded our existing partnerships and began collaborations with a number of fantastic new partners. In total, we launched 13 new forecasting tournaments offering a total of $77,500 in prize money, with a number of programs delivering policy-relevant predictions on public health, biosecurity, and nuclear-risk. Outside of these core themes (each of which I’ll touch on later), Forecasting Our World In Data was a standout launch: a project that “probes the long-term future, delivering predictions on topics like global investment in AI, world life expectancy, CO2 emissions, and more on time horizons from one to 100 years.” Another favorite was our final launch of the year, the Sagan Tournament, focused on all things space-related — from technology, to scientific discovery, to governance.
  • Ukraine War Response: After accurately predicting the war in Ukraine, the Metaculus community continued to closely monitor the conflict, responding rapidly to new developments in a forecasting tournament launched just 48 hours after the invasion. With nuclear security expert Peter Scoblic, we deployed the Red Lines in Ukraine project as an early-warning system gauging the likelihood Russia would make use of nuclear weapons.
  • Biosecurity & Public Health: 2022 saw us increasing our impact in public health and biosecurity, executing the $25,000 Biosecurity Tournament with the Institute for Progress and Guarding Against Pandemics, as well as the Real-Time Pandemic Decision Making tournament with UVA’s Biocomplexity Institute to aid COVID computational modeling efforts. In May when clusters of Mpox virus infections were observed across multiple countries, our community responded immediately through our Mpox Series, contributing ensemble predictions and providing a crucial health tool for assessing widespread community transmission when data were sparse.

We grew our forecasting community and connected with a wider network.

We upgraded our infrastructure and shipped new features.

  • Upgraded Tech Stack: In 2022, we rewrote nearly the entire Metaculus application, modernizing the Metaculus tech stack to support our 2023 product roadmap. We have a number of enhancements and new features in store, big and small, and we can’t wait to share them with the community.
  • Question Groups & Fan Graphs: We released question groups and fan graphs, enabling the grouping of related questions and the visualization of their forecasts in series.

  • Private Forecasting Spaces & Language Localization: To support partner projects, including internationally, we developed private forecasting spaces that enable confidential forecasting for a group or organization of any size, as well as the ability to translate the Metaculus interface into any language.
  • Tournament Scoring & Leaderboards: We updated our tournament scoring and leaderboard systems to bring greater rigor and clarity to how we reward and incentivize forecasting skill.

We collaborated on research and published reports on the biggest potential risks of 2022.

  • Nuclear Escalation in Ukraine: After the invasion of Ukraine, policymakers and the public became increasingly concerned about the prospect of nuclear escalation. We recruited a team of Metaculus Pro Forecasters to make their judgments on key questions and provide their rationales, all of which were drawn up in a full nuclear risk report.
  • Predicting the Omicron BA.1 Wave: In partnership with the University of Virginia’s Biocomplexity Institute and the Virginia Department of Health, we co-authored a paper demonstrating the accuracy and robustness of using Metaculus’s COVID-19 Omicron variant forecasting ensembles in combination with computational models by providing valuable real-time forecasts.
  • Mpox Rapid Information Aggregation: When an unexpected number of Mpox cases were reported in early May 2022, we organized a rapid forecasting response to gauge the potential scope and impact of the outbreak. Working with our research partner Tom McAndrew at Lehigh University, we co-authored a paper on the efficacy of rapid human judgment forecasting, which was published (in record time) in The Lancet Digital Health.
  • Forecasting the US-China AI & Nuclear Landscape: With our partners at the Institute for Security and Technology, we launched an initiative evaluating intervention points in the US-China nuclear relationship, with a special focus on the integration of AI into nuclear command, control, and communications systems. The resulting report combines insights from nuclear and policy subject matter experts and Metaculus Pro Forecasters.

I’m extremely proud of what we’ve accomplished as a team, and I’m deeply grateful for the support of our partner organizations and the forecasting community. If you’re excited by what we’re doing and would like to get in touch, please do feel free to grab some time with me, or shoot our team a note.

Onward,

Gaia Dempsey
CEO, Metaculus

25

0
0

Reactions

0
0

More posts like this

Comments2
Sorted by Click to highlight new comments since: Today at 8:34 PM

Thanks for this update. I like Metaculus and have started forecasting more in 2022.

Something I would enjoy seeing is the ability to have a very quick UI for creating private questions, similar to what Nathan proposes here: https://www.super-linear.org/prize?recordId=recYHpvvGFmiFq9tS

Here is what I imagine this could look like: 

  1. I pull up https://quick.metaculus.com 
  2. I only see an empty command line, and when I type in, e.g. "Will Putin still be President of Russia at the end of 2023? 1y", it instantly creates a private binary question that closes at the end of today and puts the resolve date to one year from today. The question title is filled out as a random combination of the words from the question text. It just fills in a dot for background info, resolution criteria,  and fine print. 
  3. The format could be "[question text]? [resolve date]" where the question mark serves as the indicator for the end of the question text, and the resolve date part can interpret things like "1w", "1y", "eoy", "5d"
  4. In a perfect world, this would also integrate with Alfred on my mac so that it becomes extremely easy and quick to create a new private question

This feature would allow me to quickly create a binary private forecast when I am thinking of it and improve my calibration over time. 

Also, I have a question about the calibration plot that Metaculus creates on my profile: Does it only take into account the last forecast I made on a question or somehow integrates all forecasts I made on one question over time and puts that into the calibration plot? 

It's a great idea, and I like how you've fleshed it out. I'll pass this along to our Product team. 

For your calibration plot, you can actually use the 'evaluated at' dropdown and watch your plot adjust on the fly.