Hide table of contents

A forecasting digest with a focus on experimental forecasting.

  • You can sign up here.
  • You can also see this post on LessWrong here

The newsletter itself is experimental, but there will be at least five more iterations. Feel free to use this post as a forecasting open thread.

Why is this relevant to EAs?

  • Some items are immediately relevant (e.g., forecasts of famine).
  • Others are projects whose success I'm cheering for, and which I think have the potential to do great amounts of good (e.g., Replication Markets).
  • The remaining are relevant to the extent that cross-polination of ideas is valuable.
  • Forecasting may become/is becoming a powerful tool for world-optimization, and EAs may want to avail themselves of this tool.

Conflict of interest: With Foretold in general and Jacob Laguerros in particular. This is marked as (c.o.i) throughout the text.


  • Prediction Markets & Forecasting platforms.
    • Augur.
    • PredictIt & Election Betting Odds.
    • Replication Markets.
    • Coronavirus Information Markets.
    • Foretold. (c.o.i).
    • Metaculus.
    • Good Judgement and friends.
  • In the News.
  • Long Content.

Prediction Markets & Forecasting platforms.

Forecasters may now choose to forecast any of the four horsemen of the Apocalypse: Death, Famine, Pestilence and War.

Augur: augur.net

Augur is a decentralized prediction market. It will be undergoing its first major update.

Predict It & Election Betting Odds: predictIt.org & electionBettingOdds.com

PredictIt is a prediction platform restricted to US citizens or those who bother using a VPN. Anecdotically, it often has free energy, that is, places where one can earn money by having better probabilities, and where this is not too hard. However, due to fees & the hassle of setting it up, these inefficiencies don't get corrected. In PredictIt, the world politics section...

The question on which Asian/Pacific leaders will leave office next? also looks like it has a lot of free energy, as it overestimates low probability events.

Election Betting Odds aggregates PredictIt with other such services for the US presidential elections.

Replication Markets: replicationmarkets.com

Replication Markets is a project where volunteer forecasters try to predict whether a given study's results will be replicated with high power. Rewards are monetary, but only given out to the top N forecasters, and markets suffer from sometimes being dull. They have added two market-maker bots and commenced and conclude their 6th round. They also added a sleek new widget to visualize the price of shares better.

Coronavirus Information Markets: coronainformationmarkets.com

For those who want to put their money where their mouth is, there is now a prediction market for coronavirus related information. The number of questions is small, and the current trading volume started at $8000, but may increase. Another similar platform is waves.exchange/prediction, which seems to be just a wallet to which a prediction market has been grafted on.

Unfortunately, I couldn't make a transaction in these markets with ~30 mins; the time needed to be included in an ethereum block is longer and I may have been too stingy with my gas fee.

Foretold: foretold.io (c.o.i)

Foretold is an forecasting platform which has experimentation and exploration of forecasting methods in mind. They bring us:

Metaculus: metaculus.com

Metaculus is a forecasting platform with an active community and lots of interesting questions. They bring us a series of tournaments and question series:

/(Good Judgement?[^]*)|(Superforecast(ing|er))/gi

Good Judgement Inc. is the organization which grew out of Tetlock's research on forecasting, and out of the Good Judgement Project, which won the IARPA ACE forecasting competition, and resulted in the research covered in the Superforecasting book.

The Open Philantropy Project has funded this covid dashboard by their (Good Judgement Inc.'s) Superforecasting Analytics Service, with predictions solely from superforecasters; see more on this blogpost.

Good Judgement Inc. also organizes the Good Judgement Open (gjopen.com)[https://www.gjopen.com/], a forecasting platform open to all, with a focus on serious geopolitical questions. They structure their questions in challenges, to which they have recently added one on the Coronavirus Outbreak; some of these questions are similar in spirit to the short-fuse Metaculus Tournament.

Of the questions which have been added recently to the Good Judgment Open, the crowd doesn't buy that Tesla will release an autopilot feature to navigate traffic lights, despite announcements to the contrary. Further, the aggregate...

  • is extremely confident that, before 1 January 2021, the Russian constitution will be amended to allow Vladimir Putin to remain president after his current term.
  • gives a lagging estimate of 50% on Benjamin Netanyahu ceasing to be the prime minister of Israel before 1 January 2021.
  • and 10% for Nicolás Maduro leaving before the 1st of June.
  • forecasts famine (70%).
  • Of particular interest is that GJOpen didn't see the upsurge in tests (and thus positives) in the US until until the day before they happened, for this question. Forecasters, including superforecasters, went with a linear extrapolation from the previous n (usually 7) days. However, even though the number of cases looks locally linear, it's also globally exponential, as this 3Blue1Brown video shows. On the other hand, an enterprising forecaster tried to fit a Gompertz distribution, but then fared pretty badly.

In the News

Long Content

Sorted by Click to highlight new comments since:

Nice to see a newsletter on this topic!

Clarification: The GJO coronavirus questions are not funded by Open Phil. The thing funded by Open Phil is this dashboard (linked from our blog post) put together by Good Judgment Inc. (GJI), which runs both GJO (where anyone can sign up and make forecasts) and their Superforecaster Analytics service (where only superforecasters can make forecasts). The dashboard Open Phil funded uses the Superforecaster Analytics service, not GJO. Also, I don't think Tetlock is involved in GJO (or GJI in general) much at all these days, but GJI is indeed the commercial spinoff from the Good Judgment Project (GJP) that Tetlock & Mellers led and which won the IARPA ACE forecasting competition and resulted in the research covered in Tetlock's book Superforecasting.

Thanks for the correction; edited.

Note that the headline ("Good Judgement Project: gjopen.com") is still confusing, since it seems to be saying GJP = GJO. The thing that ties the items under that headline is that they are all projects of GJI. Also, "Of the questions which have been added recently" is misleading since it seems to be about the previous paragraph (the superforecasters-only questions), but in fact all the links go to GJO.

Edited again. If you want, throw me a bone: what's the last explicit probabilistic prediction you've made? Also, I liked your review on How to Measure Anything, which feels relevant to the topic at hand. NNTR.

The headline looks broken in my browser. It looks like this:

/(Good Judgement?[^]*)|(Superforecast(ing|er))/gi

The last explicit probabilistic prediction I made was probably a series of forecasts on my most recent internal Open Phil grant writeup, since it's part of our internal writeup template to prompt the grant investigator for explicit probabilistic forecasts about the grant. But it could've easily been elsewhere; I do somewhat-often make probabilistic forecasts just in conversation, or in GDoc/Slack comments, though for those I usually spend less time pinning down a totally precise formulation of the forecasting statement, since it's more about quickly indicating to others roughly what my views are rather than about establishing my calibration across a large number of precisely stated forecasts.

I've written this interactive notebook in Foretold prediction platform. It is meant to be completely beginner friendly and takes about 2 hours to go through. I've used it as the basis for a workshop, and the accompanying slides can be found at the bottom of the notebook.

From the notebook:

In this interactive notebook, our goal is to actively try out forecasting and learn several basic tools. After this, you will be able to more easily use forecasts in your daily life and decision making, understand broadly how forecasters go about predicting stuff, and you should know if this is something you want to dive into deeper and how to go about that. We have 5 sections:

  1. We will start immediately with several examples.
  2. Then go on to understand how probabilities feel like, and how to be more calibrated.
  3. Work on the technique of outside view and inside view reasoning.
  4. Briefly discuss several interesting techniques - research, combining models and changing scope.
  5. Try out some actual forecasts from start to finish!
and 10% for Nicolás Maduro.

The time horizon for this is "before 1 June 2020." That seems reasonable.

Oh hey, I've seen you around on GJOPen. Thanks for the correction; edited.

Curated and popular this week
Relevant opportunities