Forecasting and estimation areis an important toolstool for improving the future, because good forecasts and estimates can help us appropriately plan interventions and assess risks. Over the past several decades there has been significant research and investment in forecasting and estimation techniques, tools, and organizations. This continues to be an area of investment for improving our ability to make good decisions.

The State of Forecasting within EA

There are some major branches of forecasting within the EA movement:

• Personal forecasting - individuals forecasting to improve their decision-making or for status and personal enjoyment
• Forecasting consultancies - EA organisations pay for forecasting by groups of top forecasters or Metaculus
• Forecasting research - Academic research on the accuracy of forecasting and how to do it better (eg by FRI)
• Institutional forecasting - Seeking for forecasting to be used inside government and large institutions
• Forecasting technology - Building new tools to quantify with (eg Squiggle)

These areas in more depth

Institutional forecasting.

Forecasting in institutions can range from predicting broad metrics to specific outcomes based on specific decisions. There can often be problems with buy-in from key stakeholders, who either see this as an unnecessary step or are concerned for their own status.

Forecasting Techniques

Forecasting is hard but many top forecasters use common techniques. This suggests that forecasting is a skill that can be learnt and practised.

Base rates

Reference Class Forecasting on Wikipedia

Suppose we are trying to find the probability that an event will occur within the next 5 years. One good place to start is by asking "of all similar time periods, what fraction of the time does this event occur?". This is the base rate.

If we want to know the probability that Joe Biden is President of the United States on Nov. 1st, 2024, we could ask

• What fraction of presidential terms are fully completed (last all 4 years)? The answer to this is 49 out of the 58 total terms, or around 84%.
• On the other hand, we know that Biden has already made it through 288 days of his term. If we remove the 5 presidents who left office before that, there are 49 out of 53 or around 92%.
• But alternately, Joe Biden is pretty old (78 to be exact). If we look up death rate per year in actuarial tables, it's around 5.1% per year, so this leaves him with a ~15% chance of death or a 85% chance of surviving his term.

These are all examples of using base rates. [These examples are taken from Base Rates and Reference Classes by jsteinhardt.]

Base rates represent the outside view for a given question. They are a good place to start but can often be improved on by updating the probability according to an inside view.

Note that there are often several reference classes we could use, each implying a different base rate. The problem of deciding which class to use is known as the reference class problem.

Calibration training

A forecaster is said to be calibrated if the events they say have a X% chance of happening, happen X% of the time.

Most people are overconfident. When they say an event has a 99% chance of happening, often the events happen much less frequently than that.

This natural overconfidence can be corrected with calibration training. In calibration training, you are asked to answer a set of factual questions, assigning a probability to each of your answers.

A list of calibration training exercises can be found here.

Question decomposition

Much like Fermi estimation, questions about future events can often be decomposed into many different questions, these questions can be answered, and the answers to these questions can be used to reconstruct an answer to the original question.

Suppose you are interested in whether AI will cause a catastrophe by 2100. For AI to cause such an event, several things need to be true: (1) it needs to be possible to build advanced AI with agentic planning and strategic awareness by 2100, (2) there need to be...

• Concepts
  • Forecasting
  • Prediction markets
  • Fermi estimate
  • Long-range forecasting
  • Estimate elicitation and crowdsourcing
• Tools and Platforms
  • Squiggle(wiki)
  • Guesstimate(wiki)
  • Metaculus(wiki)
  • Manifold Markets(wiki)
• Tutorials
  • Alex Lawsen's video series
  • Metaculus's tutorials
  • Metaculus wiki
• Organizations
  • Arb
  • Quantified Uncertainty Research Institute
  • Samotsvety Forecasting
  • Swift Centre for Applied Forecasting
• AI forecasting
• Forecasting newsletter

Forecasting and estimation are important tools for improving the future, because good forecasts and estimates can help us appropriately plan interventions and assess risks. Over the past several decades there has been significant research and investment in forecasting and estimation techniques, tools, and organizations. This continues to be an area of investment for improving our ability to make good decisions.

This is a broad topic group that captures several sub-topics:

• Concepts
  • Forecasting
  • Prediction markets
  • Fermi estimate
  • Long-range forecasting
  • Estimate elicitation and crowdsourcing
• Tools and Platforms
  • Squiggle
  • Guesstimate
  • Metaculus
  • Manifold Markets
• Organizations
  • Arb
  • Quantified Uncertainty Research Institute
  • Samotsvety Forecasting
  • Swift Centre for Applied Forecasting
• AI forecasting
• Forecasting newsletter