The forecasting community is failing because it is not focused on how the world actually works.
Who am I to make such an outlandish claims?
- I worked at HM Treasury as senior policy advisor across multiple policy areas, including international development, relations, and I led the team for HMTs economic and financial response to the war in Ukraine.
- I have an MSc in Cognitive and Decision Sciences from UCL, where I led experimental research into how to improve forecasting ability of policy makers and analysts in central government.
- I started a consultancy with one of the world leading researchers in computational psychology and causal modelling to improve predictive reasoning.
- I am now the Executive Director of the Swift Centre for Applied Forecasting, where I have led numerous forecasting workshops and small projects with government teams, including the Cabinet Office, HM Treasury, Department for Science, Innovation and Technology. I have also led projects and workshops with organisations working on AI capabilities, risks, and policy, including GovAI and frontier labs.
Through all of this I have met numerous organisations, including policy, strategy, and risk teams in the worlds largest and most powerful organisations, non-profits, tech firms, banks and financial institutions to discuss the use of forecasting.
1. We’re Obsessing Over Question Wording But Not The Question
The forecasting community has a fetish for resolution criteria. We spend weeks debating the exact definitions of words but spend far less time understanding what exact issues organisations need to grapple with.
When I speak to senior government officials, they often don’t even know which risks they should be looking at. They are operating in a fog of war where the primary challenge isn’t predicting the outcome of a well-defined event but instead it’s identifying which events even matter or they should be focused on.
We are providing high-precision answers to questions that decision-makers haven’t asked and often don’t care about. So much time has been spent forecasting headline geopolitical events or AI capabilities and risks - all interesting and academically engaging, but are so far away from the actual questions and issues decision makers are trying to weigh up.
To emphasise this, I have spoken to people (even in government), who are building AI forecasting tools. What they say after they’ve build a semi-reliable tool is all the same: “we’ve found people don’t know what issues they should be focusing on, and rather than a probability estimate, they want help to identify the most prescient questions”.
2. Transparency is the Real Value, Yet Everyone Focuses on the Accuracy Status Game
To a forecaster their probability is everything - as it should be. It’s how you prove you’re worth, it’s how you become a “Superforecaster” or get a job at a hedge fund.
But decision-makers do not care if you are 2% more accurate than the next guy. When it comes to actual decision making, the value of a probability is its ability to force transparency and to expose differences. Sure, it can’t be wildly wrong, but no one is fighting over single digit percentages.
In a standard policy meeting, people hide behind imprecise words like perhaps, likely, or could. Numbers strip that away and once you build comfort with using them, real value in the decision making process can be unlocked - better reasoning transparency, more efficiency in the decision making process, more effective options to achieve the objective you want.
The real value of forecasting is in the moment you realise two people in the same room have forecasts 40% apart. That is where the benefit occurs. But the community is so obsessed with maximising Brier scores that it ignores the fact that their quest for the most accurate predictions are often sapping time and effort away from utilising the most valuable element of forecasting: transparency.
3. The Clearance Filter
There is a naive, almost arrogant assumption that if we just give a Minister an accurate percentage, they will make a better decision.
I worked with Ministers who couldn’t read a graph properly. If you put a raw percentage into a submission for a Secretary of State, it will likely be intercepted by their Private Office or a Senior official during clearance and sent back for being too technical. If it does make it to their desk, they likely won’t know what to do with it or how it’s beneficial to them. A lot of political and organisation decision making is not based on how accurately you’ve predicted the world. Sad, but true.
Pure forecasting has a place, but it is a niche compared to what has been pushed and funded. The real win is a better-reasoned policy memo. If the final advice looks the same but the process of getting there involved structured reasoning and the exposure of hidden risks, that is a victory. The community’s refusal to understand the existing bureaucratic workflow is why it hasn’t been adopted.
4. Misallocation of Resources: Researching the Problem to Death
This part may come across as jaded or resentful. I don’t think that’s completely unfounded, but it comes from a place of truly caring about improving institutional decision making. I’ve personally spent thousands and have taken many risky career moves to work on it. I think without considerably better institutional decision making we will never navigate the risks of AI or avoid catastrophic events. So given that, and my experience as a HMT spending policy lead, I am disappointed when I see the misallocation of scarce resources.
I’ve watched funders pour tens of millions of dollars into forecasting platforms and large-scale research reports that practically no decision maker reads (at least not enough of them to justify the cost).
I have spoken to dozens of policy officials about these reports. Most give me a laugh and say they don’t have time. Others ask me what those platforms even are. Even after the UK closed its internal forecasting market and the US intelligence agency ended theirs, funders doubled down.
Meanwhile, those working on actual implementation - within the very organisations and institutions we claim to want to be using forecasting to improve decision making - struggled.
Crude example, but a couple of years ago I had the interest of the UK’s Policy Profession training team (covering 50,000 officials) and the Bank of England. We couldn’t secure funding to provide the workshops they wanted, or to even cover the six-months runway we’d need to get through their procurement process. A year later, I ended up working at the Swift Centre to help them deliver some research funding they got to investigate the blockers to forecasting. Blockers that they, and I, knew about (and I had already somewhat overcome with the policy profession etc. as above). But the default comfort was to fund further research, rather than actual delivery. We did that work, delivered record-breaking engagement, and still had to fight for a continuation while tens of millions were funnelled into more research and platforms.
5. Even the Believers Never Really Jumped Aboard
If you look at organisations across the Effective Altruism movement (the very people who champion forecasting and the core premise of reasoned decision making), you’ll see they struggle to use it in their own decision-making.
I’ve seen organisations in this space ignore the fundamentals of reasoning transparency and structured forecasting when it comes to their own organisational decisions and grantmaking.
Many in the community like to read the forecasts, or take part in tournaments, but how many actually make tangible changes to their decisions based on them?
Until we stop treating forecasting as a intellectual status symbol and start treating it as a messy, difficult integration problem for the world’s most powerful (and busy) people, we are just talking to ourselves.
Well written and pragmatic, thanks. Although forecasting does sounds great, I have no idea how animal nonprofits (to take one example) could use it to make better work.
I can actually think of one example in animal welfare: the EA Animal Welfare Fund forecasts grant outcomes.
Great, good to know !
I think this is a clear sign the community hasn’t been able to communicate its use case well at all. This is one reason I often use “predictive reasoning” as a more general concept when talking to people, interestingly especially if they are already aware of forecasting (as they’ve been conditioned to think it means prediction markets, tournaments, and resolution criteria).
Take your example of animal welfare, I don’t know the exact use case best aligned to you, but fundamentally i’m confident 95%+ of the decisions an animal nonprofit will make are based on two predictions:
1) What will the world be like in the future (insert timeline)?
2) What interventions will most impact/change that future closer to what you’d like it to be?
Forecasting, or more specifically, the processes that underpins the science of forecasting, can be used to increase the accuracy and efficiency of those two predictions.
Once you do that, a better estimation of the future world + a better estimation of the efficacy of your actions, will occur.
Yes, makes sense, shorter or longer AI timelines do change the game a lot, for instance.
For example, forecast on the potential viability of cultured meat could have several actionable implications.
.m It might be something worth directly funding, particularly if we think that certain types of investment will have a strong impact on it’s like it would be successful. It also might be something to start investing in planning for in legal messaging communications et cetera.
Also, if we anticipate a massive global shift towards culture meat and thus away from factory farming it makes lobbying for factory farming reforms and corporate welfare of pledges somewhat less valuable.
This is much of the theory of impact/theory of change behind The Unjournal’s cultured meat pivotal questions, project and our upcoming workshop. We are working in some belief elicitation and some forecasting elements. However, it’s not clear whether these are situations in which the broad Wisdom of the Crowd adds a lot of value or whether it’s dependent on small groups of experts.
Also see Support Metaculus' First Animal-Focused Forecasting Tournament which is something we are trying to support and we try to make sure that each of the forecasting questions has actionable value of information.
Trying to connect the “forecasting to decisions” or “evidence to decisions” pipeline with the “Pivotal questions”: an Unjournal trial initiative and our workshops https://uj-pq-workshops.netlify.app/
But the challenges you mention are real and this tracks my experience. Upvotes.
Many of us would assume the benefit of forecasting techniques is simply to get better forecasts. The fact that it's the process itself that provides the most value was really interesting to me - thanks for this post.
Thank you for posting this!
Interesting point. Curious whether this alludes to a broader bottleneck within forecasting around question generation itself, rather than question answering/calibration. My vague impression (though I don’t know anything about the practical workflows of forecasting organisations) is that a lot of attention/status accrues to producing forecasts, whereas identifying and specifying the right questions may be at least as important (and potentially harder?)
I'd be interested to hear how much effort (as a rough %) forecasting orgs currently devote to question discovery/specification versus answering already-defined questions.
Also wondering at what “level” this problem mainly exists. Is the issue:
the more granular forecasting-design questions like framing, timelines, resolution criteria, etc?
(or perhaps both of these are linked too tightly in practice to separate?)
So my experience is that identifying/specifying/generating the right questions is at least 50% of the benefit, if not higher. There are lots of reasons organisations struggle with this, from: organisational incentives; incoherent steers and views from senior managers; lack of accountability and ownership; to simply not recognising that they are trying to do a prediction.
This is why forecasting funding that has focused on improving forecasting accuracy is flawed, because it doesn't matter how accurate you are if your question isn't of use to the decision making process.
The problem exists at both those "levels", but the most important one to solve for an organisation is the first one. Issues with resolution criteria etc. decrease accuracy but as long as people's rationale's are explicit you can bridge that gap (i.e. I know why one person was higher and another was lower, it was because they both took the resolution to mean something slightly differently). But if the question/problem you are trying to solve in the first instance is wrong, then the whole thing is a waste of time and energy.
I do not consider this sad, but just a fact about how decision-making works in the real world. Economists model humans as-if they use probabilities to make choices between well-defined options. If you model humans as making decisions in this way, then forecasting makes sense.
But the real world is not a multiple choice test offering pre-defined options which need probabilistic estimates. In real world decision-making, problem formulation and problem solving are the same cognitive process. There are no problems out there waiting for you to find them, rather, you have to define what is even a problem in the first place and how to think about it. Figuring out how to solve a problem is a process of sensemaking until you feel like you have a grip on the situation; like you know what is relevant, how it is relevant, what the moving parts are, how they interact, what your leverage point is, and what your values are. A probabilistic estimate can never offer that. A probabilistic estimate abstracts away so much that it actually leaves you feeling like you don't know the space at all.
Dominic Cummings has mentioned this
Forecasting is based on an as-if (read: wrong) model of decision-making. You wouldn't decide when to make a left-hand-turn at a busy intersection based on some narrow probabilistic estimate (there is a 99% chance you won't get t-boned if you turn now) because you want to understand the decision yourself, and that probabilistic estimate is missing so much (Will I T-bone someone? Are there people in the cross walk? Are the cars slowing down because the light already changed colors? Or are they speeding up because it already changed colors?)
In real world decision-making, framing/modeling is everything. How you take large problems and turn them into something tractable that the human mind can comprehend and reason about. But superforecasting assumes that problem away. It assumes away the most important aspect of decision-making; understanding.
I think this misunderstands forecasting or mischaracterises decision making.
Your brain absolutely does decide when to make a left-hand-turn at a busy intersection based on a probabilistic estimate. Your brain is just pretty reliable at making said judgments that you assume some binary choice was made. Subconciously you are weighing up the likelihood of various risks based on your senses observing them and concluding that there is a very low (or an appropriately low) probability of it going wrong. The easy way to test this is go to a zip line, one with a drunk operator and one with a sober operator. The zip line may look identically safe but your brain will (hopefully) consider the former more risky - not because the impact of something going wrong is different between the zip lines (both falls would hurt, if not kill you), but because the likelihood of something going wrong is different (as you'd calculate that a drunk operator may not be as cognitively aware of what they are doing). That's a probabilistic assessment.
The difference is, most decisions we deal with in the world don't need us to sit down and do a formal prediction process to ensure good decision making. However, most organisational, and especially those on policy and governance etc., do require it. They are concretely two predictions you are making:
1. What will the world be? (through the lens of what you care about, e.g. will there be an oil crisis?)
2. What influence on the world will my policy have? (e.g. if I send Trump a friendly email will he avoid attacking another petrostate?)
Viewing forecasting as just the probabilistic estimate is missing the entire benefit. Forecasting is the entire process. Stating your view probabilistic just allows you to understand your own and others uncertainty (and to provide a purity of accountability and incentive alignment). It only adds decision fatigue if you have not put in the right processes to interpret the result (such as upfront thresholds for action). If you are following an optimal predictive decision making process, you should be making your assumptions and your weighing up of information explicit. This is how you determine what is relevant, what isn't relevant etc.
Dominic Cummings statement misses that everything he said there IS forecasting. What he's actually saying is "the point estimate from forecasters was not useful - it was the explicit reasoning about the causal chain etc. from forecasters that was useful".
However, benchmarking against accuracy ensures the incentives are correct and that decisions aren't manipulated by elements that decrease the end outcomes efficacy.
If you conclude that it'd be better if decision makers had: 1) the most accurate view of the world; and 2) the most accurate view of how their actions may influence that world towards their objectives. Then I would stand by the statement that it's sad.
Thanks for the post, James. It made sense to me.
This is "the Government of the United Kingdom’s economic and finance ministry".
You mean 40 pp apart?
Yeah 40pp, though 40% difference may also be informative depending on the question and distribution.
Regardless, “40” was just a random number. Basically the interesting thing are the areas of greatest difference, not the probability itself.