Motivation
There is already a fair amount of interest around Effective Altruism in judgemental forecasting. We think there’s a whole lot of good research left to be done.
The valuable research seems to be all over the place. We could use people to speculate on research directions, outline incentive mechanisms, try novel forecasting questions with friends, and outline new questions that deserve forecasts. Some of this requires a fair amount of background knowledge, but a lot doesn’t.
The EA and LW communities have a history of using prizes to encourage work in exciting areas. We’re going to try one in forecasting research. If this goes well, we’d like to continue and expand this going forward.
Prize
This prize will total $1000 between multiple recipients, with a minimum first place prize of $500. We will aim for 2-5 recipients in total. The prize will be paid for by the Quantified Uncertainty Research Institute (QURI).
Rules
To enter, first make a public post online between now and Jan 1, 2021. We encourage you to either post directly or make a link post to either LessWrong or the EA Forum. Second, complete this form, also before Jan 1, 2021.
Research Feedback
If you’d like feedback or would care to discuss possible research projects, please do reach out! To do so, fill out this form. We’re happy to advise at any stages of the process.
Judges
The judges will be AlexRJL, Nuño Sempere, Eric Neyman, Tamay Besiroglu, Linch Zhang and Ozzie Gooen. The details of the judging process will vary depending on how many submissions we get. We’ll try to select winners for their importance, novelty, and presentation.
Some Possible Research Areas
Areas of work we would be excited to see explored:
- Operationalizing questions in important domains so that they can be predicted in e.g., Metaculus. This is currently a significant bottleneck; it’s surprisingly difficult to write good questions. Examples in the past have been the Ragnarök or the Animal Welfare series. A possible suggestion might be to try to come up with forecastable fire alarms for AGI. Tamay Besiroglu has suggested a “S&P 500 but for AI forecasts,” i.e., a group of forecasting questions which track something useful for AI (or for other domains.)
- Small experiments where you and/or a group of people use forecasting for your own decision making, and write up what you’ve learned. For example, set up a Foretold community to decide on which research document you want to write up next. Predictions as a Substitute for Reviews is an example here.
- New forecasting approaches, or forecasting tools being used in new and interesting ways, or applied to new domains. For example, Amplifying generalist research via forecasting, or Ought’s AI timelines forecasting thread.
- Estimable or gears-level models of the world that are well positioned to be used in forecasting. For example, a decomposition informed by one’s own expertise of a difficult question into smaller questions, each of which can be then forecasted. Recent work by CSET-foretell would be an example of this.
- Suggestions for or basic implementation of better tooling for forecasters, like a Bayes rule calculator for considering many pieces of evidence, a Laplace law calculator, etc.
- New theoretical schemes which propose solutions to current problems around forecasting. For a recent example, see Time Travel Markets for Intellectual Accounting.
- Elicitation of expert forecasters of useful questions. For example, the probabilities of the x-risks outlined in The Precipice.
- Overviews of existing research, or thoughts or reflections on existing prediction tournaments and similar. For example, Zvi’s posts on prediction markets, here and here.
- Figuring out why some puzzling behavior happens in current prediction markets or forecasting tournaments, like in Limits of Current US Prediction Markets (PredictIt Case Study). For a new puzzle suggested by Eric Neyman, consider that PredictIt is thought to be limited because it caps trades at $850, has various fees, etc, which makes it not the sort of market that big, informed players can enter and make efficient. But that fails to explain why markets without such caps, such as FTX, have prices similar to PredictIt. So, is PredictIt reasonable or is FTX unreasonable? If the former, why is there such a strong expert consensus against what PredictIt says so often? If the latter, why is FTX unreasonable?
- Comments on existing posts can themselves be very valuable. Feel free to submit a list of good comments instead of one single post.
Nice! In the few minutes of reading this post I came up with five ideas for related things I could (and maybe should) write a post on. My only issue is that there's only 6 weeks of time for this, and I'm not sure if that'll be enough for me to finish even one given my current schedule. But I'll see what I can do. May even be the right kind of pressure, as otherwise I'd surely be following Parkinson's law and work on a post for way too long.
(The many examples you posted were very helpful by the way, as without them I would have assumed I don't have much to contribute here)
Do you want to make some of them public so that other people can steal them?
Sure. Those I can mention without providing too much context:
If anybody actually ends up planning to write a post on any of these, feel free to let me know so I'll make sure focus on something else.
Thanks!
Thanks! That's useful to know. I intend to host more prizes in the future but can't promise things yet.
There's no harm in writing up a bunch of rough ideas instead of aiming for something that looks super impressive. We're optimizing more to encourage creativity and inspire good ideas, rather than to produce work that can be highly cited.
You can look through my LessWrong posts for examples of the kinds of things I'm used to. A few were a lot of work, but many just took a few hours or so.