Epistemic Status: This concept is early and speculative
TLDR: Accuracy Agreements are contracts in which one party commits to remunerate the other based on a predetermined rate, proportional to the accuracy of a series of forecasts supplied by the contract owner. This incentivizes the owner to find cost-effective methods for producing calibrated and accurate forecasts.
The Basics
The problem: Prediction Markets are effective at determining outcomes for binary questions, but they struggle with probability distributions and more expressive situations, such as those involving functions. Also, often there are only a few potential organizations that manage estimation, and it’s simpler to pay one for estimation than it is to set up a market.
Proposed solution: A client publishes an "Accuracy Agreement" with an associated "Type Specification."
A simple agreement's terms are as follows:
A purchaser of the agreement provides a one-time or continuously-updating forecast that complies with the established Type Specification.
Upon reaching a prearranged future date, the Agreement will undergo Resolution, which entails:
The client will determine the answers to the forecasting questions.
The client will evaluate and assign a score to the submitted forecast.
The contract holder at the time of resolution will receive payment based on a function of a scoring rule. An example of this could be: "The average log score of the forecast minus the prior, multiplied by $10,000. Stated differently, $10,000 per bit of information.”
Holding an "Accuracy Agreement" with a specific forecast has a calculable value. If the holder trusts the forecast, the value would equal the expected resolution payment. (Easily calculated using expected loss).
One elegant way the client could ensure a reasonable price is by selling the contract, initially, to the winner of a bidding system. When a question is proposed, potential buyers bid for the Agreement. If a bidder believes their forecast has an EV of "$25,000," they might bid "$23,000" for the Agreement. The winner would either need to submit a prediction immediately, or would have time to generate one.
An Example:
A client publishes:
A list of 50 continuous variables, to be resolved in 1 year.
A set of (weak) priors for those variables.
A simple scoring function, e.g., $1K * total logScore sum.
A future date for bidding to commence (1 month out).
Bidders submit:
A list of forecasts for the 50 variables, each as a probability distribution.
A maximum price for purchasing the agreement with their forecast.
The top bidder is chosen and sold the contract.
In 1 year, the contract resolution occurs.
All (or some) of the variables are resolved.
The contracts’ forecasts get scored.
The agreement owner is paid according to the predetermined scoring rule
Potential Changes / Improvements
An active marketplace. Purchasers can trade these contracts with each other over time, after the initial purchase.
Introduce a market with multiple shares, allowing buyers and sellers to purchase only a portion of the market. This can help ensure that no one purchaser can overbid for the privilege of promoting their (bad) forecast.
Buyers can update their forecasts over time. The contracts cover the average accuracy over that time. This incentivizes buyers to continue to improve their contracts.
Forecasting questions can be resolved and scored by agreed-upon third parties.
There’s an extra reward for a forecast that comes with a good explanation or similar. This could be part of the scoring function.
In some cases, it might make sense for bidders not to have to put any money down. This might be considered gambling. One alternative, if the bidders are trusted, is that they get payments in two parts: one fixed fee, and one Accuracy Agreement. For example, some EA org puts out a call for a forecasting consultancy to forecast 1000 variables - but most of the payment is in the form of an Accuracy Agreement.
Powering Accuracy Agreements with Prediction Markets
Would information purchasers get a better return by subsidizing Prediction Markets, than they would by selling Accuracy Agreements?
I think this difference should be bounded. If/when it is the case that Prediction Markets are more efficient than “Accuracy Agreements”, then the owner of the Accuracy Agreement could just set up Prediction Markets once they own the agreement. This owner could subsidize the market in whatever way would do the best job.
For example, say a purchaser sees an agreement that could make them $50,000 profit without a Prediction Market, or $100,000 if they spend $20,000 on setting up and subsidizing a corresponding Prediction Market. They might bid $60,000 on the contract, and if they get it, then spend $10,000 on the market.
This setup would help encourage both good Prediction Markets, and encourage actors who are savvy about properly setting up and subsidizing said markets.
Bottlenecks
The first bottleneck to actually selling Accuracy Agreements is the legality. It might be tricky to do in a way that’s not considered gambling.
The second bottleneck is finding buyers. Accuracy agreements will be somewhat complicated assets. You’d need sophisticated estimation agencies to do competent estimation and to have the flexibility to take the necessary financial risk.
The third challenge is just the infrastructural and logistical expenses. Setting up software to make this process smooth, and making sure that questions get resolved by third parties, would require some overhead, especially early on.
Next Steps
To make progress, I’d like to see this sort of setup be tested in simple ways. First, talk with lawyers to best maneuver around the first bottleneck. If any readers of this have insight here, please leave comments!
Second, experiment with small contracts (maybe <$5k) and small consultancies. This should be achievable with a small team with a decent reputation with potential estimation consultancies. I could see QURI doing this in the future but I would be happy to see other groups do so first. Early contracts don’t need to allow for themselves to be sold, they can be simple agreements with individual consultancies.
One nice property of these contracts is that they should be able to scale gracefully, in that you could do them on small scales or large scales. So, start small, and as we gain experience, we’ll be able to better scale these setups with larger sums and more diverse Accuracy Agreement purchasers.
Thanks to Javier Prieto and Nuño Sempere for feedback
Based on reading through this, it seems promising - the question is whether it's worth someone overcoming the startup costs / bottlenecks to run a pilot, and I'm not sure it is unless there is a specific promising use case that makes this seem much better than current approaches. Any ideas for specific markets where this is valuable?
1. CEA / Open Phil wants to contract [Metaculus/Manifold/Good Judgement/Epoch] to come up with forecasts for 100 continuous questions regarding clean meat. They just care that the forecasts are good, they don't care much for which specific methods are used. They also would like to see new forecasting players join the fray and compete for cost-effective work.
The put out an open call for proposals, and say that much of the money will come in the form of a simple Accuracy Agreement (one that perhaps can't be re-sold, for legal simplicity reasons, for now).
2. A government agency wants to outsource "Predictions of construction costs, for a megaproject". This set of predictions will be very expensive to produce and update. Instead of paying consultants in the traditional way, they organize much of this as an Accuracy Agreement.
3. In 2030, in a hypothetical future world, there are many big forecasting players and purchasers. Complicated forecasting Type Definitions are routinely created for things. Like, "For every US regional area, and every point in time for the next 5 years, how many of each disease will be monitored/registered in that area?". Forecasting purchasers can expect that a public bidding market will return a reasonable result, similar to some current large project bidding procedures now.
I see this approach as a more complex but powerful alternative to the model of buyers now just contracting with specific firms like Metaculus or Good Judgement, which has started to happen in the last few years (for judgemental questions).
One weakness that these Agreements have is that they require the client (or a third party) to ensure that questions are written and scored, instead of the consultant.
This is a similar issue that Prediction Markets have, but not one that existing forecasting contracts often have. These contracts often have the forecasting contractors do the work of question specification and resolution.
So, Accuracy Agreements are probably in-between Prediction Markets and current contractor agreements, in complexity.
For big government construction projects, I believe some firms/agencies will do a lot of preparation and outlining, before a bidding process might begin. Getting things specific enough for a large bidding process is itself a fair bit of work. This can be useful for large projects, or in cases where the public has little trust in the key decision makers, but is probably cost-prohibitive for other situations.
I'm not sure about this, but there is a possibility that this sort of model violated US online gambling laws. (These laws, along with those against unregulated trading of securities, are the primarily obstacles to prediction markets in the US.) IIRC, you can get into trouble with these rules if there is a payout on the outcome of a single event, which seems like it would be the case here. There's definite gray area, but before setting up such a thing one would definitely want to get some legal clarity.
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
I think right now EAs might be making a significant mistake by paying insufficient attention to the political realm. As EAs we tend to figure out what’s most impactful for us to work on and focus hard. That’s great! But there are various actions that are ‘non-delegatable’ - the extent to which an individual can do the action is limited (like voting, going to a protest, making hard money contributions to particular campaigns). It might be useful if we were all more in the habit of doing variou...
New Video from AI in Context: The Fall and Rise of Sam Altman
If you want to skip straight to the video, here it is!
AI in Context is excited to be back with our fourth video! For those just hearing from us, we make videos for 80,000 Hours, telling stories about transformative AI...
Based on reading through this, it seems promising - the question is whether it's worth someone overcoming the startup costs / bottlenecks to run a pilot, and I'm not sure it is unless there is a specific promising use case that makes this seem much better than current approaches. Any ideas for specific markets where this is valuable?
Some ideas:
1. CEA / Open Phil wants to contract [Metaculus/Manifold/Good Judgement/Epoch] to come up with forecasts for 100 continuous questions regarding clean meat. They just care that the forecasts are good, they don't care much for which specific methods are used. They also would like to see new forecasting players join the fray and compete for cost-effective work.
The put out an open call for proposals, and say that much of the money will come in the form of a simple Accuracy Agreement (one that perhaps can't be re-sold, for legal simplicity reasons, for now).
2. A government agency wants to outsource "Predictions of construction costs, for a megaproject". This set of predictions will be very expensive to produce and update. Instead of paying consultants in the traditional way, they organize much of this as an Accuracy Agreement.
3. In 2030, in a hypothetical future world, there are many big forecasting players and purchasers. Complicated forecasting Type Definitions are routinely created for things. Like, "For every US regional area, and every point in time for the next 5 years, how many of each disease will be monitored/registered in that area?". Forecasting purchasers can expect that a public bidding market will return a reasonable result, similar to some current large project bidding procedures now.
I see this approach as a more complex but powerful alternative to the model of buyers now just contracting with specific firms like Metaculus or Good Judgement, which has started to happen in the last few years (for judgemental questions).
One weakness that these Agreements have is that they require the client (or a third party) to ensure that questions are written and scored, instead of the consultant.
This is a similar issue that Prediction Markets have, but not one that existing forecasting contracts often have. These contracts often have the forecasting contractors do the work of question specification and resolution.
So, Accuracy Agreements are probably in-between Prediction Markets and current contractor agreements, in complexity.
For big government construction projects, I believe some firms/agencies will do a lot of preparation and outlining, before a bidding process might begin. Getting things specific enough for a large bidding process is itself a fair bit of work. This can be useful for large projects, or in cases where the public has little trust in the key decision makers, but is probably cost-prohibitive for other situations.