A Manifold Market "Leaked" the AI Extinction Statement and CAIS Wanted it Deleted

History may look back on CAIS’s AI risk statement as a pivotal moment IF we survive AI.

But, what if I told you that AI researchers “leaked” info about the statement on Manifold Markets over a week before it was published? This justifiably led to CAIS requesting the market to be deleted, concerned that it could damage the impact of the statement.

To provide some background, the statement was signed by the likes of Sam Altman, Demis Hassabis, Dario Amodei and other prominent AI figures. Short and with gravitas it read:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Imagine if the CEOs of major oil companies banded together in the 1900s and told their respective governments, “We need to be careful about building infrastructure relying heavily on fossil fuels. It could be catastrophic for the environment long term.”

This may seem like an extreme comparison to make, but the reality is leading AI companies consider the risk significant enough to compromise on profit and progression. Well, that’s what their words say, time will tell how it is reflected in corporate decision-making.

I had an opportunity to talk to some of the insider traders and Dan Hendrycks from CAIS, and hope you enjoy a breakdown of one of our craziest markets and what we’ve learnt from it.

Background on Manifold Markets

Prediction markets allow people who lack expertise about certain topics to form precise models of what the future could look like thanks to the live-updating probabilities that are generated by traders. Trades can buy YES or NO shares, which fluctuate in price depending on the current probability (similar to sports betting odds).

Manifold Markets uses play money, which leads some to express skepticism of its efficacy. However, data suggest Manifold Markets are incredibly accurate!

Here is a graph from an analysis of Manifold Market’s calibration carried out by Vincent Luczkow. This places markets into buckets based on their probability at the middle of their lifespan. It then looks at the percentage of markets in each bucket that resolve to yes. To be well-calibrated, you want markets in the 10% bucket at their mid-point to resolve yes 10% of the time and no 90% of the time (this ideal is represented by the green line on the graph). Pretty accurate! Also, check out our calibration page or our performance on the midterms.

Manifold Markets is unique from other prediction platforms as its markets are all user-generated questions. Users have used our markets to make predictions about everything you can think of from global nuclear risk to their personal romantic endeavours. The core denominator between these markets is the mechanism used to generate the probability that predicts an unknown future event.

But what happens when you defy this, and a market is created by someone that does know the future?

The Tale of the Statement on AI Risk

It all started on May 20th, when a user called Quinesweeper created 3 markets titled, “Will there be another well-recognized letter/statement on AI risk by May [June, July] 31, 2023?” Whenever I write, “the/this market”, I will be referring to the market which was asking if the statement would be released before the end of May.

I’m including a screenshot of the traders who won the most profit and an annotated graph below which will be referenced throughout.

Initial trading

All markets start at 50% when created, but this market was quickly bid down to 10% by traders who had no evidence that a major statement would be published in the next 10 days. However, within the first few hours, there was already upwards buying pressure. On the first day, the only person betting yes was Firstuserhere, whose trades can be seen marked by the yellow dots.

Although Firstuserhere works on AI, most users on our site know him for being an active Manifold user, so didn’t default to assuming he had insider info. It’s also worth noting that at this point in time, the market had very little liquidity. The spikes caused by Firstuserhere were a result of bets around 300 Mana, the equivalent of $3USD. Because of this traders continued to push the probability of the market back down each time Firstuserhere bought YES shares.

Things became more suspicious when Quinesweeper made his first trade marked by the red dots on May 21st. Now there were 2 individuals placing increasingly large orders on YES, one of whom was the market creator.

Users started commenting on the market, with weak speculation that there could be insider trading. But because this was Quinesweeper’s first market, others feared he was planning to load up on YES shares and fraudulently misresolve the market to yes regardless of the outcome. The Manifold team assured users that we would fix the resolution if it was resolved incorrectly so that users could trade on their true beliefs, and not have to factor into their probability the chance of fraud.

CAIS requests the market is deleted

CAIS privately reached out to Quinesweeper and requested he delete the market. They didn’t want to risk the letter being prematurely leaked which would diminish its effectiveness, particularly as they needed time to onboard signees and prepare for press release. A reasonable request, and we will revisit why we decided not to and what we’ve learnt from it later.

In response to this request, Quinesweeper sold his Yes shares (causing the dip marked by A).

However, the market’s visibility only increased from here, as at mark B, another user known as Hyperion started participating in the market with insider information. A week later after the AI statement was released, he said in Discord,

“From my PoV, I saw the letter, list of signees, and release date early and told @mkualquiera and @dmayhem93 about it on the 23rd. We all bet YES heavily, then the market seemingly exploded in popularity after the people betting it back down put it on the front page.

It was also pretty obvious to me even early on that some of the YES bettors were people in the AI sphere and the NO bettors were largely not, if you know who people are.

A lesson I would recommend learning is that many AI markets on Manifold are heavily insider traded..."

It is unclear to me whether CAIS requested the takedown of the market before or after this peak in activity, but regardless, although Quinesweeper had already exited the market at this point, he had yet to unlist the market and it soon became trending.

He did quickly unlist it (making it only accessible through having the URL) and reached out to Manifold Markets to ask whether the market could be deleted. Manifold responded by stating that “We don’t have a policy of deleting markets due to infosec reasons,” and suggested alternate ways to draw less attention to the market.

One of these suggestions included using creator privileges to hide comments to reduce the visibility of comments that might leak information. However, we did state that we don’t necessarily recommend doing this as it could inadvertently draw more attention to the market.

Quinesweeper decided that he did want to hide some of the comments as shown above. The content of hidden comments can still be viewed by clicking them. Quinesweeper also changed his name (it had previously been his real name).

Inevitably, the combination of all these actions just brought more attention to the market as users perceived it all to be very strange and began discussing it in our Discord server and sharing links to it.

This brought more attention to firstuserhere, who held the most YES shares and was most well-known amongst YES bettors.

When researching for this article, I spoke extensively with Firstuserhere. He explained to me that at this point in time, he had reason to suspect that there was a letter, but didn’t have any information about release dates. Thus, he felt comfortable stating he didn’t have insider info. He was being truthful about allocating a budget between the 3 markets with different time periods to learn how to spread his budget between them.

As the market gained traction, he began noticing people trading on it who he had previously worked with and knew were likely to have insider info in the AI space. He privately reached out to them and they were able to confirm his suspicions and the planned release date. People continued to assume that his original statement still held true and that he didn’t hold insider information, so he decided to stop commenting but continue to amass YES shares through large limit orders.

At this point, the Yes traders with insider info decided to do some damage control. It is not clear to me whether this was to encourage people to continue betting on NO so they could continue to buy more YES shares, or if they were actually concerned about a leak and were trying to protect CAIS.

Regardless, Hyperion’s trio began this copypasta in the comments. Others soon joined in and people thought it was just a meme to buy some YES shares and paste the copy pasta.

Turning the market into a meme

The strategy to decrease the visibility of the market had failed, so instead it was time to sow chaos and invalidate speculation of insider trading by turning everything into one big meme. And so, insider Mkualquiera and others began producing these banging memes.

It was here at point C that Mira decided to join the market and began market-making to capture value from the volatility of the market. This is done by setting up limit orders for both YES and NO shares (eg. buy NO at 60% and Yes at 40%). These shares cancel each other out, but you capture profit due to the difference in price between the shares at 40% and 60%. MayMeta, the number 3 profiter, had also been market-making the entire time and continued to do so until the final few hours when they switched to YES.

This led to the market converging to the Schelling point, aka 50%, which tends to happen in these big markets when no one has any idea what is going on. This isn’t just due to, “Oh we don’t know, bet it to 50%”, but rather the math behind the pricing of shares on the market.

Firstuserhere has also informed me that a few YES insiders agreed to not buy the market above 55% so they could conceal their continued accumulation of cheap shares. They really were working together to maximise how much profit they could make off of the ignorant NO bettors.

From what I can tell, turning the market into one big meme worked surprisingly well, and some users genuinely believed there was no statement, and that the only reason the probability was so high was “for the memes”.

Other users still suspected there was a letter and insider trading due to the strong signals from the YES traders prior to the memes. However, the release date was still obfuscated due to the creator making markets for June and July and suspected insiders buying shares in all 3.

This led our top trader and forecaster Jack, to buy NO shares on the May market but YES shares on the June and July markets following the Kelly criterion to size his bets. This led to him losing a sizable amount of mana, although he was able to recover a portion of it thanks to his YES shares on the other markets.

This is what Jack had to say in Discord after the market concluded,

“I was 95% sure there was inside info of some form on the market. I was fairly confident that firstuserhere was trading on a mix of gossip/rumor and deduction based on the more well informed insiders' trading. I also observed that firstuserhere was buying up to 90% on the June/July markets and only up to about 30-40% on the May market (until the last couple days when they bought up to 70%).

I therefore deduced that the timing of the letter was likely uncertain, and predicted accordingly. As of yesterday [May 29th] my prediction for by May, taking into account all information, was 55% and I made a Kelly bet accordingly. In retrospect I probably should have updated further up. 

Part of the problem for me trading on this prediction was that it was far easier to get large fills on my orders when buying NO than when buying YES. I also predicted 90-95% for by June/July, but unfortunately nobody was buying much NO over there.”

Jack then made a market on if he will believe the AI risk letter markets were priced rationally. He resolved this to 80%, reasoning that the June and July markets were reasonably priced, but the May market was underpriced. His final reflections can be read in the comments of the market linked above.

The Final day

The memes continued until resolution. However, a few traders did change their strategies towards the end. Mira was mostly profiting from market-making, but did also hold YES shares. However, at point D Mira decided to abandon their market-making strategy and went all-in on YES shares, investing over 150,000 mana worth of YES shares ($1,500). Firstuserhere later revealed that Mira privately confronted him before making this decision.

Katja Grace, a known writer who frequently discusses AI (check out her recent piece in the Time magazine!) entered the market with her first trade buying the market from 59% to 92%. Her bets were placed at 10:06am, whereas the statement was shared by CAIS on Twitter at 10:30am.

I reached out to her about whether she had insider information or if she was simply the fastest to react to the news, and this is what she told me:

“A combo—I had insider information (I signed the thing earlier), including having heard a rumor that it would be released at 2am [10:00 BST]. I stayed up to 2am refreshing the page, and as soon as I saw it become public I bet on it.”

And with that, the market came to an end. It was encouraging to see how seriously it was perceived compared to the previous letter. It headlined the news and has been a vital step in initiating talks within government bodies about AI risk. Knowing that thanks to our markets, several non-insiders were able to infer that the statement was going to be released 1-2 weeks before it actually did, was fascinating to see.

Should Manifold have taken down the market?

Quinesweeper first messaged Manifold on the 23rd, asking if the markets could “be deleted at the request of an organisation”.

Austin responded by informing him that we don’t have a policy for deleting markets for infosec reasons. Manifold Markets believes in radical transparency - almost all of our Notion work documents, salaries, and internal meeting notes are public.

CAIS never directly reached out to us and it was only after the statement was published that we learnt it was about them. I’m not sure what we would have done if we had the full-context and the request was made directly.

However, now that we have the full picture, I agree that if the market had caused the statement to prematurely enter the news cycle, it could have diminished its effect. The statement had been mistakenly indexed by Google, so was discoverable if someone was prompted to dig hard enough.

When reaching out to Dan Hendrycks, Executive & Research Director at CAIS, for comment he said,

“We were unhappy about the market, as it seemed to provide an unnecessary risk regarding the statement…

We felt it was crucial in having the time to prepare our media efforts. If the statement leaked early, then we might not have had a press release or statement prepared for the media…

As an example, sloppily rushing a book or an article would not improve epistemics. Likewise, rushing out the signatory letter would harm its overall effect. As such, even though the market would improve epistemics around whether a signatory letter would be released, we strongly feel that this would have reduced epistemics overall.”

That being said, under different circumstances this question COULD have had more upsides than downsides:

• If the probability was very low, it could prompt someone to start working on one.
• Building anticipation and hype resulting in a more prominent announcement.

Unfortunately in this case, the first point was null, as the market was created by an insider who already knew there was a statement in the works. The second point is a weaker one, particularly with Manifold’s current size.

A market’s inherent net value is only clear after it resolves (if even then), and thus for now Manifold likely will continue to not delete markets for infosec reasons. But, we haven’t had the chance to fully discuss this within the team, so don’t take this to be a hard stance.

Deleting a market is unprecedented, so even if we were convinced that the market does more harm than good, it probably wouldn’t be the solution due to the attention it draws.

What could have been asked instead?

Most people don’t know what information is actually valuable to know and thus the right questions to ask. I think this is often seen by hyper-fixation on the end result, and not having an understanding of the key factors that have the largest influence. One of the reasons forecasting the timeline and probability of extinction from AI is so challenging is that we aren’t even sure what some of the key factors that could cause/prevent it are. We need to be able to predict this, before even beginning to evaluate their respective probabilities and their effect on the end result’s probability! See the relevant market created by Eliezer Yudkowsky.

Here are some examples of questions with higher potential value when asked at an appropriate time. I’m sure someone with a firmer understanding of the AI space could think of even better ones!

• “Will the next AI statement make a significant positive impact?”
  • Nuanced resolution criteria would have to accompany this question for it to be useful. But, you can see how predicting this could help people decide whether working on a statement is an effective use of resources.
• “If another AI statement is released in May/June/July, will there be evidence it prompts discussion in the US government?”
  • This could help determine when the most impactful time to release a statement.
  • You could also make sub-questions such as: “If another AI statement is released in 2023 AND signed by Sam Altman, will there be evidence…” This could help determine who to prioritise to sign the statement.
• “Which organisation is most likely to release the next AI statement?”
  • This could help people coordinate without necessarily revealing sensitive information.
• “Will the next AI statement mention X thing?”
  • Help spark discussion on what is worth mentioning, with people predicting accordingly what they believe the leaders in the space will decide is valuable to include.

When maximising the value of a question, you should ask yourself two things:

• Is the information I learn from this actionable in some way?

And if not:

• Will this provide informative data on something that is being asked/discussed that is important for people to perceive correctly?

I think most markets do fulfil point 2 in some way, but the extent of which is quite opaque.

Conclusion

First off, I want to apologise on behalf of the Manifold team to CAIS for any stress our markets may have caused.

Even though prediction markets hold a wealth of untapped potential, we need to be careful to avoid pitfalls. Users creating markets shouldn’t overlook the types of impact their market may have, and ultimately, the Manifold team is responsible to make sure users aren’t perversely incentivised. If a market is created because AI researchers want to pretend to be politicians’ spouses for a day, then there is room for improvement!

That being said, the vast majority of our markets are created with good incentives, and it still astounds me that we can conjure accurate probabilities about most future events so easily using prediction markets.

Here are a few examples you should check out:

• Raise awareness by providing a grounded, quantified figure. After reading other media which leans into uncertainty and hypes up worrying possibilities, it’s always refreshing to visit Manifold to see a realistic outside view. Although, in the case of AI extinction perhaps less so…

• “Personal” market that provides helpful information to the creator to help with managing planning/expectations. Can also act as additional incentive and accountability in some cases.

• Provides an outside view to improve people’s understanding of the world.

• It would be remiss of me to not include this market, which just resolved Yes last Friday after one of our founders got married (they were single when Manifold first started 1.5 years ago)!

The net value each individual market provides is often hard to foresee, but the fact we are already at a stage where their potential for impact warrants vigilance is a bullish signal. We will continue to work to improve epistemic practices, build a community that facilitates thinking in probabilities, and make forecasting accessible and fun to the average person.

If you felt inspired by this article to create some questions for your own organisation, but want to do so in a contained environment to minimise risk, then check out our recently released feature - private groups!

Thanks for reading ^^