Austin

Cofounder @ Manifold Markets
636San Francisco, CA, USAJoined Jul 2020

Bio

Hey there~ I'm Austin, currently building https://manifold.markets. Always happy to meet people; reach out at akrolsmir@gmail.com, or find a time on https://calendly.com/austinchen/manifold !

Comments
76

Manifold Markets ran a prediction tournament to see whether forecaster would be able to predict the winners! For each Cause Exploration Prize entry, we had a market on "Will this entry win first or second place?". Check out the tournament rules and view all predictions here.

I think overall, the markets did okay -- they managed to get the first place entry ("Organophosphate pesticides and other neurotoxicants") as the highest % to win, and one of the other winners was ranked 4th ("Violence against women and girls").  However, they did miss out on the two dark horse winners ("Sickle cell disease" and "shareholder activism"), which could have been one hypothetical way markets would outperform karma. Specifically, none of the Manifold forecasters placed a positive YES bet on either of the dark horse candidates.

 

I'm not sure that the markets were much better predictors than just EA Forum Karma -- and it's possible that most of the signal from the markets were just forecasters incorporating EA Forum Karma into their predictions. The top 10 predictions by Karma also had 2 of the 1st/2nd place winners:

And if you include honorable mentions in the analysis, EA Forum Karma actually did somewhat better. Manifold Markets had 7/10 "winners" (first/second/honorable), while EA Forum Karma had 9/10.

Thanks again for the team at OpenPhil (especially Chris and Aaron) for hosting these prizes and thereby sponsoring so many great essays! Would love to see that writeup about learnings, especially curious what the decision process was that lead to these winners and honorable mentions.

I think anime/gaming expos/conventions might be a good example actually - in those events, the density of high quality people is less important than just "open for anyone who's interested to come". Like, organizers will try to have speakers and guests lined up who are established/legit, but 98% of the people visiting are just fans of anime who want to talk to other fans.

Notably, it's not where industry experts converge to do productive work on creating things, or do 1:1s; but they sure do take advantage of cons and expos to market their new work to audiences. By analogy, a much larger EA Expo would have the advantage of promoting the newest ideas to a wider subset of the movement.

Plus, you get really cool emergent dynamics when the audience size is 10x'd. For example, if there are a 1-2 people in 1000  who enjoy creating EA art, at 10000 people you can have 10-20 of them get together and meetup and talk to each other

Also re funding -- obviously, super super biased here but I think something like "experimentation is good", "the amount of EA money that's been spent on prediction markets is quite low overall, in the single digit millions" and "it's especially unclear where the money would be better spent"

Prediction pools (like Metaculus-style systems) are maybe the solution I'm most aware of in this space, and I think executing on these could also be quite valuable; if you have good proposals on how to get better forecasts about the future, I think a lot of people would happily fund those~

FWIW my strongest criticism of prediction markets might look something like "Prediction Markets are very general purpose tools, and there's been a lot of excitement about them from a technocratic perspective, but much less success at integrating them into making better decisions or providing novel information, especially relative to the counterfactual of eg paying forecasters or just making random guesses"

Austin from Manifold here - thanks for this writeup! I do actually agree with your core thesis (something like "prediction markets get a lot of hype relative to how much value they've added"), though I think your specific points are not as convincing?

  • 1 Re: long-run reliability, this is actually a thing we think a lot about. Manifold has introduced different loan schemes such that the cost of capital of investing in a long-term market is lower, but I could imagine better market structures or derivatives that correctly get people to bet on important longterm questions
  • 2 The existence of free money is worth noting, and points out some limits of prediction markets:
    • Markets which don't get enough attention will be less accurate
    • Markets without enough liquidity (fewer traders, less $ traded) will be less accurate
    • Efficient Market Hypothesis isn't unilaterally true - markets exist on a spectrum of efficiency, and simply setting up a "market" doesn't magically make the prices/predictions good
  • That said, "hey look, these markets are clearly wrong" is painting prediction markets with an overtly broad brush, and might lead you to miss out on markets that are actually valuable. By analogy, you wouldn't hold up  a random antivaxxer's tweet as proof that all of Twitter is worthless; rather, you should think that the context and the people who are producing the tweet or market actually make a difference
  • 3 The asymmetric payout for being right in Doom scenarios has been discussed eg in this bet between Yudkowsky and Caplan. I think this is simultaneously true, and also not super relevant in practice, since it turns out the motivation (at least in Manifold markets) is often closer to "I want to make this number correct, either for altruistic info-providing reasons, or for egotistical show people I was right reasons, not completely rational bankroll-maximizing cost-benefit analyses"

+100 on this. I think the screening processes for these conferences overweight legible in-groupy accomplishments like organizing an EA group in your local town/college, and underweights regular impressive people like startup founders who are EA-curious -- and this is really really bad for movement diversity.

Yes, I might be salty because I was rejected from both EAG London and Future Forum this year... 

But I also think the bar for me introducing friends to EA-curious is higher, because there isn't a cool thing I can invite them into. Anime conventions such as Anime Expo or Crunchyroll Expo are the opposite of this - everyone is welcome, bring your friends, have a good time -- and it works out quite well for keeping people interested in the subject.

Thanks for writing this up! Really appreciate the clear and transparent writeup across hiring, output, and financial numbers, and think that more orgs (including Manifold!) should strive for this level of clarity. One thing I would have been curious to see is how much money came in from each funding source, haha.

I set up a prediction market to see how RP will do against its funding goals:

Half-baked draft that has been sitting around for a while: https://blog.austn.io/posts/why-you-should-switch-from-google-docs-to-notion

I would spend more time on this cause if I felt that EA orgs actually would listen to me on this point. And I actually do pitch this quite often to when I talk to leaders at EA orgs, haha.

Yes, Notion has been around for 6 years and has raised hundreds of millions in funding: https://www.crunchbase.com/organization/notion-so

Load More