Summary:
- Prediction markets have big implications for governance and informed decision-making.
- Anybody can suggest questions on Metaculus. Questions that open and resolve on a semester schedule would make it easier to integrate this into the classroom.
- Teachers could design a course with 10-50% of the student’s grade based on prediction market success relative to their classmates.
A few teachers have toyed around with prediction markets in their classrooms (1, 2, 3). Predictions weren’t attached to student grades, and students were given at most a pittance for successful predictions. Evidence on whether it was fun or improved student scores on tests was mixed.
My feelings about these studies are also mixed. It’s great that the teachers tried something new, and some even ran an RCT to evaluate the effects on student learning and engagement (and honestly reported the negative findings!).
Yet the studies were sort of a sideshow, or extra, poorly-rewarded work for the students. The study that did the most careful job of setting up an experiment and collecting data also gave the survey evaluating student engagement immediately before finals, when students are at their most tired and anxious. It seems to me that this sort of study design sets prediction markets up to fail. As far as I can tell, no further studies attempting this have been published since 2013.
I’d love to see a teacher go out on a limb and run a class centered on participation in real-world prediction markets. Students could be graded on a curve based on their Brier score at the end of the class. This could be 10-50% of their overall grade, and researching predictions should comprise an equivalent proportion of the students’ workload. The phrase “prediction markets” should be in the title of the class, and the grading scheme in the course description, so that it selects for informed and eager students.
This would be a way to get students making and evaluating falsifiable predictions about real-world questions. The infrastructure needn’t be complex. Students could simply be assigned new email accounts at the start of class, which would then be used to create new Metaculus accounts. Their overall Metaculus score at the end of the term could be used to develop a curved grade for this portion of the class.
Metaculus has question categories for many subjects, not just the stereotypical politics-and-economics fare that dominates public discussion. Unfortunately, many of these resolve in the far future, such as “Will synthetic biological weapons infect 100 people by 2030?” Metaculus questions seem to be chosen for being exciting and interesting. I’d like to see Metaculus have many more questions that are resolved within weeks or months of being posted. One way to approach a class would be to design a suite of questions that would be tailored to the subject, input as Metaculus questions, and assigned to the students.
Any such project should put students’ engagement and education first. But if such a project was successful, it would benefit the prediction market world as well. It would add a direct incentive, and indirect financial incentive, to making accurate predictions on play-money prediction markets, avoiding running afoul of US regulations on gambling. It would increase the range of questions being posed, and spread familiarity with prediction markets amongst the student population. And if ten classes of 30 students were running prediction markets on Metaculus, that would represent a more than 100-fold increase in forecasters on many current questions.
College teachers in the USA often make in the range of $1,000-$5,000 per class. So offering a grant to a motivated teacher to run such an experimental class wouldn't be particularly expensive. Scrape together $10,000 and offer an advance market commitment to the first teacher to offer such a class.
There are 1.5 million college faculty in the USA, and about 20 million college students. The size of the world’s educational system is many times bigger than that. Somewhere out there, maybe this is already happening. But if you, like me, don’t have a teaching position that would allow you to try this idea out, you can still help lay the groundwork.
Near the start of every semester, consider the area of your own interest and expertise, and create and commit to maintaining 3 questions on Metaculus that resolve by the end of the semester. Ideally, the questions should be thematically relevant to a class. You could look up a particular class of interest at your current or former university for inspiration, or even contact a professor to ask for their ideas.
I'm going to stick with a qualitative argument for this idea. Like any other recommendation on how to best use one's time for altruistic purposes, it's going to fit the skills, resources, and inclinations of some people better than others. My EA argument is that it’s fast, easy, neglected, possibly fun, and particularly tractable for a huge portion of the workforce who currently don’t have any good career recommendations from 80,000 Hours.
Prediction is hard; it thus seems a bit cruel to grade students based on brier score, since it can be hard for some people to improve at that (also, over the course of one class you wouldn't get much time to iterate and improve at forecasting, so you'd be grading people in part on their newbie guesses early on). This would be like teaching an intro-to-economics class and then grading students based on whether their stock picks outperformed the S&P 500 -- too difficult a task for an intro course, and too much randomness to seem fair. Maybe grading heavily on Brier score could be appropriate in an advanced class like a grad-level course for students in DC-area universities who are aiming to become intelligence-agency analysts. And of course it would be fine to have Brier score in an intro course be worth only a token portion of one's grade, like 5%, just to make things fun.
But I like the general idea of a class based around learning about forecasting and prediction markets (or maybe as one unit in a larger class on economics, civics, or statistics). The idea of assigning students to create and participate in a metaculus market seems good; I just wouldn't grade based on brier score. Instead of using students' grades to motivate them, I'd split the class into teams, give each team a private group chat where they can discuss questions, and use people's competitive instincts (individual + team leaderboards) to create motivation.
In general I think university teachers miss out on many chances to have their students get involved in doing useful work; eg having ecology students contribute to the wikipedia entry for a particular tree species rather than just write a report on that species.
That’s why I suggested the prediction market would be based on a curve relative to one’s classmates :) I may go back and emphasize that point.