I do research on the philosophy and psychology of rationality and well-being in a PhD project that is part of the interdisciplinary research project Modeling Human Happiness at the University of Oslo. I also make technologies to improve learning and reasoning through Disputas, a tech-startup I co-founded. See my website here: https://paalkvarberg.com/
Good question! There were many differences between the approaches by FFI and the GJP. One of them is that no superforecasters were selected and grouped in the FFI tournament.
Here is google's translation of a relevant passage: "In FFI's tournament, the super forecasters consist of the 60 best participants overall. FFI's tournament was not conducted one year at a time, but over three consecutive years, where many of the questions were not decided during the current year and the participants were not divided into experimental groups. It is therefore not appropriate to identify new groups of super forecasters along the way" (2022, 168). You can translate the entirety of 5.4 here for further clarification on how Beadle defines superforecasters in the FFI tournament.
Thank you for this thoughtful post! I am just about to start a PhD in philosophy on the psychology and metaethics of well-being, so I am fairly familiar with the research literature on this topic in particular. I totally agree with you that foundational issues should be investigated more deeply in EA circles. To me, it is baffling that there is so little discussion of meta-ethics and the grounds for central propositions to EA.
You are right that many philosophers, including some who write about method, think that ethics is about weighing intuitions in reflective equilibrium. However, I think that it is seriously misleading to state this as if it is an undisputed truth (like you do with the quote from Michael Plant). In the 2020 Philpapers survey, I think about half of respondents thought intuitions-based philosophy was the most important. However, the most cited authors that do contemporary work in philosophical methodology eg. Timothy Williamson and Herman Cappelen, dont think intuitions plays important roles at all, at least not if intuitions are thought to be distinct from beliefs.
I think that all the ideas you mention concerning how to move forward look very promising. I would just add "explore meta-ethics", and in particular "non-intuitionism in ethics". I think that there are several active research programs that might help us determine what matters, without relying on intuitions in a non-critical way. I would especially recommend Peter Railton's project. I have also written about this and I am going to write about it in my PhD project. I would be happy to talk about it anytime!
Hey Harrison! This post evaded me until now. I am sorry to hear that you are not going to continue working on this. I hope that you will still follow progress of other projects, and that you will chime in with feedback when some of the people working on similar things post about it i the future!
Thank you for this! I think that there are several paths to impact for a scaled up version of this, but I am not at all sure what path is most consequential. I am curious about what you think is the most important way evaluations of this sort can have an impact.
Thank you for this! I think your framework for instructional design is likely to be very useful to several projects working to create educational content about EA. I happen to be one of these people, and would love to get in touch. Here is a onepager about the project I am currently pursuing. I shared your post with others who might fint it interesting.
I look forward to seeing what you decide to do next!
I participated in an activity of this sort some years ago. I really enjoyed the structured conversation, and working towards consensus in a group. The experience was way more intense than any other context of presentation or debate that I have been a part of otherwise. I don't know whether EA groups should use the technique, but I wanted to share from my own experience:)
Thanks for writing up this idea in such a succinct and forceful way. I think the idea is good, and would like to help any way I can. However, I would encourage thinking a lot about the first part "If we get the EA community to use a lot of these", which I think might be the hardest part.
I think that there are many ways to do something like this, and that it's worth thinking very carefully about details before starting to build. The idea is old, and there is a big graveyard of projects aiming for the same goal. That being said, I think a project of this sort has amazing upsides. There are many smart people working on this idea, or very similar ideas right now, and I am confident that something like this is going to happen at some point.
Metaculus is also currently working on a similar idea (causal graphs). Here are some more people who are thinking or working on related ideas, (who might also appreciate your post): Adam Binks, Harrison Durland, David Manheim and Arieh Englander (see their MTAIR project).
Seems like I forgot to change "last updated 04.january 2021" to "last updated 04. january 2022" when I made changes in january haha.
I am still working on this. I agree with Ozzie's comment below that doing a small part of this well is the best way to make progress. We are currently looking at the UX part of things. As I describe under this heading in the doc, I don't think it is feasible to expect many non-expert forecasters to enter a platform to give their credences on claims. And the expert forecasters are, as Ian mentions below, in short supply. Therefore, we are trying to make it easier to give credences on issues while reading about them the same place you read about them. I tested this idea out in a small experiment this fall (with google docs), and it does seem like motivated people who would not enter prediction platforms to forecast issues might give their takes if elicited this way. Right now we are investigating this idea further through an mvp of a browser extension that lets users give credences on claims found in texts on the web. We will experiment some more with this during the fall. A more tractable version of the long doc is likely to appear at the forum at some point.
I'm not wedded to the concrete ideas presented in the doc, I just happen to think they are good ways to move closer to the grand vision. I'd be happy to help any project moving in that direction:)
Yes, the 60 FFI supers were selected and evaluated on the same 150 questions (Beadle, 2022, 169-170). Beadle also identified the top 100 forecasters based on the first 25 questions, and evaluated their performance on the basis of the remaining 125 questions to see if their accuracy was stable over time, or due to luck. Similarly to the GJP studies, he found that they were consistent over time (Beadle, 2022, 128-131).
I should note that I have not studied the report very thoroughly, so I may be mistaken about this. I'll have a closer look when I have the time and correct the answer above if it is wrong!