I am doing an Ask Me Anything. Work and other time constraints permitting, I intend to start answering questions on Sunday, 2020/07/05 12:01PM PDT.
__________________________
I am Top 20 (currently #11) out of 1000+ on covid-19 questions on the amateur forecasting website Metaculus. I also do fairly well on other prediction tournaments, and my guess is that my thoughts have a fair amount of respect in the nascent amateur forecasting space. Note that I am not a professional epidemiologist and have very little training in epidemiology and adjacent fields, and there are bound to be considerations I will inevitably miss as an amateur forecaster.
I also do forecasting semi-professionally, though I will not be answering questions related to work. Other than forecasting, my past hobbies and experiences include undergrad in economics and mathematics, a data science internship in the early days of Impossible Foods (a plant-based meats company), software engineering at Google, running the largest utilitarian memes page on Facebook, various EA meetups and outreach projects, long-form interviews of EAs on Huffington Post, lots of random thoughts on EA questions, and at one point being near the top of several obscure games.
For this AMA, I am most excited about answering high-level questions/reflections on forecasting (eg, what EAs get wrong about forecasting, my own past mistakes, outside views and/or expert deference, limits of judgmental forecasting, limits of expertise, why log-loss is not always the best metric, calibration, analogies between human forecasting and ML, why pure accuracy is overrated, the future of forecasting...), rather than doing object-level forecasts.
I am also excited to talk about interests unrelated to forecasting or covid-19. In general, you can ask me anything, though I might not be able to answer everything. All opinions are, of course, my own, and do not represent those of past, current or future employers.
Most of the forecasting work covered in Expert Political Judgement and Superforecasting related to questions with time horizons of 1-6 months. It doesn't seem like we know much about the feasibility or usefulness of forecasting on longer timescales. Do you think longer-range forecasting, e.g. on timescales relevant to existential risk, is feasible? Do you think it's useful now, or do you think we need to do more research on how to make these forecasts first?
What do you think helps make you a better forecaster than the other 989+ people?
What do you think makes the other ~10 people a better forecaster than you?
Hey I want to give a more directly informative answer later but since this might color other people's questions too: I just want to flag that I don't think I'm a better forecaster than all the 989+ people below me on the leaderboards, and I also would not be surprised if I'm better than some of the people above me on the leaderboard. There's several reasons for this:
- Reality is often underpowered. While medium-term covid-19 forecasting is less prone to those issues in comparison to many other EA questions, you still have a bunch of fundamental uncertainty about how actually good you are. Being correct for one question often relies on a "bet" that's loosely correlated with being correct on another question. At or near the top, there are not enough questions for you to be sure if you just got lucky in a bunch of correlated ways that others slightly below you in the ranks got unlucky on, vs you actually being more skilled. The differences are things like whether you "called" it correctly at 90% when others put 80%, or conversely when you were sufficiently calibrated at 70% when others were overconfident (or just unlucky) at 90%.
- Metacul
... (read more)This was a lot of good discussion of epistemics, and I highly valued that, but I was also hoping for some hot forecasting tips. ;) I'll try asking the question differently.
I'll instead answer this as:
- I probably answered more questions than most of them.
- I update my forecasts more quickly than most of them, particularly in March and April
- Activity has consistently been shown to be one of (often, the) strongest predictors of overall accuracy in the academic literature.
- I suspect I have a much stronger intuitive sense of probability/calibration.
- For example, 17% (1:5) intuitively feels very different to me than 20% (1:4), and my sense is that this isn't too common
- This could just be arrogance however, there isn't enough data for me to actually check this for actual predictions (as opposed to just calibration games)
- I feel like I actually have lower epistemic humility compared to most forecasters who are top 100 or so on Metaculus. "Epistemic humility" defined narrowly as "willingness to make updates based on arguments I don't find internally plausible just because others believed them."
- Caveat is that I'm making this comparison solely to top X% (in
... (read more)1.) This is amazing, thank you. Strongly upvoted - I learned a lot.
2.) Can we have an AMA with JGalt where he teaches us how to read all the news?
Non-forecasting question: have you ever felt like an outsider in any of the communities you consider yourself to be a part of?
Yes, I think part of feeling like you don't belong is just pretty normal to being human! So on the outside, this should very much be expected.
But specifically:
- I think of myself as very culturally Americanized (or perhaps more accurately EA + general "Western internet culture"), so I don't really feel like I belong among Chinese people anymore. However, I also have a heavy (Chinese) accent, so I think I'm not usually seen as "one of them" among Americans, or perhaps Westerners in general.
- I mitigate this a lot by hanging out in a largely international group. I also strongly prefer written communication to talking, especially to strangers, likely to a large part because of this reason (but it's usually not conscious).
- I also keep meaning to train my accent, but get lazy about it.
- I think most EAs are natives to, for want of a better word, "elite culture":
- eg they went to elite high schools and universities,
- they may have done regular travel in the summers,
- most but not all of them had rich or upper-class parents even by Western standards
- Some of them go to burning man and take recreational drugs
- Some of them are much more naturally comforta
... (read more)Is forecasting plausibly a high-value use of one's time if one is a top-5% or top-1% forecaster?
What are the most important/valuable questions or forecasting tournaments for top forecasters to forecast or participate in? Are they likely questions/tournaments that will happen at a later time (e.g. during a future pandemic)? If so, how valuable is it to become a top forecaster and establish a track record of being a top forecaster ahead of time?
Here's a ton of questions pick your favourites to answer. What's your typical forecasting workflow like? Subquestions:
Do you tend to make guesstimate/elicit/other models, or mostly go qualitative? If this differs for different kinds of questions, how?
How long do you spend on initial forecasts and how long on updates? (Per question and per update would both be interesting)
Do you adjust towards the community median and if so how/why?
More general forecasting:
What's the most important piece of advice for new forecasters that isn't contained in Tetlock's superforecasting?
Do you forecast everyday things in your own life other than Twitter followers?
What unresolved question are you furthest from the community median on?
A botched Tolstoy quote comes to mind:
Of course that's not literally true. But when I reflect on my various mistakes, it's hard to find a true pattern. To the extent there is one, I'm guessing that the highest-order bit is that many of my mistakes are emotional rather than technical. For example,
If the emotion hypothesis is true, to get better at forecasting, the most important thing might well to be looking inwards, rather than say, a) learning more statistics or b) acquiring more facts about the "real world."
What do EAs get wrong about forecasting?
I think the biggest is that EAs (definitely including myself before I started forecasting!) often underestimate the degree to which judgmental forecasting is very much a nascent, pre-paradigm field. This has a lot of knock-on effects, including but not limited to:
- Thinking that the final word on forecasting is the judgmental forecasting literature
- For example, the forecasting research/literature is focused entirely on accuracy, which has its pitfalls.
- There are many fields of human study that does things like forecasting, even if it's not always called that, including but not limited to:
- Weather forecasting (where Brier score came from!)
- Intelligence analysis
- Data science
- Statistics
- Finance
- some types of consulting
- insurance/reinsurance
- epidemiology
- ...
- More broadly, any quantified science needs to make testable predictions
- Over-estimating how much superforecasters "have it figured out"
- eg here on calibration precision.
- Relatedly, overestimating how much other good forecasters/aggregation platforms have things figured out.
- For example, I think some people over-estimate the added accuracy of prediction markets like PredictIt, or aggregation engines like Metaculus/GJO, or that of top
... (read more)so you've done quite a few different things - right now, would you rather go into research, or entrepreneurship, and why?
I would like to hear your thoughts on Generalist vs Specialist debate.
Hmm this doesn't answer any of your questions directly, but might be helpful context to set: My impression is that relatively few people actually set out to become generalists! I think it's more accurate of an explanation to think of some people being willing to do what needs to get done (or doing things they find interesting, or has high exploration value, or a myriad of other reasons). And if those things keep seeming like highly impactful things to do (or continues to be interesting, has high learning/exploration value, etc), they keep doing them, and then eventually become specialists in that domain.
If this impression is correct, specialists start off as generalists who eventually specialize more and more, though when they start specializing might vary a lot (Some people continue to be excited about the first thing they tried, so are set on their life path by the time they were 12. Others might have tried 30 different things before settling on the right one).
(I obviously can't speak for other EAs; these are just my own vague impressions. Don't take it too seriously, etc)
Hmm, I don't feel too strongly about this... (read more)
I vaguely recall hearing something like 'the skill of developing the right questions to pose in forecasting tournaments is more important than the skill of making accurate forecasts on those questions.' What are your thoughts on this and the value of developing questions to pose to forecasters?
Can you give your reflections on the limits of expertise?
Relatedly, on the nature of expertise. What's the relative importance of domain-specific knowledge and domain-general forecasting abilities (and which facets of those are most important)?
What should a typical EA who is informed on the standard forecasting advice do if they actually want to become good at forecasting? What did you do to hone your skill?
My guess is to just forecast a lot! The most important part is probably just practicing a lot and evaluating how well you did.
Beyond that, my instinct is that the closer you can get to deliberate practice the more you can improve. My guess is that there's multiple desiderata that's hard to satisfy all at once, so you do have to make some tradeoffs between them.
In case you're not aware of t... (read more)
How many Twitter followers will you have next week?
Why is pure accuracy overrated?
Are you using or do you plan to use your forecasting skills for investing?
Thanks for the answer. Makes sense!
Some possible insight: the NASDAQ is doing even better, at its all-time high and wasn't hit as hard initially, and the equal-weight S&P 500 is doing worse than the regular S&P 500 (which weights based on market cap), so this tells me that disproportionately large companies (and tech companies) are still growing pretty fast. Some of these companies may even have benefitted in some ways, like Amazon (online shopping and streaming) and Netflix (streaming).
20% of the S&P 500 is Microsoft, Apple, Amazon, Facebook and Google. Only Google is still down since February at their peaks before the crash, the rest are up 5-15%, other than Amazon (4% of the S&P 500), which is up 40%!
Say an expert (or a prediction market median) is much stronger than you, but you have a strong inside view. What's your thought process for validating it? What's your thought process if you choose to defer?
Lots of EAs seem pretty excited about forecasting, and especially how it might be applied to help assess the value of existential risk projects. Do you think forecasting is underrated or overrated in the EA community?
Good forecasts seem kind of like a public good to me: valuable to the world, but costly to produce and the forecaster doesn't benefit much personally. What motivates you to spend time forecasting?
When I look at most forecasting questions, they seem goodharty in a very strong sense. For example, the goodhart tower for COVID might look something like:
1. How hard should I quarantine?
2. How hard I should quarantine is affected by how "bad" COVID will be.
3. How "bad" COVID should be caches out into something like "how many people", "when vaccine coming", "what is death rate", etc.
By the time something I care about becomes specific enough to be predictable/forecastable, it seems like most of the thing I actually cared about has been lost.
Do you have a sense of how questions can be better constructed to lose less of the thing that might have inspired the question?
Meta: Wow, thanks a lot for these questions. They're very insightful and have made me think a lot, please keep the questions (and voting on them) coming! <3
It turns out I had some prior social commitments on Sunday that I forgot about, so I'm going to start answering these questions tonight plus Saturday, and maybe Friday evening too.
But *please* don't feel discouraged from continuing to ask questions, reading these questions have been a load of fun and I might keep answering things for a while.
What do you think you do that other forecasters don't do?
What news sites, data sources, and/or experts have you found to be most helpful for informing your forecasts on COVID-19?
For Covid-19 spread, what seems to be the relative importance of: 1) climate, 2) behaviour, and 3) seroprevalence?
Forecast your win probability in a fight against:
500 horses, each with the mass of an average duck.
1 duck, with the mass of an average horse.
(numbers chosen so mass is roughly equal)
How important do you think it is that your or others' forecasts are more well-understood or valued among policy-makers? And if you think they should listen to forecasts more often, how do you think we should go about making them more aware?
I'm very motivated to make accurate decisions about when it will be safe for me to see the people I love again. I'm in Hong Kong and they're in the UK, though I'm sure readers will prefer generalizable stuff. Do you have any recommendations about how I can accurately make this judgement, and who or what I should follow to keep it up to date?
As someone with some fuzzy reasons to believe in their own judgement, but little explicit evidence of whether I would be good at forecasting or not, what advice do you have for figuring out if I would be good at it, and how much do you think it's worth focusing on?
How much time do you spend forecasting? (Both explicitly forecasting on Metaculus and maybe implicitly doing things related to forecasting, though the latter I suspect is currently a full-time job for you?)
How optimistic about "amplification" forecast schemes, where forecasters answer questions like "will a panel of experts say <answer> when considering <question> in <n> years?"
I've recently gotten into forecasting and have also been a strategy game addict enthusiast at several points in my life. I'm curious about your thoughts on the links between the two:
What were your reasons for getting more involved in forecasting?
Hi Linch! So what's up with the Utilitarian Memes page? Can you tell more about it? Any deep lessons from utilitarian memes?
Do you think people who are bad at forecasting or related skills (e.g. calibration) should try to become mediocre at it? (Do you think people who are mediocre should try to become decent but not great? etc.)
What's your process like for tackling a forecast?
Do you think forecasting has a place in improving the decision making in business?
How much time do you spend on forecasting, including researching the topics?
Forecasting has become slightly prestigious in my social circle. At current margins of forecastingness, this seems like a good thing. Do you predict much corruption or waste if the hobby got much more prestigious than it currently is? This question is not precise and comes from a soup of vaguely-related imagery.
In what meaningful ways can forecasting questions be categorized?
This is really broad, but one possible categorization might be questions that have inside view predictions versus questions that have outside view predictions.
I will forecast a personal question for you e.g. "How many new friends will I make this year?" What do you want to ask me?
Thanks for doing this AMA! In case you still might answer questions, I'm curious as to how much value you think there'd be in:
E.g., if someone asked you for advice on whether to do work in academia similar to Tetlock's work, or build things like Metaculus or calibration games, or do something else EAs often think is valuable, what might you say?
(I ask in part because you wrote about judgemental forecasting being "ve
... (read more)Often times, to me it seems, machine learning models reveal solutions or insights that, while researchers may have known them already, are actually closely linked to the problem it's modelling. In your experience, does this happen often with ML? If so, does that mean ML is a very good tool to use in Effective Altruism? If not, then where exactly does this tendency come from?
(As an example of this 'tendency', this study used neural networks to find that estrogen exposure and folate deficiency were closely correlated to breast cancer. Source: https://www.sciencedirect.com/science/article/abs/pii/S0378111916000706 )
Which types of forecasting questions do you like / dislike more?