If you have one opinion, and the prevailing experts have a different opinion, should you assume that you’re incorrect? And if so, how can you determine who’s an expert, and whether or not you count as one yourself? In this whiteboard discussion from Effective Altruism Global 2018: San Francisco, Gregory Lewis and Oliver Habryka offer their contrasting perspectives. 

Below is a transcript Gregory and Oliver's talk, which we have lightly edited for clarity. You can also watch it on YouTube and read it on effectivealtruism.org.

The Talk

Gregory: Probably what we are going to try and do is, I'm going to explain to you why I'm right and then… I'm joking. I'm going to probably set up my stool for the discussion about whether or not to defer when someone's come in the community to discuss epistemic modesty. I guess Ollie will then…

Oliver: Bring my perspective into it, contrast it a bit, and that will be a starting point for the whole discussion.

Gregory: By way of introduction, there's many cases where deferring to what other people think, or people are who are maybe more informed than you think, is pretty inarguable and pretty obvious. I think that's common knowledge amongst most people here. A common example given is, you know, if you're taking your bins out or taking the trash out on a particular day of a week and you see none of your neighbors have left their trash out on the same day of a week, you probably think like, "I've made a mistake here" rather than "everyone else's made a mistake." And then change your mind because of that. And likely if you read a textbook and you see something in that it looks a little bit surprising to you it's probably more likely that you just don't understand it rather than the textbook makes a mistake. That doesn't mean textbooks are infallible, but I mean you should probably like… if you have to guess which one's right, you or the textbook, you should probably guess the textbook in most cases.

So that's all the really inarguable stuff, which I think Ollie is going to be arguing against very vociferously for the next hour. Alas for me. But where it gets a little more complicated, there's lots of areas where we do often take ourselves or license ourselves to disagree. So like in politics, in various complicated factual matters, in religion and all sorts of other things besides. And there we think, even if they're experts, we still think we are within our rights epistemically to go actually these guys are wrong and we're right. I guess my view on this is like, I'm probably like a modesty radical, immodestly modest, you might say, insofar as I think, in essence, what you should always do in each possible condition of things is you should hew your own view towards an idealized expert consensus. So you weigh up different expert classes, work out which ones are more relevant, and do all of that stuff. And then you get an answer based on that.

And that's what you should think is actually true. So your credence should be driven by this, and your own personal impression of where the balance of reason lies counts in most cases for virtually nothing. It counts no more than another person similarly well informed as you, coming to the view they have. And so that typically means in my day to day life, I end up sort of deferring quite a lot. And so, I don't have a particularly strong political views or like, hot button topics because the fact there is controversy means I probably shouldn't be very confident that one side is right or the other and so on and so forth. I guess the EA relevance of this is in many cases, EAs do take views which like outside side or often against expert consensus, and how often is that a reasonable thing to do or not reasonable? And obviously given my own ideology here, I tend to be leaning in favor of thinking that the EA community should be a lot more cautious and more deferential to existing bodies of expertise. So that's like me as it were setting out my stool. I look forward to Ollie pelting me with rotten fruit, as appropriate. I'll hand over to him for a bit.

Oliver: I think ultimately a lot of what Greg said is right. I think there is an important sense in which it is obviously really important to take in due account other people's epistemic states and other people's observations, and their current opinions in forming your own opinions. But I think there are various problems with trying to do this naively. I think the perspective that I'm taking, is not necessarily have something deeply flawed with something like the complete, philosophically rigorous view of what Greg is saying, but more something where I'm saying that if you do naively the thing that Greg recommends people do, you will end up running into various problems, and I think these problems are kind of illustrated by three of the topics we hope to cover today. There is the question of who are you going to count as an expert, which is I think is underlying just if I'm going to go by expert consensus there is this key question of, who I will let trust as an expert? In questions of religion should I trust theologians? They seem to have thought most about it. Does mean that God definitely exists given that most theologians seem to think God exists? But then we are like, ah, maybe philosophers. So we have this interesting question of who do we trust as our experts?

Similar, a related question is when are you yourself an expert? At what point do you cross the threshold to think that you are the world expert on a relevant topic? Maybe you are the person who has done the most research in the world of all the people that exist into a certain form of quantum mechanics or in certain questions about online discussion platform design (which I hope I have achieved). And in those situations, I think that is just a question where I expect where if you do the modest thing, and if you kind of carry forward the attitude of modesty into that situation, you will usually make bad decisions.

And third of all, there are important second order effects, where if you have a group of people who are generally primarily making decisions based on other people's opinions, you run into many, many information cascade problems that we often see in markets, often see in various real life situations, panics, where everyone else is primarily paying attention to what everyone else does. And then you get stuff like bystander effect, you get stampede and stuff like that. And that is a problem that I think can be avoided with very rigorous and careful thinking about how other people came to their opinions. But it's something that if you're going to naively take the modesty stance, I think you're going to be on average doing worst than I expect most people in this room to be doing. Okay. I think those are my three big perspectives and objections that I have with the broader framework because I think philosophically, I agree with a lot of things that Greg says, but I think the question is more about, how are we going to actually implement these insights about epistemology and understanding.

Gregory: Can we go with the first things first, maybe? So the question of what counts as expertise. Right? And so I often, in fact, I've written at length, at great length on modesty online. My idea is, I often like to use is that maybe somewhat loosely, subject matter expertise is basically the same thing as expertise per se. So to answer "is it good to have a death penalty" look for people who work in criminal justice, is a good to increase or decrease minimum wage, maybe we should talk to economists, I say as one enters the room just now. If you want to know if you're sick or not maybe consult a physician. I think there were many cases where people would agree that's a useful first pass heuristic.

But there's a key question as to, in many cases it seems like this often co-locates with social status. So I used to be a doctor, doctors have this quite good reputation, present company notwithstanding, and that means that you're taken maybe more seriously than you deserve to be. There are like lots of stories actually in medicine going further where people are overly deferential to doctors who are doing really stupid things. Or like the nurse, who is not in the same high status profession the doctor is, sort of let people like me make terrible mistakes and cause damage. So that's obviously like where it can go wrong and definitely what I would want to avoid.

The question is we have some data, some interesting data Ollie wanted to cover. So I'm stealing his thunder, on where very simple scoring rules can beat commercial expertise. So there was like reason rules from Kahneman, there's prediction markets and all of this stuff. And so it seems you don't necessarily need lots of subject matter knowledge, beyond being an academic, being a professor, to actually get better answers than the supposed expert class. Do you want to elaborate more on that, so I'm not making your point before you can do an even better job of what I'm attempting to do.

Oliver: I think like one of the most interesting questions that illustrates this most is the super forecasting literature, where you have this interesting thing where you have the super forecasters trying to make judgments about global politics, trying to make judgments about technological advances. And those are people without subject matter expertise. They are people who usually are generally well read, and can do Wikipedia searches, and then just try to apply kind of good forms of… good rules of reasoning, that are pretty domain general, like they don't really depend on a specific domain. And in the super forecasting literature, that has shown to very often outperform experts. And now you have this problem where like okay we now… we have the experts are forecasting, which we can think of as kind of the experts at answering certain forms of predictive questions. And we have domain experts who are hopefully good at answering domain expertise questions. But when domains widows intersect, which is quite a bit obviously, we are kind of stuck if we try to do naive expert aggregation.

Gregory: I think it's worth… maybe just on the second point. So it's obviously the case that if you're trying to defer in the epistemic sense, your criteria for expertise is just like predictive accuracy. So in a world where the super forecasters and intelligent analysts are predicting something and you know the super forecaster has got a better track record, even if the analysts have a higher social status, the modest thing is to go with the super forecasters and not with the intelligence analysts. The problem is, of course, that we may find, which we may go on to later, is that maybe the general skills required to make good predictions about various events is like this single key skill, and doesn't really have very much to do with particular domains of expert knowledge, and so it shouldn't be like deferring very much to these sorts of people.

I guess my own view, to like till my hand preemptively, is that although it may be the case for your super forecasting, you might be able to sort of take issue with intelligence services or similar things like that, if you're a typical person in the audience, you're probably not a super forecaster. And so even if there's even better people out there, you should be deferring to, instead of an expert class, a typical subject matter expert has got advantages over you, or so I would hope, but that may lead us into the next thing a little bit too early. On which, more later.

I guess the other thing say, that Ollie alluded to, it is a lot harder than it sounds just to defer to the experts because, taking your example further, like if you ask theologians whether God exists, they all agree. If you ask people in general, most people believe God exists, worldwide. That's so-called the common consent argument for God's existence. We also see, but then you go, yeah, fine, but maybe smart people are expert class on this. So you stratify by IQ. People with higher IQs trend atheist compared to general population. Okay. That might be something. But wait a minute, Well, what about philosophers? Philosophers themselves also, philosophers generally also trend atheist. Philosophers of religion, though, who spend lots of time thinking about whether God does or doesn't exist, they actually trend strongly theistic, at least in the US.

And then you go, well, who should we undercut? You go to underlying features. Maybe like people are really just naturally religious gravitate to philosophy of religion. They weren't persuaded by the arguments, and you can have lots of very complicated discussions over who counts as an expert, and like make a rod for my own back. There's a worry whereby you can liberally gerrymander who you count as an expert to get the answer you wanted to have in the first place. So, weigh these people slightly more highly, and there's lots of fudge factor, so you can twist the dials to, to get the answer you'd like. Obviously, epistemically virtuous people like me would never do anything like that, but you know, better off for the rest of you guys. So whatever. It's harder, it is very hard to do. I would still insist or suggest, I'll suggest it can still be better than trying to strike out on your own and try and form your own impressions of it, which is perhaps where we begin to route back into a topic of disagreements.

Oliver: Yes. So I think one area that I'm curious about kind of exploring, that I think could be really relevant here, is I expect us to have some disagreements about to what degree do we think there are a set of key underlying, think of them as general rationality skills, or you could think of them as general philosophy skills or something like that. Where you can easily imagine a world, and I like to always… there's a straw man mathematician perspective where you study mathematics, and mathematics has like eight branches. You'll study algebra, you'll study linear algebra, you study some of the more graph theoretic stuff, and now you really have eight models of the world and everything is really just an application of this. Like after you notice eight models, like voting theory, no problem, you just apply some graphs to it. Database design, no problem, you just apply some algebraic constructions to it. And so, there's this idea that we might be living in a world where there are very few simple models that we can generally apply to a lot of different things, and that ultimately expertise in those models dominates the ability to predict accurately in that domain over this specific subject expertise.

I don't think this is truly as extreme as the naive mathematician's view suggested, but I think it has quite a lot of merit in general, which we see both in super forecasting domain, but also in the domain of thinking rigorously and reasoning rigorously, which isn't necessarily only as an are, a super forecasting domain. I think in many ways I would describe it as something like philosophical competence, where you have a concrete example.

One of the big problems that I heard Open Phil has often run into, in their discussions with various experts, is they try to make a decision of like how likely are various risks, and they go to the relevant experts and they asked them, “how likely are those risks?” And they answer them with “well very high.” And you're like, “well what do you mean with very high?” "Well, like definitely high." And you're like "is it 50 percent or 70 percent?" It's like, “I don't know. I can't really put a number on it. High.” And you end up kind of in this situation where, I think there's an important sense in which general skills about how you interact with your beliefs, and even being able to make precise statements about the world that are general in the sense, like, could screen off just the sense of domain expertise. And I expected we probably disagree about the degree to which there is specific domain expertise in detail to these different areas.

Gregory: Like can I just reach the summit of the epistemic function with all creation arrayed before you, is the only way you can make good progress. So that's like, yes. So I think you do anticipate correctly. I'd be like, "well I'm not entirely sure about that." I mean, it seems the case that there's like something which super forecasters have which involves doing lots of different things. They have to be able to predict. There are specialist politics forecasters, there's some of that. They often look quite good in various domains, so definitely the transfer capability set, it's worth saying that they have some background knowledge. They're not usually doing things literally from nothing but it's like not very much, like something like a reasonably low time level expertise, rather than spend like two decades studying like dynamics of politics in a certain area.

I guess there are also other areas you want to look to besides super forecasting, because like you could say there's like civilizational inadequacy, whereby people aren't trying to protect certain very important events. And so maybe experts haven't really like been trying to develop expertise yet in that area. Well it's like expert performance. It may be the case that it's quite widely separated, so people who are really good at Go on really good at Chess necessarily, or like Poker, and things like that. And we have pretty good evidence that actually, when they can play each other. There's lots of things you'd like to learn, to become really good at chess, which aren't just general skills. Things like IQ and stuff help, but it helps to the degree you'd expect. So I guess my view is that lots of bits of reality have this property whereby you have to like pile in lots and lots of precise knowledge to make good progress in an area. We might just be like at a very early stage of forecasting. It's like in a field where you can become an expert in. Maybe the forthcoming and, one hopes, Tetlock-infication of governments will lead more in that area but who knows? Maybe I'm wrong.

Oliver: Yes. So whenever we can find maybe a set of predictions or something like that, where we completely expect different groups of people to give different quality of answers. That would help us maybe tease out some of the things.

Gregory: Yeah.

Oliver: So I wonder if we could find the edge of like where I expect super forecasters to stop being good. Where just like you expect super forecasters to stop being good, where maybe it works, where like I can imagine that there is a world where you… Like let's say we go into the domain of quantum physics or something like that, and where I think I still need to reflect a bit more about it, but naively would expect that, I think like my trust in super forecasters would extend further into domain of like narrow quantum physics, and getting questions such as like, "Is cold fusion possible?" Something you talked about it in one of your blog posts, and we ask physicists, maybe even a pretty good physicist from a top university, and we ask a super forecaster, and who would we trust on that? And I think I would be pretty torn. I really do. I think I would have a good chance, especially if I have a super forecaster work on it for a week, I would have a pretty good chance that I would trust them more than the average physicists. At least PhD students.

Gregory: I definitely agree with the direction of where we differ. I'm just not quite sure if we'd like can crispy like, find out exactly where, because not necessarily, if you have a fairly wacky thing like cold fusion, physicists wouldn't necessarily usually work on cold fusion. Or have any knowledge of it, like they'd just think it seems weird to me. It's effectively outside my… I agree in the general sense that obviously, you'd want several key core skills, like having good accuracy of beliefs. You would expect these things to apply more broadly than I would. If you're trying to predict whether the answer is like Ramsey 55, or Ramsey 77, I would be much more willing to trust my intuitions about super forecasters to give more evidence for that, or similar things like that.

It would generally be nice, I guess… One interesting area perhaps is like, data on rationality training, like calibration training. There's not really crisp evidence of improving like job performance is my understanding. I'm not very acquainted with this literature, but my standard understanding is, we do this stuff. They learn these cognitive biases, and yet you haven't gotten better. Now it could be a skill like this doesn't transfer. If you just drill it you might become smarter, but it could be something like, it's not really training the underlying skill. But if it was improving the underlying skill and yet you don't see improvement in performance where we think expertise would matter, we would give different predictions about how likely that is to be the case. But again this might be a little bit too woolly to really pin down.

Oliver: I actually think this is a very important domain, and I think one of the things that kind of underlines a lot of my edifice of opinion here is, there is this interesting question, where at some point in the 17th century, humanity started being able to solve problems that it was never be able to solve before, remotely. Like we started out like being able to kind of, if we really tried, build a simple city, and then 200 years later or 300 years later we build rockets that flew to the moon. We built computers, we built the Internet, and it really seems like something… like, we see something like the Flynn effect. You can imagine maybe something like general competence increased and people got more intelligent. People had less like nutritional deficiencies and maybe that's the reason why we can do all of those things.

But I think the case for that is not strong enough. I think ultimately there is a strong argument for something like, there was a quantification and mathematification of a lot of these fields of science, that allows us to build the methods of science, build out physics as a domain, and then start just solving lots and lots of problems with this pretty small set of tools. And like, if you look at all of human knowledge at the time, this at the time, if you look at the 17th century, like what you would study at university, it was primarily rhetoric. It was primarily like lots and lots of theology. And then you would maybe spend something like 10% of your time studying mathematics, maybe 15% of your time studying mathematics. But that branch turned out, that kind of relatively small aspect of human knowledge, turned out to be very widely applicable, and to solve a really large domain of problem. And I think this highlights an important sentence in which we have, I think we have an answer to discretion of whether there are domain general skills that allow you to get better at solving problems in a large range. And I think the answer is, a lot of like modern quantified science and the evidence we have for that is the scientific and industrial revolution. And I wonder if like…

Gregory: I think there's part of the point I agree with, but I'm not sure how much of the variance it explains in the classical challenge, here. I mean like inferring, I mean like, there's a perennial problem in the field of like, macro-historical judgment is very hard. Too early to tell what caused the scientific revolution. It might be too early to say we're confident with the scientific revolution was because we discovered a maths, when there was like lots of precursors. Now we discovered calculus reasoning just before, and I think it's quite hard to work out things like intellectual traditions. But I agree in the general sense. We have like sort of really crisp historical data showing we made this like advance in thinking tools, and this great wealth of understanding opened up in front of us.

I think that's all very favorable evidence. It's like been, to a very large degree, these very key skills you can apply to lots of stuff, rather than my view where it's like, you have like lots of like hedgehogs. Hedgehogs or foxes? I think hedgehogs. You have to burrow you way into your area to like, get good answers. I'm just wondering. We might be thinking of moving to point B? You get the last word in, but…

Oliver: No, I think that's pretty natural stopping point here. Okay, cool. So next thing that we wanted to cover is when are you an expert, and it covers a lot of very similar ground. I wonder whether you have like some initial thoughts.

Gregory: Yes. I think like where we can go next is we can, we can discuss often, in the general sense, like how the different classes of cognizers are like, and how much there's like a single axis of thinking skills. Like with people who were top, who are able to get lots of things, or if it's like very, very dimensional. Like, if you're an expert at math, you can make good predictions at math. If you're an expert in politics, you can make good predictions in politics. I know there's lots of different things and therefore if you're like, but then what do you do, and if you're a typical person who is not an expert in many things, or in my case an expert in anything, what you can do instead is say, if I'm like pretty mediocre at everything, I can steal the knowledge out people have got, and steal their predictions and use that instead. And that just seems like a strategy-stealing, superior approach. And then I guess I'd then say even if we… even if we disagree or how much there's like, one key skill, I'll say you can often maybe like surpass what's a typical individual in this community can do, if we defer more, rather than saying I've read, I've looked into stuff and rationality, I've read various works on like how to by rational. I see myself equipped and have very confident pronouncements to make, on let's say macro economic theory or public health and stuff like that. And that's the typical principle of mediocrity rather than modestly. The typical person, even in EA, who are very clever people, of course you all are, generally would always almost always do better if they're deferring to someone. Whether that's a super forecaster or a domain expert is unclear. But I do think we are unexceptional cognizers in this whole sense, or at least not so exceptional. We can always bank on ourselves to want to stake out our own views constantly, but maybe that's not quite getting to the crux of the matter, as you might say.

Oliver: So I think this is… I feel like this is almost responding to your previous point and I kind of wanted pull on this, but it's kind of something that's been… where I have this thing. I have this feeling that we both agree that the question of who is an expert and how should we aggregate opinion, and something like when exactly should we believe someone else's opinion and how should we integrate with us? Like if we both agree this is a very rich and important domain, in a sense that it's important that we figure out the general rules of this space, we figure out how to aggregate knowledge like that. But I feel… I don't know what it is, but I have sometimes the feeling that when you advocate for modesty, that you advocate not for the study of a rich field.

I feel you're not advocating necessarily being like, something that… I think the thing that I would be definitely deeply behind is something like: I think the question of how to aggregate opinion of the people around you is important, and the answer is not obvious. Like the answer is, here are some interesting considerations that we have. And then you would like maybe need examples, and then somebody would want to start getting expertise in this domain, meta-expertise. That would be, let's look at the example of super forecasters with physicists, analyze the structure of that, and then come to some conclusions, and then maybe try to extract some general principles from that. And then I would be like, yep, those are good principles. But like kind of making it clear that there is an art and a skill here that needs to be built up over time, where I'm worried that when you usually say modesty, you're like just, be less arrogant or something like that.

And I agree that maybe some people have like an aura of arrogance, but it's like a very… something like trying to tell a physicist that they're not making a lot of progress, because they always write too large in their notebook, and so they run out of space before they finished thinking on a current notebook page. And I'm like, I guess I can see how this is maybe an error that someone is making, but it really doesn't seem like the most important dimension, or something like that. Like arrogance or something feels almost like, it's a relevant variable, that I think is important to calibrate, but it doesn't strike me as one of the key variables. And I wonder whether you might disagree with that.

Gregory: I didn't know. It may be a little bit hard to litigate the examples I have in mind, which may not be very helpful to discuss. I think my view is, if maybe the case that it's not necessarily straightforward, that doing like the modesty thing well is in itself a bit of a skill. But I think it is also the case that maybe I can't identify really clear mistakes. So there's the case I got from a friend where someone said by way I've disproved the continuum hypothesis: I've god this notebook, would you like to have a look, and it just feels like in that sort of world, we answer, no, I can just tell you with probability 1 minus epsilon you're wrong about this, and you should have thought the same because there's lots of proofs of this, and the rest of it.

Oliver: Yes.

Gregory: So I see like I definitely feel like there's like steps you make to do better. I often see like sort of like really obvious mistakes, and then the simple word, is please stop being so epistemically arrogant. But let me say what you do instead is a bit more involved. But as a crude rule of thumb, I often find this seems to do better is I guess my view. But also the other thing whereby like epistemic immodesty, and immodesty in general are not exactly the same thing, and they can co-locate. You might get annoyed or people being like a bit cocky. So for me, I'm like cocky all the time, and I'm like preaching epistemic modesty, so these things do come apart in various respects. But in the same way, it's important to stress that one shouldn't poison one's epistemics by personal dislike you have for a certain communication style, or anything like that. I may not always practice what I preach here, but I do at least recognize I may be at fault.

Oliver: Yeah. I do think this is something that I probably expect to be useful to clarify. Like we both agree and we kind of had an earlier conversation a bit about this, where there's just this really important distinction between what we think of as social expertise, where there's like, I guess the degree to which our gut judgment says that somebody is an expert, and something like the actual epistemic expertise. The actual track record of how much will they reliably get questions in this domain right. And just that those are definitely two different things, in different domains, that can get completely different answers.

Gregory: What made how completely might be another…

Oliver: Yes, that might be the-

Gregory: I might be more deferential. It's like sort of cultural expertise. Like okay, if I've got an expertise it's good. If you're an expert in this area, even if you're not a super forecaster, you might still be worth deferring to in many cases. Probably like obviously that disagreement over how well we can correlate may not be a large part of a wider discussion I guess.

Oliver: Yes. So I do think that… I think the example that you mentioned, somebody thinks they just proved the continuum hypothesis, is very good, because also contrast like naturally with somebody comes in and says… I don't know, let's go back to like, when was it, 2012? Of like Iraq has nuclear weapons or something like that, or chemical weapons. Where there was this… there was a situation where we had experts in the field, who was the CIA at the time, who made predictions, and now somebody comes and is like, you know, actually, I Googled myself for a while for the relevant evidence, and I think it's pretty obvious that they don't. And then we, I think in that case I wouldn't go say probability one minus epsilon at all that they are wrong. I would probably say, if I don't know them very well, like 30% that they're right at maximum, maybe 20% that they are right.

But I would have a very different attitude towards that kind of thing. And I'm very worried that if we try to paint this very broad brush of modesty, that we'll put those in the same bucket as the continuum hypothesis and mathematics, which we know has… is in a framework, with several proofs. And is in a framework in which we can expect to be rigorous, and I think it's very important. Like is it a framework that we've seen work? Or something like that. Where like, I do think mathematics is really important. We have seen mathematics work over and over again, and so there's a sense in which when somebody disproves a major mathematical theorem that we expect to be true, or like at least that many, many people have thought about a lot, then we can trust that their effort was something productive. Whereas if we're talking about international politics and situations where I have never heard of a large class of experts who reliably got those questions right. And so in those situations I don't think we should apply the same standard of modesty.

Gregory: So I think you'll obviously fair to say that picking mathematics is the easiest example of my case. We actually know the ground truth. I think I would say though, but well, I would go on to say something like this. So suppose you're this person who Googles for maybe like two hours, whether Iraq has chemical weapons, whatever it was, and you decide like probably not. It seems to be the approach a typical person would take. It's like if you're like a third party to this conversation, like somebody walks up to you and says oh, I've done some Googling. Actually Iraq doesn't have chemical weapons after all.

That evidence to you should be like aggregated with basically zero weight. Because there's lots… like thousands of thousands of people, with all of their views saying it has or it hasn't. Experts have spent loads and loads of time on it. And now it may be the case they are right. Because it's very hard to tell as you say, there's no really good, robust demonstration of evidence. But the mere fact that someone's telling you "I've Googled this for a while" or similarly cursory efforts, shouldn't really massively update you one way or the other, I would allege. And then you can sort of anticipate my next idea, which is well if that's true, then by indifference, you shouldn't care whether you are that person or you're hearing it from a third party. Because if you're a person who spent two hours Googling it or whatever, then you also shouldn't be updating any more than if someone else was doing, who's similarly situated to you. And that, I would argue, would always lead to considerable deference in most cases.

Oliver: I think that really depends because, usually, let's say I followed… at the time I don't think I was doing this. But let's imagine we, in the present day, we have the same Iraq nuclear weapons question. And I've been casually following the news and the answer is like, okay, I've heard. I've definitely heard the fact that there are people who have opinions on both sides. But I wasn't given any evidence about the methods of the people on either side. Like I wasn't given access to the methods that they used to arrive at their opinion, but now a friend of mine comes over and says, I said no. Now I agree that if we just count the number of people who have opinions in here, this might be, he might just be a very small drop in the bucket, but if we assume that people with the better methods reliably get better answers, and I have some expectation that my friend has better methods than the average person, which I think I can expect in the domain of policy forecasters, then this is actually a major update.

Because now it is the first time that I heard evidence, with the methods that were being used. And I think a really dangerous… like a really important dimension of all of this, is that in scientific fields it is very often extremely hard to figure out what methods people actually use to come to their opinions. If I read a physics journal that describes an experiment, I only really see the experiment that they ran, and maybe some mathematical analyses they did, but I didn't get any insight into the thinking. I didn't get any insight into how did they go about hypothesis generation? How did they go about… how many alternative hypotheses did they consider and weigh? And for my friends and for the people who are very close to me, I have that evidence much more often because I know their methods and processes. And so I actually think updating on that to a large degree can very often be the correct choice.

Gregory: Ah, right. So we do definitely disagree on that. So I think where I go is, I think I would suggest/allege/whatever, that it seems you have a very parochial view of the typical epistemic qualities of the people around you. So I agree. Like if someone, like if your friend like comes up to you, somebody who you respect, and they give you their reasoning as to why we should think Iraq doesn't have chemical weapons and you go, "I know this person is telling me what they think, and they're definitely very reasonable." But you go, "look man, there's professors of international relations." Now you might go like, "okay but they may not necessarily be super forecasters." No, granted, but you can probably anticipate if you manage to get to talk to one of these people, they're like, you would hope, I'd suggest you should expect, if you chatted with someone who'd worked in security studies in the Middle East, they'd give you a reasoned judgment as to why we think it's more or less likely that Iraq has chemical weapons or not.

And so it may be the case that you think… if you think like the relevant, like meta-epistemic qualities are very rare to find, such that your friend who has them, you'd put them above some relevant domain experts, we have like implicit knowledge of what they think. You retire what you see the US government saying, you retire what you see academics are saying. You retire all these talking heads, or, well maybe they aren't experts, but other people who are experts on the news saying things, and so on and so forth. I'll say that you shouldn't necessarily go, "but because I, I've seen with my own eyes what they're thinking" in the case of your friend, you should weigh that more heavily. Maybe you can't have access to the generating methods, but you should like offer like some, you should be willing to trust these people. They do actually have some there if you are able to investigate, because that's where I would like want to go I guess.

Oliver: But I do think that like the big assumption that was here was that an average chosen professor of foreign studies at maybe a major university, will they have methods that are at all reasonable to come to good conclusions? And I think the answer is like, no, in the sense that I think the average professor of foreign relations at a major university does not have something like the calculus of reason to actually make correct probabilistic statements about that kind of stuff. Like, it doesn't matter. Even if you're exposed to a lot of the data, if you don't have a sense… if you don't reliably can keep apart causation from correlation, if you don't reliably keep apart like the social biases that you have and the status hierarchies and the environment from your epistemic beliefs, it doesn't… I don't think it really matters how much precisely you know about the specific foreign relations of the country you're talking about, because that evidence won't be like… those ideas won't be able to build on one another. You're not actually given me a reasonable opinion that I can trust. And I think, like, I want to go back to kind of the mathematical reasoning, because I think there's a strong analogy here of, you can imagine you have, there is this question, let's say we have…

I think, let's go into the domain of construction, where only in the 18th or 19th century did we really start to have any form of a systematic method of building things. Like we did some simple static analysis before that, but mostly when we built things, we stacked things on top of each other, saw whether they fell down, noted which things didn't fall down, and then built those things again, and maybe combined them in smaller ways. And so we had these people who definitely were exposed a lot to the domain of building things. They architected many buildings with this method, and now we have modern physicists encountering them. And they have access to the modern calculus of physics and understand modern statics. And now let's say they've only participated in building a single building.

They've only seen a single thing constructed, but they've kind of seen all the basic parts of how it is to construct a building. What I'm arguing for is that they will be able to integrate that evidence much, much better, because they will have the language to describe those kind of things. They will be able to describe statics, they will be able to generalize, when they see an upstanding arc, why that arc stands up and how much weight it will have, and that ultimately the next building is the one that I would trust a physicist much, much more than a person who just built a building.

Gregory: So I think it's interesting because I think where we differ is maybe like at a baseline level or at a baseline difference between two groups. So I'm definitely not going to defend the claim, that like a typical professor of foreign policy has got really good like epistemic norms, more than a typical person, because we have lots and lots and lots of data on heuristics and biases that apply to everyone, no matter how allegedly degreed or prestigious they are. I guess where I differ is I'm not sure, although you keep excellent company, whether a particular friend of yours is in expectation going to be much better than domain experts in this area, such that you go, "even though this person doesn't have the same level of expertise as this person, we have a pro tanto reason to trust them slightly more at least." Because they have the other metacognitive skills, I should count them much higher than the prevailing view of an expert class, because they are much higher than the average level of like epistemic quality in this area. I guess is like the thing.

If that could be made concrete whereas you'd test people, and see how good they are at predictions, if it turns out for example, that bay area community is chock full of super forecasters, then yes I'd that's obviously like a really good reason to take their views much more seriously than for example, what you typically see expert groups doing. I guess where I sort of differ is, although I think we're great, I'm not sure we're that great, in this sort of way.

Oliver: I think I'm, I'm not arguing that precisely the people I'm surrounding myself.,, I think I hope that the people have chosen as my friends are good at thinking; I think that is one of the primary things I select them for. But, I don't want to make the argument that that is the thing that ultimately, it's not the fact that they're my friends that's the primary selection criteria here, but I'm saying something like, it is, there are many, many people who are not my friends who if they give me advice, if they make a judgment in a certain case, then I will trust him much more than my friends. I'm not making the argument that my friends are…

Gregory: Maybe like we can get more crisp idea. So I agree it doesn't really matter whether you like these people or they're friends of yours, but I guess like I don't see what you might think of as like… If you're going to treat certain individuals like, sort of epistemically… "oracle" is more dismissive than I mean, but sort of like people with very good judgment applied across multiple domains, so you would actually have good reason to trust them over what prevailing experts seem to say, then it seems like you can test, we can just look at their like track record and say, "look, we just keep getting stuff right, again and again and again, in all these different areas."

But I guess I don't usually see this evidence, or at least there is no public version of this evidence, for when I see this often done widely, and it may be the case for you do have this pro-evidence, which is great. I wonder, out of interest, how carefully you try to investigate the track record of people you take to trust so highly. I also wonder in the typical use case of people doing this stuff, how often they are doing it as well. Because maybe, you epistemic virtuous par excelsis, are already good at this. But a typical person might massively overweight what their peer group perhaps thinks. That that makes sense if someone they quite like happens to say and so on and so forth. So maybe even if the angels can do what you're doing, maybe the great unwashed like myself should be quietly deferring away.

Oliver: So, I think the evidence that we have that certain forms of reasoning vastly outperform other forms of reasoning is not evidence that is very hard to access. I don't think like the fact that these gradients exist, and seeing the rough structure of it, and seeing that it's a very lopsided, kind of very heavy tailed domain where we see certain forms of reasoning massively outperforming other forms of reasoning, I think it's something that is fairly accessible just looking at the trajectory of history. I agree that historical judgments are hard to make, but I do think that across the board we've seen a few specific forms of reasoning just show up as successful again and again. So I don't think that it's about like the degree to which I have like a special ability to judge other people's judgment in some major like…

Gregory: I'm not saying you have special access rather than like, do you in fact see these people out there? Is it the case that you can find them. There's a few super forecasters, not many super forecasters, and…

Oliver: The thing that I'm saying is that the natural variation does exist out there, and it has existed over human history, and already makes a massive difference. I'm seeing that whether you… when trying to answer a question, make simple quantitative estimates or not is something we see massive amounts of variation in the population. Just like it probably is about like something like 5% to 15% of the population that I think very frequently if they try to answer a question, we'll go and make a simple quantitative estimate of the problem, and there's about 85% of the population who expect will not do this. This is a variation that we normally see. It's nothing special, it's not an extremely high bar of epistemic excellence. So it's just like a pretty standard skill that some people have and some people don't. And I think we see reliably the degree to which we make these short, small quantitative estimates tends to be highly predictive of the ability to make reasonable judgments in almost all domains. And as such, I felt that the thing you were just saying is that there aren't that many people who really live up to this high bar of excellence. And I'm saying no, I think even small gradient degrees to which you are slightly better at reasoning make a massive difference in your ability to predict the future. And those was a small enough that we see natural variation in different scientific fields, in different people, and so on.

Gregory: Yes, okay. So I guess like perhaps where I would go is, I would want to like say something like this. So it may be the case, per like the previous topic, like the main variance in like how good your predictions are is like how good you are at these other skills you've been highlighting, like making Fermi estimates, or like metacognitive skills are a really big factor. But compared to that, often domain expertise like has like a bit of a boost, especially initially, that rapidly diminishes. So it's often the case you must put your faith in epistemic qualities over mere domain expertise. But then there's also the question of how the distribution of epistemic qualities goes. I definitely agree, you do like a little bit better by doing Fermi estimates, and things like that. I guess I'll say is that often we take the expert class of people who work in academia, for example, like implicitly higher IQ, and we know IQ correlates rationality skills as well.

And so it seems to me that even if we can find people who are somewhat better, who are really good at these sorts of skills, I think the typical academic is somewhat better, if not really, really good at these sort of metacognitive skills. There's a lot of room at the top for which they could reach. Now I want to say that maybe the room at the top, there's not many people in this like far tail of this distribution. And so then, I want to make feet this observation I made before about, how many of these people do we have? And so on, and so forth.

Oliver: Yes. My model predicts something like, the effect of this continues to be pretty large, like all throughout the board, up until, if I have a single research organization, or if I have an academic field that has these skills generally much more widely distributed, even identifying the individual people… at every given step, I think it's highly valuable to identify people who have those epistemic skills, and not. And I'm worried that this gives rise to something like… I wouldn't necessarily say that we disagree on data mentioned that much, on like if somebody would put a gun to her head and like what would we say in the moment, what the correct answer is, but something like the degree to which we expect additional evidence about people's expertise to continue updating our opinions in this space. Where I think like my model is something like if you give me a rough guess of, let's say we go back does Iraq have nuclear weapons question, and you tell me most university professors with foreign studies, or let's say like 70% of them think this is correct.

And like, okay, I guess I'm going to make it like an update on that because that seems… let's say they think they don't have weapons. Okay, I'm going to make an update on that. I don't know. I feel that I would drastically change my mind if I would now hear that a specific university that I think has like larger track record, let's say something like, I don't know, I don't have much meta-domain expertise of like, foreign relations. But if I would see someone who I think has demonstrated good judgment here, maybe this would come up as a question in the good judgement project, I would expect that this would drastically change my opinion again, where I would expect that it would change this quite a bit. And as such I suspect that there's this common phenomenon that I see with people taking the modest approach, where they kind of saying, just trust the average of the experts.

And then I'm like no, look, there's this sense of confidence that comes with that, where you're like, ah, the average of the experts is as best as we can do. And like, no. The average of the experts is definitely not the best that we can do. The average of finding out the epistemic practices of every single person in the field and precisely understanding them, and also understanding the subject domain is the best we can do, and both of those are really interesting directions to go, and there's much more work to be done. You can't just stand here and say that the problem is like as much as we can possibly solve, solved just because kind of you have made a rough expert survey.

Gregory: Maybe we should move on the next topic.

Oliver: Yes.

Gregory: We have 15 minutes. But I'm going to briefly defend myself from your slander. So it's definitely the case that it's not going to be that all experts are created equal, even through common sense with the epistemics that had been previously cashed out in mind. And I agree as you say, you can do much, much better than the average of the experts, but the average method may do much, much better for your first order impressions of view. And now it may be the case amongst like very sophisticated cognizers, they end up sort of getting stuck in the middle of being a little bit too modest, a little bit too stuck. But I think definitely in the general world, even I think in the EA community, you see much more, I have this impression, I read this a little while, I'm pretty certain this is the right answer. And they don't really consult the experts, and sort of ask what the experts think, and all the rest of it. And I always want to say, there's much further you could do in terms of aggregate agreement and things like that.

But I think that asking the experts the last thing you want to do. But probably I think often it's a step forward among prevailing… as a fundamental thing, in terms of weighing the variance, it's like you can imagine sort of back tracing prediction markets and betting markets, and see how big an impact is like updating different releasing information to have to update it. And then if we buy that, that there's a stronger sense of how elitist we should be in terms of weighing different sorts of expertise, is a hard question. We mentioned earlier this idea of like gun to your head versus second order. So maybe I'll let you take it away, with respect to that.

Oliver: I think think one of the things that influences my perspective on all of this the most, it's kind of… I think we've been so far answering the question of okay, let's see. You have to give an answer to what the likelihood of Iraq having the weapons is, in the next five minutes. That's it. Now you need to make a decision. For some reason somebody is putting a gun to your head and you need to give the answer. Usually you're not in that environment. I think usually you're in an environment, hopefully, where you are trying to explore, you're trying to understand the topic, and I think there's the question of what attitude should we have towards the topic while we're trying to understand the relevant subject matter? And I think a classical example that I would give here is let's say you're a mathematics undergraduate, and you are trying to understand the basic theorems of mathematics. Let's say you're in linear algebra. Let's say you're in real analysis or something like that, and you can easily imagine a world where you hear like there's Zorn's lemma, or like kind of like one of the most highly confusing axioms of real analysis. You go okay, I guess lots of people find this reasonable. So I guess that's what I should believe. And then you continue doing your proofs in mathematics and you dutifully write your proofs on. But I don't think like this will actually produce a great mathematician. I think the great mathematician will be produced who will react to that being like, what? Why would you be able to have an infinite chain of things that bottoms out at some maximum, and be able to always be sure that it has this weird attribute? That doesn't make any sense! Like I can imagine here that's also equivalent to this other thing!

Like there's a sense in which I expect emotional engagement with the topic, to heavily depend the degree to which they learn, and that I think can very easily be mistaken for something like immodesty, where like when I studied mathematics I will have emotions about it. So like, there's no way this is true, this is obviously false, and it has nothing to do with my degree of… I mean it's something like… maybe a part of me has that credence, but it is an additive to have while learning, and I'm definitely worried that… I really enjoy this aspect of scientific fields, of physicists and mathematicians, and also many people in this community are people being able to have these engaged conversations. Of feeling like they're really interacting with their models, and I'm worried that you are mistaking that for something like immodesty. Or maybe we disagree, maybe disagree that it's a good idea.

Gregory: No, I think in this case it's a good idea. I mean, we may disagree on how we interpret it. It's obviously the case that you could be like an epistemic free ride with modesty, so you spend all your time trying to weigh up for the expert classes and never form object-level beliefs. You're just like, I'm really good like calibrating expertise now, I do that all day long, and I have like no object level views on anything beyond what I can do. Right? And that might be a really useful skill for the wider community. It could be like a really useful thing to do. Like maybe what a super super forecaster could be like, like a valuable trait for the community, on which much more later, but it's obviously the case that if you want to become a mathematician, that's like not the right approach. Your job is to actually do object level work, like object level progress, and that requires like actually engaging with the object level matters of things, like full impressions about things, make arguments about various bits and pieces, and I think you can do this with…

You can marry this with modesty or, marry this modestly to modesty, in the sense that you could like, go well, in my own personal view, I do philosophy occasionally in my spare time. I'll have a little bit of a look at this topic. I'm probably not going to make anything groundbreaking to argue in favor of this particular view, and then hopefully come up with something interesting. Even if like you put a gun, what's the most plausible view on this topic? I may end up like just deferring… well, philosophy is hard to defer to. But, the pervading view. Maybe a better example, is like Scott Sumner, who's like a master economist, talks about a two level view of approaches. Well, from my own personal perspective, I just spend all my time doing stuff like doing object level work, like hopefully proving my theory is right, but if you put a gun to my head I would actually do the overall thing, and then like if there's lots of like sort of like back and forth, like testing out models, then I'm pretty cool with that.

I guess I get the impression, like usually people, if you put a gun to their head, they'll say basically the same thing as they were saying just now. And it's like less of like the whole, less of a two-level approach, which I see. If they were doing lots of that, I'm really happy. Or like reasonably happy, I'm pretty curmudgeonly anyway, but like more happy than I usually am. I guess I just don't see as much as I'd like to see naturally. I sort of get this impression with us, we're overconfidently holding first impressions, and there's like benefits like doing this normal thing, but I almost feel, not for you guys, but some people I've talked to other, use it as like an excuse, like, well I'm just like trying to be really cocky here, but excuse this.

Oliver: Yes. I think there's like one, one perspective where I expect we both, at least my model of our two models, we'll both agree that one of the things that is most valuable here, that I think is definitely correct, which is that like this habit and pattern of people actually betting their beliefs, and then having bets with one another that are evaluated one way or another, and putting probabilities on that, and having those bets proportional to those probabilities, is like a norm that I'm very excited about. And like, would love to see spread much more. I'm really excited about people doing more like, prediction market-like things, especially because I think it actually gives you the evidence of whether we see people being overconfident. I think we could have the conversation about when we look at the track record of the people. I think this is like a conversation probably for another day.

Q&A

Question: How do we navigate expert advice coming from domains plagued by sub-optimal incentive structures such as medicine or academia? And similarly, does the instinct to trust friends whose model you have access to over experts make more sense when considering that some of these experts may have perverse incentives that lead them to promote certain beliefs?

Gregory: So the answer is, like, doing it is really hard. I'm not saying it's easy to discount heavily by like who has skin in the game. Who has biases, and things like that. I guess like I have like a very long discussion of this sort of thing. I feel like I can rehearse what I typically say so it's like… so okay. So you think someone's like, you think expert classes… so economists may be biased, or medics are often biased about various things for a variety of reasons. But a question like, hey, is this bias like selectively toxic, like versus you or versus your friend, because politics gets in the way. Why are your friends less likely to be biased, from the outside view even you would be. And that's like okay, fine. So maybe it does count against them but it seems like, unless they're really riven by biases, maybe on balance they're more likely to be right.

Because it's easy if you're like, a pharma company doing something, it's easier to get your drug to market if it actually works than if it doesn't. You can still do it, but at least it's harder. So it's solvent in that sort of way. So I guess I want to often say that, yeah, I know, I'm not sure either sometimes. But I guess my view would be in many cases, that you can undercut expertise with claims of bias, but I think often this selective undercutting effect versus you or versus your friends, is not so large as to outweigh the other stuff. Like oh, maybe I should study this for 20 years. Like maybe that does count for something in many cases. So I'll say that often like, wins the day. It doesn't always, but it often does.

And if you thing like the expert class should be got rid of, maybe should just defer to other people. So sort of like, people who read really widely in this topic on the Internet, or some other group who might not have this bias but also might have thought about it much longer than you have.

Oliver: My perspective on that is qualitatively, I probably agree. Quantitatively I kind of go much further in the direction of like, the bias of relevant expert fields. I expect, okay actually I think I want to give a slightly different answer. A different answer is something like, there is a very dangerous framing in thinking of expert fields as being biased. I think there's like a general problem that we have. This like this is named for the modern fallacies and biases field, that alternatively it's the field of modern cognitive heuristics. And there's two framings of the same field, that I think highlight very importantly different perspectives. Where, when you think of fallacies and biases, you think of how are they deviating from a rational actor who is perfectly epistemic and doing everything right, and you're like, what is that really like? I don't know how to implement it on a human mind.

But in a cognitive heuristic domain, I'm asking myself what procedures are they actually executing to get to the answer and, now you're not, you're not trying to answer the question of deviation, which I think is really hard, but instead of just trying to answer the question of, well, what kind of questions do we expect that set of heuristics to reliably answer correctly, which ones will it reliably answer badly, and just trying to build a model of how they actually come to their opinions. And kind of similarly to how I much prefer the framing of cognitive heuristics over the framing of cognitive fallacies and vices, do I also much prefer the framing of building models where experts come to their opinions, as opposed to asking ourselves the question of how experts are biased. I think trying to answer the question of how experts are biased without first having built a model of how precisely they will come to their opinions, is the doomed endeavor, and instead I would usually try to answer the question of, well, let's just ask ourselves the question of what questions are they incentivized to answer? What things like, what do they output given those incentives? And then ask ourselves whether that is the kind of process that we would trust in outputting the correct answer. And have a very different framing.