Comments
Here are choice parts of my model of deference:
- Whether you should defer or not depends not only on your estimation of relative expertise but also on what kind of role you want to fill in the community, in order to increase the altruistic impact of the community. I call it role-based social epistemology, and I really should write it longly at some point.
- You can think of the roles as occupying different points on the production possibilities frontier for the explore-exploit trade-off. If you think of rationality as an individual project, you might reason that you should aim for a healthy balance between exploring and exploiting due to potential diminishing returns to either one. But if you instead take the perspective of "how can I coordinate with my community in order to maximize the impact we produce?" you start to see why specializing could be optimal.
- If you are a Decision-Maker, you're optimizing for allocating resources efficiently (e.g. money, work, power, etc.), and the impact of your allocation depends on how accurate your related beliefs are. And because accurate beliefs are so important to your decisions, you should opportunistically defer to people whenever you think they might have better information than you (Aumann-agreement style), as long as you think you're decently calibrated and you're deferring to advice with sufficient bandwidth. You should be Exploiting existing knowledge and expertise by deferring to it. But because you frequently defer to others, you may not be safe to defer to in turn due to potential negative externalities associated with information cascades that can be hard to correct.
- If you are an Explorer, your job is to optimize for the chance of discovering important insights that can help the community make progress on important open problems. This is fundamentally a different project compared to just trying to acquire accurate beliefs. Now, you want to actively avoid ending up with the same belief states as other people to some extent. Notice that the problems are still open, which means that existing tools and angles-of-attack may be insufficient for the task. Evaluate paradigms/approaches for how neglected they are. Remember, it doesn't matter whether you're right about what other people are right about as long as you are extremely right about what other people are wrong about. So if you want to maximize the chance that the community ends up solving the problem, you want to coordinate with other explorers in order to search separate parts of the idea-tree. What matters is that the right fruits are picked, not that you end up picking them. We're in a parallel tree search paradigm, and this has implications for how we individually should balance the explore-exploit trade-off.
- If you are an Expert/Forecaster, your job is to acquire accurate beliefs that are safe to defer to. If there's a difficult and important question (crucial consideration) for which better forecasts could marginally improve the careers/donations of a lot of people, this could be an important way to produce impact. Your impact here depends on the accuracy of your beliefs, so unlike the Explorer, you don't have strong reasons to avoid common belief states. Your impact also depends on how safe you are to defer to, because you can potentially do a lot of harm by reinforcing false information cascades. And these considerations are newcomblike, so you should act by that rule which, when followed by the proportion of other experts you predict will follow it due to the same reasoning as you, maximizes community impact. Sometimes that means you want to report your independent impressions, and sometimes that means you want to share and elicit likelihood ratios instead of posterior beliefs. A common failure mode here is to over-optimize for making your beliefs legible, which in extreme cases turns into a race to the bottom, and in median cases turns into myopic empiricism where you predictably end up astray because you refuse to update on a large class of illegible (but Bayesian) evidence.
- The limiting case of a Decision-Maker always reporting their independent impressions is (roughly) an Expert. But only insofar as it's psychologically feasible to maintain a long-term separation between independent and all-things-considered impressions, and I have my doubts.
- What kind of knowledge-work you want to do depends not only on your comparative advantages but also on your model of how the community produces altruistic impact. If on your model community impact is marginally bottlenecked by insights, you should probably consider aiming for ambitious insight-production. If on the other hand, you think you can have more impact by contributing to marginally better forecasts about what problems are most important to work on, maybe consider aiming for producing deference-safe predictions. And if you just happen to have a bunch of money lying around, you just don't have the luxury of recklessly diverging from expert consensus, and you should use everything in your toolbox to make sure you're allocating them efficiently.
- No one is pure any of these. The roles are separated by what optimization criteria they use, and you optimize for different things in different areas of your life, and over your lifetime. But I think it's usefwl to carve out the roles, so you can notice when you need to put which hat on, and the different things that implies for how you should play.
EA has a high deference culture? Compared to what other cultures? Idk but I feel like the difference between EA and other groups of people I've been in (grad students, City Year people, law students...) may not be that EAs defer more on average but rather that they are much more likely to explicitly flag when they are doing so. In EA the default expectation is that you do your own thinking and back up your decisions and claims with evidence*, and deference is a legitimate source of evidence so people cite it. But in other communities people would just say "I think X" or "I'm doing X" and not bother to explain why (and perhaps not even know why, because they didn't really think that much about it).
*Other communities have this norm too, I think, but not to the same extent.
EA has a high deference culture compared to the epistemic norms it claims to adhere to = compared to the standards it aspires and claims to follow, I'd say. This can be true independently of the difference with other groups of people that you described (which I think is also a true description).
Yeah, I agree with that. On the margin I think more EAs should defer less. I've been frustrated with this in particular on topics I know a lot about, such as AI timelines.
tl;dr, I think deference is more concerning for EA than other cultures. Relative to how much we should expect EAs to defer, they defer way too much.
1) We should expect EA to have much less deference culture than other cultures, since a lot of EA claims are based on things like answers to philosophical questions, long term future predictions, ect. These kinds of things are really hard to answer, and I don't think it's the case that most experts have a much better shot at answering these than some relatively smart and quantitative University students. Questions about moral philosophy are the exact kinds of questions you expect to have a super wide range of answers to, so the number of EAs that claim they're longtermist is kind of surprising and unexpected. I think this is a sign there's more deference than their should be.
On the other hand, for more concrete and established scientific fields where experts do have a much better chance at making decisions than students, it makes way more sense to defer to them about what things are important.
2) EAs are optimizing for altruism, so decisions on what to work on require lots of thought. I'm guessing most non-EA people choose to work on things they enjoy or are emotionally invested in.
I can easily tell you, without any evidence or deference, what things I thing are fun and am emotionally invested in. But it takes a lot more time and research to come up with what I think is the most impactful.
I think EAs having more evidence and reasoning to back up what we're working on just naturally arises from being an EA, and doesn't necessarily mean we have better epistemics than other communities.
3) Explicitly saying when you're deferring to someone seems like it does a better job of convincing people "wow! these EA people seem more correct than most other communities" and does a worse job at actually being more correct than most other communities. Being explicit about when we defer to people still means we might defer way too much.
4) Edit: I think this point is not actually about deference. Also, I know very little about MIRI and have no idea if this is in any way realistic. I'm guessing you could replace MIRI with some other org and this kind of story would be true, but I'm not totally sure.
Also, idk I feel like some things that look like original, detailed thinking actually just ends up being closer to deference than I'd like. I think perhaps a story that's happened before is "MIRI researcher thinks hard about AI stuff, and comes up with some original thoughts with lots of evidence. Writes on alignment form. Tons of karma, yay."
Sure, the thinking is original, has evidence to back it up, and looks really nice, pretty, and useful. That being said, even if this is original thinking, I'm guessing if you looked at how this person was using the opinions of other people to shape their own opinions, it would look like
So even if this thinking looks really original and intelligent, this still seems like a problem with deference. Not deferring to other MIRI researchers an unhealthy amount probably looks more like getting more insight from mainstream academia and non-EAs.
I guess the point here is that it's much easier to look like you're not deferring to people too much than to actually not defer to people too much.
5) I think people in general defer way too much and do not think hard enough about what to work on. I think EAs defer too much and occassionally don't think hard enough about what to work on. Being better than the latter doesn't really mean I'm satisfied with the former.
FWIW I agree that EAs should probably defer less on average. So e.g. I agree with your point 5.
I don't like the example you gave about MIRI -- I think filter bubbles & related issues are real problems but distinct from deference; nothing in the example you gave seems like deference to me. (Also, in my experience the people from MIRI defer less than pretty much anyone in EA. If anyone is deferring too little, it's them.)
Yeah you're right, it does seem separate, although sort of an adjacent problem? I think the larger problem here is something like "EA opinions are influenced by other EAs more than I'd like them to be". Over-deference and filter bubbles are two ways where I think getting too sucked into EA can create bad epistemics.
I didn't mean to call out MIRI specifically, and just tried to choose an EA org where I could picture filter bubbles happening (since MIRI seems pretty isolated from other places). I know very little about what MIRI work *actually* looks like. I'll change the original comment to reflect this.
I also think that EA consensus views are often unusually well-grounded, meaning there are unusually strong reasons to defer to them. (But obviously this may reflect my own biases.)
Fwiw I think many effective altruists defer too little rather than too much.
Could you a few specific examples of times you have seen EAs deferring too little?
My view is that when you are considering whether to take some action and are weighing up its effects, you shouldn't in general put special weight on your own beliefs about those effects (there are some complicating factors here, but that's a decent first approximation). Instead you should put the same weight on yours and others' beliefs. I think most people don't do that, but put much too much weight on their own beliefs relative to others'. Effective altruists have shifted away from that human default, but in my view it's unlikely - in the light of the general human tendency to overweight our own beliefs - that we've shifted as far in the direction of greater deference as we ideally should. (I think that it may not be possible to attain that level of deference, but it's nevertheless good to be clear over what the right direction is.) This varies a bit within the the community, though - my sense is that highly engaged professional effective altruists, e.g. at the largest orgs, are closer to the optimal level of deference than the community at large.
I won't be able to give you examples where I demonstrate that there was too little deference. But since you asked for examples, I'll point to some instances where my opinion is that there's too little deference.
Whether you think someone deferred too little or too much regarding some particular decisions will often depend on your object-level views on what's effective. In my view, quite a few interventions pursued by effective altruists are substantially less effective than the most effective interventions; and those who pursue those less effective interventions would normally increase their impact if they deferred more, and shifted to interventions that are closer to the effective altruist consensus. But obviously, readers who disagree with my cause priorities (i.e. longtermism, of a fairly conventional kind) may disagree with that analysis of deference as well.
Relatedly, one pattern that I've noticed is that people on the forum - including people who aren't deeply immersed in effective altruist thinking - criticise some longstanding effective altruist practices or strategies by arguments that are unconvincing to me. In such cases, my reaction tends to be that they should have another go and think "maybe they've thought more about this than I have - maybe there is something that I've missed?" More often than not, very smart people have thought very extensively about most such issues, and it's therefore unlikely that someone who has thought substantially less about them would be more likely to be right about them. I think that perspective is missing in some of the forum commentary. But again, whether you agree with my on this will depend on your view on the object-level criticisms. If you think these criticisms are in fact convincing, then you're probably less likely to believe that the critics should defer to the effective altruist consensus.
Hey Stefan,
Thanks for the comment, I think this describes a pretty common view in EA that I want to push back against.
Let's start with the question of how much you have found practical criticism of EA valuable. When I see posts like this or this, I see them as significantly higher value than those individuals deferring to large EA orgs. Moving to a more practical example; older/more experienced organizations/people actually recommended against many organizations (CE being one of them and FTX being another). These organizations’ actions and projects seem pretty insanely high value relative to others, for example, a chapter leader who basically follows the same script (a pattern I definitely personally could have fallen into). I think something that is often forgotten about is the extremely high upside value of doing something outside of the Overton window, even if it has a higher chance of failure. You could also take a hypothetical, historical perspective on this; e.g. if EA deferred to only GiveWell or only to more traditional philanthropic actors, how impactful would this have been?.
Moving a bit more to the philosophical side, I do think you should put the same weight on your views as other epistemic peers. However, I think there are some pretty huge ethical and meta epistemic assumptions that a lot of people do not realize they are deferring to when going with what a large organization or experienced EA thinks. Most people feel pretty positive when deferring based on expertise (e.g. “this doctor knows what a CAT scan looks like better than me”, or “Givewell has considered the impact effects of malaria much more than me”). I think these sorts of situations lend themselves to higher deference. Something like “how much ethical value do I prescribe to animals”, or “what is my tradeoff of income to health” are; 1) way less considered, and 2) much harder to gain clarification on from deeper research. I see a lot of deferrals based on this sort of thing e.g. assumptions that GiveWell or GPI do not have pretty strong baseline ethical and epistemic assumptions.
I think the amount of hours spent thinking about an issue is a somewhat useful factor to consider (among many others) but is often used as a pretty strong proxy without regards to other factors; e.g. selection effects (GPI is going to hire people with a set of specific viewpoints coming in), or communication effects (e.g. I engaged considerably less in EA when I thought direct work was the most impactful thing, compared to when I thought meta was the most important thing.). I have also seen many cases where people make big assumptions about how much consideration has in fact been put into a given topic relative to its hours (e.g. many people assume more careful, broad-based cause consideration has been done than really has been done. When you have a more detailed view of what different EA organizations are working on, you see a different picture.).
On the philosophical side paragraph - totally agree; this is why worldview diversification makes so much sense (to me). The necessity of certain assumptions leads to divergence of kinds of work, and that is a very good thing, because maybe (almost certainly) we are wrong in various ways, and we want to be alive and open to new things that might be important. Perhaps on the margin an individual's most rational action could sometimes be to defer more, but as a whole, a movement like EA would be more resilient with less deference.
Disclaimer: I personally find myself very turned off by the deference culture in EA. Maybe that's just the way it should be though.
I do think that higher deference cultures are better at cooperating and getting things done - and these are no easy tasks for large movements. There have also been movements that have done terrible things in the past, accidentally, with these properties. There have also been movements that have done wonderful things, with these properties.
I'd guess there may be a correlation between people who think there should be more deference being in the "row" camp and people who think less in the "steer" camp, or another camp, described here.
I worry a bit that these discussions become a bit anecdotal; and that the arguments rely on examples where it's not quite clear what the role of deference or its absence was. No doubt there are examples where people would have done better if they had deferred less. That need not change the overall picture that much.
Fwiw, I think one thing that's important to keep in mind is that deference doesn't necessarily entail working within a big project or or org. EAs have to an extent encouraged others to start new independent projects, and deference to such advice thus means starting an independent project rather than working within a big project or org.
I think there are several things wrong with the Equal Weight View, but I think this is the easiest way to see it:
Let's say I have O(H)=2:1 which I updated from a prior of 1:6. Now I meet someone who A) I trust to be rational as much as myself, and B) I know started with the same prior as me, and C) I know cannot have seen the evidence that I have seen, and D) I know has updated on evidence independent of evidence I have seen.
They say O(H)=1:2.
Then I can infer that they updated from 1:6 to 1:2 by multiplying with a likelihood ratio of 3:1. And because C and D, I can update on that likelihood ratio in order to end up with a posterior of O(H)=6:1.
The equal weight view would have me adjust down, whereas Bayes tells me to adjust up.
This post seems to amount to replying "No" to Vaidehi's question since it is very long but does not include a specific example.
> I won't be able to give you examples where I demonstrate that there was too little deference
I don't think that Vaidehi is asking you to demonstrate anything in particular about any examples given. It's just useful to give examples that illustrate your own subjective experience on the topic. It would have conveyed more information and perspective than the above post.
Agreed--except that on the margin I'd rather encourage EAs to defer less than more. :) But of course some should defer less, and others more, and also it depends on the situation, etc. etc.