Doing a lot of good has been a major priority in my life for several years now. Unfortunately I made some substantial mistakes which have lowered my expected impact a lot, and I am on a less promising trajectory than I would have expected a few years ago. In the hope that other people can learn from my mistakes, I thought it made sense to write them up here! I will attempt to list the mistakes which lowered my impact most over the past several years in this post and then analyse their causes. Writing this post and previous drafts has also been very personally useful to me, and I can recommend undertaking such an analysis.
Please keep in mind that my analysis of my mistakes is likely at least a bit misguided and incomprehensive.
It would have been nice to condense the post a bit more and structure it better, but having already spent a lot of time on it and wanting to move on to other projects, I thought it would be best not to let the perfect be the enemy of the good!
To put my mistakes into context, I will give a brief outline of what happened in my career-related life in the past several years before discussing what I consider to be my main mistakes.
Background
I came across the EA Community in 2012, a few months before I started university. Before that point my goal had always been to become a researcher. Until early 2017, I did a mathematics degree in Germany and received a couple of scholarships. I did a lot of ‘EA volunteering’ over the years, mostly community building and large-scale grantmaking. I also did two unpaid internships at EA orgs, one during my degree and one after graduating, in summer 2017.
After completing my summer internship, I started to try to find a role at an EA org. I applied to ~7 research and grantmaking roles in 2018. I got to the last stage 4 times, but received no offers. The closest I got was receiving a 3-month-trial offer as a Research Analyst at Open Phil, but it turned out they were unable to provide visas.
In 2019, I worked as a Research Assistant for a researcher at an EA aligned university institution on a grant for a few hundred hours. I stopped as there seemed to be no route to a secure position and the role did not seem like a good fit.
In late 2019 I applied for jobs suitable for STEM graduates with no experience. I also stopped doing most of my EA volunteering.
In January 2020 I began to work in an entry-level data analyst role in the UK Civil Service which I have been really happy with. In November, after 6.5mon full-time equivalent worked, I received a promotion to a more senior role with management responsibility and a significant pay rise.
First I am going to discuss what I think I did wrong from a first-order practical perspective. Afterwards I will explain which errors in my decision making process I consider the likely culprits for these mistakes - the patterns of behaviour which need to be changed to avoid similar mistakes in the future.
A lot of the following seems pretty silly to me now, and I struggle to imagine how I ever fully bought into the mistakes and systematic errors in my thinking in the first place. But here we go!
What did I get wrong?
-
I did not build broad career capital nor kept my options open. During my degree, I mostly focused on EA community building efforts as well as making good donation decisions. I made few attempts to build skills for the type of work I was most interested in doing (research) or skills that would be particularly useful for higher earning paths (e.g. programming), especially later on. My only internships were at EA organisations in research roles. I also stopped trying to do well in my degree later on, and stopped my previously-substantial involvement in political work.
-
In my first year after finishing my degree and post-graduation summer internship, I only applied to ~7 roles, exclusively at EA organisations. That is way too small a number for anyone who actually wants a job!
-
1.5 years after graduating, I gave up hoping for any EA org role. I started to apply for ordinary jobs, but then accepted a support role for an EA researcher on a grant after a personal introduction, only working part time. This was despite the fact that there were few outside view signs that this would be a good idea except it being an EA role, and no clear plan how this would result in a real job [or impact].
These mistakes were not created equal - the first and second had a much larger negative impact than the third. The combination of the second and third mistake had the direct impact of me being unemployed or underemployed for over 2 years when I wanted to work. When I finally started a ‘real job’, it had been almost 3 years since I graduated.
Which systematic errors in my decision-making likely caused these mistakes?
While I tried to group my assessment of the underlying causes of my mistakes by theme to make them easier to read, they often tie into each other. I am uncertain in my assessments even now, so please read the below in that light.
When I thought about how I want to do the most good in my life, I prioritised being cooperative and loyal to the EA community over any other concrete goal to have an impact. I think that was wrong, or at least wrong without a very concrete plan backing it up.
I put too much weight on what other people thought I should be doing, and wish I had developed stronger internal beliefs. Because I wanted to cooperate, I considered a nebulous concept of ‘the EA community’ the relevant authority for decision-making. Around 2015-2019 I felt like the main message I got from the EA community was that my judgement was not to be trusted and I should defer, but without explicit instructions how and who to defer to. I did plenty of things just because they were ‘EA’ without actually evaluating how much impact I would be having or how much I would learn.
I thought that my contributions (outreach activities, donations & grantmaking, and general engagement) would ensure that I would get the benefits of being dedicated, like a secure role within the EA structure once it seemed like the EA community was no longer financially constrained. I did not distinguish between ‘professional career development’ and ‘volunteering’, because I viewed everything under the EA community umbrella.
There are many examples of me taking what other EAs said much too seriously, but here are some of the more impactful ones:
When I joined the community, I received plenty of extremely positive feedback. I trusted these statements too much, and wrongly had the impression that I was doing well and would by default be able to do a lot of good in the near future. I also over-weighted occurrences like being invited to more exclusive events. When an organisation leader said to me I could work at their organisation, I interpreted it literally. When other senior EAs asked me to do something specific, I thought I should do as told.
I stopped doing political work (in ~2014/2015) as I had the impression that it was EA consensus that this was not particularly valuable. I now regret this, it might have opened high impact routes later on. The network I had there was great as well, some of the people I used to work with have done very well on the political ladder.
When I received a trial offer from OpenPhil as a Research Analyst in 2018, I thought this would mostly end my job search. Even though I could not do the trial for visa reasons, I thought the offer would make it easy to find a job in the EA sphere elsewhere. This was both based on things Open Phil told me and the very high regard the community seemed to hold this application process and opportunity in. That you could succeed to get to the trial stage but still not be able to find a job in the EA sphere caught me off-guard.
I also focused far too much on working at EA organisations. In 2015, talk about how Effective Altruism was talent-constrained became popular. Up until that point, I had been prepared to aim for earning-to-give later, take on an advocacy role, or go into academia. But at that point I started to take it for granted that I would be able to work at EA orgs after my degree. I did not think enough about how messages can be distorted once they travel through the community and how this message might not have been the one 80,000 Hours had intended. I might have noticed this had I paid more intention to their writing on the topic of talent-constraints and less to the version echoed by the community. Paying more attention to their written advice, I could have noticed the conflict between the talent-constrained message as echoed by the community with the actual 80,000 Hours advice to keep your options open and having Plan A, B and Z.
Similar things can be said about the risks newly started projects could possibly entail. While I think the reasoning e.g. 80,000 Hours brought forth on the topic is sound, again I did not appreciate how messages get distorted and amplified through the community. My interpretation was that my judgement generally was not to be trusted, and if it was not good enough to start new projects myself, I should not make generic career decisions myself, even where the possible downsides were very limited.
I was too willing to take risks.
Some level of risk-taking is good and necessary, but my deference to the EA community made me blind towards the risks I was taking. I did not think carefully enough about the position I would be in if focusing on EA jobs failed: that of a long-unemployed graduate with no particular skills.
The advice to be willing to take more risks prominent within EA was geared towards ‘talented, well-credentialed graduates without responsibilities’ - whether talented or not, I am not well-credentialed and have dependents. Therefore I should have questioned more how much this advice really applied to me.
I stopped trying to do well in my degree, as good grades seemed unnecessary if I was just going to work at EA organisations later anyway. I thought the time could be much better invested on community building or better donation decisions. I also did not try to do any kind of research effort despite this still being the path I was most interested in.
I put much less effort into developing my broader capabilities and intellectual interests. I did not think about the fact that most of my EA volunteering activities would bring me little career capital. I should have paid more attention to the fact that it would be especially hard for me to enter academia given my grades or other direct work paths which usually require years of up-front investment.
Unfortunately, even once I understood that direct work is not about working at EA orgs, I am not really qualified to start on any of the most-discussed routes without substantial upskilling which in turn is not easily accessible to me.
I underestimated the cost of having too few data points.
This one sounds a bit nebulous, there are a few different aspects I am trying to get at.
Something I struggled with while trying to find a job was making sense of the little information I had. I was endlessly confused why I seemed to have done so well in some contexts, but was still failing to get anywhere. Often I wondered whether there was something seriously wrong with me, as I would guess is often the case for outwardly qualified people who are underemployed regardless. I now think there was nothing to explain here - most people who want a job as much as I did apply to more than one highly competitive job every month or two.
While I knew on some level that a lot of randomness is involved in many processes, including job applications, I still tried to find meaning in the little information I had. Instead, my goal should have been to gather much more data, especially as I got more desperate. To be fair to my past self, I would have been keen to apply to more jobs, but as I was only interested in EA org jobs, there were way too few to apply to.
It was obvious to me that I was missing out on lots of career capital including valuable skills while not working: true almost by definition. But I do not think I appreciated how valuable work is as a calibration exercise. Whenever people talked about ‘figuring out what you are good at’, I didn’t understand why this was so valuable - while there would be some information I would gain, this did not seem that important compared to just getting better at things.
Now I think I mostly misunderstood what people were trying to get at with ‘figuring out what you are good at’. What you are good at is mostly about relative not absolute performance. For me, learning ‘what I am good at’ this year has mostly not looked like discovering I am better or worse at a skill than I thought, but instead discovering how good other people are at the same skills. Particularly useful are comparisons to people who are otherwise similar and might work in a similar profession. I have gotten some positive feedback on traits I thought I was weak on, but was apparently still better than other analysts. I have also found out about some skill axes that I never realised there was any meaningful variance on.
I did not notice my ignorance around how some social structures operate.
I found it really difficult to understand how I was supposed to navigate the ‘professional’ EA community and had a poor model of what other people’s expectations were.
I had no luck applying the advice to ‘talk to other people’ when trying to find a job through informal networks. It did work for people around me, and I still don’t really know why; probably the whole conversation needs to be framed in a very specific way. The couple of times I tried to be more direct I made a botch of it.
I also had the wrong framework when it came to interactions with potential employers, and wider experience with applying to jobs (as well as running more application processes myself) has helped me see that. My understanding of what potential employers would judge is whether I was a generally smart and capable person. This was wrong, a better focus would have been whether I can help them solve their very specific problems. I probably would have approached some interactions with potential employers differently if I had internalized this earlier. I failed to model other people’s preferences in these interactions as separate from my own and did not try hard enough to understand what they are.
I thought having no strong preferences for a role within the EA community would be considered a good thing, as it proved that I was being cooperative. But most employers probably want to hear about how their role fits your particular skills and that you are really excited about it, including within the EA sphere.
I underestimated the cost to my motivation and wellbeing, and how these being harmed could curb my expected impact.
By late 2018, I had been waiting for opportunities for a year and felt pretty bad. At that point, my first priority should have been to get out of that state. When I accepted the research assistant role, I was insufficiently honest with myself about whether I would be able to do well given how burnt out I felt.
As there was no clear path from being a research assistant on a grant into something more secure and well defined, I just stayed in a burnt out state for longer. In autumn 2019 I thought it would be better for me to mentally distance myself from the EA community, which did make me feel a bit better.
I was still often profoundly miserable about my employment situation. The big shift here came after starting my data analyst job in January 2020 and my misery which had reduced me to tears each week for over 2 years was basically cured overnight.
While the direction of the change is not surprising, it has been astounding to me how much more productive I have been this year compared to previous years.
Being miserable also hindered my ability to assess my prospects rationally. It took me a long time to properly give up on my EA job prospects: whenever I thought this path might not work out for me at all, the thought was too horrifying to seriously contemplate. Having to start again at zero with my investments having been in vain just seemed too awful. Perhaps this would deserve its own mention as a high level systematic error: When confronted with failure, I had left no line of retreat.
What next?
As mentioned, I have been much, much happier since I started working in the Civil Service, especially now with the promotion. It is really great for me to be in an environment in which I feel encouraged to take as much responsibility as I want and solve problems by myself.
My main goal this year has been to become more enthusiastic and excitable, especially regarding my ability to have an impact, and I am glad to report that this has been going very well! I have also felt much more in control of my future and have been able to be strategic about my goals.
For the near future my main aim in my job is still to gain more skills and get much better calibrated on what my strengths and weaknesses are relative to other people.
I also want to get much better calibrated on what might be possible for me in the medium to long term, as I still want to consider options broadly.
I am still in the process of figuring out what my personal beliefs are on where I can have the most impact in addition to the personal fit considerations. This year I have spent a lot of time thinking about how large a role I want doing good to play in my life as well as moral considerations on what I consider doing good to be.
Next year I hope to make more progress on my beliefs around cause prioritisation as well as practical considerations on how to do a lot of good. Ironically, mentally distancing myself from the EA sphere a bit is just what I needed to make this a plausible goal.
A critical assessment of what I have written here is very welcome! Please point it out if you think I forgot some mistakes or misanalysed them.
Special thanks to AGB, Richard Ngo, Max Daniel and Jonas Vollmer who gave feedback on drafts of this post.
I also get a lot of this vibe from (parts of) the EA community, and it drives me a little nuts. Examples:
Being epistemically modest by, say, replacing your own opinions with the average opinion of everyone around you, might improve the epistemics of the majority of people (in fact it almost must by definition), but it is a terrible idea on a group level: it's a recipe for information cascades, groupthink and herding.
In retrospect, it's not surprising that this has ended up with numerous people being scarred and seriously demoralized by applying for massively oversubscribed EA jobs.
I guess it's ironic that 80,000 Hours—one of the most frequent repeaters of the "don't accidentally cause harm" meme—seems to have accidentally caused you quite a bit of harm with this advice (and/or its misinterpretations being repeated by others)!
That last paragraph is a good observation, and I don’t think it’s entirely coincidental. 80k has a few instances in their history of accidentally causing harm, which has led them (correctly) to be very conservative about it as an organisation.
The thing is, career advice and PR are two areas 80k is very involved in and which have particular likelihood of causing as much harm as good, due to bad advice or distorted messaging. Most decisions individual EAs make are not like this, and it’s a mistake if they treat 80k’s caution as a reflection of how cautious they should be. Or worse, act even more cautiously reasoning the combined intelligence of the 80k staff is greater than their own (likely true, but likely irrelevant).
I don't think any of 80k's career advice has caused much harm compared to the counterfactual of not having given that advice at all, so I feel a bit confused how to think about this. Even the grossest misrepresentation of EtG being the only way to do good or something still strikes me as better than the current average experience a college graduate has (which is no guidance, and all career advice comes from companies trying to recruit you).
I think the comparison to "the current average experience a college graduate has" isn't quite fair, because the group of people who see 80k's advice and act on is is already quite selected for lots of traits (e.g. altruism). I would be surprised if the average person influenced by 80k's EtG advice had the average college graduate experience in terms of which careers they consider and hence, where they look for advice, e.g. they might already be more inclined to go into policy, the non-profit sector or research to do good.
(I have no opinion on how your point comes out on the whole. I wasn't around in 2015, but intuitively it would also surprise me if 80k didn't do substantially more good during that time than bad, even bracketing out community building effects (, which, admittedly, is hard))
(Disclaimer: I am OP’s husband)
As it happens, there are a couple of examples in this post where poor or distorted versions of 80k advice arguably caused harm relative to no advice; over-focus on working at EA orgs due to ‘talent constraint’ claims probably set Denise’s entire career back by ~2 years for no gain, and a simplistic understanding of replaceability was significantly responsible for her giving up on political work.
Apart from the direct cost, such events leave a sour taste in people’s mouths and so can cause them to dissociate from the community; if we’re going to focus on ‘recruiting’ people while they are young, anything that increases attrition needs to be considered very carefully and skeptically.
I do agree that in general it’s not that hard to beat ‘no advice’, rather a lot of the need for care comes from simplistic advice’s natural tendency to crowd out nuanced advice.
I don’t mean to bash 80k here; when they become aware of these things they try pretty hard to clean it up, they maintain a public list of mistakes (which includes both of the above), and I think they apply way more thought and imagination to the question of how this kind of thing can happen than most other places, even most other EA orgs. I’ve been impressed by the seriousness with which they take this kind of problem over the years.
Yeah, totally agree that we can find individual instances where the advice is bad. Just seems pretty unlikely for that average to be worse, even just by the lights of the person who is given advice (and ignoring altruistic effects, which presumably are more heavy-tailed).
I think I probably agree with the general thrust of this comment, but disagree on various specifics.
'Intelligent people disagree with this' is a good reason against being too confident in one's opinion. At the very least, it should highlight there are opportunities to explore where the disagreement is coming from, which should hopefully help everyone to form better opinions.
I also don't feel like moral uncertainty is a good example of people deferring too much.
A different way to look at this might be that if 'good judgement' is something that lots of people need in their careers, especially if they don't follow any of the priority paths (as argued here), this is something that needs to be trained - and you don't train good judgement by always blindly deferring.
Yeah, and besides the training effect there is also the benefit that while one person who disagrees with hundreds is unlikely to be correct, if they are correct, it’s super important that those hundreds of others get to learn from them.
So it may be very important in expectation to notice such disagreements, do a lot of research to understand one’s own and the others’ position as well as possible, and then let them know of the results.
(And yes, the moral uncertainty example doesn’t seem to fit very well, especially for antirealists.)
I'd say that "Intelligent people disagree with this" is a good reason to look into what those people think and why - I agree that it should make you less certain of your current position, but you might actually end up more certain of your original opinion after you've understood those disagreements.
See also answers here mentioning that EA feels "intellectually stale". A friend says he thinks a lot of impressive people have left the EA movement because of this :(
I feel bad, because I think maybe I was one of the first people to push the "avoid accidental harm" thing.
"Stagnation" was also the 5th most often mentioned reason for declining interest in EA, over the last 12 months, when we asked about this in the 2019 EA Survey, accounting for about 7.4% of responses.
Thanks, David, for that data.
There was some discussion about the issue of EA intellectual stagnation in this thread (like I say in my comment, I don't agree that EA is stagnating).
Yeah, I think it's very difficult to tell whether the trend which people take themselves be perceiving is explained by there having been a larger amount of low hanging fruit in the earlier years of EA, which led to people encountering a larger number of radical new ideas in the earlier years, or whether there's actually been a slowdown in EA intellectual productivity. (Similarly, it may be that because people tend to encounter a lot of new ideas when they are first getting involved in EA, people perceive the insights being generated by EA as slowing down). I think it's hard to tell whether EA is stagnating in a worrying sense in that it is not clear how much intellectual progress we should expect to see now that some of the low hanging fruit is already picked.
That said, I actually think that the positive aspects of EA's professionalisation (which you point to in your other comment) may explain some of the perceptions described here, which I think are on the whole mistaken. I think in earlier years, there was a lot of amateur, broad speculation for and against various big questions in EA (e.g. big a priori arguments about AI versus animals, much of which was pretty wild and ill-informed). I think, conversely, we now have a much healthier ecosystem, with people making progress on the myriad narrower, technical problems that need to be addressed in order to address those broader questions.
Thanks David, this is more or less what I was trying to express with my response to Stefan in that thread.
I want to add that "making intellectual progress" has two different benefits: One is the obvious one, figuring out more true things so they can influence our actions to do more good. As you say, we may actually be doing better on that one.
The other one is to attract people to the community by it being an intellectually stimulating place. We might be losing the kind of people who answered 'stagnation' in the poll above, as they are not able to participate in the professionalised debates, if they happen in public at all.
On the other hand, this might mean that we are not deterring people anymore who may have felt like they need to be into intellectual debates to join the EA community. I don't know what the right trade-off is, but I suspect it's actually more important not to put latter group off.
I actually think the principles of deference to expertise and avoiding accidental harm are in principle good and we should continue using them. However, in EA the barrier to being seen as an expert is very low - often its enough to have written a blog or forum post on something, having invested less than 100 hours in total. For me an expert is someone who has spent the better part of his or her career working in a field, for example climate policy. While I think the former is still useful to give an introduction to a field, the latter form of expertise has been somewhat undervalued in EA.
I guess it depends on what topics you're referring to, but regarding many topics, the bar for being seen as an expert within EA seems substantially higher than 100 hours.
I gave a talk about this, so I consider myself to be one of the repeaters of that message. But I also think I always tried to add a lot of caveats, like "you should take this advice less seriously if you're the type of person who listens to advice like this" and similar. It's a bit hard to calibrate, but I'm definitely in favor of people trying new projects, even at the risk of causing mild accidental harm, and in fact I think that's something that has helped me grow in the past.
If you think these sorts of framing still miss the mark, I'd be interested in hearing your reasoning about that.
I'm somewhat sympathetic to the frustration you express. However, I suspect the optimal response isn't to be more or less epistemically modest indiscriminately. Instead, I suspect the optimal policy is something like:
I think this way one can largely have the best of both worlds.
(I vaguely remember that there is a popular post making a similar recommendation, but couldn't quickly find it.)
Something I want to add here:
I am not sure whether my error was how much I was deferring in itself. But the decision to defer or not should be made on well defined questions and clearly defined 'experts' you might be deferring to. This is not what I was doing. I was deferring on a nebulous question ('what should I be doing?') to an even more nebulous expert audience (a vague sense of what 'the community' wanted).
What I should have been doing instead first is to define the question better: Which roles should I be pursuing right now?
This can then be broken down further into subquestions on cause prioritisation, which roles are promising avenues within causes I might be interested in, which roles I might be well suited for, etc, whose information I need to aggregate in a sensible fashion to answer the question which roles I should be pursuing right now.
For each of these subquestions I need to make a separate judgement. For some it makes more sense to defer, for others, less so. Disappointingly, there is no independent expert panel investigating what kind of jobs I might excel at.
But then who to defer to, if I think this is a sensible choice for a particular subquestion, also needs to be clearly defined: for example, I might decide that it makes sense to take 80k at their word about which roles in a particular cause area are particularly promising right now, after reading what they actually say on their website on the subject, perhaps double-checking by asking them via email and polling another couple of people in the field.
'The community' is not a well defined expert panel, while the careful aggregation of individual opinions can be, who again, need to be asked well defined questions. Note that this can true even if I gave equal weight to every EA's opinion: sometimes it can seem like 'the community' has an opinion that only few individual EAs hold if actually asked, if any. This is especially true if messaging is distorted and I am not actually asking a well defined question.
Taking into account specific facts or arguments made by other people seems reasonable here. Just writing down e.g. "person X doesn't like MIRI" in the "cons" column of your spreadsheet seems foolish and wrongheaded.
Framing it as "taking others' views into account" or "ignoring others' views" is a big part of the problem, IMO—that language itself directs people towards evaluating the people rather than the arguments, and overall opinions rather than specific facts or claims.
I think we disagree. I'm not sure why you think that even for decisions with large effects one should only or mostly take into account specific facts or arguments, and am curious about your reasoning here.
I do think it will often be even more valuable to understand someone's specific reasons for having a belief. However, (i) in complex domains achieving a full understanding would be a lot of work, (ii) people usually have incomplete insight into the specific reasons for why they hold a certain belief themselves and instead might appeal to intuition, (iii) in practice you only have so much time and thus can't fully pursue all disagreements.
So yes, always stopping at "person X thinks that p" and never trying to understand why would be a poor policy. But never stopping at that seems infeasible to me, and I don't see the benefits from always throwing away the information that X believes p in situations where you don't fully understand why.
For instance, imagine I pointed a gun to your head and forced you to now choose between two COVID mitigation policies for the US for the next 6 months. I offer you to give you additional information of the type "X thinks that p" with some basic facts on X but no explanation for why they hold this belief. Would you refuse to view that information? If someone else was in that situation, would you pay for me not giving them this information? How much?
There is a somewhat different failure mode where person X's view isn't particularly informative compared to the view of other people Y, Z, etc., and so by considerung just X's view you give it undue weight. But I don't think you're talking about that?
I'm partly puzzled by your reaction because the basic phenomenon of deferring to the output of others' reasoning processes without understanding the underlying facts or arguments strikes me as not unusual at all. For example, I believe that the Earth orbits the Sun rather than the other way around. But I couldn't give you any very specific argument for this like "on the geocentric hypothesis, the path of this body across the sky would look like this". Instead, the reason for my belief is that the heliocentric worldview is scientific consensus, i.e. epistemic deference to others without understanding their reasoning.
This also happens when the view in question makes a difference in practice. For instance, as I'm sure you're aware, hierarchical organizations work (among other things) because managers don't have to recapitulate every specific argument behind the conclusions of their reports.
To sum up, a very large amount of division of epistemic labor seems like the norm rather than the exception to me, just as for the division of manual labor. The main thing that seems somewhat unusual is making that explicit.
I note that the framing / example case has changed a lot between your original comment / my reply (making a $5m grant and writing "person X is skeptical of MIRI" in the "cons" column) and this parent comment ("imagine I pointed a gun to your head and... offer you to give you additional information;" "never stopping at [person X thinks that p]"). I'm not arguing for entirely refusing to trust other people or dividing labor, as you implied there. I specifically object to giving weight to other people's top-line views on questions where there's substantial disagreement, based on your overall assessment of that particular person's credibility / quality of intuition / whatever, separately from your evaluation of their finer-grained sub-claims.
If you are staking $5m on something, it's hard for me to imagine a case where it makes sense to end up with an important node in your tree of claims whose justification is "opinions diverge on this but the people I think are smartest tend to believe p." The reason I think this is usually bad is that (a) it's actually impossible to know how much weight it's rational to give someone else's opinion without inspecting their sub-claims, and (b) it leads to groupthink/herding/information cascades.
As a toy example to illustrate (a): suppose that for MIRI to be the optimal grant recipient, it both needs to be the case that AI risk is high (A) and that MIRI is the Best organization working to mitigate it (B). A and B are independent. The prior is (P(A) = 50, P(B) = 50). Alice and Bob have observed evidence with a 9:1 odds ratio in favor of A, so think (P(A) = 90, P(B) = 50). Carol has observed evidence with a 9:1 odds ratio in favor of B. Alice, Bob and Carol all have the same top-line view of MIRI (P(A and B) = 0.45), but the rational aggregation of Alice and Bob's "view" is much less positive than the rational aggregation of Bob and Carol's.
It's interesting that you mention hierarchical organizations because I think they usually follow a better process for dividing up epistemic labor, which is to assign different sub-problems to different people rather than by averaging a large number of people's beliefs on a single question. This works better because the sub-problems are more likely to be independent from each other, so they don't require as much communication / model-sharing to aggregate their results.
In fact, when hierarchical organizations do the other thing—"brute force" aggregate others' beliefs in situations of disagreement—it usually indicates an organizational failure. My own experience is that I often see people do something a particular way, even though they disagree with it, because they think that's my preference; but it turns out they had a bad model of my preferences (often because they observed a contextual preference in a different context) and would have been better off using their own judgment.
I think I perceive less of a difference between the examples we've been discussing, but after reading your reply I'm also less sure if and where we disagree significantly.
I read your previous claim as essentially saying "it would always be bad to include the information that some person X is skeptical about MIRI when making the decision whether to give MIRI a $5M grant, unless you understand more details about why X has this view".
I still think this view basically commits you to refusing to see information of that type in the COVID policy thought experiment. This is essentially for the reasons (i)-(iii) I listed above: I think that in practice it will be too costly to understand the views of each such person X in more detail.
(But usually it will be worth it to do this for some people, for instance for the reason spelled out in your toy model. As I said: I do think it will often be even more valuable to understand someone's specific reasons for having a belief.)
Instead, I suspect you will need to focus on the few highest-priority cases, and in the end you'll end up with people X1,…,Xl whose views you understand in great detail, people Y1,…,Ym where your understanding stops at other fairly high-level/top-line views (e.g. maybe you know what they think about "will AGI be developed this century?" but not much about why), and people Z1,…,Zn of whom you only know the top-line view of how much funding they'd want to give to MIRI.
(Note that I don't think this is hypothetical. My impression is that there are in fact long-standing disagreements about MIRI's work that can't be fully resolved or even broken down into very precise subclaims/cruxes, despite many people having spent probably hundreds of hours on this. For instance, in the writeups to their first grants to MIRI, Open Phil remark that "We found MIRI’s work especially difficult to evaluate", and the most recent grant amount was set by a committee that "average[s] individuals’ allocations" . See also this post by Open Phil's Daniel Dewey and comments.)
At that point, I think you're basically in a similar situation. There is no gun pointed at your head, but you still want to make a decision right now, and so you can either throw away the information about the views of person Zi or use it without understanding their arguments.
Furthermore, I don't think your situation with respect to person Yj is that different: if you take their view on "AGI this century?" into account for the decision whether to fund MIRI but have a policy of never using "bare top-level views", this would commit to to ignoring the same information in a different situation, e.g. the decision whether to place a large bet on whether AGI will be developed this century (purely because what's a top-level view in one situation will be an argument or "specific" fact in another); this seems odd.
(This is also why I'm not sure I understand the relevance of your point on hierarchical organizations. I agree that usually sub-problems will be assigned to different employees. But e.g. if I assign "AGI this century?" to one employee and "is MIRI well run?" to another employee, why am I justified in believing their conclusions on these fairly high-level questions but not justified in believing anyone's view on whether MIRI is worth funding?)
Note that thus far I'm mainly arguing against taking into account no-one's top-level views. Your most recent claim involving "the people I think are smartest" suggests that maybe you mainly object to using a lot of discretion in which particular people's top-level views to use.
I think my reaction to this is mixed: On one hand, I certainly agree that there is a danger involved here (e.g. in fact I think that many EAs defer too much to others EAs relative to non-EA experts), and that it's impossible to assess with perfect accuracy how much weight to give to each person. On the other hand, I think it is often possible to assess this with limited but still useful accuracy, both based on subjective and hard-to-justify assessments of how good someone's judgment seemed in the past (cf. how senior politicians often work with advisors they've had a long work relationship with) and on crude objectives proxies (e.g. 'has a PhD in computer science').
On the latter, you said that specifically you object to allocating weight to someone's top-line opinion "separately from your evaluation of their finer-grained sub-claims". If that means their finer-grained sub-claims on the particular question under consideration, then I disagree for the reasons explained so far. If that means "separately from your evaluation of any finer-grained sub-claim they ever made on anything", then I agree more with this, though still think this is both common and justified in some cases (e.g. if I learn that I have rare disease A for which specialists universally recommend drug B as treatment, I'll probably happily take drug B without having ever heard of any specific sub-claim made by any disease-A specialist).
Similarly, I agree that information cascades and groupthink are dangers/downsides, but that they will sometimes be outweighed by the benefits.
If 100 forecasters (who I roughly respect) look at the likelihood of a future event and think it's ~10% likely, and I look at the same question and think it's ~33% likely, I think I will be incorrect in my private use of reason for my all-things-considered-view to not update somewhat downwards from 33%.
I think this continues to be true even if we all in theory have access to the same public evidence, etc.
Now, it does depend a bit on the context of what this information is for. For example if I'm asked to give my perspective on a group forecast (and I know that the other 100 forecasters' predictions will be included anyway), I think it probably makes sense for me to continue to publicly provide ~33% for that question to prevent double-counting and groupthink.
But I think it will be wrong for me to believe 33%, and even more so, wrong to say 33% in a context where somebody else doesn't have access to the 100 other forecasters.
An additional general concern here to me is computational capacity/kindness-- sometimes (often) I just don't have enough time to evaluate all the object-level arguments! You can maybe argue that until I evaluate all the object-level arguments, I shouldn't act, yet in practice I feel like I act with lots of uncertainty* all the time!
___
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you're making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don't think it's unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
__
* Narrow, personal, example that's basically unrelated to EA: I brush my teeth with fluoride toothpaste. I don't floss. Why? Cochrane review was fairly equivocal about flossing and fairly certain about toothbrushing. Maybe it'd be more principled if I looked at the data myself and performed my own meta-analysis on the data, or perhaps self-experimented like Gwern, to decide what dental hygiene activities I should take. But in practice I feel like it's a reasonable decision procedure to just defer to Cochrane review on the empirical facts of the matter, and apply my own value judgments on what activities to take given the facts available.
I'm not sure if we have a principled disagreement here, it's possible that I just described my view badly above.
I agree that one should act such that one's all-things-considered view is that one is making the best decision (the way I understand that statement it's basically a tautology).
Then I think there are some heuristics for which features of a decision situation make it more or less likely that deferring more (or at all) leads to decisions with that property. I think on a high level I agree with you that it depends a lot "on the context of what this information is for", more so than on e.g. importance.
With my example, I was also trying to point less to importance per se but on something like how the costs and benefits are distributed between yourself and others. This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one's own direct stake is small. I used a decision with large effects on others largely because then it's not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Apologies for the long delay in response, feel free to not reply if you're busy.
Hmm I still think we have a substantive rather than framing disagreement (though I think it is likely that our disagreements aren't large).
Perhaps this heuristic is really useful for a lot of questions you're considering. I'm reminded of AGB's great quote:
For me personally and the specific questions I've considered, I think considering whether/how much to defer to by dividing into buckets of "how much it affects myself or others" is certainly a pretty useful heuristic in the absence of better heuristics, but it's mostly superseded by a different decomposition:
So I think a big/main reason it's bad to defer completely to others (say 80k) on your own career reasons is epistemic: you have so much thought and local knowledge about your own situation that your prior should very strongly be against others having better all-things-considered views on your career choice than you do. I think this is more crux-y for me than how much your career trajectory affects yourself vs others (at any rate hopefully as EAs our career trajectories affect many others anyway!).
On the other hand, I think my Cochrane review example above is a good epistemic example of deference. even though my dental hygiene practices mainly affect myself and not others (perhaps my past and future partners may disagree), I contend it's better to defer to the meta-analysis over my own independent analysis in this particular facet of my personal life.
The other main (non-epistemic) lens I'd use to privilege greater or lower humility is whether the explicit and implicit social expectations privilege deference or independence. For example, we'd generally** prefer government bureaucrats in most situations to implement policies, rather than making unprincipled exceptions based on private judgements. This will often look superficially similar to "how much this affects myself or others."
An example of a dissimilarity is when someone filling out a survey. This is a situation where approximately all of the costs and benefits are borne by other people. So if you have a minority opinion on a topic, it may seem like the epistemically humble-and-correct action is to fill out the poll according to what you believe the majority to think (or alternatively, fill it out with the answer that you privately think is on the margin more conducive to advancing your values).
But in all likelihood, such a policy is one-thought-too-many, and in almost all situations it'd be more prudent to fill out public anonymous polls/surveys with what you actually believe.
Agreed, though I mention this because in discussions of epistemic humility-in-practice, it's very easy to accidentally do double-counting.
*I don't like this phrase, happy to use a better one.
**I'm aware that there are exceptions, including during the ongoing coronavirus pandemic.
Thanks! I'm not sure if there is a significant difference about how we'd actually make decisions (I mean, on prior there is probably some difference). But I agree that the single heuristics I mentioned above doesn't by itself do a great job of describing when and how much to defer, and I agree with your "counterexamples". (Though note that in principle it's not surprising if there are counterexamples to a "mere heuristics".)
I particularly appreciate you describing the "Role expectations" point. I agree that something along those lines is important. My guess is that if we would have debated specific decisions I would have implicitly incorporated this consideration, but I don't think it was clear to me before reading your comment that this is an important property that will often influence my judgment about how much to defer.
I think that in theory Max is right, that there's some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it's probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think "I'll make up my mind on small decisions, and defer on big decisions." But then they'll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake - there's now enough money in EA that, in my opinion, there should be many $5M grants which aren't strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that's because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I'm not confident about that; the hypothesis that we just haven't internalised the "hits-based" mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
I think this is part of the reason. A plausibly bigger reason is that VC funding can't result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of "terrible startup for the future of sentient beings," the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
Good point, I agree this weakens my argument.
I agree it's possible that because of social pressures or similar things the best policy change that's viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that's in fact true.
(Note that when implemented well, the "best of both worlds" policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding 'bad, unilateral action' is a key reason in favor of epistemic deference, it could actually "destigmatize" iconoclastic views if it's common knowledge that an iconoclastic pre-deference view doesn't imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I'm not sure if you think my mistake was mainly to consider a $5M stake a "large-scale" decision or something else, but if it's the former I'm happy to concede that this wasn't the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).
I strongly agree that "the optimal response isn't to be more or less epistemically modest indiscriminately", and with the policy you suggest.
If I recall correctly, somewhat similar recommendations are made in Some thoughts on deference and inside-view models and EA Concepts: Share Impressions Before Credences.
I disagree Max. We can all recall anecdotes of overconfidence because they create well-publicized narratives. With hindsight bias, it seems obvious that overconfidence was the subject. So naturally we overestimate overconfidence risks, just like nuclear power.
The costs of under confidence are invisible and ubiquitous. A grad student fails to submit her paper. An applicant doesn't apply. A graduate doesn't write down her NGO idea. Because you can't see the costs of underconfidence, they could be hundreds or thousands of times the overconfidence costs.
To break apart the question