Ask Rethink Priorities Anything (AMA)

Hi Jonas,

Since we last posted our longtermism budget, we've raised ~$89,500 restricted to longtermism for 2021 (with the largest being the grant recommendation from the Survival and Flourishing Fund). This means we will enter 2021 with ~$121K restricted to longtermism not yet spent. Overall, we'd like to raise an additional $403K-$414K for longtermist work by early 2021.

For full transparency - note that, if necessary, we may also choose to use unrestricted funds on longtermism and that this is not factored into these numbers. We currently have ~$273K in unrestricted funds, though we will likely have non-longtermism things we will need to spend this money on.

Given that we are currently just raising money to cover the salaries of our existing longtermist staff (including operations support) as well as start an longtermism intern program, we expect we will be able to deploy longtermist money quickly. We also have a large talent pool of longtermist researchers we likely could hire this year if we ended up with even more longtermism money.

I did internal modeling/forecasting for our fundraising figures, and at least on the first pass it looked like our longtermist work was more likely to be funding constrained than our other priority cause areas, at least if "funding constrained" is narrowly defined as "what's the probability that we do not raise all the money that we'd like for all planned operations to run smoothly."

My main reasoning was somewhat outside-viewy and focused on general uncertainty: our longtermist team is new, and relative to other of Rethink's cause areas, less well-established with less of a track record of either a) prior funding, b) public work other than Luisa's nuclear risk work, or c) a well-vetted research plan. So I'm just generally unsure of these things.

Three major caveats:

1. I did those forecasts in late October and now I think my original figures were too pessimistic. 

2. Another caveat is that my predictions were more a reflection of my own uncertainty than a lack of inside view confidence in the team. For context, my 5th-95th percentile credible interval spanned ~an order of magnitude across all cause areas.

3. When making the original numbers, I incorporated but plausibly substantially underrated the degree that  the forecasts will change and not just reflect reality. For example, Peter and Marcus may have prioritized different decisions accordingly due to my numbers, or this comment may affect other people's decisions.

Hi Dan,

Thanks for your questions. I'll let Marcus and Peter answer the first two, but I feel qualified to answer the third.

Certainly, the large number of invertebrate animals is an important factor in why we think invertebrate welfare is an area that deserves attention. But I would advise against relying too heavily on numbers alone when assessing the value of promoting invertebrate welfare. There are at least two important considerations worth bearing in mind:

(1) First, among sentient animals, there may be significant differences in capacity for welfare or moral status. If these differences are large enough, they might matter more than the differences in the numbers of different types of animals.

(2) Second, at some point, Pascal's Mugging will rear its ugly head. There may be some point below which we are rationally required to ignore probabilities. It's not clear to me where that point lies. (And it's also not clear that this is the best way to address Pascal's Mugging.) There are about 440 quintillion nematodes alive at any given time, which sounds like a pretty good reason to work on nematode welfare, even if one's credence in their sentience is really low. But nematodes are nothing compared to bacteria. There are something like 5 million trillion trillion bacteria alive at any given time. At some point, it seems as if expected value calculations cease to be appropriately action-guiding, but, again, it's very uncertain where to draw the line.

Thanks for the questions!

On (1), we see our work in WAW as currently doing three things: (1) foundational research (e.g., understanding moral value and sentience, understanding well-being at various stages of life), (2) investigating plausible tractable interventions (i.e., feasible interventions currently happening or doable within 5 years), and (3) field building and understanding (e.g., currently we are running polls to see how "weird" the public finds WAW interventions).

We generally defer to WAI on matters of direct outreach (both academic and general public) and do not prioritize that area as much as WAI and Animal Ethics do. It's hard to say more on how our vision differs from WAI without them commenting, but we collaborate with them a lot and we are next scheduled to sync on plans and vision in early January.

On (2), it's hard to predict exactly what additional restrict donations do, but in general, we expect them to increase in the long run how much we spend in a cause by an amount similar to how much is donated. Reasons for this include: we budget on a fairly long-term basis, so we generally try to predict what we will spend in a space, and then raise that much funding. If we don't raise as much as we'd like, we likely consider allocating our expenses differently; and if we raise more than we expected, we'd scale up our work in a cause area. Because our ability to work in spaces is influenced by how much we raise, generally raising more restricted funding in a space ought to lead to us doing more work in that space.

I can answer 6, as I’ve been doing it for Wild Animal Welfare since I was hired in September. WAW is a new and small field, so it is relatively easy to learn the field, but there’s still so much! I started by going backwards (into the Welfare Biology movement of the 80s and 90s) and forwards (into the WAW EA orgs we know today) from Brain Tomasik, consulting the primary literature over various specific matters of fact. A great thing about WAW being such a young field (and so concentrated in EA) is that I can reach out to basically anyone who’s published on it and have a real conversation. It’s a big shortcut!

I should note that my background is in Evolutionary Biology and Ecology, so someone else might need a lot more background in those basics if they were to learn WAW.

Hi Denis,

Lots of really good questions here. I’ll do my best to answer.

1. Thinking vs reading: I think it depends on the context. Sometimes it makes sense to lean toward thinking more and sometimes it makes sense to lean toward reading more. (I wouldn’t advise focusing exclusively on one or the other.) Unjustified anchoring is certainly a worry, but I think reinventing the wheel is also a worry. One could waste two weeks groping toward a solution to a problem that could have been solved in afternoon just by reading the right review article.

2. Self-consciousness: Yep, I am intimately familiar with hopelessly inchoate thoughts and notes. (I’m not sure I’ve ever completed a project without passing through that stage.) For me at least, the best way to overcome this state is to talk to lots of people. One piece of advice I have for young researchers is to come to terms with sharing your work with people you respect before it’s polished. I’m very grateful to have a large network of collaborators willing to listen to and read my confused ramblings. Feedback at an early stage of a project is often much more valuable than feedback at a later stage.

3. Is there something interesting here?: Yep, this also happens to me. Unfortunately, I don’t have any particular insight. Oftentimes the only way to know whether an idea is interesting is to put in the hard exploratory work. Of course, one shouldn’t be afraid to abandon an idea if it looks increasingly unpromising.

4. *Survival vs. exploratory mindset: Insofar as I understand the terms, an exploratory mindset is an absolute must. Not sure how to cultivate it, though.

5. Optimal hours of work per day: I work between 4 and 8 hours a day. I don’t find any difference in my productivity within that range, though I imagine if I pushed myself to work more than 8, I would pretty quickly hit diminishing returns.

6. Learning a new field: I can’t emphasize enough the value of just talking to existing experts. For me at least, it’s by far the most efficient way to get up-to-speed quickly. For that reason, I really value having a large network of diverse people I can contact with questions. I put a fair amount of effort into cultivating such a network.

7. Hard problems: I’m fortunate that my work is almost always intrinsically interesting. So even if I don’t make progress on a problem, I continue to be motivated to work on it because the work itself is so very pleasant. That said, as I’ve emphasized above, when I’m stuck, I find it most helpful to talk to lots of people about the problem.

8. Emotional motivators: When I reflect on my life as a whole, I’m happy that I’m in a career that aims to improve the world. But in terms of what gets me out of bed in the morning and excited to work, it’s almost never the impact I might have. It’s the intrinsically interesting nature of my work. I almost certainly would not be successful if I did not find my research to be so fascinating.

9. Typing speed: No idea what my typing speed is, but it doesn’t feel particularly fast, and that doesn’t seem to handicap me. I’ve always considered myself a slow thinker, though.

10. Obvious questions: Yeah, I think there is a general skill of “noticing the obvious.” I don’t think I’m great at it, but one thing I do pretty often is reflect on the sorts of things that appear obvious now that weren’t obvious to smart people ~200 years ago.

11. Tiredness, focus, etc.: Regular exercise certainly helps. Haven’t tried anything else. Mostly I’ve just acclimated to getting work done even though I’m tired. (Not sure I would recommend that “solution,” though!)

12. Meta: I’d like to see others answer questions 1, 3, 6, 7, and 10.

2. Personally, I’m very self-conscious about my work and tend to wait to long too share it. But the culture of RP seems to fight that tendency— which I think is very productive!

1. Thinking vs. reading. 

Another benefit of thinking before reading is that it can help you develop your research skills. Noticing some phenomena and then developing a model to explain it is a super valuable exercise. If it turns out you reproduce something that someone else has already done and published, then great, you’ve gotten experience solving some problem and you’ve shown that you can think through it at least as well as some expert in the field. If it turns out that you have produced something novel then it’s time to see how it compares to existing results in the literature and get feedback on how useful it is.

This said, I think this is more true for theoretical work than applied work, e.g. the value of doing this in philosophy > in theoretical economics > in applied economics. A fair amount of EA-relevant research is summarising and synthesising what the academic literature on some topic finds and it seems pretty difficult to do that by just thinking to yourself!

3. Is there something interesting here?

I mostly try to work out how excited I am by this idea and whether I could see myself still being excited in 6 months, since for me having internal motivation to work on a project is pretty important. I also try to chat about this idea with various other people and see how excited they are by it.

4. Survival vs. exploratory mindset. 

I also haven’t heard these terms before, but from your description (which frames a survival mindset pretty negatively), an exploratory mindset comes fairly naturally to me and therefore I haven’t ever actively cultivated it. Lots of research projects fail so extreme risk aversion in particular seems like it would be bad for researchers.

5. Optimal hours of work per day. 

I typically aim for 6-7 hours of deep work a day and a couple of dedicated hours for miscellaneous tasks and meetings. Since starting part-time at RP I’ve been doing 6 days a week (2 RP, 4 PhD), but before that I did 5. I find RP deep work less taxing than PhD work. 6 days a week is at the upper limit of manageable for me at the moment, so I plan to experiment with different schedules in the new year.

6. Learning a new field.

I’m a big fan of textbooks and schedule time to read a couple of textbook chapters each week. Lesswrong’s best textbooks on every subject thread is pretty good for finding them. I usually make Anki flashcards to help me remember the key facts, but I’ve recently started experimenting with Roam Research to take notes which I’m also enjoying so my “learning flow” is in flux at the moment.

8. Emotional motivators. 

My main trick for dealing with this is to always plan my day the night before. I let System 2 Dave work out what is important and needs to be done and put blocks in the calendar for these things. When System 1 Dave is working the next day, his motivation doesn’t end up mattering so much because he can easily defer to what System 2 Dave said he should do. I don’t read too much into lack of System 1 motivation, it happens and I haven’t noticed that it is particularly correlated with how important the work is, it’s more correlated with things like how scary it is to start some new task and irrelevant things like how much sunlight I’ve been getting.

9. Typing speed. 

I struggle to imagine typing speed being a binding constraint on research productivity since I’ve never found typing speed to be a problem for getting into flow, but when I just checked my wpm was 85 so maybe I’d feel different if it was slower. When I’m coding the vast majority of my time is spent thinking about how to solve the problem I’m facing, not typing the code that solves the problem. When I’m writing first drafts, I think typing speed is a bit more helpful for the reasons you mention, but again more time goes into planning the structure of what I want to say and polishing, than the first pass at writing where speed might help.

11. Tiredness, focus, etc. 

My favourite thing to do is to stop working! Not all days can be good days and I became a lot happier and more productive when I stopped beating myself up for having bad days and allowed myself to take the rest of the afternoon off.  

12. Meta. 

The questions I didn’t answer were because I didn’t have much to say about them so I’d be happy to see answers to them!

#9 Typing speed: I think my own belief is that typing speed is probably less important than you appear to believe, but I care enough about it that I logged 53 minutes of typing practice on keybr this year (usually during moments where I'm otherwise not productive and just want to get "in flow" doing something repetitive), and I suspect I still can productively use another 3-5 hours of typing practice next year even if it trades off against deep work time (and presumably many more hours than that if it does not). 

#10 Obvious questions. I suspect that while sometimes ignoring/not noticing "obvious questions/advice" etc is coincidental unforced errors, more often than not there is some form of motivated reasoning going on behind the scenes (eg because this story will invalidate a hypothesis I'm wedded to, because it involves unpleasant tradeoffs, because some beliefs are lower prestige, because it makes the work I do seem less important, etc). I think training myself carefully to notice these things has been helpful, though I suspect I still miss a lot of obvious stuff. 

#11 Tiredness, focus, etc..I haven't figured this out yet and am keen to learn from my coworkers and others! Right now I take a lot of caffeine and I suspect if I were more careful about optimization I should be cycling drugs over a weekly basis rather than taking the same one every day (especially a drug like caffeine that has tolerance and withdrawal symptoms). 

Hi Denis,

Thanks again for these questions. I'll share my answers in a few comments. This context and disclaimer - including that I only started with Rethink a month ago - should be borne in mind.

1. Thinking vs reading

I don't think I really have explicit policies regarding balancing reading against thinking myself and recording my thoughts. Maybe I should. 

I'm somewhat inclined to think that, on the margin and on average (so not in every case), EA would benefit from a bit more reading of relevant literatures (or talking to more experienced people in an area, watching of relevant lectures, etc.), even at the expense of having a bit less time for coming up with novel ideas. 

I feel like EA might have a bit too much a tendency towards "think really hard by oneself for a while, then kind-of reinvent the wheel but using new terms for it". It might be that, often, people could get to similar ideas faster and in a way that connects to existing work better (making it easier for others to find, build on, etc.) by doing some extra reading first.  

Note that this is not me suggesting EAs should increase how much they defer to experts/others/existing work. Instead, I'm tentatively suggesting spending more time learning what experts/others/existing work has to say, which could be followed by agreeing, disagreeing, critiquing, building on, proposing alternatives, striking out in a totally different direction, etc.

(On this general topic, I liked the post The Neglected Virtue of Scholarship.)

10. “Obvious questions”

(Just my personal, current, non-expert thoughts, as always. Also, I’m not sure I’m addressing precisely the question you had in mind.)

A summary of my recommendations in this vicinity:

1. If people want to do research and want a menu of ideas/questions to work on, including ideas/questions that seem like they obviously should have a bunch of work on them but don’t yet, they could check out this central directory for open research questions, and/or an overlapping 80,000 Hours post. 
2. If people want to discover “new” instances of such ideas/questions, one option might be to just try to notice ideas/variables/assumptions that seem important to some people’s beliefs, but that seem debatable and vague, have been contested by others, and/or haven’t been stated explicitly and fleshed out.
   • One way to do this might be to have a go at rigorously, precisely writing out the arguments that people seem to be acting as if they believe, in order to spot the assumptions that seem required but that those people haven't stated/emphasised. 
   • One could then try to explore those assumptions in detail, either just through more fleshed-out “armchair reasoning”, or through looking at relevant empirical evidence and academic work, or through some mixture of those things.
   • I think this is a big part of what I’ve done this year.
     • Here’s one example of a piece of my own work which came from roughly that sort of process. 

I’ll add more detailed thoughts below.

---

I interpret this question as being focused on cases in which an idea/open question seems like it should’ve been obvious, or seems obvious in retrospect, yet it has been neglected so far. (Or the many cases we should assume still exist in which the idea/question is still neglected, but would - if and when finally tackled - seem obvious.)

It seems to me that there are two major types of such cases:

• Unnoticed: Cases in which the ideas/open questions haven’t even been noticed by almost anyone
  • Or at least, almost anyone in the relevant community/field. 
    • So I'd still say an idea counts as "unnoticed" for these purposes even if, for example, a very similar ideas has been explored thoroughly in sociology, but no one in longtermism has noticed that that idea is relevant to some longtermist issue, nor independently arrived at a similar idea.
• Noticed yet neglected: Cases in which the ideas/open questions have been noticed, but no one has really fleshed them out or tackled them much
  • E.g., a fair number of longtermists have noticed that the question of how likely various types of recovery are from various types of civilizational collapse. But as far as I’m aware, there was nothing even approaching a thorough analysis of the question until some recent still-in-progress work, and there's still room for much more work here.
    • More thoughts and notes on this here and here.
  • Another example is questions related to how likely global, stable totalitarianism is; what factors could increase or decrease the odds of that; and what to do about this. Some people have highlighted such questions (including but not only in the context of advanced AI), but I’m not aware of any detailed work on them.

This is really more a continuum than a binary distinction. In almost all cases, there’s probably been someone in a relevant community who’s at least briefly noticed something relevant. But sometimes it’ll just be that something kind-of relevant has been discussed verbally a few times and then forgotten, while other times it’ll be that people have prominently highlighted pretty precisely the relevant open question, yet no one has actually worked on it. (And of course there'll be many cases in between.)

---

For "noticed yet neglected" ideas/questions, recommendation 1 from above will be more relevant: people could find many ideas/questions of this type in this central directory for open research questions, and just get cracking on them. 

That directory is like a map pointing the way to many trees that might be full of low-hanging fruit that would’ve been plucked by now in a better world. And I really would predict that a lot of EAs could do valuable work by just having a go at those questions. (I’m less confident that this is the most valuable thing lots of EAs could be doing, and each person would have to think that through for themselves, in light of their specific circumstances. See also.)

So we don’t necessarily need all EA-aligned researchers to try to cultivate a skill of “noticing the ideas that should’ve been tackled/fleshed out already” (though I’m sure some should). Some could just focus on actually exploring the ideas that have been noticed but still haven’t been tackled/fleshed out.

---

For "unnoticed" ideas/questions, recommendation 2 from above will be more relevant. 

I think this dovetails somewhat with Ben Garfinkel calling for[1] more people to just try to rigorously write up more detailed versions of arguments about AI risk that often float around in sketchier or briefer form. (Obviously brevity is better than length, all else held equal, but often a few pages isn’t enough to give an idea proper treatment.)  

---

There are at least two other approaches for finding "unnoticed" ideas/questions which seem to have sometimes worked for me, but which I’m less sure would often be useful for many people, and less sure I’ll describe clearly. These are:

• Trying to sketch out causal diagrams of the pathway to something (e.g., an existential catastrophe) happening
  • I think that doing something like this has sometimes helped me notice there there are:
    • assumptions or steps missing in the standard/fleshed-out stories of how something might happen,
    • alternative pathways by which something could happen, and/or 
    • alternative/additional outcomes that may occur
  • See also
• Trying to define things precisely, and/or to precisely distinguish concepts from each other, and seeing if anything interesting falls out
  • Here’s an abstract example, but one which matches various real examples that have happened for me: 
    • I try to define X, but then notice that that definition would fail to cover some cases of what I’d usually think of as X, and/or that it would cover some cases of what I’d usually think of as Y (which is a distinct concept). 
    • This makes me realise that X and/or Y might be able to take somewhat different forms or occur via different pathways to what was typically considered, or that there’s actually an extra requirement for X or Y to happen that was typically ignored. 
  • I feel like it’d be easy to misinterpret my stance here. 
    • I actually think that definitions will never or almost never really be “perfect”, and I agree with the ideas in this post (see also family resemblance). And I think that many debates over definitions are largely nitpicking and wasting time. 
    • But I also think that, in many case, being clearer about definitions can substantially benefit both thought and communication. 

---

I should again mention that I’m only ~1.5 years into my research career, so maybe I’ll later change my mind about a bunch of those points, and there are probably a lot of useful things that could be said on this that I haven’t said.

[1] See the parts of the transcript after Howie asks "Do you know what it would mean for the arguments to be more sussed out?"

I don't work at Rethink Priorities but I couldn't resist jumping in with some thoughts as I've been doing a lot of thinking on some of these questions recently

Thinking vs. reading. I’ve been playing around with spending 15-60 min sketching out a quick model of what I think of something before starting in on the literature (by no means a consistent thing I do though). I find it can be quite nice and help me ask the right questions early on. 

Self-consciousness. Idk if this fits exactly but when I started my research position I tried to have the mindset of, ‘I’ll be pretty bad at this for quite a while’. Then when I made mistakes I could just think, ‘right, as expected. Now let’s figure out how to not do that again’. Not sure how sustainable this is but it felt good to start! In general it seems good to have a mindset of research being nearly impossibly hard. Humans are just barely able to do this thing in a useful way and even at the highest levels academics still make mistakes (most papers have at least some flaws). 

Optimal hours of work per day. I tend to work about 4-7 hours per day including meetings and everything. Including only mentally intensive tasks I probably get around 4-5 a day. Sometimes I’m able to get more if I fall into a good rhythm with something. Looking around at estimates (Rescuetime says just ~3 hours per day average of productive work) it seems clear I’m hitting a pretty solid average. I still can’t shake the feeling that everyone else is doing more work. Part of this is because people claim they do much more work. I assume this is mostly exaggeration though because hours worked is used as a signal of status and being a hard worker. But still, it's hard to shake the feeling.

Learning a new field. I just do a lot of literature review. I tend to search for the big papers and meta-analyses, skim lot’s of them and try to make a map of what the key questions are and what the answers proposed by different authors are for each question (noting citations for each answer). This helps to distill the field I think and serves as something relatively easy to reference. Generally there’s a lot of restructuring that needs to happen as you learn more about a topic area and see that some questions you used were ill-posed or some papers answer somewhat different questions. In short this gets messy, but it seems like a good way to start and sometimes it works quite well for me.  

Hard problems. I have a maybe-controversial take that research (even in LT space) is motivated largely by signalling and status games. From this view the advice many gave about talking to people about it sounds good. Then you generate some excitement as you’re able to show someone else you’re smart enough to solve it, or they get excited to share what they know, etc. I think if you had a nice working group on any topic, no matter how boring, everyone would get super excited about it. In general, connecting the solution to a hard problem to social reward is probably going to work well as a motivator by this logic. 

Emotional motivators. I’ve been thinking a lot recently about what I’m calling ‘incentive landscaping’. The basic idea is that your system 2 has a bunch of things it wants to do (e.g. have impact). Then you can shape your incentive landscape such that your system 1 is also motivated to do the highest impact things. Working for someone who shares your values is the easiest way to do this as then your employer and peers will reward you (either socially or with promotions) for doing things which are impact-oriented. This still won’t be perfectly optimized for impact but it gets you close. Then you can add in some extra motivators like a small group you meet with to talk about progress on some thing which seems badly motivated, or ask others to make your reward conditional on you completing something your system 2 thinks is important. Still early days for me on this though and I think it’s a really hard thing to get right. 

Typing speed. At least when I'm doing reflections or broad thinking I often circumvent this by doing a lot of voice notes with Dragon. That way I can type at the speed of thought. It’s never perfect but ~97% of it is readable so it’s good enough. Then if you want to actually have good notes you go through and summarize your long jumble of semi-coherent thoughts into something decent sounding. This has the side of effect of some spaced repetition learning as well! 

Tiredness, focus, etc. I’ve had lot’s of ongoing and serious problems with fatigue and have tried many interventions. Certainly caffeine (ideally with l-theanine) is a nice thing to have but tolerance is an issue. Right now what seems to work for me (no idea why) is a greens powder called Athletic Greens. I’m also trying pro/prebiotics which might be helping. Magnesium supplementation also might have helped. A medication I was taking was causing some problems as well and causing me to have some really intense fatigue on occasion (again, probably…). It’s super hard to isolate cause and effect in this area as there are so many potential causes. I’d say it’s worth dropping a lot of money on different supplements and interventions and seeing what helps. If you can consistently increase energy by 5-10% (something I think is definitely on the table for most people), that adds up really quickly in terms of the amount of work you can get done, happiness, etc. Ideally you’d do this by introducing one intervention at a time for 2-4 weeks each. I haven’t had patience for that and am currently just trying a few things at once, then I figure I can cut out one at a time and see what helped. Things I would loosely recommend trying (aside from exercise, sleep, etc): Prebiotics, good multivitamins, checking for food intolerances, checking if any pills you take are having adverse effects.
I do also work through tiredness sometimes and find it helpful to do some light exercise (for me, games in VR) to get back some energy. That also works as a decent gauge for whether I'll be able to push past the tiredness. If playing 10 min of Beatsaber feels like a chore, I probably won't be able to work.
How you rest might also be important. E.g. might need time with little input so your default mode network can do it’s thing. No idea how big of a deal this is but I’ve found going for more walks with just music (or silence) to maybe be helpful, especially in that I get more time for reflection. 
I’ve also been experimenting with measuring heart rate variability using an app called Welltory. That’s been kind of interesting in terms of raising some new questions though I’m still not sure how I feel about it/how accurate it is for measuring energy levels.

Hi Denis, thanks for these questions. I'll give my answers to a bunch of them tomorrow. Just jumping in early with a clarifying question: Could you explain what you mean by "Survival vs. exploratory mindset", and/or provide a link that explains that distinction? I haven't heard those terms before, and Google didn't immediately show me anything that looked relevant.

(Is it perhaps related to exploring vs exploiting?)

11. Tiredness, focus, etc.

I find that being tired makes my mind wander a lot when reading longform things (e.g., papers, posts, not things like Slack messages or emails), so when I'm tired I usually try to do things other than reading. 

If I'm just a bit or moderately tired, I usually find I'm still about as able to write as normal. If I'm very tired, I'll still often be able to write quickly, but then when I later read what I wrote  I'll feel that it was unclear, poorly structured, and more typo-strewn than usual. So when very tired, I try to avoid writing longform things (e.g., actual research outputs).

Things I find I'm still pretty able to do when tired include commenting on documents people want input on (I think I'm more able to focus on this than on regular reading because it's more "interactive" or something), writing things like EA Forum comments, replying to emails and Slack messages and the like, doing miscellaneous admin-y tasks, and reflecting on the last week/month and planning the next. So I often do a disproportionate amount of such tasks during evenings or during days when I'm more tired than normal, and at other times do a disproportionate amount of reading and "substantive" writing.

Also, I'm fortunate enough to have flexible hours. So sometimes I just work less on days when I'm tired (perhaps spending more time with my wife), and then make up for it on other days.

2 and 3. Self-consciousness and Is there something interesting here?

These questions definitely resonate with me, and I imagine they’d resonate with most/all researchers.

I have a tendency to continually wonder if what I’m doing is what I should be doing, or if I should change my priorities. I think this is good in some ways. But sometimes I’d make better decisions faster if I just actually pursued an idea more “confidently” for a bit, to get more info on whether it’s worth pursuing, rather than just “wondering” about it repeatedly and going back and forth without much new info to work with. Basically, I might do too much self-doubt-style armchair reasoning, with too little actual empirical info. 

Also, pursuing an idea more “confidently” for a bit will not only inform me about whether to continue pursuing it further, but also might result in outputs that are useful for others. So I try to sometimes switch into “just commit and focus mode” for a given time period, or until I hit a given milestone, and mostly minimise reflection on what I should prioritise during that time. But so far this has been like a grab bag of heuristics and habits I use, rather than a more precise guideline for myself.

See also When to focus and when to re-evaluate.

Things that help me with this include, and/or some scattered related thoughts, include:

• Talking to others and getting feedback, including on early-stage ideas
  • I liked David and Jason’s remarks on this in their comments
  • A sort-of minimum viable product and quick feedback loop approach has often seemed useful for me - something like: 
    • First getting verbal feedback from a couple people on a messy, verbal description of an idea
    • Then writing up a rough draft about the idea and circulating it to a couple more people for a bit more feedback
    • Then polishing and fleshing out that draft and circulating it to a few more people for more feedback
    • Then posting publicly
    • (But only proceeding to the next step if evidence from the prior one - plus one’s own intuitions - suggested this would be worthwhile)
  • Feedback has often helped me determine whether an idea is worth pursuing further, feel more comfortable/motivated with pursuing an idea further (rather than being mired in unproductive self-doubt), develop the idea, work out which angles of it are most worth pursuing, and work out how to express it more clearly
• Reminding myself that I haven’t really gathered any new info since the last time I thought “Should this really be what I spend my time on?”, so thinking about that again is unlikely to reveal new insights, and is probably just a stupid part of my psychology rather than something I’d endorse. 
  • I might think to myself something like “If a friend was doing this, you’d think it’s irrational, and gently advise them to just actually commit for a bit and get new info, right? So shouldn’t you do the same yourself?”
• Remembering Algorithms to Live By drawing an analogy to a failure mode in which computers continually reprioritise tasks and the reprioritisation takes up just enough processing power to mean no actual progress on any of the tasks occurs, and this can just cycle forever. And the way to get out of this is to at some point just do tasks, even without having confidence that these “should” be top priority.
  • This is just my half-remembered version of that part of the book, and might be wrong somehow.
• Remembering that I’d be deeply uncertain about the “actual” value of any project I could pursue, because the world is very complicated and my ambitions (contribute to improving the long-term future) are pretty lofty. The best I can do is something that seems good in expected value but with large error bars. So the fact I feel some uncertainty and doubt provides basically no evidence that this project isn’t worth pursuing. (Though feeling an unusually large amount of uncertainty and doubt might.) 
• Remembering that, if the idea ends up seeming to have not been important but there was a reasonable ex ante case that it might’ve been important, there’s a decent chance someone else would end up pursuing it if I don’t. So if I pursue it, then find out it seems to not be important, then write about what I found, that might still have the effect of causing an important project to get done, because it might cause someone else to do that important project rather than doing something similar to what I did.

Regarding your Typing speed question, Tom Chivers (a journalist) was asked in a recent EA Forum AMA "How one should go about learning how to write high-quality material? And what is the way to get it published?"

His reply:

I wish I had a better answer to the first one than "become good at writing". My own pathway was reading loads and loads, and writing loads and loads, and then essentially mimicking the writing that I liked (mainly Pratchett tbh) until eventually I noticed that I'd stopped doing that and had a recognisable style of my own. I sometimes go through my old emails from before I was a journalist and see I've just written needlessly long show-offy emails to friends, which I cringe about a bit now, but they were clearly practice for when I had to do it for real.

Actually, also, I did philosophy at uni and MA, and I found that the way I learnt to structure an argument in those essays has been really helpful.

Oh and this might sound silly but become good at typing. If you can type as fast as you think then when the ideas are flowing quickly then they just sort of appear on the page. I used to work as a medical secretary for a long time and I swear that helped me an awful lot, not least in transcribing interviews but also just in being able to get ideas down quickly.

As for getting it published: pitch! Ideally start by developing a relationship with some editor somewhere. It might be a good idea to blog as well, so that you can point people to stuff you've written. [emphasis added]

7. Hard problems

I’m not actually sure if the precise problem you’re describing resonates with me. I definitely often feel very uncertain about: 

• whether the goal I’m striving towards really matters at all
• even if so, whether it’s a goal worth prioritising
• whether I should prioritise it (is it my comparative advantage?)
• whether anything I produce in pursuing this goal will be of any use to anyone

But I’m not sure there have been cases where, for a week or more, I didn’t feel like I was at least progressing towards: 

• having the sort of output I had planned or now planned to produce (setting aside the question of whether that output will be useful to anyone), and/or
• deciding (for good reason) to not bother trying to create that sort of output

Note that I’d count as “progress” cases where I explored some solutions/options that I thought might work/be useful for X, and all turned out to be miserable wastes of time, so I can at least rule those out and try something else next week. I'd also count cases where I learned other potentially useful things in the process of pursuing dead ends, and that knowledge seems likely to somehow benefit this or other projects.

It is often the case that my estimate of how many remaining days something will take is longer at the end of the week than it was at the beginning of the week. But this is usually coupled with me thinking that I have made some sort of progress - I just also realised that some parts will be harder than I thought, or that I should do a more thorough job than I’d planned, or something like that.

(But I feel like maybe I'm just interpreting your question differently to what you intended.)

8. Emotional motivators

(Disclaimer: I'm just reporting on my own experience, and think people will vary a lot in this sort of area, so none of the following is even slightly a recommendation.)

In general:

• Personally, I seem to just find it pretty natural to spend a lot of hours per week doing work-ish things
• I tend to be naturally driven to “work hard” (without it necessarily feeling much like working)  by intellectual curiosity, by a desire to produce things I’m proud of, and by a desire for positive attention (especially but not only from people whose judgement I particularly respect)
  • That third desire in particular can definitely become a problem, but I try to keep a close eye on it and ensure that I’m channeling that desire towards actions I actually endorse on reflection
• I do get run down sometimes, and sometimes this has to do with too many hours per week for too many weeks in a row. But the things that seem more liable to run me down are feeling that I lack sufficient autonomy in what I do, how, and when; or feeling that what I’m doing isn’t valuable; or feeling that I’m not developing skills and knowledge I’ll use in future
  • That last point means that one type of case in which I do struggle to be motivated is cases where I know I’m going to switch away from a broad area after finishing some project, and that I’m unlikely to use the skills involved in that project again. 
    • In these cases, even if I know that finishing that project to a high standard would still be valuable and is worth spending time on, it can be hard for me to be internally motivated to do so, because it no longer feels like doing so would “level me up” in ways I care about. 
• I seem to often become intensely focused on a general area in an ongoing way (until something switches my focus to another area), and just continually think about it, in a way that feels positive or natural or flow-like or something
  • This happened for stand-up comedy, then for psychology research, then for teaching, then for EA stuff (once I learned about EA)
    • (The other points above likewise applied during each of those four “phases” of my adult life)

Luckily, the sort of work I do now:

• is very intellectually stimulating
• involves producing things I’m (at least often!) proud of
• can bring me positive attention
• allows me a sufficient degree of autonomy
• seems to me to be probably the most valuable thing I could realistically be doing at the moment (in expectation, and with vast uncertainty, of course)
• involves developing skills and knowledge I expect I might use in future

That means it’s typically been relatively easy for me to stay motivated. I feel very fortunate both to have the sort of job and “the sort of psychology” I’ve got. I think many people might, through no fault of their own, find it harder to be emotionally motivated to spend lots of hours doing valuable work, even when they know that that work would be valuable and they have the skills to do it. Unfortunately, we can’t entirely choose what drives us, when, and how.

(There’s also a scary possibility that my tendency so far to be easily motivated to work on things I think are valuable is just the product of me being relatively young and relatively new to EA and the areas I’m working in, and that that tendency will fade over time. I’d bet against that, but could be wrong.)

9. Typing speed

I'd be surprised if typing speed was a big factor explaining differences in how much different researchers produce, or in their ability to produce certain types of output. (But of course, that claim is pretty vague - how surprised would I be? What do I mean by "big factor?")

But I just did a typing test, and got 92wpm (with "medium" words, and 1 typo), which is apparently high. So perhaps I'm just taking that for granted and not recognising how a slower typing speed could've limited me. Hard to say.

6. Learning a new field

I don’t know if I have a great, well-chosen, or transferable method here, so I think people should pay more attention to my colleagues’ answers than mine. But FWIW, I tend to do a mixture of: 

• reading Wikipedia articles
• reading journal article abstracts
• reading a small set of journal articles more thoroughly
• listening to podcasts
• listening to audiobooks
• watching videos (e.g., a Yale lecture series on game theory)
• talking to people who are already at least sort-of in my network (usually more to get a sounding board or “generalist feedback”, rather than to leverage specific expertise of theirs)

I’ve also occasionally used free online courses, e.g. the Udacity Intro to AI course. (See also What are some good online courses relevant to EA?)

Whether I take many notes depends on whether I'm just learning about a field because I think it might be useful in some way in future for me to know about that field, or because I have at least a vague idea of a project I might work on within that field (e.g., "how bad would various possible types of nuclear wars be, from a longtermist perspective?"). In the latter case, I'll take a lot of notes as I go in Roam, beginning to structure things into relevant sub-questions, things to learn more about, etc. 

Since leaving university, I haven’t really made much use of textbooks, flashcards, or reaching out to experts who aren’t already in my network. It's not really that I actively chose to not make much use of these things (it’s just that I never actively chose to make much use of these things), and think it’s plausible that I should make more use of these things. I’ll very likely talk to a bunch of experts for my current or upcoming research projects.

These are fascinating, I would love to see answers to all of these questions!

Wow! Thanks for all the insightful answers, everyone!

Would anyone mind if I transfer these into a post on my blog (or a separate post in the EA Forum) that is linear in the sense that there is one question and then all answers to it, then the next question and all answers to it, and so on? That may also generate more attention for these answers. :-)

What are the biggest mistakes Rethink Priorities did?

I can’t speak for the entire organization, but I can talk about what I see as my biggest mistakes since I started working at Rethink Priorities:

1. Writing articles about interventions I think are promising and thinking that my work is done once the article is published. Examples are baitfish (see the comment above), fish stocking, rodents farmed for pet snake food. The way I see things now, if I think that something should be done, I should express that opinion very clearly and with fewer caveats, find funders who want to fund it, find activists that want to do it, and connect them. Or something like that. And that is the kind of work I am doing at the moment, even though I think I am much better at writing articles than at doing this.
2. Avoiding expressing opinions too much. It’s related to the point above. I think that in the past I was too afraid of writing something that could later turn out to be wrong. Hence, I wrote articles in such a way that sometimes the reader could not even know what I think about a problem I am writing about, how important I think it is in the context of other things, etc. I wanted decision makers to read my articles and form their own opinions based on what I said. I now think that this is not ideal because decision makers may not have the time to form nuanced opinions based on subtle details in my long articles. But someone has to form actionable opinions, and it is me who has the context and the time for that. So I want to try to write more articles of the “This is what I think you should do and I’m going to explain why” type, rather than the “Here is a 40 page summary of everything I've ever read on this topic” type. I sometimes want to write articles of the former type because then my managers, funders and myself can all clearly see what I’ve been working on for all this time. But my end goal is making an impact so I try to not think about that too much. Note that if I pledged to only ever write articles that are purely of the former kind, I might end up not writing a single paragraph all year. I don’t think I should go that far.
3. Spending too much time on finishing articles that I know won’t have that much impact. In some cases, it’s better to just drop them, admit to yourself that you wasted some time, and move on to the next project. That said, there were some articles that I had strongly considered abandoning, but in the end I was happy I finished them.
4. Spending any time on details that I know right away won’t be that important. There are some examples of this in Estimates of global captive vertebrate numbers article. Did I really need to write about pets, civet farming, and other relatively minor problems that I know effective altruists won’t work on? I guess I wanted the list to be complete, but I don’t know why. It wasted not only my time, but also the time and the attention of the readers.
5. Being too frugal. In the beginning of working at Rethink Priorities, I wanted to either take a low salary, or spend as little money as I can and donate the rest. But the problems that it caused made me less productive and possibly decreased my impact. Now I allow myself to spend more and I think I'm better off because of it.
6. Not doing more to address some of my productivity problems, especially negative self-talk about myself and my work. Almost every day I hate myself for not doing enough work. It is exhausting, and it tires me out more quickly and hence I become even less productive. I still haven’t found a good way to deal with it. I tried therapy multiple times but I never emphasized this specific issue so that is on my to-do list. I also want to try more meditation, maybe that can help.

What new charities do you want to be created by EAs?

For me it's a lobbying organization against baitfish farming in the U.S. I wrote about the topic two years ago here. Many people complimented me on it but no one did anything. I talked with some funders who said they would be interested in funding someone suitable pursuing this, but I haven’t found who could this be. The main argument against it used to be that the industry is declining. But the recently released aquaculture census suggests that it is no longer declining (see my more recent thoughts on numbers here).

Using fish as live bait is already prohibited in some U.S. states (see the map in Kerr (2012)). Many other states have import and movement restrictions (see this table). It seems that all of this happened due to environmental concerns. And the practice is banned in multiple other countries. To me this shows that it is plausible to make progress on this.

Take a look at this graph I made of the number of animals farmed in the U.S. at any time.

I used yellow and black colours to represent ranges. So for example, I think that there are between 1 billion and (5+1=)6 billion baitifsh farmed in the U.S. at any time. It’s more likely to be closer to 1 billion than to 6 billion though. Still, if we wanted to decrease the number of vertebrates farmed for the U.S. consumption by say 500 million, it would seem very difficult to make Americans decrease their chicken and egg consumption by 25%, or decrease their farmed fish consumption by 13%-42%. Decreasing baitfish production by a 500 million might also be difficult but I think it is much more easily achievable.

I am doing a bit more research on this right now (in parallel with other projects), and I might make another EA forum post about it at some point but I don’t know if that is what is needed to make this happen. I think that at this point someone should just try to do it.

If anyone is interested, please schedule a meeting with me here or write to me at saulius at rethinkpriorities dot org .

If one is only concerned w/ preventing needless suffering, prioritising the most extreme suffering, would donating to Rethink Priorities be a good investment for them, and if so, how so?

Contrary to organizations like OPIS, Center for Reducing Suffering, and Center on long-term risk, we don't have reducing extreme suffering set as our only priority. We sometimes work on reducing suffering that may not be classified as extreme (arguably, our work on cage-free hen campaigns fall into this category). And perhaps some other work is not directly about reducing suffering at all. Since preventing extreme suffering is not our only priority, I think that we are unlikely to be the best donation opportunity for this specific goal. That said, when I look at the list of our publications, I think that almost all the articles we write contribute to the goal of preventing needless and extreme suffering in some way, although in many cases it is quite indirect. In the end, we are not able to compare whether or not Rethink Priorities is a better donation opportunity for this purpose than other organizations in an unbiased way.

Thanks for the questions!

If one is only concerned w/ preventing needless suffering, prioritising the most extreme suffering, would donating to Rethink Priorities be a good investment for them, and if so, how so?

I think this depends on many factual beliefs you hold, including what groups of creatures count and what time period you are concerned about. Restricting ourselves to the present and assuming all plausibly sentient minds count (and ignoring extremes, say, less than 0.1% chance), I think farm and wild animals are plausibly candidates for enduring some of the worst suffering.

Specifically, I'd say it's plausible some of the worst persistent current suffering is plausibly in farmed chickens and fish, and thus work to reduce the worst aspects of those is a decent bet to prevent extreme suffering. Similarly, wild animals likely experience the largest share of extreme suffering currently because of the sheer numbers and nature of life largely without interventions to prevent, say, the suffering of starvation, or extreme physical pain. For these reasons, work to improve conditions for wild animals plausibly could be a good investment.

Still restricted to the present, and outside the typical EA space altogether, I think it's plausible much of the worst suffering in the world is committed during war crimes or torture under various authoritarian states. I do not know if there's anything remotely tractable in this space or what good donation opportunities would be.

If you broaden consideration to include the future, a much wider set of creatures plausibly could experience extreme suffering including digital minds running at higher speeds, and/or with increased intensity of valenced experience beyond what's currently possible in biological creatures. Here, what you think is the best bet would depend on many empirical beliefs again. I would say, only, that I'm excited about our longtermism work and think we'll meaningfully contribute to creating the kind of future that decreases the risks of these types of outcomes.

What new charities do you want to be created by EAs?

I don't have any strong opinions about this and it would likely take months of work to develop them. In general, I don't know enough to suggest that it is desirable that new charities work in areas I think could use more work rather than existing organizations up their work in those domains.

What are the biggest mistakes Rethink Priorities did?

Not doing enough early enough to figure out how to achieve impact from our work and communicate with other organizations and funders about how we can work together.

If one is only concerned w/ preventing needless suffering, prioritising the most extreme suffering, would donating to Rethink Priorities be a good investment for them, and if so, how so?

I like the answers Marcus and Saulius gave to this question. I'll just add two things those answers didn't explicitly mention.

EA movement-building

• Rethink has done and plans to do work aimed at improving efforts to build the EA movement and promote EA ideas
  • E.g., Rethink's work on the EA Survey, or its plans related to:
    • "Further refining messaging for the EA movement, exploring different ways of talking about EA to improve EA recruitment and increase diversity.
    • Further work to explore better ways to talk about longtermism to the general public, to help EAs communicate longtermism more persuasively and to increase support for desired longtermist policies in the US and the UK."
• And building the EA movement and promoting EA ideas seems like plausibly one of the best interventions for reducing needless/extreme/all suffering
  • E.g., building the EA movement could increase the flows of talent and funds to existing suffering-focused EA organisations (such as CLR), lead to the creation of new ones, or lead to talented people using their careers to effectively reduce suffering in other ways (e.g., through specific roles in government or AI labs)
  • E.g., promoting EA ideas (even without "building the EA movement") could lead to a general shift in voting, policies, behaviours towards reducing suffering

 Forecasting

• Rethink plans to "Use novel econometric methods to better understand our ability to reliably impact the long-term future", as well as to "Improve our ability to forecast the short-term and long-term future."
• Improving our ability to forecast events and impacts, and improving our understanding of when and how much to trust forecasts, would presumably be about as useful for reducing suffering as for all other efforts to improve the world. (And I think it'd plausibly be very useful for such efforts.)
  • This seems especially true in relation to: 
    1. efforts to reduce suffering in the long-term future, and
    2. decisions about how much to focus on reducing suffering in the long-term future vs reducing suffering in the nearer term.

Caveats

I'm not necessarily arguing that Rethink is where someone should donate if they wish to reduce suffering. That would depend on things like precisely how effective Rethink's movement-building work would be for EA movement-building, precisely how useful EA movement-building is for reducing suffering, etc. 

I'm not in a good position to make those judgements, for reasons including that: 

• I don't take a primarily suffering-focused perspective myself (so I haven't thought about it a great deal - though I did work at the Center on Long-Term Risk for 3 months)
• I now work for Rethink, so I might be biased
• I've only worked at Rethink for a month, and don't work on the EA movement-building or forecasting stuff myself

But hopefully this is useful food for thought anyway :)

Hi Neel,

We'd obviously be very excited to take 10x our budget if you're offering ;)

Right now, 10x our budget would be ~$14M, which would still be 8x smaller than large think tanks like the Brookings Institution. I think if we had 10x the budget, the main thing we would do is expand our research staff as rapidly as non-financial constraints (e.g., management, operations, and team culture) allow.

There are definitely many more areas of research we could be working in, both within our existing cause areas (currently farmed animal welfare, wild animal welfare, invertebrate welfare, longtermism, and EA movement building) and other cause areas we aren't working in yet. We'd also need more operations staff and management to facilitate this.

As for specific research questions, I think we have a much clearer vision of what we would do with 2x the money than 10x the money. I personally (speaking for myself not the rest of the org) would love to see us hire staff to work more directly on farmed animal welfare policy and to investigate meat alternatives, do much more to understand EA community health and movement building, do more fundamental research (e.g., like our work on moral weight and investigating well-being metrics), and potentially investigate new charities that could be launched (similar to CE's work). But that is just a wishlist and it would change as I talk to more people.

We're already working a lot to prioritize what questions we want to tackle - our longtermist and wild animal departments, for example, just recently expanded beyond one person and we're in the process of making new research agendas, so it is hard to recommend ideas in those areas right now. 

One benefit of hiring more people, though, is we'd have more people to do the important work of figuring out what it is we should do!

Hi Arushi,

I am very hopeful the internship program will let us identify, take on, and train many more staff than we could otherwise and then either hire them directly or be able to recommend them to other organizations.

While I am wary of recommending unpaid labor (that's why our internship is paid), I otherwise think one of the best ways for a would-be researcher to distinguish themselves is writing a thoughtful and engaging EA Forum post. I've seen a lot of great hires distinguish themselves like this.

Other than open more researcher jobs and internships, I think other EA orgs could perhaps contribute by writing advice and guides about research processes or by offering more "behind the scenes" content on how different research is. done.

Lastly, in my personal opinion, I think we should also do more to create an EA culture where people don't feel like the only way they can contribute is as a researcher. I think the role gets a lot more glamor than it deserves and many people can contribute a lot from earning to give, working in academia, working in politics, working in a non-EA think tank, etc.

I’m happy to see an increase in the number of temporary visiting researcher positions at various EA orgs. I found my time visiting GPI during their Early  Career Conference Programme very valuable (hint: applications for 2021 are now open, apply!) and would encourage other orgs to run similar sorts of programmes to this and FHI’s (summer) research scholars programme. I'm very excited to see how our internship program develops as I really enjoy mentoring.

I think I was competitive for the RP job because of my T-shaped skills, broad knowledge in lots of EA-related things but also specialised knowledge in a specific useful area, economics in my case. Michael Aird probably has the most to say about developing broad knowledge given how much EA content he has consumed in the last couple of years, but in general reading things on the Forum and actively discussing them with other people (perhaps in a reading group) seems to be the way to develop in this area. Developing specialised skills obviously depends a lot on the skill, but graduate education and relevant internships are the most obvious routes here.

What do you think individuals could do to become skilled in this kind of research and become competitive for these jobs?

There are some relevant answers in here and here.

What do you think individuals could do to become skilled in this kind of research and become competitive for these jobs?

I think this is a relatively minor thing, but trying to become close to perfectly calibrated (aka being able to put precise numbers on uncertainty) on some domains seem like a moderate-sized win, at very low cost. 

I mainly believe this because I think the costs are relatively low. My best guess is that the majority of EAs can become close to perfectly calibrated on trivia numerical questions in much less than 10 hours of deliberate practice, and my median guess is for the amount of time needed  is around 2 (eg practice here).

I want to be careful with my claims here. I think sometimes people have the impression that getting calibrated is synonymous with rationality, or intelligence, or judgement. I  think this is wrong:

1. Concretely, I just don't think being perfectly calibrated is that big a deal. My guess is that going from median-EA levels of general calibration to perfect calibration on trivia questions is an improvement in good research/thinking by 0.2%-1%. I will be surprised if somebody becomes a better researcher by 5% via these exercises, and very surprised  if they improve by 30%.
2. In forecasting/modeling, the main quantifiable metrics include both a) calibration (roughly speaking, being able to quantify your uncertainty) and b) discrimination (roughly speaking, how often you're right). In the vast majority of cases, calibration is just much less important than discrimination. 
3. There are generalizability issues with generalizing from good calibration on trivia questions to good calibration overall. The latter is likely to be much harder to train precisely, or even precisely quantify (though I'm reasonably confident that going from poor calibration on trivia to perfect calibration should generalize somewhat, Dave Bernard might have clearer thoughts on this)
4. I think calibration matters more for generalist/secondary research (much of what RP does) than for things that either a) require relatively narrow domain expertise, like ML-heavy AI Safety research or biology-heavy biosecurity work, or b)  require unusually novel thinking/insight (like much of crucial considerations work). 

Nonetheless, I'm a strong advocate for calibration practice because I think the first hour or two of practice will pay off by 1-2 orders of magnitude over your lifetime, and it's hard to identify easy wins like that (I suspect even exercise has a less favorable cost-benefits ratio, though of course it's much easier to scale).

Misc thoughts on "What do you think individuals could do to become skilled in this kind of research and become competitive for these jobs?"

There was some relevant discussion here. Ideas mentioned there include: 

• getting mentorship outside of EA orgs (either before switching into EA orgs after a few years, or as part of a career that remains outside of explicitly EA orgs longer-term)
• working as a research assistant for a senior researcher

I think the post SHOW: A framework for shaping your talent for direct work is also relevant.

Hi Arushi,

Good questions! I'll split some thoughts into a few separate comments for readability. 

Writing on the Forum

I second Peter's statement that

one of the best ways for a would-be researcher to distinguish themselves is writing a thoughtful and engaging EA Forum post. I've seen a lot of great hires distinguish themselves like this.

(Though in some cases it might make sense to publish the post to LessWrong instead or in addition.)

This statement definitely seems true in my own case (though I imagine for some people other approaches would be more effective):

I got a offer for an EA research job before I began writing for the EA Forum. But I was very much lacking in the actual background/credentials the org said they were looking for, so I'm almost certain I wouldn't have gotten that offer if the application process hadn't included a work test that let me show I was a good fit despite that relevant lack of background/credentials. (I was also lucky that the org let me do the work test rather than screening me out before that.) And the work test was basically "Write an EA Forum post on [specific topic]", and what I wrote for it did indeed end up as one of my first EA Forum/LessWrong posts.

And then this year I've gotten offers from ~35% of what I've applied to, as compared to ~7% last year, and I'd guess that the biggest factors in the difference were:

1. I now had an EA research role on my CV, signalling I might be a fit for other such roles
2. Going from 1FTE non-EA stuff (teaching) in 2019 to only ~0.3FTE non-EA stuff (a grantwriting role I did for a climate change company on the side of my ~0.7FTE EA work till around August) allowed me a lot of time to build relevant skills and knowledge
3. In 2020 I wrote a bunch of (mostly decently/well received) EA Forum or LessWrong posts, helping to signal my skills and knowledge, and also just "get my name out there"
   • "getting my name out there" was not part of my original goal, but did end up happening, and to quite a surprising degree.
4. Writing EA Forum and LessWrong posts helped force and motivate me to build relevant skills and knowledge
5. Comments and feedback from others on my EA Forum and LessWrong posts sometimes helped me build relevant skills and knowledge, or build my ideas of what was worth thinking and writing about
   • See also this other comment of mine from this AMA

Factors 1 and 2 didn't depend on me writing things on the EA Forum or LessWrong. But factors 3-5 did. So it seems that writing for the Forum and LessWrong really helped me out here. It also seems plausible that, if I'd started writing for the Forum/LW before I got my first EA job offer, that might've led to me getting an offer sooner than I in fact did.

(But I'm not sure how generalisable any of these takeaways are - maybe this approach suited me especially well for some reason.)

On this, I'd also recommend Aaron Gertler's talks Why you (yes, you) should post on the EA Forum and How you can make an impact on the EA Forum.

My own story & a disclaimer

(This is more of a tangent than an answer, but might help provide some context for my other responses here and elsewhere in this AMA. Feel free to ignore it, though!)

I learned about EA in late 2018, and didn't have much relevant expertise, experience, or credentials. I'd done a research-focused Honours year and published a paper, but that was in an area of psychology that's not especially relevant to the sort of work that, after learning about EA, I figured I should aim towards. (More on my psych background here.) I was also in the midst of the 2 year Teach For Australia program, which involves teaching at a high school, and also wasn't relevant to my new EA-aligned plans.

Starting then and continuing through to mid 2020 ish, I made an active effort to "get up to speed" on EA ideas, as described here. 

In 2019, I applied for ~30 EA-aligned roles, mostly research-ish roles at EA orgs (though also some non-research roles or roles at non-EA orgs). I ultimately got two offers, one for an operations role at an EA org and one for a research role. I think I had relevant skills but didn't have clear signals of this (e.g., more relevant work experience or academic credentials), so I was often rejected at the CV screening stage but often did ok if I was allowed through to work tests and interviews. And both of the offers I got were preceded by work tests. 

Then in 2020, I wrote a lot of posts on the EA Forum and a decent number on LessWrong, partly for my research job and partly "independently". I also applied for ~11 roles this year (mostly research roles, and I think all at EA orgs), and ultimately received 4 offers (all research roles at EA orgs). So that success rate was much higher, which seems to fit my theory that last year I had relevant skills but lacked clear signals of this. 

So I've now got a total of ~1.5 years FTE of research experience, ~0.5 of which (in 2017) was academic psychology research and ~1 of which (this year) was split across 3 EA orgs. That's obviously  not enough time to be an expert, and I still have a great deal to learn on a whole host of dimensions.

Also, I only started with Rethink roughly a month ago.

Research training programs, and similar things

(You said "Besides offering similar internships". But I'm pretty excited about other orgs running similar internships, and/or running programs that are vaguely similar and address basically the same issues but aren't "internships". So I'll say a bit about that cluster of stuff, with apologies for sort-of ignoring instructions!)

David wrote:

I’m happy to see an increase in the number of temporary visiting researcher positions at various EA orgs. I found my time visiting GPI during their Early Conference Career Programme very valuable (hint: applications for 2021 are now open, apply!) and would encourage other orgs to run similar sorts of programmes to this and FHI’s (summer) research scholars programme. I'm very excited to see how our internship program develops as I really enjoy mentoring.

I second all of that, except swapping GPI's Early Conference Career Programme (which I haven't taken part in) for the Center on Long-Term Risk's Summer Research Fellowship. I did that fellowship with CLR from mid August to mid November, found it very enjoyable and useful. 

I recently made a tag for posts relevant to what I called "research training programs". By this I mean things like FHI and CLR's Summer Research Fellowships, Rethink Priorities' planned internship program, CEA's former Summer Research Fellowship, probably GPI's Early Career Conference Programme, probably FHI's Research Scholars Program, maybe the Open Phil AI Fellowship, and maybe ALLFED's volunteer program. Readers interested in such programs might want to have a look at the posts with that tag. 

I think that these programs might be one of the best ways to address some of the main bottlenecks in EA or at least in longtermism (I've thought less about areas of EA other than longtermism). What I mean is related to the claim that EA being vetting-constrained, and to Ben Todd's claim that some of EA's main bottlenecks at the moment are "organizational capacity, infrastructure, and management to help train people up". There was also some related discussion here (though it's harder to say whether that overall supported the claims I'm gesturing at).

So I'm really glad a few more such programs have recently popped up in longtermism. And I'm really excited about Rethink's internship program (which I wasn't involved in the planning of, and didn't know about when I accepted the role at Rethink). And I'd be keen to see more such programs emerge over time. I think they could take a wide variety of forms, including but not limited to internships.

And I'd strongly recommend aspiring or early-career researchers consider applying to such programs. See also Jsevillamol's post My experience on a summer research programme.

(As always, these are just my personal views, not necessarily the views of other people at Rethink.)

Hi Roger,

There are different possible scenarios in which invertebrates turn out to be sentient. It might be the case, for instance, that panpsychism is true. So if one comes to believe that invertebrates are sentient because panpsychism is true, one should also come to believe that robots and plants are sentient. Or it could be that some form of information integration theory is true, and invertebrates instantiate enough integration for sentience. In that case, the probability that you assign to the sentience of plants and robots will depend on your assessment of their relevant level of integration.

For what it's worth, here's how I think about the issue: sentience, like other biological properties, has an evolutionary function. I take it as a datum that mammals are sentient. If we can discern the role that sentience is playing in mammals, and it appears there is analogous behavior in other taxa, then, in the absence of defeaters, we are licensed to infer that individuals of those taxa are sentient. In the past few years I've updated toward thinking that arthropods and (coleoid) cephalopods are sentient, but the majority of these updates have been based on learning new empirical information about these animals. (Basically, arthropods and cephalopods engage in way more complex behaviors than I realized.) When we constructed our invertebrate sentience table, we also looked at plants, prokaryotes, protists, and, in an early version of the table, robots and AIs of various sorts. The individuals in these categories did not engage in the sort of behaviors that I take to be evidence of sentience, so I don't feel licensed to infer that they are sentient.

Hi Vaidehi,

What kind of research do you plan on doing to answer these questions?

I will be working on both of these projects with David Moss. Our plan is to run surveys of the general public that describe EA (or longtermism) and ask questions to gauge how people view the message. We'd then experimentally change the message to explore how different framings change support, with the idea that messages that engender more support on the survey are likely to be more successful overall. For EA messaging, we'd furthermore look at support broken down by different demographics to see if there are more inclusive messages out there. We did a similar project we did for animal welfare messaging on live shackle slaughter, which you can look at to get a sense of what we do. We also have a lot of unpublished animal welfare messaging work we're eager to get out there as soon as we can.

~

Did you consider other areas of EA movement building apart from messaging before choosing this one, and if so how did you narrow down your options?

As you know, we do run the EA Survey and Local Groups Survey. Right now, our main goal is to stay within analysis of EA movement building rather than work to directly build the movement like other groups (e.g., CEA, 80K, GWWC, TLYCS) already do. We see these messaging studies as a good next step. However, we have not systematically compared opportunities yet as we don't have the staff or funding right now to do such a search.

~

Do you see general EA messaging as part of your longtermist focus, or is this a separate category?

We see these as separate projects in separate cause areas, though there will definitely be a lot of cross-cause learning. Note that we also do this for farmed animal welfare as well and may also do so in wild animal welfare in the near future. It is a very useful thing to do for all sorts of causes!

~

Either ways, how do you figure out how to allocate resources to this movement building-related efforts? 

Right now we just have allocated funding from restricted funding and a portion of our unrestricted funding. We will also likely fundraise for this work specifically from interested donors.

(Speaking for myself and not others on the team, etc)

 At a very high level, I think I have mostly "mainstream longtermist EA" views here, and my current best guess would be that AI Safety, existential biosecurity, and cause prioritization (broadly construed) are the highest EV efforts to work on overall, object-level. 

This does not necessarily mean that marginal progress on these things are the best use of additional resources, or that they are the most cost-effective efforts to work on, of course.

This is not a satisfying answer but right now I think the longtermist effort with the highest expected value is spending time trying to figure out what longtermist efforts we should prioritize.

I also think we should spend a lot more resources on figuring out if and how much we can expect to reliably influence the long-term future, as this could have a lot of impact on our strategy (such as becoming less longtermist or more focused on broad longtermism or more focused on patient longtermism, etc.).

I don't have a third thing yet, but both of these projects we are aiming to do within Rethink Priorities.

(Just my personal views, as always)

Roughly in line with Peter's statement that "I think the longtermist effort with the highest expected value is spending time trying to figure out what longtermist efforts we should prioritize", I recently argued (with some caveats and uncertainties) that marginal longtermist donations will tend to be better used to support "fundamental" rather than "intervention" research. On what those terms mean, I wrote: 

It’s most useful to distinguish intervention research from fundamental research based on whether the aim is to:

• better understand, design, and/or prioritise among a small set of specific, already-identified intervention options, or
• better understand aspects of the world that may be relevant to a large set of intervention options (more)

See that post's "Key takeaways" for the main arguments for and against that overall position of mine.

I think I'd also argue that marginal longtermist research hours (not just donations) will tend to be better used to support fundamental rather than intervention research. (But here personal fit becomes quite important.) And I think I'd also currently tend to prioritise "fundamental" research over non-research interventions, but I haven't thought about that as much and didn't discuss it in the post.

So the highest-EV-on-the-current-margin efforts I'd pick would probably be in the "fundamental research" category.

Of course, these are all just general rules, and the value of different fundamental research efforts, intervention research efforts, and non-research efforts will vary greatly. 

In terms of specific fundamental research efforts I'm currently personally excited about, these include analyses, from a longtermist perspective, of: 

• totalitarianism/dystopias,
• world government (see also),
• civilizational collapse and recovery,
• "the long reflection", and/or
• long-term risks from malevolent actors

Basically, those things seem like variables that might (or might not!) matter a great deal, and (as far as I'm aware) haven't yet been looked into from a longtermist perspective much. So I expect there could be some valuable low-hanging fruit there.

Maybe if I had to pick just three, I'd bundle the first two together, and then stamp my feet and say "But I want four!"

(I have more thoughts on this that I may write about later. See also this and this. And again, these are just my personal, current views.)

Hey Josh, thanks for the question!

From first principles, our allocation depends on talent fit, the counterfactual value of our work, fundraising, and, of course, some assessment of how important we think the work is, all things considered.

At the operational level, we set targets as percentage of time we want to spend on each cause area based on these factors and we re-evaluate based on that as our existing commitments, the data, and as changes in our opinions about these matters warrant.

I think it would always be good to have more senior researchers, but they seem rather hard to find. Right now, my personal view is that the best way to build senior researchers is to hire and train mid-level or junior-level researchers. We hope to keep doing this with our past hires, existing hires, and our upcoming intern program.

If you're interested in funding researcher talent development, I think funding our intern program is a very competitive opportunity.

I haven’t read that report in full, but I imagine that it's such a big issue in the X-risk research because it grew very quickly from an obscure field, to a field with a lot of funding available and a lot of people wanting to work in it. I think it’s a rare situation, and I don't feel that it's a significant problem in the kind of research that I do (farmed animal welfare). I remember hearing that it is a problem in cultured meat R&D though, and it makes sense, the situation is similar.

Hey Edo,

I definitely receive valuable feedback on my work by posting it on the Forum, and the feedback is often most valuable when it comes from people outside my current network. For me, the best example of this dynamic was when Gavin Taylor left extensive comments on our series of posts about features relevant to invertebrate sentience (here, here, and here) back in June 2019. I had never interacted with Gavin before, but because of his comments, we set up a meeting, and he has become an invaluable collaborator across many different projects. My work is much improved due to his insights. I'm not sure Gavin and I would ever have met (much less collaborated) if not for his comments on the Forum.

Hi Edo,

I definitely think I’ve received valuable feedback on my work on the EA Forum, as well as on LessWrong. This feedback came in the form of upvotes/downvotes, comments on my posts, and private messages/discussions that people had with me as a result of me having posted things.

It’s harder to say: 

1. In what ways was that feedback valuable?
2. Precisely how valuable was it? How does that compare to other sources of feedback?
3. How did the value vary by various factors (e.g., EA Forum vs LessWrong, posts that are more like summaries vs posts that are more like “original research”)?
4. What proportion of that came from people in my existing network?

Some thoughts on those points follow. (But first I should flag that I think there are also good reasons to post to the EA Forum/LW other than to get feedback, including to share potentially useful ideas, to signal one’s skills/knowledge to aid in later job applications, and make connections with other EAs; more on this in this other comment of mine.) 

Note that all of the following relates to posts I made before joining Rethink, as I haven't yet posted anything related to my work with Rethink.

Q1: Valuable in what ways?

• Maybe the main way the feedback was useful was in helping me get an overall sense of how I was doing as an EA researcher, how I was doing as a macrostrategy researcher, and how valuable the kinds of work I was doing were, to inform whether to carry on with those things.
  • That said, I think votes and comments provided less useful feedback on these points than I’d have expected. That feedback basically just seemed to indicate “You’re probably neither a terrible fit for this nor an amazing wunderkind, but rather somewhere in the vast chasm in between.” Which I guess did narrow my uncertainty slightly, but not very much.
  • But there was one case in which my posts led to a more experienced researcher learning that I existed, perceiving me as having strong potential, and reaching out to me to chat, and I think that that conversation substantially informed my career plans. And since then I’ve had further conversations with that person that have also informed my career plans.
• Another way the feedback was useful was via some comments on posts informing my specific choices about what research or write about next, or what shape those next posts should take.
  • If I recall correctly, this happened a few times with my first series of posts (on moral uncertainty).
    • Some comments helped me determine what threads it’d be interesting/necessary to explore more (e.g., because people were still confused about those things).
    • I think that’d happen less often now, because now I’m better at writing posts in general and I have more opportunities for feedback pre-posting (e.g., from my colleagues at Rethink).
  • The best example was that this comment from MichaelStJules on a post of mine prompted me to make my database of existential risk estimates.
    • Maybe I would’ve eventually ended up making such a database anyway, but I don’t think I’d explicitly thought of doing so before seeing that comment.
    • I think that that database is probably in the top 5 most valuable things I’ve publicly posted this year (out of probably ~35 posts, if we exclude things like question posts and link posts). And I think it was more valuable than the post of mine which MichaelStJules commented on.
    • I think this is an interesting case, because making that database required no special skills (it was just a weirdly overlooked low-hanging fruit that anyone could’ve plucked already), and the relevant part of MichaelStJules’ comment was just one sentence, and they just gestured in the general direction of what I ended up doing rather than clearly outlining it. So it feels sort-of like this was an “easy win” that just required a space for some accidental public brainstorming.
• The way the feedback was most often valuable, but which is less important than the above two things, was via helping me improve specific posts. I often edited posts in response to comments.
• Finally, I imagine feedback sometimes helped me improve my research or writing style.
  • Off the top of my head, I can’t remember that happening. But maybe it happened early on and I’ve just forgotten.

Q2: How valuable (compared to other things)?

• I’d probably describe how useful the feedback was by saying “Maybe less valuable than I’d have idealistically expected, but valuable enough to be a noticeable extra perk of posting publicly.”
• I think the two most valuable sources of feedback for me in 2019 and 2020, which I think were much more valuable than feedback from the Forum, were (1) results from job applications, and (2) conversations with people who were further along in various career paths.
  • This is partly because what I needed most was an overall sense of which pathway I should be heading in.
  • But as noted above, my Forum posts did lead to one instance of (2) - i.e., conversations about my career plans with a more experienced person.
• Regular sources of feedback on things I wrote on the Forum also probably tended to be somewhat less useful than the results from a survey I ran about the quality and impact of my writing on the Forum and LessWrong.

Q3: How did the value vary?

• I think I probably got a similar amount of value per unit of feedback on the EA Forum and LessWrong
  • But I think the case in which my posts prompted a useful conversation about my career plan was prompted by my Forum rather than LessWrong posts.
  • And feedback on LessWrong was less pleasant, on average (more often needlessly blunt or snarky - but still better than most of the internet, and still often the substance of what people were saying was useful).

Q4: What proportion was from my existing network?

• I think almost all of the value came from people “outside of my existing network” (here meaning “people I hadn’t interacted with 1-1, though maybe I’d had public comment exchanges with them”).
  • This is probably partly because:
    • My network of EA researchers / Forum users / similar happened to be quite small at the start of this year
    • I wrote across a wide range of topics this year, so the set of people who’d be able to give useful input on something I wrote is quite wide and diverse, making it harder to have them all in my network and individually solicit their feedback
    • If people were already in my network (e.g. if they were coworkers), I’d be more likely to get feedback from them before/without posting to the Forum/LW
  • The first two of those points have become less true over time, so I imagine from now on I might tend to get a higher proportion of my feedback in ways that don’t require posting to the Forum.

I did try it on some occasions with people who wanted to do research similar to the kind of research that I do. I think that it saved me less time than the time it took me to think of good questions to outsource and explain everything, and so on. This might be partly because there is a skill in outsourcing that I haven't mastered yet. I don't know if it helped anyone to decide whether they should pursue this type of career. If it did, then it was very much worth it.

One way I used volunteers (and friends whom I forced to volunteer) productively was making them read texts that I wrote and asking to comment aloud (not in writing) on everything that is at least slightly unclear. Then I didn't explain, but rewrote that part, and asked them to read again and asked if they understand it now. I found that this is important for texts that contain some complicated ideas/reasoning. E.g., it was very useful for the explanation of optimizer's curse and other things in this article. Not important for simple texts.

Hey Edo, thanks for the question!

We've had some experience working with volunteers. In the past, when we had less operational support than we do now, we found it challenging to manage and monitor volunteers but we think it's something that we're better placed to handle now so may explore again in the coming years, though we are generally hesitant about depending on free labor.

We've not really had experience publicly outsourcing questions to the EA community, but we regularly consult wider EA communities for input on questions we are working on. Finally, and I'm not sure this is what you meant, but we've also partnered with Metaculus on some forecasting questions.