Learnings about literature review strategy from research practice sessions

Earlier this year, myself and a few other early-career researchers decided to try creating sessions where we could deliberately practice certain research skills by creating a high-feedback environment where we could build intuitions and refine techniques. This grew out of a frustration with the slow feedback loops typical of research, the exciting prospect of improving skills that should pay off consistently through an entire career, and a lack of good resources on how to actually do research. 

This document collects some of the lessons learned from our first research practice experiment, a series of five literature review practice sessions with between 3-6 longtermist researchers. Note though that many of the benefits were from improved intuitions or small details about how to do literature reviews and were hard to boil down into concrete advice. For example, we found this to be a useful environment to explore the finer details of how to do research that do not otherwise come up easily in conversation (e.g. how many searches on Google Scholar do you do before you start reading papers?), build intuitions on research taste (i.e. judging trustworthiness of a paper quickly), and to better understand what we actually do when we do literature reviews (think for a moment, do you actually know what steps you take when starting a review?). 

As such, I think it is likely worthwhile to try the exercise for yourself with a group of peers. The structure we used was as follows:

• Identify an interesting question (before starting): These can be almost anything so long as the question is well-structured (e.g. try to avoid having two questions in one). It’s worth playing around with the type of question too and seeing how that affects the literature review strategy.
• Do a quick literature review on the topic (50-60 minutes): Focus on answering the question and producing a readable output. A good framing for the output is to make something you could confidently come back to after a month of not thinking about the topic and be ready to continue the project/review.
• Read and comment on each others’ work (15 minutes): While reading and critiquing others’ work, each person would focus also on what they could have done better given what they see others have accomplished
• Come together to discuss (45 minutes): This would likely touch on what the others found, how they found it, ways we got stuck, and potential methods to improve on the technique which was used. We would generally not spend much time on the subject matter and try to maintain focus on methods rather than content once the review was done.

The following are techniques we identified as being useful. We believe that these should be valid for literature reviews on almost any topic. Note though that we focused primarily on questions within the social sciences, so take this advice with a grain of salt if you are doing technical work. 

• Focus on breadth first! It's best to have a pile of papers ready to look at before diving too deep into any given paper. Some papers are far more valuable than others and if you dive into the first you find, you might miss the highest value papers. Searching smartly is often more effective than going down the citation trail (unless you find the right meta-analysis that is). Be aware however that it’s possible to spend too much time on this step (like any of the others), though we tend to think people usually spend too little. It might even be worth setting a timer for this step to make sure you don’t cut it off too soon or too late. For the exercise I’d suggest (with high uncertainty) 15 minutes, for a real literature review it really depends on context.
• Try a variety of search terms: It’s almost always useful to spend a few minutes generating synonyms and potentially useful terms to search for early on. You probably want to be thinking about good search terms consistently in the early stages of your lit review. It's easy to think you've searched for all the relevant terms when you've already found something a bit promising, so it's good to think a bit outside the box early on and keep looking for new terms to use for search throughout the process. Some search terms can yield much better results than others, sometimes surprisingly. For example, we once found that “private tutoring effect” yielded irrelevant papers while “one-on-one tutoring effect” gave exactly what we were looking for.
• Think before you search: For some questions it can help to make a rough model/writeup of how you would answer the question first or to break down the question into its constituent parts so that you can search through the topic in a way that makes sense to you and/or answers questions of interest posed by the model. E.g. if you are researching what makes people good researchers you might find it useful to first write out what you think the answer is, which factors seem most important, etc. While this may seem to carry the risk of biasing your search, I’d argue that it’s better to have explicit, known possibilities of bias than to let implicit assumptions drive your search without you being aware of them. This way you can:
  • Be wrong and notice that you were wrong and thus update your thinking. Otherwise you might read through new information and think you already knew it, possibly leading to your not incorporating the new information well
  • Consciously look for information that contrasts with what you think (as opposed to the possibility of having an implicit bias in your search)
  • Find useful search terms by breaking down the area into component parts which seem more likely to have a solid academic literature around them
• Connectedpapers.com is a useful search tool in almost all occasions once you’ve found a relevant paper to feed into it. Keep in mind though that it sometimes fails to make the right connections so it’s good practice to generate graphs with a few different papers.
  • In order to find meta-analyses with this tool it can be helpful to use the derivative works tab once you’ve generated a graph with a useful paper.
• To summarize findings I (Alex) generally make a list of potential answers to the question and include the citations for each. This might be worth trying, though it was not proven to reliably be the ideal method of consolidating a literature review

Thanks Nora Amman, Jan Brauner, and Ben Snodin for feedback. Thanks to Nora Amman, Ondrej Bajgar, Jan Brauner, Lukas Finnveden, Chris van Merwijk, and others for attending the sessions, coming up with some of the above ideas, and helping improve the process!