I think this discussion is a bit too abstract. It could be helpful with concrete examples of non-academic EA research that you think should have been published in academic outlets. It would also help if you would give some details of what changes they would need to make to get their research past peer reviewers.
I just stumbled across this post:
https://forum.effectivealtruism.org/posts/KGqAvdxN6KypkmdPb/effective-self-help-a-guide-to-improving-your-subjective
Which I strongly think would be of interest outside of the small bubble that is EA. It should be published as a pre-print or review article.
Thank you for your comment. This is a good point. I thought it was obvious, but it indeed isn't. A perfect example would be:
https://www.openphilanthropy.org/research/what-a-compute-centric-framework-says-about-takeoff-speeds/
This model is a fully-fledged analysis by an "EA-research" team that would probably gain interest and scrutiny by academic researchers in economics and artificial intelligence. It even has, despite the fact that it is not published in any conventional research outlets that would be picked up by search machines.
I think it could be uploaded to arXiv with little to no change. I think this would greatly enhance the viewership and reach of the research.
Here’s an article by @Brian_Tomasik enumerating, and briefly discussing, what strike me as most of the relevant considerations for and against publishing in academia.
The ‘citability’ consideration also applies to Wikipedia, which requires that all claims be supported by “reliable sources” (and understands that notion quite narrowly). For example, many concepts developed by the rationalist and EA communities cannot be the subject of a Wikipedia article merely because they have not received coverage in academic publications.
There's a third option to get your research in Google Scholar etc. and get feedback and credible ratings and evaluation without all of the "tiresome process" and "extractive and time consuming" (and format-limiting) features of traditional journals.
Post your work on the web, link it/add i9t to an archive, get a DOI (through Zenodo etc.)
Submit it to be publicly evaluated by The Unjournal or another journal-independent evaluation system (Peer Communities In also have some good options).
Relevant Forum Wiki Link
Your research is in all likelihood built upon decades, or perhaps centuries of academic research. Researchers actively browse the academic literature in the areas that they are interested in. If you do not publish your research in a fashion that is noticed by academic databases such as Google Scholar, you will lose a lot of readers. Furthermore, these researchers might have used your research as a building block for their research. In other words, you are actively slowing down future academic research.
The vast majority of academic contributions get read by approximately no one. The average high-effort LessWrong post or EA Forum post likely gets an order of magnitude more readers than they would if the same effort were put into an academic paper.
Almost nobody searches academic databases, even academics and researchers mostly use normal Google search when researching things.
Academia has benefits, but reaching a larger audience is not one of them, as far as I can tell (there are of course exceptions and some publications are much better suited for being published in a journal than a blogpost, but by and large academia does not have a good way of actually driving readership).
The vast majority of academic contributions get read by approximately no one.
Academia has benefits, but reaching a larger audience is not one of them, as far as I can tell (there are of course exceptions and some publications are much better suited for being published in a journal than a blogpost, but by and large academia does not have a good way of actually driving readership).
I agree that most academic research is a bad ROI but I find that a lot of this sort of 'nobody reads research' commentary is equating reads with citations which seems completely wrong. By that metric most forum posts would also not be read by anyone.
I do think it is somewhat fair to say that you will miss out on audiences if you only post on a forum because a forum post probably won't reach certain audiences.
Almost nobody searches academic databases, even academics and researchers mostly use normal Google search when researching things.
This has not been my experience as an academic. For several years my job was to trying to find and synthesise evidence for industry/government decision makers. Those reports involved searching academic databases and largely omitted unpublished literature and never included any forum posts, as far as I recall. I think that this was also true of my colleagues.
Literature reviews are also published and read a lot and these usually do not include unpublished work.
I think we should maybe consider doing some research to test the reach and credibility of different communication modalities to have a better sense of when they are worthwhile. For instance, an experiment to test if people (ideally policymakers or similar demographics) differ greatly in their assessments of content in different formats (e.g., preprint v forum post). Or to see where key decision makers get their information from and what they trust.
Otherwise, we run the risk that a lot of very detailed and helpful content on places like LW and EAF is largely overlooked by people outside those communities, including many important decision makers.
I agree that most academic research is a bad ROI but I find that a lot of this sort of 'nobody reads research' commentary is equating reads with citations which seems completely wrong. By that metric most forum posts would also not be read by anyone.
I agree-for one, the studies I've seen saying that the median publication is not cited are including conference papers, so if one is talking about the peer-reviewed literature, citations are significantly greater. I've estimated the average number of citations per paper is around 30 for the peer-reviewed literature. Furthermore, from what I've seen, the number of reads on places like ResearchGate and Academia.edu tend to be one to two orders of magnitude greater than the number of citations. So I think a reasonable expectation for a peer-reviewed paper is hundreds or thousands of reads.
The academic publishing system is deeply flawed and we need to push for change from an EA standpoint.
As of now, we are losing thousands upon thousands of top-grade, peer-reviewed research articles that are stuck in obscure journals or self-published and read by nobody. This is due to the high cost of translation, publication, and open access fees, which are particularly prohibitive for researchers in the third world. We simply can't afford to publish in high-impact journals unless we do self-funding or beg for foreign funds.
I believe the EA community has compassion for the injustice in the world, and I believe that we can be the push for that societal change. We need to prioritize knowledge and information above the corporate greed of publishers.
I completely agree that academic publishing is completely flawed and incentives are a mess. However, anyone that has actually sought to improve upon that must confess that changing this is difficult and will take time. I think it is an important cause.
Not publishing in academic fashion is not a viable or even working solution to the problem of academic publishing.
I understand, It's a huge task to replace the 150 year systems set in place, we can't do it overnight, but I do believe that not enough academics talk about it. I am not sure why.
I agree with you on the second point too, I have never source-cited anything that wasn't peer reviewed and published in a journal, I know other academics will agree. I was just giving a perspective on why academic writers in general might go through the alternative routes of publishing.
Just saw this from @soroushjp - it seems very relevant.
Almost nobody searches academic databases, even academics and researchers mostly use normal Google search when researching things.
As an academic and researcher, I don't agree with this. I will use regular google or wikipedia if I want the basics of a subject, or am researching something that has no related academic field. If I want high quality research on an established field, I will use google scholar, and from there, look through the citations of relevant papers to find other relevant research.
The vast majority of academic contributions get read by approximately no one. The average high-effort LessWrong post or EA Forum post likely gets an order of magnitude more readers than they would if the same effort were put into an academic paper.
This is the wrong comparison. The counterfactual case here is not "write an academic paper and don't tell anyone about it", it's "write an academic paper, then link it on lesswrong/EAforum". In the latter case, you get both general public reads and academic searchability.
The other point is that we don't care about every read equally. If you can get the worlds leading expert in a subject to engage with your research, the result will be orders of magnitude more productive in terms of feedback and future research than the average Lesswrong user who doesn't really understand what you're talking about. Very, very few subjects have their leading experts reading this forum.
As an academic and researcher, I don't agree with this. I will use regular google or wikipedia if I want the basics of a subject, or am researching something that has no related academic field.
Yep, agree that if you are writing things that could straightforwardly fit into an academic field, publication is often pretty decent (though I still expect that people will find the average engagement their paper gets to be highly disappointing, given the levels of effort involved).
My sense is only a very small minority of work on the EA Forum or LessWrong would straightforwardly fit into an existing scientific field.
The counterfactual case here is not "write an academic paper and don't tell anyone about it", it's "write an academic paper, then link it on lesswrong/EAforum". In the latter case, you get both general public reads and academic searchability.
I agree this helps a bit, though also beware that link posts, especially to long pdfs with bad readability on mobile also tend to lose about 80% of their readership on LW and the EA Forum.
Yeah, I agree with these points. To me the best practice would be:
If the research doesn't really fit within an established field or journal, write it up as a post.
If the research does fit, do a cost benefit analysis on whether it's worth the extra effort to make it publication worthy. Remember that the benefits aren't just in terms of academic readership: the process of making it publication worthy may allow you to spot mistakes or shortcomings, which are also valuable.
If you do try and publish, put it on Arxiv or equivalent and link it on the forum, ideally with a good, readable summary of the findings.
I would say this is largely true, but there are a number of notable exceptions where publishing gets more views and is more influential than writing a forum post. Here are just a couple.
For example this article on health mapping in Uganda is published in a pretty low ranked journal, but is cited by an impressive 40 people and is part of the reason why I connected with the author to make our health AIM tool which has huge potential to impact healthcare provision in remote areas.
https://equityhealthj.biomedcentral.com/articles/10.1186/s12939-020-01371-5
Yes, it is true that a forum post may get you more readers, and more importantly perhaps, a select readership. But I am advocating for also publishing your research in a way so that it shows up in a Google Scholar query. You should probably publish it here too, if it is important.
I do a bit of research myself and my impression is that most researchers rarely use a plain Google search, but rather use Google Scholar, Scopus or tools like ResearchRabbit and Elicit. None of these returns non-scholarly articles.
Furthermore, it is not only about reaching a larger audience, but rather making your research accessible to those who search. Publishing to ArXiv allows for someone who is actively searching for your article to find it. Conversely, the article is of most value to these readers. Lastly, ArxiV and other scholarly servers are more likely to be used for decades into the future, increasing the number of readers in the future relative to other options.
It is not uncommon in the EA-sphere to publish your research on your own website, GitHub, LessWrong or the EAForum. However, I think that more people should consider publishing their research as is usual in academia.
If you are publishing on select webpages only:
Your research is in all likelihood built upon decades, or perhaps centuries of academic research. Researchers actively browse the academic literature in the areas that they are interested in. If you do not publish your research in a fashion that is noticed by academic databases such as Google Scholar, you will lose a lot of readers. Furthermore, these researchers might have used your research as a building block for their research. In other words, you are actively slowing down future academic research.
Even worse, your research might be forgotten. [1]
Most policy-makers, civil servants and the general public seem to value academic credentials and academic research. I find it likely that you are not going to be invited to speak on a topic, if you have not published your research in an outlet that signals credibility to policy-makers. If you want to make a difference with your research by bringing it into policy, you would be better off not (just) publishing it on your blog.
A journal submission may result in low-cost feedback by other researchers. [2] This strikes me as a useful thing to have, particularly as articles such as the post on interest rates and AGI seem to be influential, but would have been rejected by many economists. [3]
Most people that do not publish their research in a way so that it emerges in academic databases name two main objections.
This would have been a fair objection in 2005, but I do not think that it is any longer. Many researchers read and use research published as a pre-print on webservers such as arXiv, as long as it is good research. [4]
It does not take a lot of time to do this and your article is guaranteed to be found by scholarly search machines. You do not need to go through the tiresome process of submitting to academic journals in order for your research to be read by most researchers in your field.
This objection is a good one. However, I think that it applies only to a very small set of research. Namely, research institutes and think tanks who target policy-makers directly and already know that they will be taken seriously. However, if their research may be interesting for others too, they should probably consider publishing it openly as well.
I think that the benefits of publishing in an academic fashion usually far outweigh the costs. So why do people publish their research on non-academic websites?
My very speculative account of why this happens is best explained through an example research paper that I recently came across (on a personal website). The author wrote:
I am choosing to publish [this here] because journals tend to be extractive and time consuming, and because I am in a position to not care about them.
I think the author has done great research and I do not want to criticise that particular decision. But I think that this sentence may reveal some hidden motives.
Of course, academic journals are all about prestige and people with an "EA-mindset" may be inclined to reject this notion. However, I think that the sentence "because I am in a position to not care about them" carries an important message. If the author is in a position to reject prestige-via-journal-publication, this is itself a way of signalling status among the rationalist and EA-community. Maybe it is simply seen as cool to publish your research straight to LessWrong and not to Probability&Statistics Letters.
Did you kow that Europeans have rediscovered Vitamin C at least seven times in around 500 years only to forget about this piece of research quickly after discovery? This information was incredibly important since treating scurvy was very cost-effective back then.
Of course, peer review is often shitty and slow. But often it is also helpful. Hey, it is free!
Interest rates are not only affected by the way people discount the future. Interest rates in most industrialised countries are largely controlled by the central bank and therefore contain little foresight regarding AGI Timelines.
See as an example this work of Aubrey de Grey, which is a simple post to arXiv. The article is minimalistic, and surely did not take a lot of time to typeset. This research has sparked immense work in mathematics despite the fact that it has never been published in an academic journal, the author is not a formally trained mathematician and not affiliated with an organization related to maths.
some related discussion: https://www.lesswrong.com/posts/TYTEJxzeK3jBMq2TZ/your-posts-should-be-on-arxiv and the comments on that post
I think whether a post would be a good fit for an academic journal depends a lot on the concrete article, and might be not worthwhile for some. Maybe you can encourage authors of specific posts directly and point them towards a fitting academic journal?
+1 for this suggestion; it seems promising for someone with a good understanding of different journals' focus areas, submission requirements, and readership to suggest potential destinations to the authors of high-quality posts. Ideally, they would offer some coaching/assistance to the author to help them through the often arduous process of submitting.
I haven't done a BOTEC but it seems potentially good to have more EA thought published in more traditional venues, if only to lend some credibility to the non-published stuff.