Our website:
What is Let’s Fund?
Let’s Fund helps you discover, learn about and fund breakthrough research, policy and advocacy projects.
We’re Hauke (formerly at CEA and the Center for Global Development) and Henry (software engineer, founder of EA Work Club). We created Let’s Fund as an alternative to “safe” donation options – to find high-risk, high-reward projects that are in need of more funding.
Our first crowdfunding campaign, Let’s Fund #1, is now online. We're raising funds for Professor Chris Chambers at Cardiff University. Chambers is a leading proponent of a new and better way of doing and publishing research, called ‘Registered Reports’, where scientific papers are peer-reviewed before the results are known.
This might:
- make science more theory-driven, open and transparent
- find methodological weaknesses prior to publication
- get more papers published that fail to confirm the original hypothesis
- increase the credibility of non-randomized natural experiments using observational data
The funds will free up his time to focus on accelerating the widespread adoption of Registered Reports by leading scientific journals.
This ‘meta-research’ project might be exceptionally high-impact because we can cause a paradigm shift in scientific culture, where Registered Reports become the gold standard for hypothesis-driven science.
Check out our website to read our in-depth research on the value of science funding, meta-research, discussion of Registered Reports advocacy, and a cost-effectiveness estimate of Chris Chambers’ advocacy efforts.
Project Rationale
We want to move money to effective projects and spread EA memes. We give people an option to donate to high-risk, high-reward projects over established, risk-averse charities outside the mainstream EA cause areas. It’s also for those people who want to learn about where their money goes in advance.
We are looking for help from the EA community. In particular, we're looking for:
- Concrete feedback on our research
- High-level feedback on our first project in particular
- High-level feedback about the idea of Let's Fund in general
- Donations to our first project
- Help in promoting our first campaign
We are happy to answer any questions there or in the comments below.
I haven't reviewed this in depth, but this strikes me as exactly the right type of thing that EAs should now be funding in the space of international development. There is much more potential in the meta- space that hasn't been explored, and this example being supported, and Hauke's experience, seem promising in that direction.
For example, other meta ideas I'd expect to be highly valuable:
etc.
My project http://sogive.org is gathering a charity dataset focused on key indicators of cost-effectiveness.
Hi, I really like this (shared this in my social media channels). I am working on another new research methods project and great to see others working in this area. ( we ( regivolution.org ) presented our work in EA Global London last month, and we hope to make forum post in here soon as well about us as well as our rigorous ultra-rapid review methods ) We might even use this as an inspiration for some materials we will generate in future (We will of course cite/link to this resource)!
General feedback about the project and content:
Some improvement suggestions - Note: In general I want to say that I really like the quality and I didn't see almost any red flags in the quality of your work, you have gone in to detail, and cite pretty much similar material I would cite myself as well and have good material I wasn't even aware of before. Also, these comments are at general level, I did not go through individual studies, except that I noticed one likely quite a problematic study in the page why fund research. I added this comment straight to the google docs document
Are you aware of the Oxford prioritisation project’s short assessment of open science also posted to EA forums? They pretty much stated that there would be in general no room for funding for open science including registered reports. However, they mentioned things that could change their mind and I think your case falls to these categories. I think this could be worth mentioning.
See here: https://forum.effectivealtruism.org/posts/x87EKumG8gS4aH7xP/daniel-may-open-science-little-room-for-more-funding
Transparency of literature review methods: I would like to hear more about the procedure you had for choosing this meta-research cause over other causes. Now it seems to me that you were interested about the possibilities of meta-research did some preliminary, very fast review of 4 options, then you chose that this researcher and pre-registration would be the best bet and then wrote detailed analyses only about this topic and nothing about others. It would be interesting to get a better impression of your "protocol/pre-notes" (:D ) what you planned, and what post-hoc changes you did during the way. I think this would be important, but this would be particularly important as you are advocating for pre-registration policies. (of course, I would not expect full-pre-registration but just some even very rough description of your process along the way, no matter how chaotic it was). One other thing you should probably be quite transparent is how you assess the quality of your references, if you do not assess each one of them carefully this is fine (based on this one quite clearly problematic study I noticed, this is probably likely), but in my opinion you should be transparent about this.
One alternative funding option (note: I have intellectual conflicts of interest as I am involved in both initiatives mentioned, also I think that it is not necessarily better than this very carefully chosen the one you have): Advocating for the use of evidence frameworks/systems. My opinion currently is that in the end for reaching superb scientific research impact we need to advocate for evidence frameworks/systems. The rationale is similar to why we should advocate for meta-science. Meta-science has potential to improve multiple research areas simultaneously. Evidence frameworks have potential to improve multiple methodological aspects simultaneously in a flexible way that can easily and transparently be improved, (e.g. when new meta-research evidence comes up) these can also be applicable to multiple research areas, improve the level of collaboration and improve the transparency of research findings. GRADE in evidence-based medicine is one quite successful example of this although it was still a lot of issues. Curate science is a database and framework in development that specifically designed to assess the credibility of replication studies (particularly in psychology)
European Union has also funded meta-science quite aggressively recently (see this quite large Mirror, research project)
http://miror-ejd.eu/ec-documents/
Things I am less certain about and you might have considered them already
What would make you change your mind: I think this is more concrete and actionable than just listing out what are problems.
The degree of confidence: I personally like the way to present ones confidence to claims e.g. 80% certain.
Functionality of the website
Small thing - feature where you can see a footnote when you hover a mouse over number would be good, also confirmation to move another website was slightly irritating, when you wanted to look what the references actually are in laptop fast, but I think these may be more preference issues.
As I mentioned, in general, I think your work is good, and I didn't see many red flags in the quality of your work, you have done detailed work, and cite pretty much similar material I would cite myself as well. Even though I have read quite a bit of research in this area there was even new interesting stuff for me.
Thanks so much for the detailed comments and feedback!
1) I'm aware of Oxford Prio's research into this and had read their research. They looked more into Open Science which is related but slightly different from meta-research. There are many more funders funding transparancy, openness and data sharing than meta-research. I do not recall that Oxford prio looked Registered Reports specifically. I think it's an interesting analysis but I think the funders they listed in their analysis are neglecting Registered Reports.
2) re: the literature review methods: we do have a section titled 'Our Research process' in one of our reports. I did a literature review and based on expertise in science created a shortlist of potentially very impactful projects within meta-research (some other projects I considered are listed in the report linked above). Then I interviewed about 5 people in the meta-research field and asked them about their opinions on these projects. Registered Reports seemed most promising of all of these projects. How assess the quality of the papers I cite: as a general rule I look into the studies corresponding to how much weight I place on the point I'm trying to make. So if the study is crucial to the point I'm trying to I look into it more. I also try to find evidence from many different sources so that no argument rests on a single source (also see Sequence vs. Cluster Thinking by Givewell), and generally see whether there are many counter arguments and cite those. Thanks also for flagging the conflict of interest in the cancer study I cite, I absolutely see where you're coming from and have added a footnote to the analysis. In this particular case, I disagree that the study should be completely dismissed and has little value, only because this study was partially funded by a pharma cooperation. I believe that the results can broadly be informative and give a general idea of the impact of new drugs, though it might be an a biased overestimate. This because the study was more descriptive as opposed to an experimental, is not advocating for a particular drug, was peer-reviewed, pushlished and partially funded by the impartial National Institute on Aging. It also does fits with my priors that new cancer drugs often do work quite well.
Evidence Frameworks such as GRADE are interesting and I think they deserve to be funded more. I think they are in practise applied much more in clinical science and would not have such a generalized impact as Registered Reports.
re: EU funding for meta-research: It's encouraging that they have funded this specific doctoral programme for meta-research, and that space should definitely be watched, but I can't see that there is a big general pot of funding available for meta-research, that would hint at this area not stillbeing relatively neglected.
In the future we might add explicit numerical estimates of the degrees of confidence in particular claims for now we opted for careful language and hedging where appropriate. It's a great idea to add a section on what would 'falsify' the case for this grant, we might do this in the future (from the top of my mind: if one could provide evidence that RR advocacy would be bad in the sense that getting people to donate to this over conspicious consumption would be net bad for the world, or if there'd be much better high-risk, high-reward funding opportunity that smaller donors would give to that we're distracting donors from then I would definitely change my mind).
Thanks also for the website suggestions! Generally your feedback is very much appreciated.
I think this is a valuable project -- I imagine there must be plenty of donors looking for exciting projects like this, and having the level of detail that is found on the lets-fund site is great. Some more questions about the project:
-- how will go about finding high-impact projects (I think you mentioned to me that you have some in mind already, but do you have a method for generating a sustainable flow of projects or is there a risk that you might run out soon?) -- how will it be funded?
Thanks for the comment!
Our KPI is quality-adjusted money moved. So we not only need a good fundraising ratio but also keep the research quality very high (our first campaign is backed up by ~17k words of analysis). We already have some research on climate change (https://lets-fund.org/research/climate-change/) from my time doing research at the Center for Global Development for our second campaign. But we want to do more and better research, especially into GCR reduction projects (potentially evaluated/peer-reviewed by other (EA) researchers for QA). If you have a really good idea for what we could look into next, we can discuss it in the comments here.
There are many high impact projects out there, one just needs to do all the research to find them. We're currently doing the YCombinator startup school and they tell you 'do things htat don't scale' (see http://paulgraham.com/ds.html).
Because we're not taking a cut of the money donated (in fact, we can't do that becasue we're not handling any money, it's all through Facebook and GoFundMe's crowdfunding platforms), we will have to find donors who think that our work is valuable.