https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html
Via Gwern.
In this post I will talk about an approach to research (and other projects that involve high uncertainty) that has substantially improved my productivity. Before implementing this approach, I made little research progress for over a year; afterwards, I completed one project every four months on average. Other changes also contributed, but I expect the ideas here to at least double your productivity if you aren't already employing a similar process.
Many EA type activities could benefit from this framework!
Even smart people will often intuitively (that is to say, without realizing it, or only dimly realize it) shy away from the part of the project that would provide information that would tell them they're doing the wrong thing. This is part of the value of things like gantt charts and other project maps in that even though the plans they are typically used to generate fail when colliding with reality, they can alert you to ways you are fooling yourself about the most uncertain parts of a project.
My own approach i describe as multiobjective optimization but more based on simulated annealing/statistical mechanics) and deals with 'stopping times' rather than 'fail rates' though they are closely connected. I think maybe many EA affiliated people will not go through that whole paper--at least the few i've met. (I was told to get a CS degree either at UCSF where i had a job in theoretical biology or stanford, so i chose the 'stopping time' or 'fail rate'. I was pretty succesful at failing. Completed failing at 4 projects in 4 months. Condoleeza Rice also teaches at Stanford now---she helped win the war in Afghanistsan, Iraq, etc. No, good deed goes unrewarded.
Could you please elaborate on how I could apply this? Or where can I learn more?
I just noticed your question, since I've only recently started looking at EA forums, and I mostly look at the discussions on science, economics, climate change, and on EA methodology and practice (eg the recent one about basic income projects in Malawi by Givedirectly or some other similarily named group). This is one reference https://en.wikipedia.org/wiki/Stopping_time . I am mostly self-educated in stochastic processes, but this is a standard topic in texts. It basically means if you are doing a search --or many searches -- you try to estimate how much time/resources you will spend pursuing one search (or allocate time and resources among several alternative searches), before you will call it a success, or a failure, and give up.
I sort of know this 'intuitively' from hiking in mountains---i sometimes have had to check out several different paths to get to where i want to go, so you try following one for a while, and then decide whether to keep going because it seems to going the right place, or you go back and repeat the search on other trails. (It has been the case that at times all the trail options turned out to 'fail' (headed to cliffs that were impassable to me) so you end up going nowhere at least for awhile.
Multiobjective optimization is another standard topic --utility maximization in economics is one example (often solved via calculus of variations, or for more complex problems via computer algorithms). Intuitively for me this is like a hike where have several attractive choices to go to (different scenic high spots, waterfalls, valleys, or areas with special kinds of flora and fauna --I'm a sort of amateur naturalist) , and usually you can't go everywhere (especially with time constraints) , so have to select some subset which is 'optimal' (and maybe save the ones you missed for another day).
Since my math skills are 'suboptimal' I have been trying to develop my own formalism, which is a 'toy model' (like many in stochastic processes---eg random walks, urn models, etc.) but may capture essence of more complex ones. Its a 'labor of love' and may go nowhere and is sort out of the mainstream. Also its an attempt to make these tropes of problems relativley simple so you dont need a PhD to get the idea, and maybe even apply it. The analogy might be a GPS on your phone or in your car--give you directions on where to go and what to do. .
I have been trying off and on to find people interested in this model---possibly as collaborators (but the few people I've talked either work on their own models, or else work using standard heavy duty computational or high level math formalisms). Also few of them work on applications of the kind I am interested in (which are close to some EA proejcts) --more often they are into investing, sometimes product development, or on allocating resources to best find terrorist cells and such.
Thank you!!