Strong +1 to this. I think I have observed people who have really good academic research taste but really bad EA research taste

Taste is huge! I was trying to roll this under my "Process" category, where taste manifests in choosing the right project, choosing the right approach, choosing how to sequence experiments, etc etc. Alas, not a lossless factorization

These exercises look quite neat, thanks for sharing!

Thanks Seb. I don't think I have energy to fully respond here, possibly I'll make a separate post to give this argument its full due.

One quick point relevant to Crux 2: "I can also think of many examples of groundbreaking basic science that looks defensive and gets published very well (e.g. again sequencing innovations, vaccine tech; or, for a recent example, several papers on biocontainment published in Nature and Science)."

I think there are many-fold differences in impact/dollar between the tech you build if you are trying to actually solve the problem and the type of probably-good-on-net examples you give here.

Other ways of saying parallels of this point:

• Things which are publishable in nature or science are just definitively less neglected, because you are competing against everyone who wants a C/N/S publication
• The design space of possible interventions is a superset of, and many times larger than the design space of interventions which also can be published in high impact journals
• We find power-laws in cost effectiveness lots of other places, and AFAIK have no counter-evidence here. Given this, even a small orthogonal component between what is incentivized by academia and what is actually good will lead to a large difference in expected impact.

I bet it is! The example categories I think I had in mind at time of writing would be 1) people in ML academia who want to be doing safety instead doing work that almost entirely accelerates capabilities and 2) people who want to work on reducing biological risk instead publish on tech which is highly dual use or broadly accelerates biotechnology without deferentially accelerating safety technology.

I know this happens because I've done it. My most successful publication to date (https://www.nature.com/articles/s41592-019-0598-1) is pretty much entirely capabilities accelerating. I'm still not sure if it was the right call to do this project, but if it is, it will have been a narrow edge revolving on me using the cred I got from this to do something really good later on.

This is interesting and also aligns with my experience depending on exactly what you mean!

• If you mean that it seems less difficult to get tenure in CS (thinking especially about deep learning) than the vibe I gave, (which is again speaking about the field I know, bioeng) I buy this strongly. My suspicion is that this is because relative to bioengineering, there is a bunch of competition for top research talent by industrial AI labs. It seems like even the profs who stay in academia also have joint appointment in companies, for the most part. There isn't an analogous thing in bio? Pharma doesn't seem very exciting and to my knowledge doesn't have a bunch of PI-driven basic research roles open. Maybe bigtech-does-bio labs like Calico will change this in the future? IMO this doesn't change my core point because you will need to change your agenda some, but less than in biology.
• If you mean that once you are on the Junior Faculty track in CS, you don't really need to worry about well-received publications, this is interesting and doesn't line up with my models. Can you think of any examples which might help illustrate this? I'd be looking for, e.g., recently appointed CS faculty at a good school pursuing a research agenda which gets quite poor reception/ crickets, but this faculty is still given tenure. Possibly there are some examples in AI safety before it was cool? Folks that come to mind mostly had established careers. Another signal would be less of the notorious "tenure switch" where people suddenly change their research direction. I have not verified this, but there is a story told about a Harvard Econ professor who did a bunch of centrist/slightly conservative mathematical econ who switched to left-leaning labor economics after tenure.

"Working backwards" type thinking is indeed a skill! I find it plausible a PhD is a good place to do this. I also think there might be other good ways to practice it, like for example seeking out the people who seem to be best at this and trying to work with them.

+1 on this same type of thinking being applicable to gathering resources. I don't see any structural differences between these domains.

This is an excellent comment, thanks Adam.

A couple impressions:

• Totally agree there are bad incentives lots of places
• I think figuring out what existing institutions have incentives that best serve your goals, and building a strategy around those incentives, is a key operation. My intent with this article was to illustrate some of that type of thinking within planning for gradschool. If I was writing a comparison between working in academia and other possible ways to do research I would definitely have flagged the many ways academic incentives are better than the alternatives! I appreciate you doing that, because it's clearly true and important.
• In that more general comparison article, I think I may have still cautioned about academic incentives in particular. Because they seem, for lack of a better word, sneakier? Like, knowing you work at a for-profit company makes it really transparently clear that your manager (or manager's manager's) incentives are different from yours, if you want to do directly impactful research. Whereas I've observed folks, in my academic niche of biological engineering, behave as if they believe a research project to be directly good when I (and others) can't see the impact proposition, and the behavior feels best explained by publishing incentives? In more extreme cases, people will say that project A is less important to prioritize than project B because B is more impactful, but will invest way more in A (which just happens to be very publishable). I'm sure I'm also very guilty of this, but its easier to recognize in other people :P -I'm primarily reporting on biology/ bioengineering/ bioinformatics academia here, though consume a lot of deep learning academias output. FWIW, my sense is there is actually a difference in the strength and type of incentives between ML and biology, at least. From talking with friends in DL academic labs, it seems like there is still a pressure to publish in conferences but there are also lots of other ways to get prestige currency, like putting out a well-read arxiv paper or being a primary contributor to an open source library like pytorch. In biology, from what I've seen, it just really really really matters that you publish in a high impact factor journal, ideally with "Science" or "Nature" on the cover.
• It also matters a whole lot who your advisor is, as you mention. Having an advisor who is super bought in to the impact proposition of your research is a totally different game. I have the sense that most people are not this lucky by default, and so would want to optimize for the type of buy-in or, alternatively, laissez-faire management which I pattern match to the type of research freedom you're describing.

All of this said, I think my biggest reaction is something like "there are ways of finding really good incentives for doing research"! Instead of working in existing institutions-- academic, for-profit research labs, for-profit company-- come up with a good idea for what to research and how, and just do it. More precisely: ask an altruistic funder for money, find other people to work with, make an organization if it seems good. There are small and large versions of this. On the small scale you can apply for EA grants or another org which grants to individuals, and if you're really on to something you ask for org-scale funding. I'm not claiming that this is always a better idea: you will be missing lots of resources you might otherwise have in e.g. academia.

But compared to working with a funder who, like you, wants to solve the problem and make the world be good, any of the other institutions mentioned including academia look extremely misaligned. And IMO its worth making it clear that relative to this, almost any lab/ institute's academic incentives suck. Once this DIY option is on the table I think it is possible to make better choices about whether you like the compromise of working at another institution or whether you will use this option to get specific resources that will make the "forge your own way" option more tractable. E.g.: don't have any good ideas for a research agenda? Great, focus on figuring this out in your PhD. Don't know any good people you might recruit for your project? Great, focus on building a good network in your PhD. Etc etc

I'm curious if you still feel like incentives are misaligned in this world, or whether it feels too impractical to be included in your list, or disagree with me elsewhere?

Thanks again :)

Yeah this is great; I think Ed probably called them sleeping beauties and I was just misremembering :)

Thanks for the references!

Appreciate your comment! I probably won't be able to give my whole theory of change in a comment :P but if I were to say a silly version of it, it might look like: "Just do the thing"

So, what are the constituent parts of making scientific progress? Off the cuff, maybe something like:

1. You need to know what questions are worth asking / problems are worth solving
2. You need to know how to decompose these questions in sub-questions iteratively until a subset are answerable from the state of current knowledge
3. You need to have good research project management skills, to figure out what order it makes sense to tackle these sub-questions and most quickly make progress toward the goal which is where all the impact is
4. You need people to have smart ideas to guess the answers to sub-questions and generate hypotheses
5. You need people to do or build things, like run experiments, code, or fab physical objects
6. You need operations and logistics to turn money into materials and people, and to coordinate the materials and people
7. You need managers to foster productive environments and maintain healthy relationships
8. You need advisors to hold you accountable to the actual goal
9. You often need feedback loops with the actual goal, in case you've decomposed the problem incorrectly or something else in the system has gone awry.
10. You need money

I'm making this up, but do you see what I mean?

Then my advice would be to figure out which subset of these are so constraining that you can't start the business of doing the thing, and to solve those constraints e.g. by cultivating instrumental resources like research ability. Otherwise, set yourself up with the set of 1-10 which maximize your likelihood of succeeding at the thing, and start doing the thing. Figure the rest out as you go.

It's totally conceivable that an academic lab is the best place available to you. But I would want you to come to that conclusion after having thought hard about it, working backward from the actual goal.

Assuming the aspects of 1-10 which are research skills are covered, my object level sense is that academia goes wrong on 1,3,5,6,7,8,9.

All told my algorithm might be something like:

1. What other existing entities/ groups look good on these inputs to the scientific progress machine? These might be existing companies, labs, random people on the internet, non-profits, whatever. Would also include looking for academic opportunities that look better on the above. Don't think about made up categories like "non-profit" when doing this. Just figure out what it would look like to work at/with this entity to accomplish the goal.
2. What levers do I have to tweak things such that my list of existing places looks even better?
3. What would it look like for me to make my own enterprise to directly do the thing? What resources am I missing?
4. What opportunities do I have to pursue instrumental goods/ resources that don't look like doing the thing?
5. With bias toward doing the thing, see which of working with existing collections of people, pushing existing collections of people to be different in some way, starting your own thing, and gathering instrumental resources you are missing looks like it will lead to the best outcomes.
6. Do that thing. Periodically reevaluate.

This probably isn't very helpful, but I don't know of any tricks! I could say more stuff about "industry" vs. "academia" but for the most part I think those conversations are missing the point unless you can drill way more into the specifics of a situation.

Good luck :) remember that lots of other people are trying to figure the same kind of thing out. In my experience they are the best people to learn from

Thanks Charles! I think of your two options I most closely mean (1). For evidence I don't mean 2: "Optimize almost exclusively for compelling publications; for some specific goals these will need to be high-impact publications."

My attempt to restate my position would be something like: "Academic incentives are very strong and its not obvious from the inside when they are influencing your actions. If you're not careful, they will make you do dumb things. To combat this, you should be very deliberate and proactive in defining what you want and how you want it. In some cases this might involve pushing against pub incentives, in other cases it might involve optimizing for following them really really hard. What you want to avoid is telling yourself the reason for doing something is A, while the real reason is B, where B is usually something related to academic incentives. Publishing good papers is not the problem, deluding yourself is."