Epoch researchers recently shared a great blog post titled “Most AI value will come from broad automation, not from R&D”. You should read it! This seems like a big-if-true claim given how much focus goes into the ‘intelligence explosion followed by an industrial explosion’ narrative. Below, I summarise their post and then offer some critiques and responses.

Summary of Epoch

In the post, Ege Erdil and @Matthew_Barnett  argue (contrary to the standard narrative) that automating R&D tasks, in particular AI R&D, will not be central to the economic significance of AI. Instead, they argue general diffusion of AI through the economy by automating non-research jobs will be the more important source of economic growth. This is because:

• Historical evidence. R&D has only contributed about 20% of labour productivity growth in the US since 1988. Some of the main non-R&D factors are capital accumulation (buying more and fancier machines can make the same worker more productive) and knowledge diffusion (e.g. learning new techniques and institutional practices from more productive firms).
  • This gives us a (maybe weak) prior that R&D isn’t the main deal. Indeed, the US spends about $5T on capital accumulation and $1T on R&D, roughly proportional to the estimated shares of productivity gains.
• Software singularity may fizzle. A common reason to think that AI will be different than the historical trend of R&D playing a small role in productivity growth is that we will automate AI R&D itself, get recursive improvements to reach superintelligence, and only then rapidly automate all other remote work. Erdil and Barnett think this is possible, but somewhat unlikely, given that algorithmic insights may get progressively harder to find at a steep rate.
  • In this scenario, compute and data are not growing very rapidly, so progress must come from algorithms. Probably, there will be strong complementarity between data/compute and algorithms, such that just better algorithms can only very imperfectly substitute for more data and compute.
  • Historically, automating parts of R&D (e.g. calculations and, to some extent, coding) has not led to dramatic scientific acceleration.
• R&D is hard. Non-AI R&D often requires significant physical labour (e.g. in wet labs for biology) which will be slow to automate. Even AI R&D requires agentic computer-use skills (and long-term planning and research taste) which will be slower to automate than pure abstract reasoning tasks. As such, by the time AIs are able to automate much of R&D, they will have already been previously able to automate many other remote jobs, and will be broadly diffused in the economy. This broad diffusion will be the main driver of economic growth.

A key strategic implication of this is that we will likely have widespread AI diffusion contributing a significant fraction of GDP before we have recursive improvement and superintelligence.

Response to Epoch

Epistemic note: I have thought about this probably a lot less than Epoch people, and also have a less relevant background. Which makes me think I should defer at least moderately to them.

But it still seems valuable to give my object-level thoughts. I mainly don’t directly dispute Epoch’s points, instead giving countervailing reasons that push against their conclusion.

Firstly, I think AI R&D may be relatively early to be automated.^[1] It seems very likely that many physical jobs (e.g. nursing) will be automated relatively late, but even compared to other fully remote jobs:

• AI researchers command high salaries. Particularly if inference-compute scaling continues, the first automated remote workers may be quite expensive to run.^[2] As such, supply may be quite limited, and the first use cases would (all else equal) be in fields with very high human salaries. AI researchers often have very high salaries, so replacing them will be a top priority.^[3]

• R&D skill scales strongly with cognitive ability.^[4] We may get models that are weakly superhuman abstract reasoners before we have models that are good at doing a wider range of computer and physical tasks. R&D jobs seem to have some of the best returns to intelligence - e.g. a 2 standard deviations smarter researcher might be vastly more innovative and productive than her average colleague. This seems different to many other professions where raw intelligence seems like a smaller fraction of the overall job.^[5]

• AI R&D is amenable to trial and error. We may get models that have flashes of brilliance before we get models that reliably never make mistakes. Science is a strong-link problem where what matters most is your very best ideas, not how many mistakes you make or bad ideas you come up with.^[6] Fields like surgery that require very high robustness may be automated later than R&D.

• AI R&D is not heavily regulated. Plenty of fields already do or likely will require humans to do particular tasks. Since AI R&D is less regulated, that will make it easier to automate early.
• AI companies are experts in AI R&D. It is easier to automate what you know lots about, and have proprietary internal training data (e.g. history of GitHub repos). As Erdil and Barnett note, leading AI companies seem bullish on automating AI R&D. Maybe this is misguided or just hype, but I place a bit more weight on that being an honest assessment of AI companies' plans. These predictions may also be self-fulfilling, as AI companies could have quite a bit of discretion about where to direct their automation efforts first.

Secondly, even if broad diffusion of AI through the economy contributes more to general economic growth, I think R&D might be more important strategically. In particular, even if broad automation of labour throughout the economy leads to higher economic growth, tightly scoped automation of R&D in strategic sectors - AI, chip design, cyber offense/defense, military technology - would lead to a larger increase in national power. Competitive geopolitical pressures will likely force countries to focus first on bolstering their military-industrial might rather than broader consumer welfare.^[7] 

Thirdly, the post critiques a ‘software intelligence explosion’ but does not discuss a ‘chip technology’ IE. As Davidson, Hadshar, and MacAskill argue, automating chip technology research is another feedback loop that could, together with software, create accelerating progress. The third type of ‘full stack’ IE involving general capital accumulation and investment in semiconductor manufacturing is closer to what Erdil and Barnett think is likely.

Overall, I still think it is more likely than not that targeted AI R&D automation will be the main game, and broader labour automation will be downstream of this and less strategically important.

—

Thanks to Nathan Barnard, Ryan Greenblatt, and Rose Hadshar for helpful comments.

1. ^{^}

    Tom Davidson has raised some similar points here.

2. ^{^}

    However, inference costs tend to decline rapidly over time.

3. ^{^}

    Other fields with huge salaries, like quant trading, will also presumably be focuses of early remote work automation. This would fit in with the trend of increasingly automated trading anyway.

^{^}

 @Jackson Wagner  made some similar useful points in a comment on the blog post.

^{^}

 E.g. consider an architect. This job seems liable to be automated because (I think?) it could be fully done remotely. But my guess is the most intelligent/good at abstract reasoning architects aren’t many multiples more productive than average architects.

^{^}

 Although, compute to run experiments to work out which ideas are promising will be scarce, so this picture is a simplification, and having lots of costly-to-rule-out bad ideas is still problematic.

^{^}

 For this factor to be important the government would likely need to play a key role in allocating AI to industries. This level of command and control may not happen

Show all footnotes