I write The Roots of Progress, a blog about the history of technology and the philosophy of progress. Some of my top posts:
I am also the creator of Progress Studies for Young Scholars, an online learning program for high schoolers; and a part-time adviser and technical consultant to Our World in Data, an Oxford-based non-profit for research and data on global development.
My work is funded by grants from Emergent Ventures, Open Philanthropy, the Long-Term Future Fund, and Jaan Tallinn (via the Survival and Flourishing Fund).
Previously, I spent 18 years as a software engineer, engineering manager, and startup founder.
Ask me anything!
UPDATE: I'm pausing for now but will come back and I will try to get to everyone, thanks for all the questions!
Oh, I should also point to the SSC response to “ideas getting harder to find”, which I thought was very good: https://slatestarcodex.com/2018/11/26/is-science-slowing-down-2/
In particular, I don't think you can measure “research productivity” as percent improvement divided by absolute research input. I understand the rationale for measuring it this way, but I think for reasons Scott points out, it's just not the right metric to use.
Another way to look at this is: one generative model for exponential growth is a thing that is growing in proportion to its size. One way this can happen is that the growing thing invests a constant portion of its resources into growth. But in that model, you expect to see the resources used for growth to be exponentially increasing. IMO this is what we see with R&D.
Another place you can see this is in the growth of a startup. Startups can often grow revenue exponentially, but they also hire exponentially. If you used a similar measure of “employee productivity” parallel to “research productivity”, then you'd say it is going down, because an increasing number of employees is needed to maintain a constant % increase in revenue.
Further, what these examples ought to make clear is that exponentially increasing inputs to create exponential growth is actually totally sustainable. So, I don't see it as a cause for alarm at all, but rather (as Scott says) the natural order of things.