This is an article from New Things Under the Sun, a new website by Prof. Matt Clancy that aims to be "a living literature review on social science research about innovation." If you think economic growth and technological progress are important for the greater good (for example, as a means of benefiting future generations), then you'll benefit from this resource.
I've copied and pasted the full text of the article below, since it is licensed under Creative Commons Attribution 4.0. However, the live version on the website may be updated in the future, so this version will eventually be out of date. This version is current as of October 3, 2021.
This essay is a map of the content on New Things Under the Sun that relate to accelerating technological progress. For this essay I’ll basically take it as a given that technological progress is desirable. While I think that’s true, I don’t necessarily think it is obvious (though see the article “What are the returns to R&D?” for one theoretical argument for why we should be spending more on R&D). The main message is that there are lots of factors that collectively set the rate of technological progress - it’s not down to any one thing.
All of the below is founded on a base of evidence derived from academic studies. That doesn’t mean it’s correct: academic knowledge isn’t received from on high; it is always contested and always evolving. But it does represent a real effort to think through these issues with data and careful thought by a community of experts. So I think it’s worth hearing.
The raw inputs of innovation: human beings
Innovation comes from human beings (at least for now). So one of the most important inputs to innovation is, simply, more people. And indeed, we do have some evidence that larger populations innovate at a faster rate, though most of this evidence relates to innovation that predates the modern world. The article “More people leads to more ideas” reviews some evidence consistent with this view from isolated islands, lab experiments, and the history of the human race.
So anything that increases our population could increase the rate of technological progress.
Of course, not all people become innovators. One reason may be that they lack the resources to innovate (see below, on incentives). But another might be that they lack the very idea of trying new things. People take their cues about what range of options they should consider from the people around them. And if the people around you simply accept problems as unsolvable, you might do the same and not try to innovate yourself. That’s one way that innovation can become self-perpetuating: if lots of people are trying to innovate, that can inspire others to view problems as things to be solved rather than mitigated. The best evidence we have that this notion is true comes from a variety of studies about entrepreneurship. As discussed in “Entrepreneurship is contagious”, we tend to find entrepreneurs in social clusters, and when we expose people who are not themselves predisposed to be entrepreneurs to entrepreneurship, it seems to increase the probability they go on to found businesses. “The ‘idea’ of being an entrepreneur” provides further related evidence: the impact of entrepreneurial peers is highest when they occupy a similar social position, but low when someone probably already has the “idea” of being an entrepreneur. That also suggests people might obtain the “idea” of trying new things from seeing people like themselves try new things.
That implies one way to accelerate technological progress is to expose people to examples of people like themselves innovating.
Innovation with and without a map
So now you’ve got some people who are open to new and better ways of doing things. How successful are they likely to be?
People can actually get pretty far innovating without deep causal knowledge of how things “really” work. This occurs via a process analogous to biological evolution; tinker and retain any improvements. The article “Learning curves are tough to use” looks into one particular example of this phenomenon, wherein experience in a particular production process drives progress (though note that a lot of the literature on this kind of innovation greatly overstates how predictable and useful it is - see “Standard evidence for learning curves is not good enough”).
So, to encourage innovation (at least to some extent), just encourage people to get started using new production processes.
But to move really quickly and to make the kinds of big jumps that you simply can’t arrive at via a process of blind tinkering, it helps to have a map of the terrain. Knowledge can be that map. The article “Science as a map of unfamiliar terrain” reviews evidence from a few papers that patents which use science are more valuable, especially when the patented invention occupies relatively unfamiliar knowledge terrain.
More generally, while modern science has its share of problems, on the whole it generates knowledge that helps us better understand the world. A series of natural experiments where scientific progress was rapidly and unexpectedly sped up or slowed down, discussed in “More science leads to more innovation,” goes on to show that basic scientific discoveries are, indeed, spun out into new and valuable technologies. And even technologies that do not directly rely on new scientific discoveries frequently rely indirectly on them. Evidence for this, suggested by evidence from patent citations, is discussed in “Ripples in the river of knowledge.”
That means another obvious way to accelerate technological progress is to increase our knowledge of how the world works by expanding basic scientific research.
Choosing what science to fund
More science is better, but assuming there are limits, we might want to target areas of science that are most likely to be useful for technological progress. But this is actually quite challenging. The article “Knowledge spillovers are a big deal” looks at several examples where the intended beneficiary of a specific research project accounted for only a small part of the total benefits. In other words, the R&D had many unintended benefits, implying that precise targeting is challenging.
One of the most important ways we actually allocate funds to different scientific priorities at present is via academia. The academic ecosystem has created a system of incentives that pushes researchers to do novel work within a framework of methods and background knowledge sufficiently well understood by a community of experts that its merit can be evaluated. It’s not perfect by any stretch, but it seems to do alright. The article “Science is good at making useful knowledge” surveys some evidence that this is the case, mostly by looking at cases where the knowledge produced by one scientific community is useful to other communities as well (including inventors of technology). Moreover, while the article “Publish-or-perish and the quality of science” looks at some evidence that the scientific incentive system can go awry, the magnitudes involved are not so high that they scuttle the rationale for the whole endeavor. Still - we should push for improvements, and a future essay will look at this issue more closely.
One last issue to be aware of, when it comes to using science to accelerate technological progress, is that the process is a slow one. The article “How long does it take to go from science to technology” looks at citations from patented inventions to academic work, as well as statistical correlations between scientific research and industry productivity, to estimate that the time between a scientific discovery and a technological implementation is often on the order of 20 years.
All told, given a long enough time horizon, scientific funding should not strive to be too targeted to specific problems; as a default, the existing scientific ecosystem can be improved but works tolerably well at allocating funding.
Humans + knowledge = faster innovation
Now, we’ve got people looking to solve problems, and knowledge to help them find solutions. That knowledge isn’t much good if it’s not in the head of the people who need it. For that reason, it’s also important to circulate knowledge. You can do this in a variety of ways.
The article “Free knowledge and innovation” looks at several experimental and quasi-experimental studies on different forms of free knowledge depositories (think libraries and wikipedia) to show making knowledge freely accessible has a measurable positive impact on the rate of innovation.
One way to accelerate innovation is to create free knowledge resources.
Not all knowledge is codified in text though; some of it lives primarily in people’s heads. One way to get at this knowledge is just to pick knowledgeable people up and drop them down in a community that lacks the requisite knowledge. The article “Importing knowledge” looks at this in the context of immigration, examining a variety of cases where an influx of immigrants with a specific knowledge advantage led to that knowledge advantage being picked up in the host country.
Policies that encourages high skilled immigration, especially from people with different knowledge advantages, can also accelerate innovation.
Lastly, the article “Urban social infrastructure and innovation” looks at some evidence that creating ways for people to mix locally - in bars, cafes, and in dense neighborhoods - helps circulate local knowledge. The article “Why proximity matters: who you know” complements this article, highlighting the ways that proximity seems to be important especially for encouraging meetings between people who might not normally interact. Lastly, “Academic conferences and collaboration” indicates that social mixing doesn’t only mean where you live - it looks at how conferences encourage the formation of new collaborative relationships.
Another way to accelerate innovation is to encourage social mixing of people with different sets of knowledge.
Incentivizing Innovation
Once you’ve got people who are open to trying to solve problems, and who have the knowledge to help them innovate, innovation is now in their choice set. But for them to choose to innovate (instead of doing some other thing in their choice set), you’ve either got to lower the costs of innovating or raise the benefits of doing so.
One simple way to lower the costs of innovation is simply to provide the funding for innovation activities, e.g., from the government.
That can be risky. The government might not know who to fund, and recipients might not use the money as intended. But we do have pretty good evidence that this is not an insurmountable problem for governments. “An example of high returns to publicly funded R&D” looks at some quasi-experimental evaluations of programs like the US government’s small business innovation and research grants. These studies show the funds make a difference, and probably have quite good returns on investment.
With some caveats, we also have pretty good evidence that private sector R&D is pretty responsive to market signals, at least in health. “Medicine and the limits of market-driven innovation” looks at a number of studies in medicine that show the R&D of pharma and other private sector health companies is quite responsive to market signals. When there’s more money in curing a disease, there’s more R&D. But at the same time, that R&D tends to be limited to stuff that’s already close to the market. More fundamental research doesn’t seem very responsive to market signals.
Still, to rapidly pull technologies close to viability onto the market, making it more profitable to do so seems to work.
Competitive markets are good at efficiently allocating resources in an economy, by aggregating information and equating the value people derive from a product with the marginal cost of producing it. But not all products are well served by the market system, especially public goods (of which knowledge is just one example). It can be hard to efficiently allocate resources to public goods without market signals. One alternative that has been proposed is Buterin, Hitzig, and Weyl’s proposal for a mechanism to fund public goods in a decentralized fashion, discussed in the article “Optimal Kickstarter.”
As a general principle, the way we incentivize innovation is itself a (social) technology, and we should try to make progress on this technology as well, by trying new things.
Organizing Innovators
Now you’ve got knowledgeable people open to solving problems and incentivized to do so. Can we say anything about the way you should approach solving a problem?
One basic factor to consider is the size of the team that works on a problem. As described in the article “Are ideas getting harder to find because of the burden of knowledge?” there has been a trend towards greater use of teams in the innovation process, possibly because the amount of knowledge that must be directed at a problem tends to grow over time. The article “Highly cited innovation takes a team” surveys some work on the efficacy of innovative teams in different domains, where impact is mostly measured by creating highly cited patents or projects. It tends to be the case that bigger teams are associated with more impact, with some evidence that avoiding “weak links” in a team is more important than including a super-productive “star.” We also have some work (just one study, so be cautious putting too much weight on it) that while teams are good at producing highly cited work, they aren’t so good at producing breakthrough innovations. But a big caveat here is that all this evidence is correlational - we don’t have much in the way of experiments (natural or artificial) to draw on here.
Many problems seem to be best solved with a team.
Summary
To summarize. Innovation happens when people apply knowledge to problems. There are lots of points where we can nudge that process along faster. More people. More people considering innovation. More knowledge. More communication of knowledge. More resources to innovation. More benefits from innovating. Better management of innovators. Each of those might only have a small impact individually; but they stack up. Moreover, they compound, year on year.
There is so much more to say about all this; the academic literature is vast. So this essay will grow, bit by bit, as New Things Under the Sun gets more and more new articles added. But hopefully this has given us some places to start.
Note: this essay is continuously updated as relevant articles are added to New Things Under the Sun. To keep abreast of updates, subscribe to the site newsletter.
Outline
Introduction
The raw inputs of innovation: human beings
More people leads to more ideas
Entrepreneurship is contagious
The ‘idea’ of being an entrepreneur
Innovation with and without a map
Learning curves are tough to use
Standard evidence for learning curves is not good enough
Science as a map of unfamiliar terrain
More science leads to more innovation
Ripples in the river of knowledge
Choosing what science to fund
Knowledge spillovers are a big deal
Science is good at making useful knowledge
Publish-or-perish and the quality of science
How long does it take to go from science to technology
Human + Knowledge = faster innovation
Urban social infrastructure and innovation
Why proximity matters: who you know
Academic conferences and collaboration
Incentivizing Innovation
An example of high returns to publicly funded R&D
Medicine and the limits of market-driven innovation
Organizing Innovators
Are ideas getting harder to find because of the burden of knowledge?