A frequent worry is that our scientific institutions are risk-averse and shy away from funding transformative research projects that are high risk, in favor of relatively safe and incremental science. Why might that be?

Let’s start with the assumption that high-risk, high-reward research proposals are polarizing: some people love them, some hate them. If this is true, and if our scientific institutions pay closer attention to bad reviews than good reviews, then that could be a driver of risk aversion. In this podcast, I look at three channels through which negative assessments may have outsized weight in decision-making, and how this might bias science away from transformative research.This podcast is an audio read through of the (initial version of the) article Biases Against Risky Research, originally published on New Things Under the Sun.Articles mentionedGross, Kevin, and Carl T. Bergstrom. 2021. Why ex post peer review encourages high-risk research while ex ante review discourages it. PNAS 118(51) e2111615118. https://doi.org/10.1073/pnas.2111615118

Krieger, Joshua, and Ramana Nanda. 2022. Are Transformational Ideas Harder to Fund? Resource Allocation to R&D Projects at a Global Pharmaceutical Firm. Harvard Business School Working Paper 21-014. 

Jerrim, John, and Robert Vries. 2020. Are peer reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications. The Social Science Journal 60(1): 91-109. https://doi.org/10.1080/03623319.2020.1728506

Lane, Jacqueline N., Misha Teplitskiy, Gary Gray, Harder Ranu, Michael Menietti, Eva C. Guinan, and Karim R. Lakhani. 2022. Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation. Management Science 68(6): 3975-4753. https://doi.org/10.1287/mnsc.2021.4107

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

This is an audio read-through of the initial version of How Common is Independent Discovery?

Like the rest of New Things Under the Sun, the underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

This is an audio read-through of the initial version of Science is getting harder.

Like the rest of New Things Under the Sun, the underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

This is an audio read-through of the initial version of When Extreme Necessity is the Mother of Invention. To read the initial newsletter text version of this piece, click here.

Like the rest of New Things Under the Sun, this underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

This is an audio read-through of the initial version of Steering Science with Prizes. To read the initial newsletter text version of this piece, click here.

Like the rest of New Things Under the Sun, this underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

This is an audio read-through of the initial version of Progress in Programming as Evolution. To read the initial newsletter text version of this piece, click here.

Like the rest of New Things Under the Sun, this underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here.

Note: An audio version of New Things Under the Sun is now available on all major podcast platforms. Apple, Spotify, Google, Amazon, Stitcher

Think of new technologies as proceeding through a set of stages:

* Basic scientific research that explores phenomena

* Applied research to better understand how to harness certain phenomena

* Technology development to capture and orchestrate phenomena for a purpose

* Marketing and diffusion of the new technology

The real world can be more complicated with back-and-forth interplay between the stages, but this is a fine place to start. If you want to shape the direction of technology, you can intervene early in this process and try to push the kinds of technology you want onto the market, by subsidizing research. Or you can intervene at the end of the process and try to pull the kinds of technology you want into existence by shaping how markets will receive different kinds of technology.

One specific context where we have some really nice evidence about the efficacy of pull policies is the automobile market. Making fuel more expensive or just flat out mandating carmakers meet certain emissions standards seems to pretty reliably nudge automakers into developing cleaner and more fuel efficient vehicles. We’ve got two complementary lines of evidence here: patents and measures of progress in fuel economy. In this post, first I’ll go over the evidence, and then I’ll talk a bit what I think we should take away from it. In my view, we have strong evidence that pull policies work well for incremental progress, but the case for their efficacy at promoting radical innovation is a lot shakier.

Patents, Progress, and Pull Policies

One pull policy is a tax on undesirable technologies, since, all else equal, that makes the taxed technology less profitable to develop and alternatives more profitable to develop. A carbon tax is the most famous example of this kind of policy. Aghion et al. (2016) are interested in how a carbon tax might change innovation in the auto sector, but given how rarely anyone actually tries to implement a carbon tax, they can’t directly study the question. Instead, they do the next best thing. From the perspective of a carmaker, one of the main effects of a carbon tax is to raise the price of fuel. So how do carmakers respond to higher fuel prices?

To answer that question, they need a way to measure innovation. More specifically, they want to measure the kind of innovation a firm decides to do: do carmakers focus on clean technology (electric, hydrogen fuel cell, and hybrid vehicle) or conventional fossil fuel innovation? Patents are a useful dataset for this kind of problem, since they are correlated with inventive effort and can be easily categorized into different kinds of technology. One problem though is that patents vary tremendously in how valuable they are. Some are very important, but a lot are junk. So Aghion and coauthors focus on the subset of patents for which the patent-holder sought patent protection in three big markets: the USA, the European Union, and Japan. Since it’s costly to apply for a patent in each market, doing so in all these markets is a signal that the inventor thinks the patented invention is sufficiently valuable to be worth protecting in multiple large markets. So in this paper, you can think of them measuring innovation by counting the number of valuable patents for different kinds of automobile technology.

In an ideal setting, if we really wanted to assess the impact of fuel prices on the innovation decisions of carmakers, we would want to randomly assign some carmakers to face higher fuel prices than others. Then we could compare the subsequent patenting behavior across groups facing different fuel prices. And if we wanted to establish this robustly, we would want to do this kind of experiment many times.

We can’t do that. But Aghion and coauthors do something that gets you closer to this ideal. The price of fuel varies a lot from year-to-year, thanks to fluctuations in the price of oil (see left figure below), but it also varies a lot from country-to-country, because countries vary substantially in the size of their taxes on fuel (see right figure below).

Moreover, carmakers typically sell to multiple countries, but have different footprints in different countries. Aghion and coauthors reason that carmakers are more sensitive to (tax-inclusive) fuel prices in the countries where they have a larger share of their total sales. For example, in the figure above it’s clear that the UK raised taxes pretty substantially over the 1990s, while taxes remained flat in the USA. In other words, if we have two carmakers, one with most of its sales in the USA and some in the UK, and another with most of its sales in the UK rather than the USA, then these carmakers are effectively facing different fuel prices. The carmaker selling primarily to the UK sees an increase in effective fuel prices, and we can compare their behavior to the one selling primarily in the USA.

For every carmaker, Aghion and coauthors construct an “effective” fuel price that is specific to that carmaker, by weighting the fuel price in each country by the carmaker’s exposure to that country. They estimate this exposure from the share of patents the carmaker seeks protection for in that country over 1965-1985, because generally you don’t bother seeking patent protection in countries where you don’t plan to operate in the future. They then look to see how the patents of carmakers differ in the subsequent 20 years (1985-2005) as each carmaker faces a different effective fuel price. We can then compare the behavior of firms that were established in markets that went on to have higher fuel prices to the behavior of firms that were established in markets that went on to have lower fuel prices.

(Note that they estimate the markets where a firm is established over 1965-1985, but look at the effect of fuel prices over 1985-2005; this prevents their results from being driven by innovative fuel efficient companies entering markets when they raise taxes, or from their measure of exposure to different markets from being whipped around by subsequent patenting activity)

Aghion and coauthors find fuel prices exert a powerful impact on innovation. A 10% increase in the effective price of fuel (that a specific carmaker is exposed to) is associated with roughly 5% fewer patents of the conventional “fossil fuel” type, and a 10% increase in clean energy patents.

Rozendaal and Vollebergh (2021) adapt this strategy to study the impact of emissions standards on auto innovation (in addition to fuel prices). Emissions standards are essentially a requirement that a carmaker’s average CO2 emissions per mile fall below some target by some date. It’s not actually quite that simple; but that’s the gist of the idea. Just as fuel prices differ across time and space, so to do regulatory standards. And just as a carmaker is more likely to care about the fuel prices in markets where it has a lot of sales, so too is a carmaker more likely to care about the emissions standards of markets where they have a lot of sales.

Rozendaal and Vollebergh make one of those kinds of observations that is obvious in retrospect but which some previous papers apparently missed. In terms of its impact on the rate and direction of innovation, what matters is not whether a country has an emissions target, or even if this target is high or low. What matters is (1) how high or low is this target relative to today’s average emissions and (2) how long do carmakers have to meet the target? If the standard is high, but you’ve already cleared it, then even though it’s high it imposes no extra incentive to innovate. On the other hand, if the standard is high and you have not met it yet, but actually have a long way to go to meet it, then it matters whether you’ve got ten years or one year to get there. If you don’t account for this kind of thing, it might look like standards don’t have much of an impact on innovation.

Rozendaal and Vollebergh construct a measure of standards that takes all this into account: it’s basically the difference between the average emissions of a country and the target, divided by the number of years left to meet the target. When the number is large, it means the average car in the country has a long way to go before it meets the target, and not a lot of time to get there. Here’s how their measure looks for the big three markets.

As with Aghion and coauthors, Rozendaal and Vollebergh construct estimates of each carmaker’s exposure to these three major markets, as of the year 2000. Those heavily exposed to Japan, for example, faced stronger incentives to innovate in the 2000s, compared to those heavily exposed to US and EU markets. But after 2010, the situation has largely reversed. Again, Rozendaal and Vollebergh are going to look at valuable patenting of clean and dirty technology in response to these measures of emission standards stringency (in addition to fuel prices).

And they find these pull policies work. A 10% increase in the stringency standards is associated with 2% more clean patents. Also importantly, Rozendaal and Vollebergh confirm Aghion and coauthor’s finding that fuel prices matter, albeit the strength of the relationship is weaker than Aghion and coauthors find. This might be because they are looking at a very different time frame, 2000-2016 compared to 1985-2005.

What I like about these studies is that they have thousands of different inventive entities, and each of these entities is, at least in principle, facing a “pull policy”of a different strength (based on the mix of markets they operate in). This lets you estimate pretty precisely how effective these policies are. Of course empirical economics is hard and these papers have a few potential weaknesses too. Among these is their reliance on patents as a measure of innovation. So let’s turn to a complementary strand of the literature that doesn’t rely on patents at all. The trade-off will be that we no longer have measures of the rate of innovation for quite as many different organizations facing different policies. Then, after that, I’ll point to a few more reasons for caution about these results.

Directly Measuring Fuel Efficiency

Presumably, the reason policymakers are interested in shaping the direction of progress in automobiles is that they want to encourage innovation that makes cars more fuel efficient. Ideally, we would want cars to be zero emission vehicles. So why not simply measure fuel efficiency directly, if that’s what we care about? We don’t actually care about patents, but results.

The challenge here is that you can improve fuel efficiency without innovating. You can make lighter cars, or cars with smaller engines with less horsepower. In general, cars have a lot of attributes and some of those attributes are associated with a reduced fuel efficiency. So if you want to measure technological progress, what we really need to do is see if cars get more fuel efficient without changing any of these other attributes. And that is precisely what another strand of literature tries to do.

Knittel (2011) has data on the models of practically all consumer vehicle models sold in the USA between 1980 and 2006: approximately 14,000 cars and 13,000 light trucks. For each of these vehicles he has data on their attributes, most importantly weight, horsepower, torque, and fuel efficiency (as measured in miles per gallon). As illustrated in the figure below, there is a tradeoff between each of these attributes and fuel efficiency. In general, models with higher weight, horsepower, or torque, have lower fuel efficiency. But what’s interesting is to compare the shape of this tradeoff at the beginning of the sample (in 1980) to the end (in 2006). For any of these attributes, if you pick a level you want, it tends to be feasible to get much higher fuel efficiency in 2006 than in 1980. The gap between these curves is a fairly direct measure of technological progress.

What Knittel ends up with is year-by-year estimate of the state of US auto fuel efficiency technology; that is, on average, how much more fuel efficient can you make a car, compared to one with the same attributes in 1980. He can then look at how this measure changes over time to assess the rate of technological progress. The results of Aghion et al. (2016) and Rozendaal and Vollebergh (2021), which were based on patents as a measure of innovation, imply that we should see the fastest technological change during periods of elevated US fuel prices and when US fuel economy standards were most stringent (relative to current levels). And that’s basically what Knittel finds - a 10% increase in the fuel price was associated with 0.3% faster technological change, and a 10 percentage point increase in fuel economy standards was associated with one percentage point faster technological progress.

Klier and Linn (2016) builds on this approach to run the sort of larger-scale statistical tests that Aghion et al. (2016) and Rozendaal and Vollebergh (2021) did with patent data. Klier and Linn estimate technological progress over 2000-2012 for US car models and 2005-2010 for Europe, a period during which the US and EU began implementing some new emissions regulations. Unfortunately for the economists, each of these regulations applied to all cars or trucks, which made them less useful for testing their efficacy, since ideally we would like to compare the behavior of firms that face regulations to those that don’t. But as with Rozendaal and Vollebergh, Klier and Linn exploit the fact that the same regulation can impose different burdens on different car companies. Specifically, manufacturers whose models are far from meeting the standards have a lot more work to do than those who are close to meeting the standard (or who already have met them). So Klier and Linn’s idea is to construct manufacturer-specific measures of stringency based on how far each manufacturer needs to tug up the fuel efficiency of its fleet, and then see if we observe faster technological progress among the cars of manufacturers who faced more stringent regulations from their point of view.

And we do. For every standard deviation increase in the stringency standards faced by a firm, technological improvement in US cars improved by an extra 0.5 percentage points per year, US trucks improved by an extra 1.5 percentage points per year, and EU cars improved by an extra 0.3 percentage points. Compared to baseline rates of improvement in the range of 1.4-2.0% per year, this isn’t bad.

So this is overall reassuring. We’re not touching patents at all with these two papers, but we still observe faster technological progress during the periods that the other patent-based papers suggest we should see it. But maybe we’re still worried that we haven’t properly controlled for everything in our statistical models. There were also recessions and sky-high fuel prices during 2000-2012, after all. Klier and Linn attempt to control for this, but it’s challenging. So let’s look at one last paper, Kiso (2019), which tries to exploit an unusually clean setting to test the effect of emission standards.

Kiso’s basic idea is to identify a setting that is very close to an experiment where we have two identical groups, and then we change one variable for one group but not the other. He looks at cars sold in the USA over the period 1985-2004. This was a twenty-year period during which US fuel prices were relatively stable and emissions standards in the US and EU were also fairly stable.

But during this period, Japan instituted emissions standards on vehicles sold there. These policies were announced in January 1993, with a deadline to hit the targets by the year 2000. In March 1999, a new set of targets were established for the year 2010. As indicated below, different vehicle weights had to hit different targets.

During this era, US and EU sales into Japan were tiny as a share of their total sales, whereas Japan was obviously a large and important market for Japanese carmakers. So obviously Japan was more incentivized over this period to improve fuel efficiency than US and EU carmakers, who are facing relatively stable prices and no major new emissions standards in their major markets. But the thing is, Japan is under no obligation to sell the same cars in Japan and the USA (and in fact it does not). So it could have been that Japan sold small fuel efficient cars in the Japanese market, and larger less fuel efficient cars in the US market, where emissions standards were relatively lax.

But the thing about ideas is they have a tendency to spill over into new applications. If you discover ways to make cars more fuel efficient without sacrificing other desirable vehicle characteristics, there is no reason you couldn’t apply those ideas to all the cars in your fleet. So Kiso’s idea is to use the methods of Knittel (2011) and Klier and Linn (2016) to compute the rate of technological progress (in fuel economy) for cars sold in the USA for Japanese carmakers and then to compare that to the rate of progress for cars sold in the USA from US and EU carmakers. If there’s a systematic difference beginning after 1993, when Japan introduced tighter standards, that’s evidence that the standards induced technological progress which spilled over into Japanese carmaker’s models in other countries.

And that is basically what we see. The figure below is the fuel efficiency gap, holding fixed car characteristics, between the US models of Japanese carmakers and the US models of US/EU carmakers. Beginning a few years after the policy announcement in 1993, the cars of Japanese carmakers have persistently better fuel economy than the US and EU cars. I’ve highlighted the three different relevant time periods in the figure below. Note the region highlighted in gray, which I call “out of sample” corresponds to a period when rising fuel prices and tightening emissions standards in the US muddy our interpretation of Japan’s emissions standards (since we now have multiple things changing at the same time).

Taken together, over 1995-2004, fuel economy technology was 2.3-3.3% higher per year for Japanese carmakers (depending on whether you include or exclude the lightest and heaviest cars) as compared to US and EU carmakers.

How far can we extrapolate this?

I think this is pretty compelling. Aghion et al. (2016) tells us fuel prices exert a strong effect on the direction of technological progress, as measured by patents. Five years later, Rozendaal and Vollebergh (2021) use the same approach on a newer slice of the data, showing that effect is still there, albeit weaker. They also show that emissions standards, measured properly, exert the same kind of effect. Then, Knittel (2011) and Klier and Linn (2016) show that measures of technological progress which are based on the actual attributes of vehicles, rather than counting patents, also accelerate during times of high fuel prices or more stringent standards. Finally, Kiso (2019) shows that even when you restrict your attention to the set of cars sold in one market with broadly stable fuel prices and emissions standards, you can detect technological progress speeding forward among the cars of automakers who are, in separate markets, facing pressure to meet higher emissions standards.

At least for cars, pull policies work.

Moreover, we can extend this conclusion beyond auto markets. There is also a slew of studies (some of them briefly mentioned here) that find higher energy prices are associated with more patenting in various renewable energy technologies (like solar, wind, or battery technologies). And in healthcare, a lot of work has also shown that firms increase R&D on diseases that become more profitable to treat. Lastly, we may want to think of Operation Warp Speed, which successfully accelerated the development of highly novel mRNA vaccines by placing huge advance orders for vaccines that had not yet been validated (among other things).

How far should we extrapolate these results? Should we conclude that pull policies usually work? That we should use them for pretty much everything?

I think there are a few things to keep in mind before going that far.

First, one big caveat with all these studies on emissions standards is we can’t quite think of these standards as being just randomly selected. It may be that more ambitious standards are set precisely when new technological opportunities make it feasible to reach those targets. That doesn’t mean the standards don’t work, since technological advances that were possible might not have happened without the standards. But it does mean we need to be cautious about extrapolating from this setting to settings where technology is in a murkier state.

In fact, at least in the context of the EU standards, a 2021 paper by Reynaert provides some evidence that the ambitions of these standards were carefully matched to what experts at the time believed was technologically feasible. For example, he notes:

The EU Commission relied on several studies to support the design of the EU emission standard [....] The policy report only includes possible technology adoptions that should be readily available for the car makers at no fixed or development costs. In designing the regulation, the policymaker clearly had the channel of technology adoption in mind.

I think this suggests the results might be overstated to some degree: to some extent more stringent policies lead to more rapid technological advance, but it’s also true that more stringent policies are also selected when more rapid technological advance is feasible.

Second, and related, recall the simplistic model of technology development I mentioned at the beginning of this piece. New technologies start with basic exploratory science, move into applied research and then technology development, before finally marketing and diffusion. In this other piece, I looked at some evidence that profit incentives worked quite well for the latter stages of health-related R&D, but relatively poorly for initial stages. Specifically, the profit motive’s effect on the development of drugs farther from being market ready appears to be pretty weak.

Can we say the same thing about cars? That this policy is great for speeding up incremental change which is already close to being commercially viable, but poorly suited for more transformative and radical technological change?

As we would expect if the regulations were designed to enable the adoption of already existing technologies , it does seem that technological change in this context was largely incremental. That said, it’s a bit tough to be sure since few papers look specifically at this. However, both Reynaert (2021) and Knittel (2011) discuss the kinds of technological changes that helped bring about higher fuel efficiency. Knittel has the following charts, showing how various new technologies get integrated into more and more cars over time. This is innovation of a sort, but this chart implies a lot of the technology gains came from more and more cars adopting existing technologies that had already been successfully proven elsewhere.

Klier and Linn (2016), as well as Reynaert (2021) doesn’t even call this “innovation” in their papers, preferring to use the phrase “technology adoption” to describe the process of car models becoming more fuel efficient. I tend to think it’s still a form of innovation - the vehicles had to be redesigned to incorporate these changes - but it’s innovation at its most incremental.

We can also turn to some other literature that attempts to describe how much different technologies rely on science for innovation. Ahmadpoor and Jones (2017) is a paper that computes how “close” a technology is to science by looking at how long is the chain of citations connecting an academic article and a patent. Auto-related technologies have pretty long chains by this metric. Patents pertaining to the internal combustion engine, motor vehicles, and electricity transmission to vehicles all having a mean distance to science exceeding 4, meaning on average the shortest link between a patent in one of these fields and a scientific journal is the patent cites a patent that cites a patent that cites a patent that cites a journal article. And a large share of patents never reach a scientific paper at all.

You can see how this stacks up against other technologies in the figure below; the horizontal axis is the length of a citation chain between patents and papers (the further the right, the further from science) and the vertical axis is the share of patents with any linkage to science (the lower, the less connected to science).

Both axes suggest auto technology is historically not dependent on science. And if you don’t like patent citations, this is also indicated by a 1994 survey of lab managers, where just 13% of projects used publicly funded science, one of the lowest shares of any industry surveyed (and as compared to a 20% average).

So although pull policies seem to have worked really well, as a general rule, innovation in auto tech has largely been of the incremental type where pull policies usually work well (at least, according to evidence from health). But there may be one important exception: electric vehicles.

The transition to an all electric vehicle fleet is a a more radical kind of innovation than the kind of incremental changes discussed above. Aghion et al. (2016) and Rozendaal and Vollebergh (2021) are the only papers that address the impact of pull policies on electric vehicles. Both papers include electric vehicle patents among their measures of clean technology, though only Rozendaal and Vollebergh actually split out electric vehicle patents from the rest to see how emissions policies affect them specifically. But when they do, they find the effect of emissions standards on electric vehicles is actually stronger than it is for other technologies. A 10% increase in standards stringency is associated with 3% more electric vehicle patents, compared to an average of 2% across all clean technologies.

One possible answer to this question is that by the time of Rozendaal and Vollebergh’s study, electric vehicle technology was, in fact, commercially viable and so pull policies worked well. Electric vehicles have been around a long time, but what made the technology much more promising in the 21st century (when their study is set) seems to have been the development of a new class of powerful lightweight batteries (lithium ion batteries). The development of these batteries, meanwhile, seems to have been largely a case of a classic knowledge spillover from the consumer electronics sector. As far as I can determine, it wasn’t the case that these batteries were developed by carmakers spurred on by the promise of profits for developing radically more fuel efficient vehicles. They just got lucky; though luck like this happens all the time in innovation, and is why you should generally push forward innovation along lots of dimensions at once.

But in terms of the efficacy of pull policies, I worry that Rozendaal and Vollebergh’s finding that emissions standards worked very well to promote more radical electric vehicle innovation is another example of high standards being set when policymakers believe the technology is already capable of meeting them. Good luck brought the auto sector good batteries, and then after Tesla motors proved you could make desirable cars on the platform, policymakers decided to push the sector to adopt this new technology. But if the sector had not had the luck to get gifted better batteries from consumer electronics, or put another way, if these pull policies had been implemented before these batteries had been developed, maybe we would not see such good results.

But it’s tough to say. I think we have unusually good evidence here that pull policies certainly work for pushing forward incremental innovation. I’m skeptical that they work nearly so well for pushing forward radical technology, but the evidence we have isn’t so great on that question.

Thanks for reading! For other related articles by me, follow the links listed at the bottom of the article’s page on New Things Under the Sun (.com). And to keep up with what’s new on the site, of course subscribe!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here.

Note: An audio version of New Things Under the Sun is now available on all major podcast platforms. Apple, Spotify, Google, Amazon, Stitcher

There’s this idea that technology is characterized by path dependency: once you start going down one technology trajectory, you kind of get locked in and it’s hard to switch to another, possibly better trajectory. That can happen for lots of reasons, but one possibility is that it’s something about the nature of knowledge itself. The more you know, the more you can learn: knowledge begets more knowledge. So whichever technology trajectory we start on becomes the one we know the most about, and therefore the one it makes most sense to stick with.

One line of evidence about this comes from dynamics of patenting. You know what’s a pretty good predictor of patent activity in the future? Patent activity in the recent past. In this post, I want to see what we can learn from a literature that directly or indirectly looks at this dynamic. But while I think this line of evidence is useful, I want to be up front that it also has significant limitations. Most importantly, in this literature we almost never get anything like an experiment. Instead, we’re reading the tea leaves in observational data.

Patent Stocks

Specifically, we’re going to look at the conceptual category of a “patent stock” (also frequently called a “knowledge stock”). To illustrate just what a patent stock is, let’s start with a practical implementation of the notion in a paper.

Aghion et al. (2016) is a paper interested in three kinds of technological progress among automobiles: clean tech (think electric cars, hydrogen fuel cells, and hybrid vehicles), fossil fuel technology (think internal combustion engine), and “grey” tech (think more fuel efficient fossil fuel technology). The paper measures innovation in these different technologies by counting valuable patents. There are about 2,500 companies and 1,000 individuals who hold one of these patents, and Aghion and coauthors construct three patent stocks for each of these inventive entities, one for each of these three flavors of technological progress. (Constructing patent stocks is hardly the only thing this paper does, but it’s what I’m focusing on today)

Constructing a patent stock basically means adding up all the patents that an inventive entity has taken out in the past, giving more weight to the more recent patents. To be very explicit, suppose Ford Motor Company had a clean technology patent stock of 1,000 last year and obtains 250 new patents for clean technology this year. Then, to construct the patent stock for current year, we take last years’ patent stock, multiply it by 80%, and then add to that the number of new patents. So this year’s patent stock is 0.8 x 1000 + 250 = 1050. Suppose we get 150 patents next year. Then the patent stock next year is 0.8 x 1050 + 150 = 990. The exact value of 80% isn’t that important and people use different numbers, though always less than or equal to 100%. The key idea is it’s telling us, in a single number, something about the prior patent activity of the Ford Motor Company. Note that last year’s patents are worth less than this year’s patents (in this case, 80% as much). And since we apply the 80% discount each year, patents from two years ago are worth even less (80% x 80% = 64%).

For the data that Aghion et al. (2016) have, for clean technology, a 10% increase in last year’s clean tech patent stock is associated with a 3% increase in this year’s new clean tech patents (for that firm). For fossil fuel technology, the link is even better: if a firm has a 10% higher patent stock in fossil fuels last year, that’s associated with 5% more new fossil fuel patents this year.

This is quite a robust (though not universal) finding. Rozendaal and Vollebergh (2021) look at the same context (clean and dirty innovation in automobiles), but with a more recent slice of data (2000-2016, instead of Aghion and coauthors’ 1986-2005). In their sample, they find an even stronger link: roughly speaking, a 10% increase in last year’s patent stock is associated with a 10% increase in patenting this year. Noailly and Smeets (2015) do something similar for innovation in renewable energy and fossil fuel energy (not automobiles). They also find that firms sought many more clean or fossil fuel patents when last year’s patent stock (of the appropriate type) was higher.

You can also go beyond firms and look at whole countries. Looking specifically at US patents, a famous 2002 paper by David Popp uses a slightly different approach to create patent stocks for 11 different technologies related to energy. He finds that a 10% increase in last year’s patent stock for a particular technology was associated with 7% more patents for that technology this year. And Porter and Stern (2000) compute patent stocks for 17 different OECD countries (looking at patents they seek in the USA, to hold consistent the definition of a patent). Again, a 10% increase in a country’s patent stock last year is associated with 8-11% more patents this year.

This isn’t a universal finding. Lazkano, Nøstbakken, and Pelli (2017) look at innovation in renewable energy, conventional energy, and storage (battery) technology. Unlike the work discussed above, they typically find the opposite result: a higher patent stock for energy storage technology last year is associated with less energy storage patenting today. Similarly for renewable energy. That said, this paper is the outlier (I’ll return to it briefly later). For now, let’s proceed with the understanding that a positive link between yesterday’s patent stock and today’s patenting is a pretty robust correlation. But before jumping from a correlation to a conclusion, we need to think a bit harder about what’s really going on here. Why is there this correlation?

(And if you feel like saying “patents don’t measure innovation!” I hear you, but bear with me for a bit)

What’s Going on Under the Hood?

One potential explanation is quite interesting. Suppose:

* Patent stocks measure how much knowledge we have about a technology

* When we have more knowledge, it’s easier to discover new things

* The new knowledge we discover gets added to our existing knowledge

If this is what’s “really” going on, then it explains why we have a positive link between last year’s patent stock and this year’s patenting. Knowledge begets more knowledge! And this is especially true of more recent knowledge, for which we have not already wrung out all the possible implications (hence the higher weight on recent patents).

If we really believe this story, we can even use the estimated statistical models to make neat little forecasts of how technologies will develop. For every year, we can use last year’s patent stock to predict how many new patents will be discovered. We then use that to predict what next year’s patent stock will be. Rinse and repeat.

There’s also a policy implication. If we can temporarily accelerate the accumulation of knowledge in a field, it can pay huge dividends. That’s because the benefits will compound, since they’ll enable more knowledge creation next year, which will enable more knowledge creation in the following year and so on.

Moreover, this model implies path dependence is a powerful force in technology. If one technology gets a minor head start, it might keep it’s lead for a very long time. In fact, if the relationship is strong enough (a 10% increase in the patent stock increases patenting by 10% or more), then all else equal a technology that is a bit behind can never catch up!

But before we get too far ahead of ourselves, we need to strongly consider some other potentially important explanations.

Let’s jettison the conceptual category “patent stock” for a minute. So far all we have really shown is that patenting in the recent past is correlated with patenting today. To think through why that might be the case, we need to think through what kinds of factors determine the number of patents in a given year (for a firm, technology, or country).

Since I’m an economist, I’m going to divide the potential factors into two categories: supply and demand. Supply factors are anything that affects the cost of getting patents (where cost is broadly construed). Demand factors are anything that affects the value of getting a patent. If it patents get cheaper or more valuable, then we should expect inventors to seek more patents. And if the things that made them cheaper or more valuable persist over time, then patenting in the recent past predicts patenting today.

Yes, it’s true that knowledge is one kind of things that makes it cheaper to discover new things and get patents. But many other non-knowledge factors might also be important. On the supply side, that might include things like more scientists/inventors; more physical capital (computers, laboratories, etc); improved access to financing; more patent lawyers; and so on. On the demand side that might include demand from consumers; new regulations favoring certain kinds of technology; or even just oddball idiosyncratic stuff like demand from a new CEO who thinks the firm should be patenting more of its existing inventions.

If we get any of these factors, we’ll get more patenting today and tomorrow (if the factor sticks around), which will deliver the correlation that patenting today predicts patenting tomorrow. But unlike the interesting “knowledge begets knowledge” theory, this doesn’t necessarily have the same policy implications, nor does it necessarily mean we have strong path dependency in technologies. If these other factors are driving the correlation, then if we increase patenting by hiring more scientists, subsidizing R&D, passing some new regulation, or whatever, the boost in patenting will last as long as the policy, but it won’t necessarily have any further knock-on effects.

So I think it’s worth poking at this correlation to see how much of it can be explained by these various factors. And while our evidence base isn’t fantastic, I think a couple lines of evidence suggest the “knowledge begets knowledge” idea is a big part of the story.

Controlling for Supply and Demand

The most straightforward way to parcel out the drivers of this patent stock/patenting correlation is to try and directly measure the plausible factors that might drive it and adjust for them wherever possible.

Let’s start with supply factors other than knowledge itself. For example, a firm that hires a lot of new R&D workers and invests in new labs might be able to crank out a lot of new (patented) inventions. As long as the firm has this elevated set of inputs, patenting might be elevated too, regardless of what’s happening to knowledge.

There certainly is a correlation between patent stocks and what we might call “inputs to invention.” Park and Park (2006)construct patent stocks for 23 different US manufacturing industries over 1976-1996 and compare them to the number of R&D workers in these sectors over the same time period, finding a correlation coefficient of about 0.3. Park and Park don’t have great measures of the non-labor inputs to research (lab equipment, computers, etc), but they do have total R&D spending, which should encompass that as well as the salaries of R&D workers. The correlation coefficient between patent stocks and R&D spending is higher, at 0.5.

So it’s true that patent stocks are partially but not totally explained by other supply side factors. Indeed, precisely one reason many papers use patent stocks is to proxy for these supply-side innovation factors, since it’s often easier to observe a firm’s patents than it’s employee headcount and R&D spend. But the correlation is a lot less than 1; we’re not explaining 100% of the variation just by looking at these supply factors.

Another strand of evidence comes from a paper that explicitly tries to adjust for the scientific labor force. Porter and Stern (2000) constructs patent stocks for 17 different OECD countries and finds a strong correlation between last year’s patent stock and this year’s patenting, even after adjusting for each country’s R&D workforce. Even within the same country, comparing two years where the scientific workforce is unchanged, if there is a 10% larger patent stock in one of those two years, they find that’s associated with 11% more patents in the following year.

So we might consider that some evidence that this isn’t entirely about supply side factors. What about demand?

We know from a variety of evidence in the life sciences that biomedical R&D is pretty responsive to changing market demand. When a disease becomes more prevalent or more profitable to treat, private sector companies respond by increasing related research. There’s no reason to think biomedical research is special in this regard, and so we should expect demand side drivers to matter for lots of technologies.

A lot of these papers are specifically concerned with the transition from fossil fuels to clean energy. In that context, the relevant demand-side factors are the price of fossil fuels and government regulations that make it more attractive to invent and use renewable energy. And lots of papers have specifically controlled for these demand side factors, and still find this patent stock / patenting correlation:

* For 11 different energy technologies, Popp (2002) finds past patent stocks are strongly correlated with present patenting after controlling for energy prices

* For renewable energy, Noailly and Smeets (2015) find the same after controlling for energy prices and the size of the energy market

* For clean and dirty auto innovation, Aghion et al. (2016) find past patent stocks are strongly correlated with present patenting after controlling for fuel prices and some proxies of government regulation

* Rozendaal and Vollebergh (2021) find the same using more recent data, fuel prices, and an improved measure of the stringency of government regulation

These measures of demand-side factors are always going to be incomplete and imperfect, but they seem like plausible candidates for the first-order drivers of demand for different kinds of energy-related innovation. So that should tilt us a bit to the side of thinking the correlation between a patent-stock and subsequent patenting isn’t just about unobserved demand.

So where do we end up? In this article, we’re interested in how much the correlation between patent stocks and subsequent patenting is driven by knowledge generating more knowledge. We’ve seen this correlation persists when we try to control for some various confounders, like the number of scientists or changing demand for different kinds of innovation. But it’s always possible we’ve missed something. Moreover, it’s a bit frustrating that we have one set of paper that looked at supply and another set that looked at demand. It might be that if we could control for both supply and demand at once, the patent stock / patenting correlation would be much smaller, or even disappear. So to close, let’s turn to a few papers that try to look more directly for positive evidence on knowledge itself.

Measuring Knowledge, not just Patents

To begin, there are a few papers that have attempted to show patent stocks really capture something like “technological know-how”. As I noted at the beginning, a potential critique up to now might have been, “patents don’t measure innovation!” For sure lots of patented stuff is junk and lots of brilliant stuff is not patented. But I think there’s signal in the noise.

For example, Park and Park (2006) and Porter and Stern (2000) both look to see how well patent stocks predict not just patenting but also “total factor productivity.” Total factor productivity - hereafter TFP - is a common (albeit highly imperfect) measure of technological capability. The basic idea behind it is if you can squeeze more economic outputs out of the same number of inputs (for example, capital and labor), then that’s a reflection of better technology. The challenge is you can’t measure TFP directly; instead, you directly measure outputs and inputs and try to predict the outputs with the inputs. Something like the gap in your predictions is your measure of technology. It’s a nice concept because it’s a general purpose measure of technology, but it’s very noisy and can be affected by things other than technological progress.

Nonetheless, it’s one of the few broad measures of technology we have. Park and Park (2006) look at how correlated the growth of TFP in an industry is with its patent stock. The correlation is there, though a bit weak; a 10% increase in the patent stock is associated with a 1.5% increase in the TFP growth rate. And Porter and Stern find similar results at the level of the country; a 10% increase in the national patent stock is associated with a 0.5% increase in the national TFP growth rate. To the extent patent stocks are correlated with a completely different measure of technological know-how (that is computed with no reference to patents), that’s some evidence patent stocks are picking up information about technological know-how. That know-how apparently affects economic performance, and so it’s not such a leap to imagine it also affects subsequent technological progress. To the extent the link is weak though, that’s either evidence that the signal about knowledge from patent stocks is pretty small, or that TFP is a crappy measure of technology, or - most likely, in my opinion - a bit of both.

Finally, a few papers try to go beyond merely counting patents, to get at some proxies for the knowledge associated with patented inventions. To the extent these papers actually do a better job of measuring knowledge (which is debatable), they tend to show more knowledge is associated with more patenting.

One of these is the previously mentioned Popp (2002). Popp attempts to use patent citations to measure the value of the knowledge created by patents. The basic idea is that if patents in a particular year have an unusually high probability of being cited, that suggests the patents in that year are particularly valuable as sources of knowledge. And if your knowledge stock contains lots of these high-knowledge patents, then it should generate more patents than another patent stock with the same number of patents but fewer of the high-knowledge type.

This is a tricky exercise; patent citations are a pretty noisy indicator of knowledge flows, though I tend to think they have some signal. Moreover, you need to be careful not to get the causality backwards, so that a surge of patents in later years causes more citations to patents in earlier years, creating this spurious correlation between the number of highly cited patents in a year and subsequent patent activity. Popp avoids this by using the probability a patent gets cited, which takes into account the number of patents that might cite it.

In any event, Popp shows yesterday’s value of this more sophisticated knowledge stock is more tightly correlated with today’s patenting than the standard knowledge stocks that make no such allowances.

Another paper that tries to measure the knowledge in patent stocks is one of my own, Clancy (2017). This paper is based on the idea that innovation is a combinatorial process - new ideas are all about finding useful combinations of older pre-existing ideas. The underlying intuition is that it’s not the number of patents in a technology field that matters, but the number of useful combinations of ideas that the field knows about.

The challenge is finding sensible proxies for these concepts, and then to test them correctly. Fortunately, the US patent and Trademark Office (USPTO) has developed a highly granular system for classifying the technologies described in patents. When I was doing the research, the USPTO had 450 different major technological “classes” and about 15,000 “mainline subclasses” which provided a more detailed description of technology. Importantly, most patents are assigned more than one of these mainline subclasses, which is one indication that two distinct “ideas” (each of which is described by one of these subclasses) have been successfully combined in the single patented invention.

Clancy (2017) defines two different measures of technological know-how for each of the 450 broad technology “classes.” The first of these is just a count of the number of patents assigned to the class, which is basically the same as the patent stocks we’ve been discussing so far. The second is a measure based on the number of pairs of mainline subclasses that are assigned to patents in that class, adjusted in a way to try and capture inventors’ depth of experience with combining the mainline subclasses. The paper then uses both of these measures to predict subsequent patenting.

Here’s an example to illustrate the basic idea. Suppose we have two different technologies: solar energy and fusion energy. Suppose they each belong to a different technology class. Pretend each has roughly the same number of patents associated with it, so that they have similar patent stocks. But suppose solar energy patents combine dozens of different technological categories and the fusion patents only combine a few different categories again and again. That might imply it would be easier to come up with new solar energy patents than fusion patents, because we only know how to do a few different kinds of things with fusion, and those things have been largely explored, while with solar we have dozens of avenues left to explore.

And that’s basically what Clancy (2017) finds. Within a given technological class, a 10% jump in this measure of “knowledge about how to combine pairs of technology” is associated with a subsequent 9% increase in patenting in that field. But now a 10% jump in the number of patents - holding fixed our measure of how much the field knows about how to combine pairs of technologies - is associated with a 30% drop in patenting. The notion here is that new patents can be good for future innovation (if they show how to do new things like combine rarely paired technologies) or bad for future innovation (if they merely exhaust possibilities that are already known). Perhaps this kind of argument can help explain why we occasionally find a higher patent stock implies less patenting, such as in Lazkano, Nøstbakken, and Pelli (2017)?

(As far as I know, Popp (2002) and Clancy (2017) are some of the only papers that adopt the patent stock approach, but experiment with developing improved measures of the “knowledge” in them. I suspect you could now do a lot more now to develop measures of knowledge in a field, for example, by using the text in patents to measure the diversity of ideas in a field, or looking at how much the field is citing new scientific research versus old research.)

The Upshot

So, to step back and appraise the situation, we’ve got this correlation that’s pretty general: yesterday’s patenting tends to predict today’s. One interesting hypothesis to explain this stylized fact is that “knowledge begets knowledge;” a policy that pumps up knowledge production today could have a long echo into the future, as new knowledge enables yet more discovery. Indeed, if the strength of this relationship is strong enough, so that a 10% increase in knowledge leads to at least a 10% increase in the production of new knowledge, then you obtain strong path dependency effects. Once you get a technological paradigm started, such as fossil fuel derived energy, innovation is self-propelling and it becomes very hard for rival paradigms to ever catch up (without active assistance).

But we should be careful about leaping to that interpretation. We don’t have anything like a nice experiment or even quasi-experiment here; just messy correlations in observational data. In particular, it’s quite likely part of this correlation is driven by factors that don’t necessarily have this self-propelling character. It might be that the correlation is driven by the fact that fields that do a lot of patenting in the recent past have a lot of scientists working in them, and so we should expect those scientists to generate a lot of patents in the future as well. But it looks like this isn’t the whole story.

At the same time, it might be that the correlation is driven by the fact that technologies that are in high demand yesterday attract a lot of inventive effort yesterday, and so long as that demand remains elevated then they will continue to attract inventive effort today. But again, at least for energy innovation, this patent stock to patenting correlation seem to exist even after taking into account those demand-side factors.

Lastly, when we do try to drill down and measure knowledge more directly, we do find some evidence consistent with this view that the knowledge is a big part of what matters. Patenting is correlated with some other measures we have of technological know-how, and improved measures of knowledge seem to predict future patenting better than unimproved measures.

Despite all that, nature is tricky and it could well be that all the confounders I’ve already mentioned, plus things we’ve overlooked, drives this correlation. But lastly, we have a few additional suggestive lines of evidence that knowledge matters for innovation, which are discussed at length elsewhere on New Things Under the Sun. For example, morescientific knowledge also seems to facilitate innovation, and knowledge spillovers seem quite important for innovation. That suggests we should approach this line of evidence already pretty open to the notion that we’ll find knowledge is an important explanatory factor.

So here is my main take-away from all this. Some part of the correlation between yesterday’s patent stock and today’s patenting is probably driven by knowledge itself, but not all of it. At the same time, we rarely find a 10% increase in yesterday’s patent stock is associated with a greater than 10% increase in today’s patenting, and when we do, it’s usually not much more than 10%. More commonly, we find a 10% increase in yesterday’s patent stock is associated with a positive but less than 10% increase in today’s patenting. Since I think knowledge probably only accounts for part of this correlation, I suspect, on average a 10% increase in knowledge (if we could measure knowledge correctly) leads to a lessthan 10% increase in knowledge discovery.

In other words, knowledge begets more knowledge, but with diminishing returns. Innovation tends to get harder. Policies that pump up knowledge do echo into the future; but they don’t echo forever. And technologies that get a head start will probably keep them for a long time; but in general, not forever.

Thanks for reading! For other related articles by me, follow the links listed at the bottom of the article’s page on New Things Under the Sun (.com).And to keep up with what’s new on the site, of course subscribe!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here.

Suppose we think there should be more research on some topic: asteroid deflection, the efficacy of social distancing, building safe artificial intelligence, etc. How do we get scientists to work more on the topic?

Buy it

One approach is to just pay people to work on the topic. Capitalism!

The trouble is, this kind of approach can be expensive. To estimate just how expensive, Myers (2020) looks at the cost of inducing life scientists to apply for grants they would not normally apply for. His research context is the NIH, the US’ biggest funder of biomedical science. Normally, scientists seek NIH Funding by proposing their own research ideas. But sometimes the NIH wants researchers to work on some kind of specific project, and in those cases it uses a “request for applications” grant. Myers wants to see how big those grants need to be to induce people to change their research topics to fit the NIH’s preferences.

Myers has data on all NIH “request for applications” (RFA) grant applications from 2002 to 2009, as well as the publication history of every applicant. RFA grants are ones where NIH solicits proposals related to a prespecified kind of research, instead of letting investigators propose their own topics (which is the bulk of what NIH does). Myers tries to measure how much of a stretch it is for a scientist to do research related to the RFA by measuring the similarity of the text between the RFA description and the abstract of each scientist’s most similar previously published article (more similar texts contain more of the same uncommon words). When we line up scientists left to right from least to most similar to a given RFA, we can see the probability they apply for the grant is higher the more similar they are (figure below). No surprise there.

Myers can also do the same thing with the size of the award. As shown below, scientists are more likely to apply for grants when the money on offer is larger. Again, no surprise there.

The interesting thing Myers does is combine all this information to estimate a tradeoff. How much do you need to increase the size of the grant in order to get someone with less similarity to apply for the grant at the same rate as someone with higher similarity? In other words, how much does it cost to get someone to change their research focus?

This is a tricky problem for a couple reasons. First, you have to think about where these RFAs come from in the first place. For example, if some new disease attracts a lot of attention from both NIH administrators and scientists, maybe the scientists would have been eager to work on the topic anyway. That would overstate the willingness of scientists to change their research for grant funding, since they might not be willing to change absent this new and interesting disease. Another important nuance is that bigger funds attract more applicants, which lowers the probability any one of them wins. That would tend to understate the willingness of scientists to change their research for more funding. For instance, if the value of a grant increases ten-fold, but the number of applicants increases five-fold, then the effective increase in the expected value of the grant has only doubled (I win only a fifth as often, but when I do I get ten times as much). Myers provides some evidence that the first concern is not really an issue and explicitly models the second one.

The upshot of all this work is that it’s quite expensive to get researchers to change their research focus. In general, Myers estimates getting one more scientist to apply (i.e., getting one whose research is typically more dissimilar than any of the current applicants, but more similar than those who didn’t apply) requires increasing the size of the grant by 40% or nearly half a million dollars over the life of a grant!

Sell it

Given that price tag, maybe a better approach is to try and sell scientists on the importance of the topic you think is understudied. Academic scientists do have a lot of discretion in what they choose to study; convince them to use it on the topic you think is important!

The article “Gender and what gets researched” looked at some evidence that personal views on what’s important do affect what scientists choose to research: women are a bit more likely to do female-centric research then men, and men who are exposed to more women (when their schools go coed) are more likely to do gender-related research. But we also have a bit of evidence from other domains that scientists do shift priorities to work on what they think is important.

Perhaps the cleanest evidence comes from Hill et al. (2021), which looks at how scientists responded to the covid-19 pandemic. In March 2020, it became clear to practically everyone in the world that more information on covid-19 and related topics was the most important thing in the world for scientists to work on. The scientific community responded: by May 2020 and through the rest of the year, about 1 in every 20-25 papers published was related to covid-19. And I don’t mean 1 in every 20-25 biomedical papers - I mean all papers!

This was a stunning shift by the standards of academia. For comparison, consider Packalen and Bhattacharya (2011), which looks at how biomedical research changed over the second-half of the twentieth century. Packalen and Bhattacharya classify 16 million biomedical publications, all the way back to 1950 and look at the gradual changes in disease burden that arise due to the aging of the US population and the growing obesity crisis. As diseases associated with being older and more obese became more prevalent in the USA, surely it was clear that those diseases were more important to research. Did the scientific establishment respond by doing more research related to those diseases?

Sort of. As diseases related to the aging population become more common, the number of articles related to those diseases does increase. But the effect is a bit fragile - it disappears under some statistical models and reappears in others. Meanwhile, there seems to be no discernible link between the rise of obesity and research related to diseases more prevalent in a heavier population.

Further emphasizing the extraordinary pivot into covid-related research, most of this pivot preceded changes in grant funding. The NIH did shift to issuing grants related to covid, but with a considerable lag, leaving most scientists to do their work without grant support. As illustrated below, the bulk of covid related grants arrived in September, months after the peak of covid publications (the NSF seems to have moved faster).

On the one hand, I think these studies do illustrate the common-sense idea that if you can change scientists beliefs about what research questions are important, then you can change the kind of research that gets done. But on the other hand, the weak results in Packalen and Bhattacharya (2011) are a bit concerning. Why isn’t there a stronger response to changing research needs, outside of global catastrophes?

It’s hard

I would point to two challenges to swift responses in science; these are also likely reasons why Myers (2020) finds it so expensive to induce scientists to apply for grants they would not normally apply for. Both reasons stem from the fact that a scientific contribution isn’t worth much unless you can convince other scientists it is, in fact, a contribution.

The first challenge with convincing scientists to work on a new topic is there need to be other scientists around who care about the topic. This is related to the model presented in Akerlof and Michaillat (2018). Akerlof and Michaillat present a model where scientists’ work is evaluated by peers who are biased towards their own research paradigms. They show that if favorable evaluations are necessary to stay in your career (and transmit your paradigm to a new generation), then new paradigms can only survive when the number of adherents passes a critical threshold. Intuitively, even if you would like to study some specific obesity-related disease because you think it’s important, if you believe few other scientists agree, then you might choose not to study it, since it will be such a slog getting recognition. There’s a coordination challenge - without enough scholars working in a field, scholars might not want to work in a field. (This paper is also discussed in more detail here)

The second challenge is that, even if there is a critical mass of scientists working on the topic, it may be hard for outsiders to make a significant contribution. That might make outsiders reluctant to join a field, and hence slow its growth. We have a few pieces of evidence that this is the case. Hill et al. (2021) quantify research “pivots” by looking at the distribution of journals cited in a scientists career and then measuring the similarity of journals cited in a new article to the journals cited in the scientist’s last three years.

For example, my own research has been in the field of economics of innovation and if I write another paper in that vein, it’s likely to cite broadly the same mix of journals I’ve been citing (e.g., Research Policy, Management Science, and various general economics journals). Hill and coauthors’ measure would classify this as being a minimum pivot of close to 0. I also have written about remote work, and that was a bit of a pivot for me; the work cited a lot of journals in fields I didn’t normally cite up until this point (Journal of Labor Economics, Journal of Computer-Mediated Communication, but also plenty of economics journals). Hill and coauthors’ measure would classify this as an intermediate pivot, greater than 0 but a lot less than 1. But if I were to completely leave economics and write something on the biology of covid-19, I might not cite any journals I’ve ever cited before; that would be measured as a the pivot maximum of 1. By this measure, most covid-related research involved a much bigger pivot than average.

Hill and coauthors then look to see what is the probability a given paper is in the top 5% for citations received in its field and year. The greater the pivot of the paper, the less likely a paper is to be highly cited.

Arts and Fleming (2018) provides some additional evidence on the difficulty of outsiders making major intellectual contributions, but among inventors instead of academics. As a simple measure of inventors entering new fields, they look at patents that are given a technology classification that has never been given to the inventor’s previous patents. As with Hill et al. (2021) they find these patents tend to receive fewer citations. One thing I quite like about this paper though is that they also go beyond citations and look at alternative measures of the value of a patent, such as whether the inventor or assignee chooses to pay the renewal fees to keep the patent active. By this measure too, patents from outsiders are less valuable. (This paper is discussed a bit more here)

Science is pretty competitive and if it’s harder to do valuable work in a new field, then it may well be in any given scientist’s best interest to stay in their lane. But that can make it hard for the system overall to respond to changing research needs.

It’s Not Impossible

While the above barriers make it harder for new scientific fields to emerge, clearly it does happen. To close, let’s look at two factors that might make change easier.

First, if you can initially solve the coordination challenge of getting a critical mass of scholars to focus attention on a new topic, then you can create a new equilibrium where pivoting into that field can be in any given individual’s self-interest. Covid-19 provides just such an example of a new equilibrium. It is too early to learn much from citations to covid-19 research, but as an early indicator Hill and coauthors look at the journals where covid-19 research gets published. They assign each journal a score based on the historical probability an article published there becomes a top-5% most cited publication for its field and year. The figure below compares the size of the pivot for covid-related and non-covid related research to the historical hit rate of the journal that publishes it.

In blue (for non-covid research) and red (for covid-related research), we can see the same pivot penalty as we observed before; article that involve a bigger research pivot are less likely to place in journals that tend to get highly cited. But the gap between these lines is also informative - because there seems to be a new consensus that covid-related research is so important, covid-related research tended to publish much better. Indeed, a pivot to covid that’s measured at around 0.7 appears to have about the same likelihood of becoming highly cited as a non-covid paper that executes a minot pivot measured at around 0.3. All else equal it’s better to make a smaller pivot to covid-related research, but large pivots are not nearly as unattractive as they previously were.

Second, if career incentives constrain scientists to stay in their lane and avoid branching out into new topics, then changing those incentives might also help. Evidence here is more mixed though.

On the one hand, we have a well-known paper by Azoulay, Graff-Zivin, and Manso (2011), which compared the recipients of Howard Hughes Medical Institute (HHMI) support to a control sample of early career prize winners in the life sciences. A key difference between these groups is that HHMI winners are relatively more insulated from the typical academic grant system: they receive at least 5 years of support that is not tied to any specific project, they have a relatively lax initial review, and are typically renewed at least once for 5 more years (and if not renewed, they get two years of funding to help keep the lab open while they search for new grants). In contrast, someone getting support on an NIH grant would often receive just three years of funding, with a comparatively low probability of being renewed.

Azoulay and coauthors use the MeSH lexicon - a standardized set of keywords assigned to biomedical papers by experts - to show HHMI investigators are more likely to explore new research topics than the control group. The MeSH words assigned to their papers tend to be of a more recent vintage, and there tends to be less overlap between the MeSH words assigned to their post-HHMI support papers and their pre-HHMI support papers when compared to the control group. That suggests being (somewhat) freed from the need to secure grant support via normal channels gives scientists the autonomy to explore new fields.

On the other hand though, we have a 2018 paper by Brogaard, Engelberg, and Van Wesep which looked at another set of career incentives that insulates a scientist from the pressure to conform: tenure. Brogaard, Engelberg, and Van Wesep look at 980 economists who at one point belonged to a top 50 economics or finance department between 1996 and 2014, and who were granted tenure by 2004. They track down these economists complete publication record in order to assess how they publish before they get tenure and in the ten years after.

For our purposes, one of their most interesting results is about whether economists use the autonomy of tenure to branch out into new journals or new areas. Unfortunately, Brogaard, Engelberg, and Van Wesep find no evidence that they do. Indeed, in some versions of their statistical tests, economists are slightly less likely to branch out after receiving tenure. And this isn’t just because economists take a breather for a year or two after getting tenure. This effect persists.

This leaves us in a bit of a muddle. For elite scientists, HHMI style support seems to have encouraged them to branch out and try new things. But for economists, the insulation of tenure did not. Is that because tenure is different from HHMI support, or because economists are different from biomedical scientists, or because elite scientists are different from everyone else? We don’t know.

Until then though, I think we can say a few things with confidence. First, the direction of research does respond to money and perceptions of intrinsic value. But it doesn’t appear to be super responsive. That might be because of some incentives scientists face to stay focused: fields may need a critical mass of sympathetic peers before it is individually rational to enter them, and even when a critical mass exists, it is challenging for outsiders to do top work in them, at least initially. Building a new field is probably pretty hard for these reasons; but if you can get the ball rolling, it’s also possible that it can continue going on it’s own momentum.

But more research is needed.

Thanks for reading! If you liked this, you might also enjoy this article about some of the forces that bias science away from novelty and towards conservatism. For other related articles by me, follow the links listed at the bottom of the article’s page on New Things Under the Sun (.com). And to keep up with what’s new on the site, of course subscribe!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here.

How do scientists and inventors decide what to work on? Part of it comes down to what they find personally meaningful. And that, in turn, can be informed by your specific life experiences.

We can actually see this in data, if we look in the right places. Every human life is unique, but in this article we’re going to collapse diverse life experiences down into something quite crude: binary gender. This can be a useful exercise because, as we’ll see in a minute, there are a variety of non-controversial strands of evidence that women are slightly more likely to work on some things than men, and that this may well reflect different perceptions on what questions are meaningful (likely derived from lived experience). But it’s also an exercise fraught with risks, because there may well be a variety of other factors that lead men and women to work on different topics.

For example, it turns out that the share of scientists who are women differs a lot across fields. Over 1990-2011, West et al. (2013) find that 41% of published authors in sociology are women, compared to just 14% of authors in economics. Should we read this as evidence that different life experiences lead men and women to have different interests in sociology, relative to economics? I don’t think so, as there are many other possible barriers to entry in one field relative to another: discrimination or bias, culture, access to networks, and so on.

But cognizant of these risks, we can still look at the choices people make within a specific, narrow, field. That won’t completely rule out the possibility that the choices of research question is constrained by bias and discrimination (even within a specific subfield), but it’s an informative place to start.

Gender and the Direction of Technological Innovation

Koning, Samila, and Ferguson (2021) analyze a set of ~400,000 US biomedical patents from 1976-2010. For each patent, they classify the gender of the inventors based on their names (they can match 98% of inventors with 95% confidence; note they have a binary classification of gender, which they point out is a not appropriate in all cases). They next run a sample of the text of these biomedical patents through an algorithm designed to classify academic articles into different biomedical categories. This algorithm classifies about 13% of these biomedical patents as being related to “female organs, diseases, physiologic processes, genetics, etc.” and another 13% as being related to male analogues. They also cross-check these classifications with a variety of alternative metrics: are patents classified as having a female focus more likely to be based on all-female clinical trials? Are they more likely to be related to diseases that have much higher incidence in women than men?

They find that patents where the majority of the inventors are men are more likely to patent male-focused patents and patents where the majority of the inventors are women are more likely to patent female-focused patents.

Einiö, Feng, and Jaravel (2019) find something similar using a variety of other datasets.

Via Nielsen data, they have information on a sample of households that purchase a wide range of specific consumer products, as well the barcode associated with each product. They can link these barcodes back to the manufacturing firm, which they can then link to patents owned by these firms. Finally, they can use the same strategy as Koning, Samila, and Ferguson to classify the inventors on these patents as men or women. This lets them draw a connection between the gender of inventors and the gender of the people purchasing products manufactured by the companies that own these patents. They find, indeed, that products that are disproportionately purchased by female households also have a higher share of female inventors.

Stepping away from patents, they also use data from Crunchbase to identify startups with female founders, and use the same Nielsen data to assess the gender of purchasers of these startups’ products. They find that startups with a female founder are also more likely to sell products with a disproportionately female consumer base.

So along three different axis, we see that inventors who are women (or at least, have names traditionally given to women) are more likely to develop new products and new technologies that appeal to women, as compared to inventors who are probably men. We can see a similar effect when we look at research papers.

Gender and the Direction of Science

Koning, Samila, and Ferguson (2021) also perform a similar exercise on biomedical science. They extract data on about 2 million biomedical research articles published between 2002 and 2020 and once again classify the authors as male or female based on their names. They focus this analysis on original research published in journals whose articles tend to be highly cited by patents (because this paper’s main focus was on patents).

There’s no need to process these articles through a machine-learning algorithm to see if they focus on female or male related topics (i.e., pertain to female/male organs, diseases, physiologic processes, genetics, etc.), since the articles had already been classified into biomedical categories as part of the publication process. Again, they find articles with more women as authors are more likely to focus on female topics. The following figure is how much more likely a given article is to be female-focused, estimated with two different statistical models.

(If interested - the black estimate comes from a basic regression, controlling for journal-year and team size-year commonalities, the gray one from a model where teams are matched to all-male control teams that are observationally similar).

Nielsen et al. (2017) looks at the influence of gender on biomedical research in a different way. Rather than seeing if papers are classified as pertaining to male or female biomedical categories, Nielsen and coauthors look to see if the gender of the author affects the probability papers incorporate gender and sex analysis, defined by them as “scientific approaches that are aimed at understanding how social and behavioral differences between women, men and gender-diverse people (gender analysis) and biological differences between female and male research subjects (sex analysis) relate to health outcomes.” The GenderMed dataset identifies approximately 5,000 papers, published between 2008 and 2015, incorporating such an analysis. Over the same period, about 1.5 million papers were published on diseases for which such an analysis can be relevant, so a gender and sex analysis is a pretty rare event (3 in 1,000 papers). Nonetheless, Nielsen and coauthors find papers with more women coauthors were more likely to include such an analysis, even within a specific disease category and country.

First and Second Order Effects of Representation

These differences are never huge, but they are persistent across quite a range of evidence. It seems to me pretty likely this reflects either a relative lack of awareness or a relative lack of empathy by male scientists/inventors about the importance of these issues. That, in turn, supplies one reason why representation in science is important. If we want research to broadly reflect the priorities of society at large, and if a part of society is not represented in the research space, the above evidence suggests we’re not going to get the kind of research we want.

That’s a first order effect: if you want research related to a specific group, the chances of getting it may be higher if that group is able to participate in the research process. But there is also a second order effect. We also have some evidence that, as representation improves, everyone’s research priorities shift.

For example, let’s look again at the probability majority male inventor teams works on a female-focused biomedical patent compared to a male-focused one (left figure below). The gap between the two has been closing. Could that be because the number of female inventors has been rising (right figure below), which is increasing male awareness of female-focused research?

Nielsen et al. (2017) also look beyond the impact of female coauthors, to the broader effect of more women in a particular subfield. They find that when there are more women studying a particular disease, this also increases the probability that studies include a gender and sex analysis, independent of the composition of the team of coauthors on any individual study. In other words, a team of men is more likely to include a gender and sex analysis if they are working on a disease where more of the scientists studying the same disease are women.

That could be because as women attain positions of influence - serving as peer reviewers, grant reviewers, PhD supervisors, etc. - the rest of the field becomes more responsive to their concerns. As discussed in “Conservatism in Science” we do have evidence that scientists constrain the feasible choice of research topics for their peers through a variety of channels like this.

But it could also be that people’s awareness and empathy can change when they are around people different from themselves.

An exposure experiment?

It’s tough to experimentally vary “awareness” and “empathy” but a 2021 dissertation by Truffa and Wong look at a situation that has parallels to this. Between 1960 and 1990 76 all-male US universities went coed and began to admit women as undergraduates. Truffa and Wong look to see what happens to the research of these universities before and after they make the switch.

They identify all the academic papers (in the Microsoft Academic Graph) with authors affiliated with these universities, and then they classify papers as being related to gender if the title or abstract includes various keywords (for example “lady”, “female”, “misogyny”, “mothers”, “sexism”). Note that this approach covers all fields, not just the life sciences. A manual inspection of 100 random papers identified as being gender-related by this keyword approach finds the method works pretty well, with only about 8% of the papers identified in this way not really related to gender. They also get similarly encouraging results when they benchmark this against other techniques (for example, by training a machine-learning algorithm on gender studies papers and papers in gender-related journals; or focusing just on biological papers and comparing the results to the biomedical classification systems discussed earlier).

When these universities began to admit women, they also began to produce more research related to gender, as illustrated in the following figure (with the year of going coed centered at zero). The effect is small in absolute terms, but large relative to the pre-existing number of papers related to gender (just 2% in 1960 at these universities).

Why does this happen?

Well, it could have nothing to do with the changing composition of the undergraduates. Maybe these institutions had already decided to take gender more seriously, and that was reflected simultaneously in research and the choice of which students to admit. But Truffa and Wong argue the actual reason for the transition from all-male undergraduates to mixed was not so high-minded. Instead, all-male schools found they were having a harder time attracting “top boys”, who increasingly preferred to attend coed universities. So, to continue attracting “top boys” these institutions (grudgingly?) began to accept undergraduate women too. In other words, the decision to go coed does not seem to have been driven by faculty itching to take gender more seriously in their research.

So why did research begin to take gender more seriously after the schools went coed? It could be that going coed was accompanied by increased hiring of women faculty. As we’ve seen above, it shouldn’t be that surprising that institutions that hire more women will end up getting more women-centered research. Truffa and Wong do find that the share of new assistant professors who were women rose from about 13% to 17% after schools went coed. But the effect goes well beyond these new hires.

Instead, a large part of this effect seems to be driven by pre-existing faculty revising their research preferences. As the figure below shows, even incumbent male professors were more likely to do gender related research after the undergraduate body included women.

These effects were bigger at colleges that admitted more women after the change. Truffa and Wong also present some evidence about a few very concrete channels through which these changes might have occurred. For example, the number of gender-related classes increased significantly after colleges went coed, and in preparing and teaching these classes, faculty might have begun to engage more with these ideas. Second, undergraduates may sometimes participate in the research process. For example, in psychology research of this era, it was common to perform experiments with undergraduate volunteers, and a coed pool of volunteers would have made it easier to do experiments that examined gender differences. Truffa and Wong show, in psychology, the increase in gender-related papers does indeed stem from experimental papers.

The Importance of Representation

The takeaway then, is that representation matters. When a group previously excluded from research enters, it may carry with it research priorities that are better aligned with the needs of the excluded group. Fortunately, as those ideas enter the bloodstream of a research community, it looks like the priorities do get taken up more widely.

But we remain a long way from gender parity in most of research. Holman, Stuart-Fox, and Hauser (2018) examine the rate at which the gender gap in science is closing. Across 115 disciplines that publish to the PubMed or arXiv paper repositories, 87 had fewer than 45% women authors in 2016. In almost every one of these fields the share of women was increasing over time, but at the current rate in most cases it would be well over a decade for authors to come within 45%, sometimes far longer (and longer still for more senior positions). Lastly, there is no reason to believe similar dynamics don’t play out with other underrepresented groups.

Thanks for reading! If you liked this, you might also enjoy this article about some of the forces that bias science away from novelty and towards conservatism. For other related articles by me, follow the links listed at the bottom of the article’s page on New Things Under the Sun (.com). And to keep up with what’s new on the site, of course subscribe!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mattsclancy.substack.com