By Rob Jenkins at Brownstone dot org.

[This article was coauthored by Michael R. Jenkins, an Assistant Professor of Marketing at Mississippi State University.]

Within academia, there seems to be a growing consensus that the peer-review system—once the backbone of academic scholarship—is broken. But is it irreparably so? Perhaps. At the very least, the breakdown of its current form is worth exploring. However, rather than abandoning the entire endeavor, we believe we have a novel solution. First, though, let us examine where the system went wrong.

In the Middle Ages, most scientific research was self-published, as scholars shared their findings among themselves. But, as the profession grew, that became impractical, and the scientific journal was born as a way of disseminating information. A scholar would have an idea, investigate, summarize his conclusions, and submit the resulting manuscript to a journal. There, the editor or editors would consider it and decide whether to publish the work as-is, request revisions, or reject it altogether. Over time, as the number of scholars continued to proliferate, all of them under increasing pressure to publish, publish, publish—in order to be hired, earn tenure, and qualify for grants—the task of journal editors became overwhelming. There were just too many submissions to give them all fair consideration.

And so they came up with the idea of farming out their evaluation of submissions to teams of unpaid reviewers, other scholars in the same field or a related field who were (theoretically, at least) qualified to judge the quality of the research under consideration. This would relieve some of the burden on the editors while also bestowing an additional stamp of legitimacy on the finished product. Whether a given piece of scholarship was worthy of publication was to be determined not just by one or two people but rather by a group of "blind" experts. Thus, the label "peer-reviewed" became the gold standard for scholarly research. A publication in a "peer-reviewed journal" has long been considered essentially unassailable, to the point that politicians and media types seem convinced they can win any argument simply by referencing a piece of "peer-reviewed research."

It was initially a pretty good system, and it worked reasonably well for a long time. But it seems to have now run its course. Tenure requirements have become more quantitative. The internet has decreased barriers to submission, encouraging more scholars to submit more articles to more journals. The number of submissions from Asian, African, and Middle Eastern universities has exploded. Even with more journals and more reviewers, the system has broken down, as all large, complex systems eventually do. We know this to be the case because of a problem first identified 20 years ago by Stanford scientist John Ioannidis, which has since come to be known as the "replication crisis."

One of the hallmarks of good science is that an experiment can be replicated—that is, another researcher using the same methodology will achieve the same result, meaning the findings are both valid and consistent. But what Ioannidis argued in his seminal 2005 article "Why Most Published Research Findings Are False" (updated in 2022) was that, well, most published research findings are flawed. The experiments can't be replicated, casting their validity into question.

Other scholars have since taken issue with Ioannidis's thesis, especially his use of the word "most." Social scientists, in particular, argue that experiments involving human subjects often can't be replicated precisely because people are themselves inconsistent. Nevertheless, scholars generally agree that the replication crisis is real, if not quite as widespread as Ioannidis suggests.

What does this have to do with peer review? Obviously, if the system were functioning as intended, with teams of bona fide experts checking and double-checking each other's work, we might expect that very fe...