library-phd-in-philosophy

Peer-review: should we get rid of it?

Share
Tweet
Share

There are way more manuscripts than opportunities for respected peer-reviewed publications (Sinhababu 2016). So many good manuscripts might never be properly reviewed (or published). This would be bad. In this post, I’ll mention a few potential solutions. Then I’ll briefly evaluate one: eliminating compulsory peer-review altogether. 

1.  Peer Review Is New

I learned from Kate Norlock that peer-review is a relatively recent thing.†

… the surprisingly short history of what we now think of as peer-review [Times Higher Ed.] … the Google ngram on peer-review: [Google ngram article] …. suggests that academics have only been so fixated on it as the measure of our worth since the 1970s.

2.  The Current Form of Peer Review Isn’t Obviously Optimal

One reason for peer-review might be that it inhibits bias. And there is some evidence that anonymous peer-review reduces bias (Budden et al 2008). However, a review of 17 studies challenges this finding (Bastian 2015; for more discussion see a post by Norlock).††

Perhaps further reviews of the evidence would undermine our other reasons for existing peer-review practices — feel free to point me to such reviews in the comments. Whatever the evidence is, if it does not obviously favor the status quo, then we should investigate our alternatives.

3.  Alternatives To The Current Peer Review

As Kate Norlock (and Aileen Fyfe) mentioned, the current norms of peer-review are relatively new. And as Hilda Bastian explains, the apparent reasons for current peer-review practices are not well supported by evidence. So let’s consider some alternatives to current peer-reviewed publication practices. Starting with the most drastic…

  1. No more journals, publishers, or top-down quality control, ever. That is, we publish stuff whenever we want, wherever we want, however we want (repositories, personal websites, etc.).
  2. Keep journals and other publication schemes, but no more top-down quality control. Journals, publishers, etc. would be for organizational purposes only — e.g., Ethics continues to publish stuff about ethics.
  3. Keep journals and other publication schemes, and keep editors, but no more referees. Journals, publishers, etc. would serve organizational and minimal quality control functions — e.g., editors could still desk reject papers that do not meet basic academic standards, but they would publish everything else.
  4. Keep journals and other publication schemes, keep editors, and keep referees, but don’t base publication decisions on referee reports. Referees still complete reports, but editors do not have access to them until after a decision has been made. Maybe editors could append (signed?) reviewer comments (ratings?) to every published paper (van Rooyen, Delamothe, and Evans 2010).
  5. Open peer review. Keep journals and other publication schemes, keep editors, and keep referees, but make reviewers sign their referee reports (van Rooyen 1999, Walsh et al 2000).
  6. Whatever we do, make publication less of a priority. For instance, we could assign less weight to publications in decisions to hire, promote, fund, invite to [whatever], etc.
  7. Whatever we do, stop attributing prestige to certain journals and publishers. Maybe just stop ranking journals altogether …or at least stop ranking them badly (see “Academic Tech: Custom Reporting & Ranking“)

(There are other possibilities that I have ignored; feel free to add to this list in the comments. )

4.  Should We Eliminate Peer Review?

Our first possibility is tantamount to eliminating compulsory peer-review altogether. We can easily imagine potential pros and cons of this proposal.

Pros

  • The opportunity to prioritize other stuff. Imagine how much time and energy would be freed up by eliminating compulsory review! Academics would not be bogged down by manuscript reviews, editors would not be bogged down by the search for reviewers, etc. Who knows, maybe this time could be better spent on something else.
  • Faster publication cycle. One of the primary reasons for delayed publication is peer-review. It takes time to find reviewers, wait for their reports, and make a decisions based on reviewers’ reports. Without a compulsory review process, this delay is eliminated. Research could be available much faster. This would have benefits for both academics and whoever consumes academic research.

Regarding the first point, it’s not entirely clear whether the value of other stuff is higher than the value of compulsory peer-review. What we need are credible analyses of the costs and benefits of each.

Something similar can be said about the second point: We need to know more about the costs and benefits of both a faster publication cycle and compulsory peer-review. Until we know more, it’s not clear whether the benefit(s) of faster publication would be a net benefit.

Cons

  • Less dialogue. If we eliminate compulsory peer-review, then there will be less dialogue between academics. This might result in — among other things — academics work being less inclusive, interdisciplinary, rigorous, etc.
  • Fewer indicators of quality. Without the stamp of approval from prestigious peer-reviewed journals, book publishers, conferences, etc., it will be harder to separate the seemingly good stuff from the seemingly not so good stuff.†††
  • More errors. With fewer people checking our work, there might be more errors in publications.

The concern about dialogue might not be warranted. There are other mechanisms for dialogue besides peer-review. Indeed, academics engage in other forms of dialogue without being compelled to do so. For example, academics are often invited (not on the basis of peer-review) to present and answer questions about their research. So there would still be opportunities for dialogue without compulsory peer-review. The question is whether the value of compulsory peer-review can be provided by these other forms of dialogue. Further, the claim that this would cause less dialogue is a testable hypothesis. I do not know of good evidence that supports the hypothesis.

The concern about indicators of quality is probably unwarranted. After all, academics can determine quality without compulsory peer-review. For instance, academics can (and do) look to the wisdom of the crowd (of readers). And they can (and do) rely on their own judgments. So it is not clear what academics would lose in the way of quality indicators if compulsory peer-review were eliminated.

I am not sure what to make of the concern about errors. After all,  additional people working on a manuscript can easily result in more errors, not less. What do the data say about the average quantity (and severity) of errors that are prevented by compulsory peer-review?

CONCLUSION

The publication of good research is delayed and even prevented by the resource-intensive demands of compulsory peer-review. One potential fix is the elimination of compulsory peer-review. In order to evaluate this proposal, we would need to determine the costs and benefits of compulsory peer-review. We’d also need to determine the costs and benefits of the alternatives.

 

Notes

 

† Kate Norlock’s original comment is here.

†† Thanks to Kate Norlock for pointing me to this review.

††† For more on prestige in academia, see this wonderful presentation: “Prestige: The first and final hurdle…“

Published by

Nick Byrd

Nick is a cognitive scientist at Florida State University studying reasoning, wellbeing, and willpower. Check out his blog at byrdnick.com/blog

One thought on “Peer-review: should we get rid of it?”

  1. Thanks to people on Reddit for helping me to look further into the literature [Reddit comments]. Here’s a preliminary list of things to check out:

    Bornmann, L., & Mungra, P. (2011). Improving peer review in scholarly journals. European Science Editing, 37(2), 41–43. (http://www.lutz-bornmann.de/icons/bornmannEssay41-43.pdf)

    Peer review in scholarly journals can be improved by masking of both authors’ and reviewers’ identities (double-blind) or by using open-to-public peer review. is essay deals with currently available options for improving peer review and o ers suggestions for enhancing the quality of publications.

    Walker, R., & Rocha da Silva, P. (2015). Emerging trends in peer review: a survey. Frontiers in Neuroscience, 9. http://doi.org/10.3389/fnins.2015.00169

    “Classical peer review” has been subject to intense criticism for slowing down the publication process, bias against specific categories of paper and author, unreliability, inability to detect errors and fraud, unethical practices, and the lack of recognition for unpaid reviewers. This paper surveys innovative forms of peer review that attempt to address these issues. Based on an initial literature review, we construct a sample of 82 channels of scientific communication covering all forms of review identified by the survey, and analyze the review mechanisms used by each channel. We identify two major trends: the rapidly expanding role of preprint servers (e.g., ArXiv) that dispense with traditional peer review altogether, and the growth of “non-selective review,” focusing on papers’ scientific quality rather than their perceived importance and novelty. Other potentially important developments include forms of “open review,” which remove reviewer anonymity, and interactive review, as well as new mechanisms for post-publication review and out-of-channel reader commentary, especially critical commentary targeting high profile papers. One of the strongest findings of the survey is the persistence of major differences between the peer review processes used by different disciplines. None of these differences is likely to disappear in the foreseeable future. The most likely scenario for the coming years is thus continued diversification, in which different review mechanisms serve different author, reader, and publisher needs. Relatively little is known about the impact of these innovations on the problems they address. These are important questions for future quantitative research.

    Bastian, H. (2014). A Stronger Post-Publication Culture Is Needed for Better Science. PLOS Med, 11(12), e1001772. http://doi.org/10.1371/journal.pmed.1001772

    Foley, J. A. (2013). Peer Review, Citation Ratings and Other Fetishes. Springer Science Reviews, 1(1-2), 5–7. http://doi.org/10.1007/s40362-013-0003-x

    Academic success or ability is often assessed through peer review or the seemingly more objective method of citation ratings. However, citation ratings may be more objective in that they offer a more automated or mechanical method of assessing quality, but it does not necessarily follow that these ratings provide an assessment of genuine scientific impact. Likewise, although peer review may provide an effective filtering system, it cannot be assumed that it provides objective critiques. This review discusses the fetishes and flaws of both methods, and suggests that future reviewing methods should involve both quantitative and qualitative methods, tailored to the specific individual subject area.

    Resnik, D. B., & Elmore, S. A. (2015). Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors. Science and Engineering Ethics, 22(1), 169–188. http://doi.org/10.1007/s11948-015-9625-5

    A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.

    Lortie, C. J., Allesina, S., Aarssen, L., Grod, O., & Budden, A. E. (2013). With Great Power Comes Great Responsibility: the Importance of Rejection, Power, and Editors in the Practice of Scientific Publishing. PLOS ONE, 8(12), e85382. http://doi.org/10.1371/journal.pone.0085382

    Peer review is an important element of scientific communication but deserves quantitative examination. We used data from the handling service manuscript Central for ten mid-tier ecology and evolution journals to test whether number of external reviews completed improved citation rates for all accepted manuscripts. Contrary to a previous study examining this issue using resubmission data as a proxy for reviews, we show that citation rates of manuscripts do not correlate with the number of individuals that provided reviews. Importantly, externally-reviewed papers do not outperform editor-only reviewed published papers in terms of visibility within a 5-year citation window. These findings suggest that in many instances editors can be all that is needed to review papers (or at least conduct the critical first review to assess general suitability) if the purpose of peer review is to primarily filter and that journals can consider reducing the number of referees associated with reviewing ecology and evolution papers.

    Robertson, C. T., & Kesselheim, A. S. (2016). Blinding as a Solution to Bias: Strengthening Biomedical Science, Forensic Science, and Law. Academic Press.

    Samir Hachani. (2015). Open Peer Review: Fast Forward for a New Science. In Current Issues in Libraries, Information Science and Related Fields (Vol. 39, pp. 115–141). Emerald Group Publishing Limited. Retrieved from http://www.emeraldinsight.com/doi/abs/10.1108/S0065-283020150000039012

    Peer review has been with humans for a long time. Its effective inception dates back to World War II resulting information overload, which imposed a quantitative and qualitative screening of publications. Peer review was beset by a number of accusations and critics largely from the biases and subjective aspects of the process including the secrecy in which the processes became standard. Advent of the Internet in the early 1990s provided a manner to open peer review up to make it more transparent, less iniquitous, and more objective. This chapter investigates whether this openness led to a more objective manner of judging scientific publications. Three sites are examined: Electronic Transactions on Artificial Intelligence (ETAI), Atmospheric Chemistry and Physics (ACP), and Faculty of 1000 (F1000). These sites practice open peer review wherein reviewers and authors and their reviews and rebuttals are available for all to see. The chapter examines the different steps taken to allow reviewers and authors to interact and how this allows for the entire community to participate. This new prepublication reviewing of papers has to some extent, alleviated the biases that were previously preponderant and, furthermore, seems to give positive results and feedback. Although recent, experiences seem to have elicited scientists’ acceptance because openness allows for a more objective and fair judgment of research and scholarship. Yet, it will undoubtedly lead to new questions which are examined in this chapter.

    Hanel, P. H. P. (2015). Why scientific publications should be anonymous. arXiv:1512.05382 [cs]. Retrieved from http://arxiv.org/abs/1512.05382

    Numerous studies have revealed biases within the scientific communication system and across all scientific fields. For example, already prominent researchers receive disproportional credit compared to their (almost) equally qualified colleagues — because of their prominence. However, none of those studies has offered a solution as to how to decrease the incidence of these biases. In this paper I argue that by publishing anonymously, we can decrease the incidence of inaccurate heuristics in the current scientific communication system. Specific suggestions are made as to how to implement the changes.

Comments are closed.