Session A14: Peer Review: History and Issues

The Curious Origins of the Scientific Referee

Where did the figure of the scientific “referee” come from, and why? Beginning in the 1820s, in the midst of reform movements in English science and in English politics, a number of British scientific societies established formal systems for reading and reporting on manuscripts by special readers, who soon became known as referees. In this personage came to be juxtaposed elements of the legal expert, the trustworthy gentleman, the state bureaucrat, and even the anonymous literary reviewer. But when the scientific referee appeared on the scene of British science, it was not certain who he was, or what he was supposed to be for. The initial impetus for the Royal Society of London’s referee system had been as much about generating publicity as making anonymous judgments. But gradually the referee report became shrouded in secrecy, and the referee was increasingly viewed as an agent for conferring rewards on authors in exchange for contributions to knowledge. The conception of the referee as primarily a gatekeeper of knowledge became dominant only later in the century, as scientific practitioners came to perceive the existence of a special set of texts called the scientific literature that could be marked off from other periodicals and which required protecting. The notion that some form of peer review -- as it came to be known during the Cold War -- was a sine qua non of legitimate scientific journals was an even later development.   

In Referees We Trust? Controversies over Grant Peer Review in the Late Twentieth Century

While many accounts of external refereeing assume that it has been a consistent part of science since the seventeenth century, the practice developed far more slowly and haphazardly than many observers realize, and it was not until after the Second World War that "peer review" became considered an essential part of scientific publishing or grant-making. This talk will explore refereeing procedures at American grant-giving organizations in the twentieth century, focusing especially on the National Science Foundation and the National Institutes of Health. The creators of the NSF and the NIH put refereeing systems in place at their foundation. However, the form and function of these systems differed from modern "peer review" in several important ways. At the NSF the initial purpose of the referee process was to advise the NSF program directors, not to dictate funding decisions. At the NIH, small "study sections" devoted to particular subjects made recommendations to the NIH leadership, which rendered final judgments. However, beginning in the 1960s a series of controversies about NIH and NSF grants placed refereeing procedures at these organizations under more intense scrutiny. These debates culminated in six days of Special Oversight Hearings into the NSF's peer review process in the summer of 1975. Following the hearings, both the NSF and NIH reformed their review processes to place more emphasis on referees' opinions about grant proposals, making peer review increasingly responsible for decision-making. These controversies illustrate that refereeing continued to undergo significant changes in form and purpose throughout the twentieth century, and further suggest that both the scientific community and the public placed increased emphasis on the role of the referee during the late twentieth century.   

Where is the trust in the peer review dynamic?

The motto of the Royal Society is ``nullius in verba'', which translates roughly as ``take nobody's word for it'', and this motto is furthermore emblematic of the conduct of science. We want facts and not opinions, verified results and not conjecture. From the time that we first started communicating scientific results, it has been recognized that scientific claims must be verified by someone who is not the maker of those claims, and who furthermore has no stake in the matter, in other words, a claim needs to be evaluated objectively. Peer review as a method of evaluation can be thought of as akin to an experiment, where the review process tests the hypothesis of a submitted paper. Peer review is however also a social process with human actors: authors, referees, and editors. As a process, peer review depends on trust, but in what way does that manifest? There are many agents in peer review: in addition to the human actors, there is also the institution that is the journal, as well as the publisher (e.g. APS) that stands behind the journal. People can also have trust in the very concept of peer review. If we accept as a proposition that publications are witnesses to science in the same way that people who attend scientific demonstrations are witnesses of an experiment, then how much do we trust this witness? A few further questions arise: \begin{itemize} \item If referees (and sometimes authors) are anonymous, what does this do to the mechanisms of trust? \item Is trust only possible between human agents, or can you trust a process or a journal in a similar way to trusting a certain car brand? \item Is an absence of trust the same as distrust? \item Is trust rational, or cognitive, or is it a practice? \end{itemize} In this paper I will attempt to locate the trust and ask how trust is earned, and, conversely, how it can be lost, using peer review as example.