Does Peer Review Matter?
The existence vel non of supportive peer-reviewed literature is often the focal point of Daubert disputes. Does peer review make any difference in the quality of the scholarly product? Daniel Enger weighs in at Slate. (Thanks to our Colorado correspondent, Jim England, for the pointer.)
1 Comments:
For readers' convenience I have reproduced below a comment I posted on the Engber article.
Daniel Engber makes a number of good points about peer review, specifically raising doubts about its effectiveness in discerning quality. But he would have been well-advised to have sought out peer review of his own article.
Three egregious errors stand out. First, Engber states that peer review is the "gold standard" for judging the quality of science. This is at best a gross exaggeration, and for non-scientist readers it is highly misleading. Peer review is a tool editors use to allocate scarce pages to competing scholars. Many articles that have no chance of surviving peer review for publication in "Science" have no difficulty passing peer review and being published elsewhere. All peer review is not equal; all authors do not face the same intensity of peer review; and all journals that practice peer review do not have the same standard of quality.
Second, Engber completely misinterpreted the text of the federal government's new policy encouraging peer review as a tool for improving the quality of influential information disseminated by federal agencies. His original text stated as follows, to which a correction was subsequently added:
"[T]he federal government recently proposed using peer review 'to improve the quality, objectivity, utility, and integrity of information' that it gives to the public for things like FDA drug requirements and HHS dietary guidelines. Ironically, most scientists hated the idea when it was first presented; the original version of the proposal stated that only researchers from private industry, who don't receive federal funding, would get to participate. (That provision has since been eliminated.*)"
" * Correction, April 8, 2005: The original version of this column missated a provision of the federal government's proposal on peer review. The revised version of the proposal would allow federally funded scientists to participate."
While it is certainly true that many scientists "hated the idea when it was first proposed," most of those who expressed opposition to the policy in public comments complained about provisions that did not exist. In particular, they thought that the policy document would have applied to them. It did not, and the text was crystal clear on that point. That so many scientists raised their voices in opposition to a phantom provision suggests that they were both misinformed by others with ulterior motives and, more disturbingly, they jumped to conclusions without bothering to read the text for themselves. Which was very non-scientific of them.
Third, Engber's original characterization of the domain of eligible peer reviewers of federal agency information products, and the larger domain implied by the correction, are both dead wrong. The original text did not state or imply that "only researchers from private industry, who don't receive federal funding, would get to participate". Rather, the text expanded the array of potential conflicts of interest that needed to be accounted for to include scientists with extensive professional and/or financial entanglements with the same federal agency whose work they would be asked to review. Previously, only financial relationships with industry were considered conflicts, even when these relationships were minimal.
The correction did not correct this error; it compounded it by adding a new one. The proposed language sought comment on the conditions under which federal scientists were sufficiently independent of federal agencies. It did not prohibit them from serving. The final text of the policy retained the capacity of federal scientists to serve as peer reviewers. What the final text did do is abandon the effort to prevent scientists financially dependent on a funding from a specific federal agency from serving as "independent" peer reviewers of the same agency's work products.
Errors like this are avoidable, and one way to avoid them is by securing competent and rigorous peer review. Daniel Engber's article, which raised a number of good questions about the value of peer review, would have been much better if it had been subjected to precisely this kind of oversight.
Post a Comment
<< Home