Talk:Peer review/Archive 1

Plautus satire's irresponsible comments are those of a crackpot.

Plautus satire's comments are idiotic nonsense. I have refereed papers for five mathematics journals, and my published papers have been refereed (and some unpublished ones that I submitted). Obviously history is consulted; obviously proof-checking is not the main point of refereeing; obviously novel ideas are examined in the context of accepted notions. To say that mathematics is not a science but a "language" ignores the fact that hundreds of journals are devoted to publishing new discoveries in mathematics. Obviously in judging publication-worthiness one considers how new discoveries may be relevant to potential future research; one considers esthetics (which for most mathematicians is the main motive for doing mathematics or for learning mathematics). Whence this loony idea that there is no need to consult history? Do you not see sections on how a new discovery fits into the historical development of the subject in many research papers in mathematics?

Plautus satire's comments are those of a crackpot. Michael Hardy 22:02, 30 Apr 2004 (UTC)

---

The strength of peer review is at the very heart of Wikipedia's philosophy.

This is quite new to me that Wikipedia uses something even similar to "peer

review". I always thought that "peer review" is at the very heart of Nupedia fiasco and Wikipedia is better off without it.
Kpjas 12:40 15 Jun 2003 (UTC)

---

The newly-added section on Patrick Michaels needs work. The connection isn't obvious between the 1996 controversy about the global warming article and the topic of peer review. If the connection is simply that this is an example where peer review failed to prevent a 'bad' article getting published, I'd argue that in that case we don't need very so much detail about the case itself. (The detail probably belongs elsewhere, perhaps somewhere in global warming.)
Toby W 15:24, 15 Jan 2004 (UTC)

(William M. Connolley 23:18, 6 Feb 2004 (UTC)) Indeed. Michaels is massively POV on global warming, and without extra detail - the name of the paper as a bare minimum - can't be trusted to report accurately.

(William M. Connolley 22:31, 8 Feb 2004 (UTC)) OK, I've removed the Michaels bit entirely. Why? Well (ignoring the low quality of the info: paper not even identified, for example), first I snipped it down to the essentials plus a link. Then I thought, OK, but if its there I should add the Soon+Baliunas case too. And then I thought: but we don't want to have the global warming wars spill over into this page by proxy. If it beongs anywhere, it belongs in GW controversy, probably.

NPOV of Peer Review Entry

I would like to challenge the neutrality of the peer review entry, particularly with regards to the following passage:

"Some sociologists of science argue that peer review makes the ability to publish susceptible to control by elites."

There are a great many who challenge the wisdom of peer review, as evidenced by the 2003 report by Tom Jefferson et al which cites the utter lack of empirical evidence that peer review contributes to the quality of published papers. Also cited as criticism of adversarial peer review (in lieu of collaboration) is the destructive nature of scientists competing for funding, which is a clear motive for fraud and abuse in any context. - Plautus satire 04:34, 24 Feb 2004 (UTC)

Note that a common policy in some domains is to find experts that are remote from the authors. With the Internet, it's just a matter of a few clicks to send a paper for refereeing in another country (where people usually do not compete for the same funding). Furthermore, there's a ethical requirement for referees to declare possible conflicts of interest to the editors.

Of course, all is not rosy and all, but this should also be taken into account. David.Monniaux 19:20, 29 Apr 2004 (UTC)

(William M. Connolley 09:37, 24 Feb 2004 (UTC)) This doesn't sound like a NPOV challenge, more an incomplete information problem. If you can find, and quote from, the TJ report, why not add it? I don't quite see the problem with funding though: this doesn't seem to have much to do with peer-review of papers for publication.

What I object to is the wording "some sociologists of science". This should be changed to reflect the more general nature of the criticisms of peer review, which do not all come from sociologists, or else criticisms like this should not be here at all. If criticisms are in the body of the entry, they should be above reproach. - Plautus satire 14:24, 24 Feb 2004 (UTC)

Let me just note I don't want more "clutter" (or whatever) added to this entry, I just want that reference to a very specific, tiny group ("sociologists of science" whatever that means) to be changed to something less startlingly tiny and strange. - Plautus satire 14:34, 24 Feb 2004 (UTC)

(William M. Connolley 15:21, 24 Feb 2004 (UTC)) "Some X..." is weaselly anyway. It would be far better replaced by specific people who have said it. If they then turn out to be people other than a few sociologists, then we'll know what to change the text to say. But if you are hoping to find some srt of academic study on the benefits or otherwise or peer review, you're likely to find it authored by sociologists, I would have though.

Well, the only reason I am reluctant to add specific "pro" or "con" references is because I have been chased out of other entries by adding "con" where there was only "pro" or "weak or approximate con" for NPOV. By "weak or approximate con" I mean a situation is set up where there is both "pro" and "con," but the "con" side is sprinkled liberally with insulting language while the "pro" side is virtually beatified. - Plautus satire 15:24, 24 Feb 2004 (UTC)

(William M. Connolley 17:12, 24 Feb 2004 (UTC)) Probably the best thing to do is to try out some of the refs you're thinking of adding, by putting them on this page first. Then we can see if they seem reasonably balanced.

Thanks for the suggestion for ways to proceed on this, I will look the piece over once again and give it my critical analysis briefly here with proposed changes. - Plautus satire 17:20, 24 Feb 2004 (UTC)

Extraneous Link to Requests for Comment on Plautus Satire

Please be aware who you are dealing with: Wikipedia:Requests_for_comment/Plautus_satire

Curps 17:27, 24 Feb 2004 (UTC)

(William M. Connolley 18:32, 24 Feb 2004 (UTC)) Indeed. However, there is nothing particularly unreasonable here.

Yet. Curps

Minor fact correction. This attempt by Curps to derail a peaceful attempt at collaboration seems to me to be a bit unreasonable. - Plautus satire 18:36, 24 Feb 2004 (UTC)

Citation Style

What work needs to be done that justifies the Template:citationstyle tag? I can't find reference to it in the discussion section. If someone could point out what should be done, I'd gladly fix it so the tag can be removed. -Verdatum (talk) 06:51, 13 December 2007 (UTC)

Archive Comment

Ha! This article failed peer review! See box above. So we better got to work on this. --Uncle Ed 22:02, 3 June 2006 (UTC)

Name of article?

That which this article, and apparently the general public, calls "peer review", is known in mathematics and statistics as "refereeing". If not for TV news broadcasts and other media addressed to the hoi polloi, I would never have heard the term "peer review". Persons whom editors of scholarly journals ask to review submitted papers are "referees". It is certainly plausible that in other fields they call it something else, but I wonder if some specificity is in order here: what are the fields in which it is called "peer review"? Michael Hardy 18:13 Feb 20, 2003 (UTC)

My understanding is that the two phrases are used in complementary ways -- a journal or an article may be called either refereed or peer-reviewed; the peer-review process involves one or more referees (that is, the reviewers are always called referees in my experience, but the process is more often called "peer review"). That said, I strongly take issue with the accuracy of this article. It is far from certain that peer-review leads to increased quality of articles; it certainly does promote conformity to general standards. And Wikipedia is most definitely not peer-reviewed in the academic sense; the whole point of Wikipedia is that anyone can be a referee, including people with widely varying levels of education in very diffeent fields. Slrubenstein

Agreed. In literature, as authors go in and out of vogue, many leave in their wake a body of books and peer-reviewed journals by and for fans of the authors. These books and journals are considered not to be authoritative by academics even though they are peer-reviewed because of the non-critical nature of the work. They reflect the qualities appreciated by the author's admirers, but often gloss over important or embarassing information about the author. --Modemx 19:59, 1 Feb 2005 (UTC)

You haven't said what field you're in. I have not heard of "peer review" in mathematics, but I've refereed some papers and had my own refereed. It's called "refereeing", not "peer review". But, if I can believe what I keep hearing, in some fields it's called "peer review". So my question is: which ones? Michael Hardy 23:24 Mar 1, 2003 (UTC)

Computer Science. See any Call for Papers from IEEE Computer society and Association for Computer Machinery.

--- ejrrjs

It is "peer review" in musicology, though "refereed" is also used from time to time. --Myke Cuthbert 23:05, 5 May 2006 (UTC)

Also, what about the anonymity issue? In mathematics, referees or usually anonymous, i.e., papers' authors don't know who the referees are, although they read the referees' reports, but referees generally do know who the authors are. I have heard that in some fields, they don't. Which ones? Maybe biology? Michael Hardy 23:26 Mar 1, 2003 (UTC)

This notion may be misunderstood, and thus may upset people, but I do not feel math is a science, it is more like a language, one that humans discovered and one that is self-consistent and extraordinarily useful while practicing science. Peer review would be useless in mathematics, as the proof is on the page, so to speak. There is no need to consult history (or "accepted" notions, which peer review does compare "new" ideas to as a basis for their validity) when "refereeing" a statistics or mathematics paper. - Plautus satire 14:30, 24 Feb 2004 (UTC)

I suspect that you do not know much about the workings of the mathematical community. First, it's inexact to state that the "proof is on the page". Real-life mathematical proofs are not machine-checkable (excluding fringe projects like Mizar) and generally require considerable expertise to be understood. Very subtle mistakes can arise; the author may have believed that some point should be skipped because it was "easy"; he may have made a definition that was subtly inconsistent; etc... For instance, checking whether Andrew Wiles' proof of Fermat's last theorem is correct is a job for only a few other high level mathematicians.

Plautus satire's comments are nonsense. I have refereed papers for five mathematics journals, and my published papers have been refereed (and some unpublished ones that I submitted). Obviously history is consulted; obviously proof-checking is not the main point of refereeing; obviously novel ideas are examined in the context of accepted notions. To say that mathematics is not a science but a "language" ignores the fact that hundreds of journals are devoted to publishing new discoveries in mathematics. Obviously in judging publication-worthiness one considers how new discoveries may be relevant to potential future research; one considers esthetics (which for most mathematicians is the main motive for doing mathematics or for learning mathematics). Where did you get this loony idea that there is no need to consult history? Do you not see sections on how a new discovery fits into the historical development of the subject in many research papers in mathematics? Michael Hardy 22:33, 1 Feb 2005 (UTC)

Second, there's an issue that many philosopher and sociologists that discuss science neglect: refereeing is not only about gauging whether a paper is scientifically correct, but also of gauging whether it's scientifically interesting. It's possible to write a whole paper of correct mathematics by writing 1=1, 2=2, 3=3... Yet, such a paper would be totally uninteresting, since it would not advance science, by saying evidences (trivial tautologies). One may for instance judge that a paper does not warrant publication in a major journal if it's about some gratuitous mathematical theory. Such judgment is of course somewhat subjective. David.Monniaux 19:20, 29 Apr 2004 (UTC)

A common argument I've had before with my students. My response is that people should actually look in a peer-reviewed journal of mathematics and then try to argue why math isn't a science. Math is "the mother of all sciences." --Modemx 20:43, 1 February 2005 (UTC)

Questionable para from fraud section

(William M. Connolley 08:37, 21 Jun 2004 (UTC)) I've moved this here for discussion:

While fraud obviously has severe negative consequences for the author of a paper, there are generally no adverse consequences to either the editor or the reviewers for recommending publication of a fraudulent paper, as detecting fraud is not a goal of the process, and the editors and reviewers almost never have enough information to detect outright fraud.

Who says that there are no consequences for editors (or reviewers - less sure). Quite likely, the editors get a black mark in their career record - what this text means probably is that nothing about that appears in the newspapers?

Hi William.

Fair point...when Nature withdrew the Schon papers, the editorial said

Nature this week finds itself in the unenviable (and unprecedented) position of formally retracting seven papers (see page 92). All share the same first author, Jan Hendrik Schön (see Nature 419, 417; 2002); in fact, this represents the entire body of work published by Schön in this journal.

which *does* look like severe negative consequences on the Journal, if not the Editor...nevertheless, the referees (being anonymous and voluntary, as much as anything) do escape censure....WMMV.

best

Robinh 10:23, 21 Jun 2004 (UTC)

Hi again. Another gem from Nature Editorial, 3 Oct 2002:

In some media reports, journalists and a few scientists who are unconnected with the Schön investigations have taken the opportunity to make potentially damaging assertions about journals, including Nature: that in order to compete or to publish exciting results, journals will cut corners in peer review, overrule hostile reviewers or select sympathetic ones.

We at Nature unequivocally reject such charges. The publication history and files of these particular papers and the editorial policies and interests of Nature are completely at odds with these assertions. Nature has nothing to gain by the pursuit of glamour at the expense of scientific quality, considering, not least, the criticisms, corrections and retractions we would then habitually be forced to publish. There is more than enough rock-solid and splendid science to publish. Furthermore, it is a strict policy of Nature that our Letters and Articles are selected for their outstanding scientific impact, sometimes also taking into account relevance to public policy issues, but never simply because the results will make headlines.

Robinh 13:16, 22 June 2004 (UTC)

Image

I added a picture from the Center for Scientific Review webpage. However, since I don't have a decent image editing program handy, the orginal that I uploaded is very big and probably should be edited for size. I'm pretty sure it's public domain, since it's from the NIH, a Federal government organization. There are several other images on the same webpage, under "images of peer review" (handy, huh?). Sayeth 19:43, 4 August 2004 (UTC)

Grammar question

"A chief rationale for peer review is that rarely is just one person, or one closely working group, able to spot every mistake or weakness ..." -- is that correct english? It seems to me that the "is" and "able" should be together, but english is not my native language. Could somebody enlighten me? -- stw (Talk) 09:52, 15 Aug 2004 (UTC)

I think this phenomenon is referred to as inversion (grammar). Inversion is the normal word order in questions (Must they feed the animals?) and can also be used after a "negative", restrictive adverb or phrase (Under no circumstances must they feed the animals. Never before have I heard such nonsense. Rarely do they eat meat. etc.)

However, you can altogether avoid this construction by not putting the adverb at the beginning of the sentence. -- Any good grammar will have more details. All the best, <KF> 10:44, 15 August 2004 (UTC)

Peer Review used to suppress opposing views

Perhaps we should mention that some scholars have charged that in environmental science, billions of dollars per year are subsidizing one viewpoint, i.e., the theory of anthropogenic global warming -- and that this financial pressure may be sufficient to corrupt the peer review process. Thus, perfectly valid results may then be dismissed from consideration on the grounds that they did not pass peer review. However, failing peer review is not proof that a paper has errors; only that the referees chosen to review it, diasagreed with it. This is science by voting, which is a self-contradiction. The only reliable criterion for the worthiness of an idea is whether others who try earnestly to verify it are able to do so. Science is replete with examples of important and valuable new ideas which were suppressed by the establishment (germ theory of disease, for example), and we should not exalt peer review as being immune to this type of abuse. --Uncle Ed 14:59, 15 Aug 2004 (UTC)

Furthermore, "Extraordinary claims require extraordinary evidence," said Carl Sagan[1]. If someone comes out with evidence to support an unpopular hypothesis (for example, "life exists on Mars" in 1975), they have to prove it beyond refute before it can even be considered seriously. -- ke4roh 16:04, Aug 15, 2004 (UTC)

Nonsense, if I submit the idea that it is possible to get to the moon by jumping on a pogo stick will the only criterion for "worthiness" be if several other scientists "try earnestly to verify it"? No, the worthlessness of some ideas are self-evident. The scientific process can be far more easily corrupted by, for example, the deep pockets of Washington think tanks. --Deglr6328 18:17, 15 Aug 2004 (UTC)

For an idea to be considered scientifically valid it also must be repeatable. For example, Einstein supported his general theory of relativity with a simply replicated experiment that proved the existence of quantum states of energy. "The establishment" initially rejected his findings because it was from someone who was "just a patent clerk." It was the repeatability of his experiment that helped put Einstein's ideas in the firmament of scientifically valid ideas. Contrast that with the researchers who railed against "the establishment" when their claims of being able to experimentally mass produce Fullerines (a.k.a. "Bucky Balls") were rejected because lack of repeatability. When someone actually found the process to mass produce Fullerines, some of those researchers were found out to simply be liars with economic rather than scientific motivations. Just because someone presents an arguably valid alternate theory about a matter of scientific controversy doesn't mean that it should be treated with equal weight to established theories because of "fairness." Fairness to the truth demands a body of repeatable results that gains the adherence of the scientific community and the standards should be high because the costs of bad science are high. --Modemx 20:28, 1 Feb 2005 (UTC)

Repeatiblity only applies to issues that can be repeated. It cannot apply to scientific theories about historical events or origin theories. e.g., we cannot reproduce the Big Bang, but we can make theories to describe and model the data.

Yes, of course, what happens if you present a new theory that could be tested by experiments but no one want to test it ? , the example is this let's suppose we have a very interesting theory about the creation of the Moon however to test it you should send a satellite to the moon ... very expensive so i do not believe they want to do that (unless you are a famous scientist), however string theory can not be tested , but it is accepted as theory by members of scientific community , what would have happened if no one have tested theory of Relativity or Quantum Mechanics just because they sound very 'odd' theories ? , a 'fair' peer review process should include the next things:

a) If the Mathematical equations are correct they paper should be published

b) If the experiments can be repeated with the same or similar results, referee should test all the experimental data, if they agree with the conclusions of the author the article should be published —Preceding unsigned comment added by 85.85.100.144 (talk) 09:56, 23 February 2008 (UTC)

The Aristotelian academics jealousy of Galileo is the classic case of Peer Review seeking to suppress opposing views. Since State, Church and the people were all supporting Galileo, they finally hatched the plot to persuade the Church that Galileo contradicted the Bible - though Galileo strongly upheld the Bible. THey eventually succeeded, and Galileo had to publish his works outside Italy. DLH 02:18, 25 June 2006 (UTC)

I added a couple of sentences to this section to point out that while control of the entire set of journals by an elite is practically impossible, the small group of "elite" journals is clearly susceptible to control by a clique. With the profusion of journals and the increasing importance of the ISI impact factor of the journals in which one publishes (especially in biology and medicine) this is becoming an increasingly likely scenario. Hence the move of Journal of Biology, for example, to non-anonymous peer review.

Note though that what was added is only science specific. The article has an extremely strong slant towards peer review issues in the sciences, seldom touching on peer review in law, humanities, and the social sciences. --Myke Cuthbert 16:38, 14 March 2007 (UTC)

The picture

(William M. Connolley 09:51, 18 Aug 2004 (UTC)) The picture at the start of this article doesn't look right to me. I don't think the chap is evaluating (in the sense of peer review) the proposals - he looks more like he is checking a pile of submitted proposals that all the boxes have been filled in.

You could be right - he/she (I can't really tell) is looking over the front page of what looks like a NRSA application. The front page has stuff like name, institution, and proposal title. However, the following pages contain the grant proposal, which is peer reveiwed. From the picture, the person looking at the grant could be just checking to make sure that boxes are filled in or he/she could be starting to read the entire grant, begining at the front page with the applicant's information. As I stated above, I got the picture from the Center for Scientific Review webpage. Looking over all the pictures they had there, I decided that this was probably best suited to the article, since many of the other pictures depicted reviewers debating grant proposals, which would be confusing to people reading about how peer review for journal articles is done anonymously. You're welcome to try to find another picture that better depicts peer review or just change the caption. Personally, I think this picture is okay, since it could be interpeted as someone starting to read a grant, but improvement is always welcome. Sayeth 15:44, 18 August 2004 (UTC)

Spoken?

(William M. Connolley 22:50, 18 April 2005 (UTC) ) The new "spoken" thing doesn't work for me... when I click on "listen to this article" I get back to the text article again. Is something broken?

Request for references

Hi, I am working to encourage implementation of the goals of the Wikipedia:Verifiability policy. Part of that is to make sure articles cite their sources. This is particularly important for featured articles, since they are a prominent part of Wikipedia, and even moreso for an article like this one. The Fact and Reference Check Project has more information. Thank you, and please leave me a message when you have added a few references to the article. - Taxman 19:18, Apr 21, 2005 (UTC)

This statement also needs a reference, particularly since "recent" is a term that loses meaning very quickly as an encyclopedic statement: 'In a recent editorial in Nature, it was stated that "in journals in those days, the burden of proof was generally on the opponents rather than the proponents of new ideas."' Rakerman 10:41, 9 June 2006 (UTC)

---

Move Physical Review from the stringent category to the laissez-faire category. I'm not sure what the acceptance rate of PR is but I think it is much closer to ApJ than to Science.

Roadrunner 9 July 2005 15:38 (UTC)

Peer review in humanities

This article is weighted heavily toward the sciences, but in the humanities peer review has an equally important role. Non-refereed journals are not considered authoritative secondary sources nor are they particularly useful for academic career advancement. In literary studies, though there are one or two exceptions--prestigious journals (such as English Literary History) that are not refereed--virtually all of the major journals (including PMLA, Studies in English Literature, Representations, Criticism. . .) are refereed, and grant and fellowship applications frequently require candidates to separate their refeered and non-refereed publications. A useful source for this issue would be the MLA Directory of Periodicals (which is distinct from the MLA Bibliography), available online through most university libraries. Chick Bowen 00:32, 5 September 2005 (UTC)

My mistake: ELH is now peer-reviewed as well. Chick Bowen

History

I'm in the early stages of a project on science in the medieval Arabic world and noticed that peer-review of a sort was occasionally practised by Avicenna (that article links to this page and talks of his 'possibly even decisive' contribution ) and others in the 11th century. I'll be happy to put something in once I've studied it a little more, but if someone knows more about it than me, they should probably jump in ahead. 69.144.83.104 17:24, 9 October 2005 (UTC) kensson

"blind" vs. "masked"

The preferred terminology is now 'masked', 'single-masked', and 'double-masked'. Many journals are adopting this term instead of 'blind'.

And yet, one of the most prominent is not – http://www.nature.com/nature/journal/v451/n7179/full/451605b.html While I personally agree with the new terminology, I have not yet, in my limited experience, ever heard of a double-masked study. And, as our job as Wikipedia editors is descriptive, rather than proscriptive writing about the world, I think both terms should be included and the difference explained in the article because both are used. —Preceding unsigned comment added by Msanford (talk • contribs) 19:36, 1 March 2008 (UTC)

On second thought, I may be getting over-zealous in my quest for perfection. I think it is well-put in the article (simply as 'alternatively'), though a citation would be nice to show the use of double-masked (as I just added one for double-blind). --Msanford (talk) 19:52, 1 March 2008 (UTC)

Cost of scientific journals?

Would it be worth mentioning that many of the specialist "peer-reviewed" medical and scientific journals are extremely expensive (thousands of dollars/year) and this presents a barrier to wide dissemination of information? Considering this along with the tremendous number of papers submitted to mainstream journals, it can be quite difficult for a controversial paper to be published in a visible peer-reviewed forum. 12.103.251.203 19:37, 7 December 2005 (UTC)

Also, would it be worth mentioning the role of a journal club in peer review? That is two fold: 1. to help share articles when they are too expensive for a group to buy (see the link above, as this was one of the first purposes of a journal club) 2. As journal clubs will discuss and debate the quality of literature after it has been "peer reviewed" - and as such, may deem that the peer review process was a success or failure. Often in a journal club, it will become clear that the peer review process was fraught with bias. —Preceding unsigned comment added by EBMdoc (talk • contribs) 19:57, 10 December 2005 (UTC)

---

> yes on both accounts. I it is clear that there is great 'cost' in peer review - and that is certainly worth delving into -- as is the role of journal clubs, and their job in 'peer review after the ink is dry'

— Preceding unsigned comment added by 69.117.14.111 (talk) 01:06, 20 December 2005 (UTC)

Journal results and scientific validity

According to the SEPP, "leading science journals are losing credibility with the media and general public. They cite a New York Times article by science reporter Nicholas Wade:

"The more fundamental issue is that journals do not and cannot guarantee the truth of what they publish. Publication of a paper only means that, in the view of the referees who green-light it, it is interesting and not obviously false. In other words, all of the results in these journals are tentative." (cited in The Week That Was)

Is this a new opinion, or has it always been the case that getting an journal article through peer review is no guarantee that the claims it makes are valid? --Uncle Ed 20:41, 22 February 2006 (UTC)

• "When reporting on journal findings, most news outlets fail to caution that studies must be replicated to be truly authenticated." [2]

Who said the above: Finfacts writers, or Donald Kennedy, the editor of Science? --Uncle Ed 20:46, 22 February 2006 (UTC)

Hillary Profita also cites NY Times reporter Nicholas Wade:

Journal editors are, of course, not without flaws. While publication in a well-respected journal offers the benefit of a peer-review, it doesn’t immediately mean that it is iron-clad truth. "Publication of a paper only means that, in the view of the referees who green-light it, it is interesting and not obviously false. In other words, all of the results in these journals are tentative,” Nicholas Wade, a science reporter for The New York Times, told Bosman. Knowing that such flaws are somewhat inherent, the best that reporters can do is call attention to those areas where skepticism is warranted. [3]

Indeed, just look at Jan Hendrik Schön for an example of peer-review not weeding out a weed. However, there is a misconception among the general public regarding the goal of peer-review (among other things): that it is to weed out non-truth, rather than to weed out research with methodological or conceptual problems. This might be worth mentioning as well. —Preceding unsigned comment added by Msanford (talk • contribs) 17:44, 26 February 2008 (UTC)

Structure of a Peer-Reviewed Paper

Although all the examples in the article are taken from the sciences, it accords mainly with what I know from my field (musicology) up to the section "Structure of a Peer Reviewed Paper". The format seems to describe "structure of a published science paper" (whether peer reviewed or not) and does not accord with the structure of any paper I have seen in the humanities. My vote would be to remove the whole section since it has little to do with peer review. (Also the title of the section lacks a hyphen). --Myke Cuthbert 23:05, 5 May 2006 (UTC)

I have removed the section. It seems to describe "structure of a published paper describing an experiment." Surely many papers do not describe experiements, even in science. For example, a meta-study, although the structure of such a paper might closely resemble one describing an experiment. But there must be theoretical papers which (roughly) propose new models (say, in cosmology? evolution? etc?) but refer to existing data only and not the results of a new experiment.

— Preceding unsigned comment added by 70.244.105.170 (talk) 14:47, 20 May 2006 (UTC)

From Talk:Intelligent design

There has recently been a lengthly discussion at Talk:Intelligent design about the statement:

To date, the intelligent design movement has yet to have an article published in a peer-reviewed scientific journal.

Please please don't respond to the claim about ID here. This page should not harbor discussions of ID vs evolution.

The conversation ended with consensus that published ID papers do exist in refereed journals, but only ones whose review process is flawed in some way. The common flaw (e.g. for the journal Rivista) is when the editorial board and referee pool are all of a similar ideological bent which is distinct from the mainstream scientific community. Consensus at ID is that such journals cannot be said to be truly "peer reviewed". Another example cited was Forest Service publications, which are reviewed and refereed in-house. Consensus is that these too are not truly peer reviewed.

If so, something to this effect should be added here. There are implications about some of these things, but no statements that would clearly rule out something like Rivista or Forest Service journals as true peer review- but such a "ruling out" is what consensus at the ID page demands. (Please ID discussion members give your input too.) Staecker 14:10, 3 June 2006 (UTC)

The dispute is over what is a proper peer review. Some people (laymen in general, as well as scientists and policymakers) tend to assert that there must be a flaw in the peer review process if it passes a scientific paper which they disagree with. (Or rejects a paper they agree with.) Thus, their standard for evaluating peer review is the results, not the method itself. Like evaluating affirmative action in nursing or firefighting by whether an equal number of men and women become nurses or firefighters.

Both sides on any controversy will usually agree (when cornered) on the criteria for a good peer review: i.e., it shouldn't be just your cronies who already agree with your position, evaluating your paper. But hardly anyone will admit a lapse in the rigor of the process. For one thing, maintaing the claim that the process is always fair allows one to make the argument that "paper X must be nonsense because it was rejected by mainstream peer-reviewed journals". This ignores the fact that papers which PASS peer review often disagree with each other. --Uncle Ed 14:59, 3 June 2006 (UTC)

Krebs cycle

Ed, you think you can do better than that horribly constructed bit on the Krebs' Cycle? The paper was rejected, but that doesn't mean that the Krebs' cycle was never subjected to peer review. What's more, the source you listed was to your hastily executed Google search and linked to a cached version of a power point presentation by a librarian. Hardly a good source to list. This kind of editting is really subpar. --ScienceApologist 22:33, 3 June 2006 (UTC)

Section removed for analysis and discussion

I have removed this section and am putting the content here in full, for further analysis as to the validity and relevance of its content:... Kenosis 05:58, 7 June 2006 (UTC) ... Fact is, all three items on this list have been extensively peer-reviewed by communities of experts since they were proposed. Galileo has been hashed over endlessly and universally accepted. Resistance of the Church to his presentation of evidence of a non-geocentric universe is today widely known and accepted as well. Watson and Crick have been hashed over endlessly and shown to have been quite correct, largely because their theory was eminently falsifiable and could have been shown wrong if it were wrong, which it turned out not to be. Interactions between the weak and strong nuclear forces, similarly have been hashed over in extreme depth, with discussion still ongoing today, the basic idea having been widely accepted today in the community of the authors' peers.... Kenosis 05:58, 7 June 2006 (UTC)

==Famous papers which were not peer-reviewed==

Because of its relatively recent status as a fixture in the scientific enterprise, many of the major breakthroughs in the history of science ironically were published without having undergone peer review. 06:04, 7 June 2006 (UTC)

1. Galileo Galilei Discourses and Mathematical Demonstrations Relating to Two New Sciences (Discorsi e dimostrazioni matematiche, intorno à due nuove scienze, 1638) Published by Lowys Elsevier in Leiden, The Netherlands.06:04, 7 June 2006 (UTC)

Even after peer review had become common practice, some famous papers have been published without review. These include:06:04, 7 June 2006 (UTC)

1. Publication of Watson and Crick's 1951 paper on the structure of DNA in Nature. This paper was not sent out for peer review. John Maddox stated that “the Watson and Crick paper was not peer-reviewed by Nature... the paper could not have been refereed: its correctness is self-evident. No referee working in the field (Linus Pauling?) could have kept his mouth shut once he saw the structure” (Nature 426:119 (2003)). The editors accepted the paper upon receipt of a “Publish” covering letter from influential physicist William Lawrence Bragg.06:04, 7 June 2006 (UTC)
2. Abdus Salam's paper "Weak and electromagnetic interactions", which elucidated the unification of the weak nuclear force with the electromagnetic force into an electroweak force. It was originally published in Svartholm: Elementary Particle Theory, Proceedings Of The Nobel Symposium Held 1968 At Lerum, Sweden (Stockholm, 1968, 367–77). Salam shared the 1979 Nobel prize, along with Steven Weinberg and Sheldon Glashow, for this work. 06:04, 7 June 2006 (UTC)

06:04, 7 June 2006 (UTC)

I agree that the Galileo one should not be on the list - peer review just wasn't done then. For W+C, I think you have the wrong idea: of course they were correct: this isn't a list of famous, wrong, papers: its just those not PR'd William M. Connolley 10:41, 7 June 2006 (UTC)

The over-arching principle, that work by one researcher is checked by other researchers, first began to be applied in the early 17th century in regards to the pioneering work in astronomy by Tycho, Copernicus, Galileo, Kepler and Newton. Technically, it would not be "peer review of a journal article" because their discoveries were not submitted to journals who used twentieth-century peer review procedures.

Back in the old days, before 1850 or so, I think they just sent each other letters or pubished their own work.

The issue being discussed here relates more to what conclusions laymen can draw from the fact that a given paper on a controversial subject (which might turn out to be protoscience or shown finally to be pseudoscience) has been accepted or rejected by the major scientific journals of our day.

If Science or Nature accept a paper for publication, can laymen then rest assured that the ideas it presents are TRUE and RELIABLE? Or should they wait a couple of years for another peer-reviewed paper which may contradict the first?

If they reject a paper, is this conclusive proof that the ideas it presents are FALSE and may therefore be dismissed from consideration?

These questions have huge implications for public policy. Some politicians will point to a single paper (which they can't even understand, since it's written in language inaccessible laymen) and say that because this paper was accepted (or rejected), we should spend tens or hundreds of billions of dollars.

It would be nice to have an encyclopedia article which explained the peer review process simply and completely, so that our readers can decide for themselves whether the procedure is being followed in respect to scientific ideas which have a bearing on public policy. --Uncle Ed 11:36, 7 June 2006 (UTC)

We all know that when public policy (read that: "politics") and/or when commercial considerations (read that: "big money") are involved, science can become a dirty business. No less so here. When Watson and Crick won the frantic race to explain DNA, they were in fact peer reviewed, quite quickly and informally, and published hastily too. The immediate consensus was that they were onto something big, and history would prove this correct.

The apparent purpose of the current intense push to include such material here in the peer review article is, however, to cherrypick the exceptions to the general rule so as to allow proponents of such concepts as intelligent design to draw on the hard-won accumulated credibility of sciences, in keeping with the wedge strategy. Ample evidence of this apparent purpose exists in the recent histories of articles such as intelligent design movement, neocreationism and talk pages thereof, and in the edit histories of the editors such as Ed Poor and DLH who are currently pushing for this material in the peer review article. The situation became so serious that the intelligent design article needed to be edit protected. I personally don't particularly begrudge Ed Poor and DLH for trying. However, this evidence of a particular agenda (read that:POV) raises the burden of justification for inclusion in the peer review article of such marginal examples of the occasional successful theory that was so compelling to be granted a free pass to the most stringent definitions of peer review.

That said, here's part of the problem as I see it. Peer review is extremely important, however abused it arguably is at times. The underlying principle, though, is "full disclosure", making the entire set of relevant methodology, data, experimental conditions, possible sources of bias, etc. publicly available for scrutiny by a community of inquirers familiar with the relevant material so it can be properly assessed for validity, reliability, statistical confidence, etc. [See, e.g., Gauch, Hugh G., Jr., Scientific Method in Practice (2003), 124 ff, esp. section on "Full Disclosure".]

Perhaps Ed Poor is onto something in advocating a yet more understandable description of relevant considerations for the layperson, for the general public if you will. I interpret this as possibly leading towards a potential improvement of the utility of the peer reveiw article.

• 1) First there is the issue I just mentioned of full disclosure being a vital underlying justification for peer review, so as to prevent the "wizard behind the curtain" from putting one over on the public (see again wedge strategy).
• 2) Then there's the issue of how the peer review process flexes to accommodate varying needs such as for a hasty publication as in the case of Watson and Crick. In their case, the peer review process was accomplished by the publication of the article by Nature. As soon as it is published it gets all the peer review in the world, because now it's open source information, so to speak. Just as the entrance requirements are occasionally waived by colleges for entrance or even for conferral of a degree, so it is the case for peer review. Some are given passes to the front of the line.
• 3)Then there's the opposite phenomenon, e.g. the Jacques Benveniste affair, where peer review was exercised prior to publication and all but James Randi ended up with mud on their face.
• 4) Tentative conclusion: Perhaps these marginal aspects of peer review are best framed in the article as a discussion of these issues, rather than just a list. Such an approach could end up informing the reader more fully without being excessively technical, merely by showing the two extremes (successful without peer review prior to publication on the one hand; vs. unsucessful despite peer review on the other extreme)...Kenosis 15:12, 7 June 2006 (UTC)

For what it's worth, I do NOT feel that Wikipedia should endorse the wedge strategy. Rather, I think our articles should DESCRIBE that strategy. Likewise, Wikipedia articles should not endorse (or condemn) the "teach the controversy" campaign - but merely describe it. And I'd like that description to say something like "Creationists are trying to wedge their point of view into US public school science classes by asserting that scientists are divided on evolution". If this is what you mean by "POV pushing", I plead guilty. But I see it more as describing one's side of a controversial subject, which (until lately) I believe is not merely permitted by NPOV policy but actually a requirement of it. --Uncle Ed 20:44, 7 June 2006 (UTC)

Obvious vs. insidious bias

There's a lot of discussion about peer-review lending scientific objectivity/credibility to the results of research. Also, the concept of biased reviewers has been discussed in the article.

Let's be careful about holding peer review up as the holy grail of scientific credibility, though. Funding applications are generally peer-reviewed, and those that are too far from the current consensus thinking on any topic are less likely to be funded, if not dismissed out of hand (either by outright prejudice, limited funding, etc.). While the final review of research results by peers may not be biased in a particular case, the research that is performed and can therefore be submitted through peer-review for publication has already been pre-selected by the initial biased peer-review funding process.

Publication in a peer-reviewed journal simply means that the research and its results reflect an acceptable methodology to the current consensus of the particular discipline. If the concept had been working several hundred years ago, Copernicus and Gallileo never would have been granted funding to perform their research, since the consensus was that the Earth was in the center and the Sun went around it. Nevertheless, C and G were correct. We need to be very careful about understanding what kind of biases are operating before we can accept as thing as believable, just because we say the words "peer-reviewed".

Also, it's interesting that the discussion has touched on public policy decisions based on single (or even multiple) published papers. Just because the results of particular research are reproducible does not mean that the underlying theory is correct or complete. Newton's physics does a pretty good job of describing how things work in the middle, but fails when we get to the extremes where relativity or quantum mechanics do a better job. Unfortunately, most politicians have little training in particular scientific disciplines or to the principles of scientific theory in general, so they sieze on the concept of peer-review as providing validity for whatever their points of view happen to be.--[David Barkhimer] 06:55 22 June 2006 (UTC)

I have been looking forward to adding a section about the documented extremes that deviate from, might we say, idealized outcomes of peer review. Haven't gotten around to it quite yet. At least a couple are mentioned in the section above, now reproduced here: ... Kenosis 08:04, 22 June 2006 (UTC)

• There's the issue of how the peer review process flexes to accommodate varying needs such as for a hasty publication as in the case of Watson and Crick. In their case, the peer review process was accomplished by the publication of the article by Nature. As soon as it is published it gets all the peer review in the world, because now it's open source information, so to speak. Just as the entrance requirements are occasionally waived by colleges for entrance or even for conferral of a degree, so it is the case for peer review. Some are given passes to the front of the line.
• Then there's the opposite phenomenon, e.g. the Jacques Benveniste affair, where peer review was exercised prior to publication and all but James Randi ended up with mud on their face.
• Tentative conclusion: Perhaps these marginal aspects of peer review are best framed in the article as a discussion of these issues. Such an approach could end up informing the reader more fully without being excessively technical, merely by showing the two extremes (successful without peer review prior to publication on the one hand; vs. unsucessful despite peer review on the other extreme) ... Kenosis 08:04, 22 June 2006 (UTC)

Personally I'd be quite comfortable with someone starting off such a section, assuming the slant is reasonably consistent with WP:NPOV and WP:VER. ... Kenosis 08:04, 22 June 2006 (UTC)

Section on Peer review failures

Happy to see this new section started. Please do not unilaterally remove citation requests without providing citation or justifying and consensusing on the talk page. I've placed an Original Research tag on it and will follow up with requests for verification per WP:VER, since there will naturally be some disagreement about the section's content. Kenosis 15:21, 27 June 2006 (UTC)

The first paragraph of the section currently reads as follows: ... Kenosis 15:21, 27 June 2006 (UTC)

• A peer review failure occurs when a peer-reviewed article contains an obvious fundamental error(s) that undermines at least one of its main conclusions. Peer review is not considered a failure in cases of deliberate fraud by authors. Letters-to-the-editor that correct major errors in articles are a common indication of peer review failures. Few journals have an effective procedure to deal with peer review failures. Journal editors do not usually acknowledge peer review failures and rarely ask peer reviewers to reconsider an article in light of published criticisms. The author is allowed a published reply to a critical letter, but the reply is usually obfuscation rather than an admission of the article's flaws. Editors sometimes refuse to publish letters correcting badly flawed articles for apparently no reason other than to cover up peer review failures. Some refereed journals have a policy to never publish letters. Even when letters reveal peer review failures, they are often not effective in alerting readers because letters are not indexed along with the articles. Editors seldom publish retractions or corrections for flawed articles unless the authors request them. 15:21, 27 June 2006 (UTC)
• (1)The first part of this paragraph currently is a made-up definition of what constitutes a peer review failure. ... Kenosis 15:21, 27 June 2006 (UTC)
• (2)("Peer review is not considered a failure in cases of deliberate fraud by authors.") This sentence is speculative without supporting literature.
• (3)("Letters-to-the-editor that correct major errors in articles are a common indication of peer review failures.") This sentence is obvious and I support it.
• (4)("Few journals have an effective procedure to deal with peer review failures.") This sentence also is understood immediately and ought easily be agreed by consensus, but still could use examples and support.
• (5)(" Journal editors do not usually acknowledge peer review failures and rarely ask peer reviewers to reconsider an article in light of published criticisms.") This sentence is unsupported and accusatory. Please support it.
• (6)("The author is allowed a published reply to a critical letter, but the reply is usually obfuscation rather than an admission of the article's flaws. ") This sentence is a ridiculously accuasatory generalization without concrete support.
• (7)("Editors sometimes refuse to publish letters correcting badly flawed articles for apparently no reason other than to cover up peer review failures.") By this point in the paragraph it is obvious that the editor who wrote this has an obvious bias and an agenda here.
• (8)Several further generalizations and accusations follow, all of which require citation and support due to their accusatory and obviously controversial nature.

This section is important to the health of the article. Please let's get it right in a way that it will remain a stable contribution to the article's informativeness, usefulness and accuracy. ... Kenosis 15:21, 27 June 2006 (UTC)

• (1)The first part of this paragraph currently is a made-up definition of what constitutes a peer review failure. ... Kenosis 15:21, 27 June 2006 (UTC)

• This is based on the "Reasons for peer review" section which says the purpose of peer review is to spot mistakes or flaws. Doesn't it logically follow that if obvious, major flaws are not identified by peer review, then it has failed? Plantguy 17:46, 27 June 2006 (UTC)

• (2)("Peer review is not considered a failure in cases of deliberate fraud by authors.") This sentence is speculative without supporting literature.

• This sentence follows from the section on Peer review and fraud which states "the process is not designed to detect fraud." That is a widely held belief. Plantguy 17:46, 27 June 2006 (UTC)

• (3)("Letters-to-the-editor that correct major errors in articles are a common indication of peer review failures.") This sentence is obvious and I support it.

• (4)("Few journals have an effective procedure to deal with peer review failures.") This sentence also is understood immediately and ought easily be agreed by consensus, but still could use examples and support.

• What kind of evidence do you want here? I agree much of this has not been studied but mainly because the peer review system was not set up to deal with peer review failures, yet they happen. I have never seen a peer-reviewed journal with a policy on how to deal with peer review failures. I have a lot of personal experiences where editors refused to correct major factual errors in referreed articles without explanation or with the excuse that there is no room. Plantguy 17:46, 27 June 2006 (UTC)

• (5)(" Journal editors do not usually acknowledge peer review failures and rarely ask peer reviewers to reconsider an article in light of published criticisms.") This sentence is unsupported and accusatory. Please support it.

• I have never seen a journal editor apologize in print for an article that contained obvious, major errors (not fraud) or announce that an article had been reexamined by reviewers and found to be lacking so is being retracted. How can you cite a negative? Plantguy 17:46, 27 June 2006 (UTC)

• (6)("The author is allowed a published reply to a critical letter, but the reply is usually obfuscation rather than an admission of the article's flaws. ") This sentence is a ridiculously accusatory generalization without concrete support.

• What kind of support do you want here? I can prepare a list of published examples where an author replied to a letter with obfuscation. One excellent example is Temple, S.A. 1979. The dodo and the tambalacoque tree. Science 203: 1364. which is a reply to the letter pointing out flaws in the dodo article by Owadally, A.W. 1979. The dodo and the tambalacoque tree. Science 203: 1363-1364. Plantguy 17:46, 27 June 2006 (UTC)

• (7)("Editors sometimes refuse to publish letters correcting badly flawed articles for apparently no reason other than to cover up peer review failures.") By this point in the paragraph it is obvious that the editor who wrote this has an obvious bias and an agenda here.

• This is supported by later citations. The editor of Bioscene refused to publish a letter or consider an article pointing out factual errors in a 1993 article. Those corrections were published elsewhere. I can provide additional examples of this type. Plantguy 17:46, 27 June 2006 (UTC)

• (8)Several further generalizations and accusations follow, all of which require citation and support due to their accusatory and obviously controversial nature.

• I provided several footnotes for the second paragraph. What more do you think is needed? This section addresses an obviously controversial topic because journals do not want to admit that peer review sometimes fails for other than author fraud, often spectacularly so. In the book, The Double Helix, it was mentioned that a former Nobel Prize winner, Linus Pauling, published an article on a possible structure for DNA, but Watson immediately recognized that it violated the basic laws of chemistry. Another example of a spectacular peer review failure. Plantguy 17:46, 27 June 2006 (UTC)

Citations are duly noted for the examples in the second paragraph – good citations at that. Problem is, the entire first paragraph is arbitrary and judgmental as a summary paragraph for the fine examples that follow in the section. The most conspicuous instances of thrown-in judgments in the current first paragraph are:

• "... the reply is usually obfuscation rather than an admission of the article's flaws",
• "... and rarely ask peer reviewers to reconsider an article in light of published criticisms."
• "... for apparently no reason other than to cover up peer review failures."

These are only the most conspicuous. By my reading of it, the paragraph currently sounds like a diatribe. Surely this can written so as not to serve as concrete proof of certain WP editors' disdain for the pre-publication peer-review process. ... Kenosis 22:18, 27 June 2006 (UTC)

• I rewrote a lot of it and tried to make it more dispassionate. Plantguy 01:48, 28 June 2006 (UTC)

The section seems to me to mainly be a list of grievances against specific cases of peer review failures which were not corrected. I am not sure that they (in detail or number) add much to the article. Also the "famous" cases seem to be only famous in the field. I know of famous cases of such failures in music research, but I don't think the casual readers, coming from many different backgrounds, will be interested.

--Myke Cuthbert 02:19, 1 November 2006 (UTC)

Stratfordians

I removed a number of paragraphs on "Stratfordians" which would, I feel, be better served in an article about disputes about Shakespeare's indentity, not in a general article about peer review. 70.25.91.244 12:43, 4 July 2006 (UTC)

Unsourced criticism

Cut:

Unfortunately "peer review" is a process that suffers from abuse by established "cliques." Worse it is not a process that assures future success. Ideas, discoveries and research should be evaluated on merit, not on whether or not they have been approved of by a disinterested circle of experts in any give field. That merit will only become apparent over time, when the ideas, discoveries and research has had an opportunity to work itself into the fabric of intellectual life and/or technology, as Arthur Schopenhauer pointed out several centuries ago. Academics and corporations who fear the publication of alternative views or technologies generally have good reasons for that fear. Reasons that are either anti-intellectual or anti-competitive.

As Charles Darwin pointed out it is "false evidence," not "false ideas" that are counterproductive.

All of this seems valid to me, but it lacks a source. What published author makes this sort of criticism? Michael Crichton, Benny Peiser, various IPCC lead authors? --Uncle Ed 15:57, 28 July 2006 (UTC)

Peer review of books

Books are only briefly mentioned in passing. Would like to see something about peer review of books, how extensive it is, what percentage of scholarly books are peer reviewed, does the process differ from that for journals, exactly what types of scholarly books are (are not) peer reviewed. Nurg 00:19, 7 August 2006 (UTC)

I found a little info on this and added a short para. Nurg (talk) 05:45, 2 February 2008 (UTC)

My appologies. I've had to revert you good faith addition per WP:COPYRIGHT (external Web site appears to be carrying work in violation of the creator's copyright). I left a note on your talk page.--Hu12 (talk) 08:01, 2 February 2008 (UTC)

Thanks. I found the original source so all is well. Nurg (talk) 09:44, 2 February 2008 (UTC)

Issues with anonymity

Does anyone have a reference discussing the use of anonymous vs. unanonymous peer review? — Preceding unsigned comment added by 128.174.115.32 (talk) 20:28, 31 October 2006 (UTC)

Does anyone have a reference that discusses lawsuits as a result of contested peer review, where the identity of the reviewer is exposed? — Preceding unsigned comment added by 128.174.115.32 (talk) 14:32, 3 November 2006 (UTC)

Semantics

It appears the controversy over peer review may be a bit semantical. Peer review is a philosophy that when executed correctly, is an excellent way to conduct scientific publishing. All the controversy appears to be when the system for implementing it messes up (biased reviewers, bad reviewers, etc.).

In that case, a "peer review" is not really obtained.

I point this out as a matter of discussion about perhaps rephrasing the sections on discussing the controversies around the subject. That is, peer review itself, as a philosophy, seems not to be the controversy. It is the way in which it is implemented. I would propose editing the "Problems with Peer Review" to be "Problems with Implementation of Peer Review and in Peer Review Itself." And then editing the text to reflect the title. — Preceding unsigned comment added by 128.174.115.32 (talk) 14:37, 3 November 2006 (UTC)

Deletion of section - let's discuss this

Earlier today I added the following paragraph to the section on Criticisms of Peer Review that has been deleted twice:

Conspicious examples of censorship in the realm of peer review are articles with potential religious connotations. Deviations from mainstream scientific thought that can be related to religious viewpoints can be quickly censored by the "elites". Even if an article passes the peer review process, its publishing can have disastrous effects for the publisher. In August of 2005, Richard Sternberg, a research associate with the Smithsonian Institution, was fired after allowing an article to be published entitled "The Origin of Biological Information and the Higher Taxonomic Categories" that cast doubt on Darwinism. Senior scientists at the Smithsonian Institution called Sternberg a “shoddy scientist” and a “closet Bible thumper,” despite his protests that he is agnostic about Intelligent Design and does not hold any sort of creationist viewpoint.

In the edit comment box I asked for help finding the actual citations (the Washington Post, Washington Times, and the UK's Independent) so that we wouldn't have to have the secondary source of the AiG article. The AiG article is a secondary source that summarized the Post and Times and Independent articles with commentary - I asked for help finding the primary sources. Please explain why we should not include this paragraph with reliable sources. standonbible 18:32, 15 November 2006 (UTC)

Evaluating peer review through fundamentalist Christian lenses of AiG is inappropriate. AiG is only a reliable source for what they believe -- they do not belong on this page nor does any analysis that follows their points belong on this page. --ScienceApologist 18:36, 15 November 2006 (UTC)

Um ... this section is discussing criticisms of the peer review process and it specifically mentions elites suppressing dissent against mainstream views. An example of this would be good - and Sternberg is perfect for this. I was careful not to include any "fundamentalist Christian" spin on it but reported the facts - facts that should be directly sourced by everyone, not suppressed just because AiG happened to report on it. Would you rather me replace the AiG ref with a {{fact}} tag? Otherwise the AiG article should serve as a guide for editors to find sources until all the sources can be found. standonbible 18:42, 15 November 2006 (UTC)

You may think you were careful, but in fact your bias shines through. First of all, firing a person has nothing to do with peer review. Second of all, claiming religious bias is a specious claim favored by AiG and backed up by zero evidence. Scientists of many different religions have no problem publishing, the only people who have problems publishing are those that contravene the scientific method and inject their religion into their papers. That's an issue for the theistic realism and philosophical naturalism pages, it doesn't belong here. In short, the entire paragraph is a red herring and represents creationist POV-creep -- it frankly doesn't belong here. --ScienceApologist 18:45, 15 November 2006 (UTC)

You said, "firing a person has nothing to do with peer review". Ahem - he was fired for allowing the article to be peer-reviewed. And I never claimed bias against religion - that is something you read into it. Don't talk about "contravening the scientific method and injecting religion into their paper - Sternberg subscribes to neither ID nor Creationism. How can the article be a red herring if the very last paragraph talks about the "elites" squelching non-mainstream research - don't you think there should be an example of this?

How's this:

Conspicious examples of censorship in the realm of peer review are articles that deviate from mainstream thought and may have religious connotations. Publishers are hesitant to pass such articles due to opposition from the mainstream; in August of 2005, Richard Sternberg, a research associate with the Smithsonian Institution, was fired after allowing an article to be published entitled "The Origin of Biological Information and the Higher Taxonomic Categories" that cast doubt on Darwinism. Senior scientists at the Smithsonian Institution immediately called Sternberg a “shoddy scientist” and a “closet Bible thumper,” despite his protests that he is agnostic about Intelligent Design and does not hold any sort of creationist viewpoint.

I removed any hint of so-called "religious bias" and even re-referenced it. Now there is no mention of AiG. Does that satisfy you? How is that "creationist POV-creep" (to use your rather blunt language)? standonbible 20:31, 15 November 2006 (UTC)

Oh, and here's what that looks like with nowiki:

Conspicious examples of [[censorship]] in the realm of peer review are articles that deviate from mainstream thought and may have [[religion|religious]] connotations. Publishers are hesitant to pass such articles due to opposition from the mainstream; in [[August]] of [[2005]], [[Richard Sternberg]], a research associate with the [[Smithsonian Institution]], was fired after allowing an article to be published entitled "The Origin of Biological Information and the Higher Taxonomic Categories" that cast doubt on [[Evolution|Darwinism]]. Senior scientists at the Smithsonian Institution immediately called Sternberg a “shoddy scientist” and a “closet Bible thumper,”<ref>Michael Powell; The Washington Post; Aug 19, 2005; A.19 [[http://pqasb.pqarchiver.com/washingtonpost/access/884160621.html?dids=884160621:884160621&FMT=ABS&FMTS=ABS:FT&fmac=&date=Aug+19%2 :C+2005&author=Michael+Powell&desc=Editor+Explains+Reasons+for+%27Intelligent+Design%27+Article]]</ref> despite his protests that he is agnostic about [[Intelligent Design]] and does not hold any sort of [[creationism|creationist]] viewpoint.<ref>David Usborne; New York; The Independent (UK); Aug 20, 2005 [[http://news.independent.co.uk/world/americas/article307079.ece]]</ref>

Thanks! standonbible 20:49, 15 November 2006 (UTC)

I tried posting what I have above and was rv'd by JoshuaZ. He said it wasn't a notable enough example. Did you even read the sources? This involved the Smithsonian Institution and went as high as President Bush - what would be notable enough? We need an example of what is being talked about - explain how that is POV. standonbible^Talk! 21:18, 15 November 2006 (UTC)

Where does it say it went as high as Dubya? Nowhere. &#0149;Jim62sch&#0149; 21:32, 15 November 2006 (UTC)

Actually, yes-where. Take a look at [[4]]. standonbible^Talk! 22:07, 15 November 2006 (UTC)

In any case the NYT had this to say about it:

August 20, 2005

Intelligent Design and the Smithsonian

The Smithsonian Institution can't seem to disentangle itself from the clutches of the anti-evolution crowd. Earlier this year, the Smithsonian's natural history museum discovered to its dismay that it had agreed to be the host and co-sponsor of a movie intended to undercut the theory of evolution and make the case for intelligent design, the idea that an intelligent agent had a hand in designing the universe. Only after intelligent-design proponents started chortling on the Internet about their stunning coup in co-opting the Smithsonian did museum officials reverse course and withdraw their sponsorship, while allowing the film to be shown.

Now comes word that a little-known government office has accused the Smithsonian of retaliating against a scientist who slipped an article promoting intelligent design into an obscure journal that has only very loose connections to the Smithsonian. That judgment, by the United States Office of Special Counsel, a federal agency set up to protect whistle-blowers, is the latest twist in a case that started with the publication of the article last year in the Proceedings of the Biological Society of Washington.

The article contended that evolution theory could not account for the great proliferation of life forms during the so-called Cambrian explosion some 530 million years ago, and that an intelligent agent was the best explanation. It set off an uproar among evolutionary biologists and was later disowned by the professional society that published it.

The editor who authorized publication, Richard Sternberg, filed a complaint contending that he had suffered reprisals. In an 11-page letter not yet officially released, the Office of Special Counsel said it had found support for his complaint but was dropping the investigation because he was not an employee of the Smithsonian, just a research associate.

E-mail notes show that several scientists and managers at the Smithsonian were extremely embarrassed and eager to push Mr. Sternberg out of his research niche, and that some dug around for material to discredit him. That may lead critics of evolution to see Mr. Sternberg as a martyr.

But those who see no place for intelligent design in the realm of science -- and that includes us -- will ruefully give him credit for maneuvering a brief for intelligent design into a peer-reviewed scientific journal, although how rigorous that review was remains a point of contention.

[[5]]

Get the point dude? &#0149;Jim62sch&#0149; 21:32, 15 November 2006 (UTC)

If that isn't a POV article, I don't know what is. But fine - put the mainstream rebuttal in there if you'd like - maybe "Other scientists doubt Mr. Sternberg's claims that he is not a proponent of ID <ref>(nytimes link)</ref>". But the facts (verifiable, etc., etc.) still say that a research associate allowed a minority viewpoint through peer review, was fired and called a "shoddy scientist" and "closet Bible-thumper", and protested that he was not a proponent of ID. As such it is a perfect example of what this section is discussing; NYT's incredibly biased and pejorative article hardly changes that. Here's another revised version that might possibly fit your personal censorship standards:

A conspicious example of such censorship of iconoclastic research took place in August of 2005 at the Smithsonian Institution. Richard Sternberg, a research associate for the Smithsonian, was fired after allowing an article to be peer-reviewed and published entitled "The Origin of Biological Information and the Higher Taxonomic Categories". The article was controversial in that it cast doubt on the scientific theory of Evolution. Senior scientists at the Smithsonian Institution immediately called Sternberg a “shoddy scientist” and a “closet Bible thumper,” despite his protests that he is agnostic about Intelligent Design and does not hold any sort of unorthodox creationist viewpoint. Other scientists, however, see Sternberg's intentions as suspect.

Supporters of the peer-review process have pointed out ...

With nowiki:

A conspicious example of such censorship of iconoclastic research took place in [[August]] of [[2005]] at the [[Smithsonian Institution]]. [[Richard Sternberg]], a research associate for the Smithsonian, was fired after allowing an article to be peer-reviewed and published entitled "The Origin of Biological Information and the Higher Taxonomic Categories". The article was controversial in that it cast doubt on the scientific theory of [[Evolution]]. Senior scientists at the Smithsonian Institution immediately called Sternberg a “shoddy scientist” and a “closet Bible thumper,”<ref>Michael Powell; The Washington Post; Aug 19, 2005; A.19 [[http://pqasb.pqarchiver.com/washingtonpost/access/884160621.html?dids=884160621:884160621&FMT=ABS&FMTS=ABS:FT&fmac=&date=Aug+19%2 :::C+2005&author=Michael+Powell&desc=Editor+Explains+Reasons+for+%27Intelligent+Design%27+Article]]</ref> despite his protests that he is agnostic about [[Intelligent Design]] and does not hold any sort of unorthodox [[creationism|creationist]] viewpoint.<ref>David Usborne; New York; The Independent (UK); Aug 20, 2005 [[http://news.independent.co.uk/world/americas/article307079.ece]]</ref> Other scientists, however, see Sternberg's intentions as suspect.<ref>New York Times; Intelligent Design and the Smithsonian; August 20, 2005 [[http://select.nytimes.com/gst/abstract.html?res=FB0A13FA3B5A0C738EDDA10894DD404482&showabstract=1]]</ref> :::Supporters of the peer-review process have pointed out ...

There. I even called darwinism "the scientific theory of evolution" so there could be no doubt about the matter. Feel free to change the last sentence (other scientists, however, see Sternberg...) as much as you would like if you want it to be more accurate. What more could you guys ask for? standonbible^Talk! 22:07, 15 November 2006 (UTC)

Heck, I could even say "that questioned key assumptions regarding the Pre-Cambrian period" if you want to make it as unbiased as possible! standonbible^Talk! 22:38, 15 November 2006 (UTC)

This is an AiG criticism and as such does not belong in this article. Standonbible wouldn't have this line of reasoning if not for AiG, therefore it qualifies as AiG propaganda and shall be excluded from the article. --ScienceApologist 23:36, 15 November 2006 (UTC)

So, Mr. Apologist, if AiG reports that the moon-dust argument (if the moon really was billions of years old it would have 6 feet of dust on it) is based on faulty assumptions and should not be used, would that validate the moon-dust argument? Your overt bias against any inconoclastic viewpoints is fatiguing. If I had found the references first, somehow I don't think you would have objected so strongly. I would appreciate it if you would refrain from so quickly attacking anything that might possibly cast doubt on your personal bias. Your syllogism lacks validity and I would like to know what about the proposed addition violates what WP policy. standonbible^Talk! 00:13, 16 November 2006 (UTC)

To possibly clarify and expand on what SA was trying to say- there is an issue of good faith and POV pushing. The fact is that you are associated with AiG and prefer to put their material in many articles. The use of the new sources was therefore a step in the right direction. However, the fundamental problem of whether this is a notable example of a problem with peer review has not been addressed. Unlike some other problems like the Sokal incident or the incident with the French brothers (what was their name again?) which received widespread coverage the Sternberg matter received almost no coverage outside the usual venues where creationism/ID are discussed. JoshuaZ 00:17, 16 November 2006 (UTC)

Thanks, Joshua. I am associated with AiG in that I have several friends who work for AiG. However I don't exactly "prefer to put their material" in articles. I was reading the peer review article for personal reasons and thought an example of the "elites" censoring an unorthodox viewpoint would be in good taste - hence the use of the AiG article on Sternberg. I don't, as you seem to think, go poking around looking for a place to insert AiG POV. Now, since that is out of the way....

I agree that the fundamental problem is whether this is a notable example. According to WP:N, a subject must have been described by multiple independent reliable sources in order to be notable. The Sternberg incident has been documented and commented upon by the New York Times, the Washington Post, the Opinion-Journal, WorldNetDaily, the Discovery Institute, NPR, the Washington Times, the World Socialist Web Site, PhysOrg.com, and the National Review (just try Googling it). Somehow I think that qualifies as "multiple independent reliable sources". Why not? standonbible^Talk! 00:55, 16 November 2006 (UTC)

WP:N describes in general what is notable enough to have an article about a topic. Hence we have an article on the Sternberg controversy. That doesn't make it notable enough to be mentioned here. JoshuaZ 01:03, 16 November 2006 (UTC)

Could you tell me what the guidelines for notability in this case would be? standonbible^Talk! 01:11, 16 November 2006 (UTC)

If you insist, I suppose we could just put "for an example of this, see the Sternberg controversy" in the paragraph right before the proposed change. I think it is more specific this way, but.... standonbible^Talk! 01:11, 16 November 2006 (UTC)

I don't know if there is a specific guideline in regard to this matter but the undue weight section of WP:NPOV may be relevant. The other issue that was brought up above is whether this is even relevant to the discussion. The Sternberg case was problematic because it avoided the standard review process. Given that, it isn't clear to me why it should be included here. JoshuaZ 01:14, 16 November 2006 (UTC)

Undue weight really deals with a minority viewpoint of an issue - not really applicable here. As far as relevancy is concerned: given the circumstances it is understandable that there are alternate views on how the review process went here. But there isn't any reason to argue about that. The fact is that this is the clearest example I have seen of the powers-that-be censoring or squelching some portion of the review process, so unless a better example can be given and cited this example really ought to be included. Otherwise the preceding paragraph is a bunch of theoretical blah-blah without much meat. standonbible^Talk! 04:54, 16 November 2006 (UTC)

I mentioned above two far more prominent examples of failure of the peer review system. Including either or both would make sense prior to this one (although I wouldn't object to all three being mentioned). JoshuaZ 04:57, 16 November 2006 (UTC)

Mentioning all three sounds good to me! I don't know much about the other two examples, though, so can I put the Sternberg example in until the other two can be added by editors more knowledgeable on the subject? Whatever you think. standonbible^Talk! 05:07, 16 November 2006 (UTC)

The Sternberg case is not an example of censorship in peer review; that it is a particular viewpoint, Sternberg's and the pro-ID/creationist one, and a viewpoint you've been promoting across the science/evolution articles of the project. There's no recognition within the scientific community or even the popular press (barring a few op-ed puff pieces from ID fellow travelers) that the Sternberg case is anything more than what the publisher says and the evidence indicates -- that Sternberg stepped outside of normal process to give something his ID cronies have desperately sought. We won't be using the Sternberg case as an example of peer review censorship here. In fact, censorship in peer review is not a common complaint at all. FeloniousMonk 05:08, 16 November 2006 (UTC)

So are the New York Times, the Washington Post, the Opinion-Journal, WorldNetDaily, the Discovery Institute, NPR, the Washington Times, the World Socialist Web Site, PhysOrg.com, and the National Review all "ID fellow travelers"? There is no hint of an ID or creationist viewpoint in the proposed edit. Why do you disagree with JoshuaZ?

I've explained why: because as the Sternberg peer review controversy article explains, it is only Sternberg's and other creationist's opinion that his imbroglio is an example of censorship in peer review. In the scientific community, there's no acceptance of that viewpoint, which they rightly point out is in line with the ID crowd's (of which Sternberg is a notable member) agenda. I'm very, very familiar with the articles and sources, and the Washington Post article, the WSJ's Opinion-Journal article, WorldNetDaily, the Discovery Institute (!! as if...) and the National Review are all ID fellow travelers. If you are unaware that the Discovery Institute is the primary promoter of ID, and that Sternberg is affiliated with it, and that its fellows have written many of the very articles you cite, then you probably shouldn't be making objections or alterations to this article's content. FeloniousMonk 05:33, 16 November 2006 (UTC)

My mistake with mentioning the Discovery Institute; I Googled it and somehow mentally confused the Discovery Institute with a branch of PBS - don't know why.

Of course the Sternberg peer review controversy was controversial! But it doesn't matter if/that what happened actually qualifies as peer review censorship - it is still a good example of peer review controversy that should go into the "criticisms of peer review" section. You can't explain that away. A controversy for a section discussing controversy. standonbible^Talk! 05:43, 16 November 2006 (UTC)

Not really... One creationist abusing his position as editor to help out his buddies does not speak to any greater issues within peer review itself. It doesn't really belong here, it's more appropriate to the ID articles. FeloniousMonk 05:48, 16 November 2006 (UTC)

One creationist research associate with ID leanings being fired for allowing a controversial article through and for exercising his authority in a controversial manner is relevant to criticisms of peer review which say that the "elite" can easily squelch minority viewpoints. I suggest we wait until JoshuaZ gets back to see what he says. standonbible^Talk! 05:53, 16 November 2006 (UTC)

Clearly you're not up to speed on the Sternberg matter: 1) He was not a "research associate" at the journal, he was an editor, 2) he was not fired from the journal, was already leaving when published Meyers paper as his one of his last acts. 3) The Smithsonian, where Sternberg was an unpaid research associate did not fire him, he quit. If you can't get the basic facts in this case right, why should your proposal carry more weight than simply being your personal opinion. FeloniousMonk 15:59, 16 November 2006 (UTC)

On the subject the journal itself said: "The Council, which includes officers, elected councilors, and past presidents, and the associate editors would have deemed the paper inappropriate for the pages of the Proceedings because the subject matter represents such a significant departure from the nearly purely systematic content for which this journal has been known throughout its 122-year history." So they disagreed not only on the grounds that he was a poor choice to evaluate the article (because he is affiliated with the DI) but because it was simply inappropriate material. This really isn't a controversy over peer review, though people have tried to inflate it to that. Had he done the right thing and put it through normal procedures it would never have become an issue in the first place. --Davril2020 12:02, 16 November 2006 (UTC)

I don't understand something. You just said that the Council rejected the paper because it was "inappropriate" - it was departing from systematic Darwinistic thought. So therefore it had nothing to do with peer review? The point that is being made in this section is that sometimes the elites may, through their power over the system, squelch unorthodox viewpoints. When something like this was published, the "elites" did exactly that by firing the publisher - and you yourself admitted that it wasn't just his method of getting it through but the contents of the paper itself. Elites-use-power-to-squelch-inappropriate-content.

Until someone can write up a good summary of the two "more prominent examples" of elites abusing the peer review system to censor "inappropriate" content, this remains the best example of that. Perhaps an RfC would be good here so that we can get some outside viewers on this debate - or we should just wait until JoshuaZ gets back. standonbible^Talk! 13:18, 16 November 2006 (UTC)

I can only assume that you are unfamiliar with the journal's response. The comment on it being inappropriate had nothing to do with the article backing of ID, but the fact that the journal simply did not publish articles of that form or type within the journal. The journal was notable for its 'systematic content' - this was its specialism and virtually all it published. The article in question was a literature review. There were purely technical reasons as well as ethical reasons for being opposed to this. Frankly, had it been a literature review supporting a particular mechanism of evolution it would probably have failed peer review as well. --Davril2020 15:52, 16 November 2006 (UTC)

You've got a good point, Davril. That was information I was unaware of. Of course, the comments made by the other scientists calling Sternberg a "closet Bible thumper" would lead me to believe that their rejection of the article and firing of Sternberg wasn't purely for structural reasons concerning the paper ... but I think that in any case this argument will not be much more productive. It would probably be good if someone could do a write-up of the examples JoshuaZ cited. standonbible^Talk! 15:59, 16 November 2006 (UTC)

If we wanted to do an RfC, it could look like this:

Controversy: Whether the Sternberg incident is a noteworthy and relevant example of the scientific elite squelching viewpoints that challenge mainstream systematic thought by any means necessary.

Side one: This is a perfect example of such censorship because all parties involved admit that the content of the paper was a consideration in the action that was taken (firing Sternberg) and, despite argumentation over the peer review process that was taken, this type of incident serves to exemplify a flaw in the peer review process. Thus, it is good for inclusion in the Criticisms of Peer review section. standonbible^Talk! 13:18, 16 November 2006 (UTC)

Side two: As the Sternberg peer review controversy article explains, it is only Sternberg's and other creationist's opinion that his imbroglio is an example of censorship in peer review. In the scientific community, there's no acceptance of that viewpoint, which they rightly point out is in line with the ID crowd's (of which Sternberg is a notable member) agenda. The Sternberg matter received almost no coverage outside the usual venues where creationism/ID are discussed. (sign name here)

Whether there are more relevant or notable examples does not need to come up. Of course you can edit side two as much as you want. I suggest that we wait until JoshuaZ returns before filing any kind of RfC but it's something that we might keep in mind as a way to get a bit more dialogue on this subject. standonbible^Talk! 13:18, 16 November 2006 (UTC)

Regarding the proposed insertion:

• "Conspicious examples of censorship in the realm of peer review are articles with potential religious connotations" . "Conspicuous examples"? More than one? Really? There's one example, and it really wasn't "conspicuous". Storm in a tea pot.
• "Deviations from mainstream scientific thought that can be related to religious viewpoints can be quickly censored by the "elites"" - again, any examples of this? The Sternberg issue is about dishonesty and abuse of his position, not about "censorship" (and by "elites"? How funny that major political players who are part of the political estabishment call underpayed academics "elites".
• "Even if an article passes the peer review process, its publishing can have disastrous effects for the publisher" - there is no evidence that the Meyers article passed peer review - neither Sternberg nor Meyers is able to provide copies of the reviews. The article lacked scientific merit and was outside of the scope of the journal.
• "In August of 2005, Richard Sternberg, a research associate with the Smithsonian Institution, was fired ..." - nope, he stuck the paper in the journal on his way out the door.
• "Senior scientists at the Smithsonian Institution called Sternberg a “shoddy scientist” and a “closet Bible thumper,” despite his protests that he is agnostic about Intelligent Design and does not hold any sort of creationist viewpoint.[1]" - what does this have to do with peer review?

So, apart from the fact that this is irrelevant inasmuch as it is a triviality that has been used as a crutch to explain the total lack of science generated by the creationist movement (and, in fact, the failure of people to even apply for grants offered to support scientific endeavours by creationists), the proposed text is factually inaccurate. Get your facts straight first, then discuss the value of including this trivia. Guettarda 13:36, 16 November 2006 (UTC)

Remember " peer reviewed " journals supports unscientific (??) theories that are not falsifiable and that can not be tested such us String Theory, Slowing time theory, Human cloning , DNA coming from space and so on , why this should be realiable ? , does they point only your mistakes or censor your ideas because they do not like them or are unpopular ? , when Einstein or other (before becoming famous) were turned down by the 'fair' peer review process was this unfair or not, what were the reviewers of Bogdanov affair or Alan Soka hoax thinking when they published such nonsenses ? , the Bible Code theory has appeared in many journals does this mean it is true --85.85.100.144 (talk) 14:39, 19 February 2008 (UTC)

Another Section in Question

Another section which deserves some attention is the section on "Peer Review Failures", in which there are about three paragraphs talking about the Center for Regulatory Effectiveness (CRE)'s dissatisfaction with an editorial published in the Journal of the American Medical Association by Linda Rosenstock. While this may be of interest to some readers, I do not believe it warrants as long an explanation as is currently being provided. Its list of specific complaints, in particular, does not seem relevant to a general discussion of peer review failures. Could we not just add an external link to the CRE's website so interested parties can find out more about the controversy regarding the JAMA editorial? It seems to me that that would be more appropriate.

(Yes, I previously removed this section for the same reason, and now see that it is back. While the poster clearly made an effort to be more objective in his/her discussion, I don't believe such a detailed discussion of this particular controversy is warranted or appropriate for a general article on peer review. Does anyone else have thoughts on this? Thanks.Chrissy385 23:15, 5 December 2006 (UTC)

peer review failures

There are quite a few additional ones to add. Before I add them, I want to consider how:

To avoid excessive controversy like the above, perhaps it might be better to make an article for each of the truly important cases, and give a link. It will focus the discussion where it belongs. The major ones are notable enough, and usually get coverage from various science news sources, not to mention newspapers. DGG 22:45, 15 January 2007 (UTC)

If an example generated controversy, then it generated notability. --Iantresman 23:14, 15 January 2007 (UTC)

Couldnt agree more--the question is simply where to best put them. I think the best way might be to give each their own article, and then have a category Peer review controversies, or a List of peer review controversies. There is a lot more to be said about peer review in general, and the individual examples and the comment will overbalance the rest. Especially when all the other go in: stem cells, Lucent, etc. etc. I'll see it if you reply here. DGG 00:49, 16 January 2007 (UTC)

I continue to be troubled by the paragraphs under the peer review failures section added by the CRE. I think this section is inappropriate for this article. Can we remove it until we decide where to add it? Chrissy385 19:40, 4 June 2007 (UTC)

external links

Cleaned up on the basis of removing self-publishing, blogs, listservs, book reviews, etc. Still needs clean-up based on quality. Even better would be to cite some of the good ones in the article and get them in the reference list. DGG 01:04, 24 January 2007 (UTC)

Lack of Evidence Favoring Peer Review As Quality Filter

Recent, authoritative meta-analyses of the scientific literature on the subject of peer review have concluded that there is no good evidence favoring it as a method to ensure (or even to increase the probability of) manuscript quality, veracity, reliability, or anything else.

According to a 2006 Cochrane systematic review -- the "gold standard" of evaluation of the biomedical literature, if anything is -- there exists an "absence of evidence" for the effectiveness of peer review for assuring manuscript quality. "We could not identify any methodologically convincing studies assessing the core effects of peer review", say the Cochrane reviewers.[1] Likewise, an earlier (2002) systematic review published in the JAMA concluded that "Editorial peer review, although widely used, is largely untested and its effects are uncertain".[2]

Most striking is the suggestion by Linkov et al in the Journal of the Royal Society of Medicine (2006) that belief in peer review is faith-based, rather than science-based: "we scientists have almost complete faith in the journal process as right and unassailable. We thus take a 'faith based' approach to research communications... Questioning peer review is like questioning the Bible, Quran or Torah."[3]

Even before the Cochrane review was published, doubts about the efficacy of peer review had mounted.[4]

I submit that if there is a lack of evidentiary basis for the value, in measurable terms, of peer review (as there is), then the invocation of it as some sort of Holy Grail of scientific reliability or veracity is inappropriate and indeed misleading. Those who bandy it about in such a way should be corrected, and referred to the relevant literature for study and self-education. All approving mentions of peer review, or mentions of it that suggest that it is essential or even important, should be disregarded as reflective of ignorance of the best and most-careful biomedical thinking and research currently available on this subject.

At this moment, both peer-reviewed and non-peer-reviewed publications have comparable status. Judgements about their credibility or quality must be made on other grounds.

Anyone who disagrees with the foregoing is encouraged to present compelling contrary documentation, comparable to what I presented; e.g. a contrasting Cochrane systematic review, or some other careful review of the literature on the subject.

References

1. http://www.cochrane.org/reviews/en/mr000016.html -- Cochrane Systematic Review: Editorial peer review for improving the quality of reports of biomedical studies

2. http://jama.ama-assn.org/cgi/content/abstract/287/21/2784 -- JAMA Vol. 287 No. 21, June 5, 2002, Effects of Editorial Peer Review: A Systematic Review

3. http://www.jrsm.org/cgi/content/full/99/12/596 -- J R Soc Med 2006;99:596-598, Scientific Journals are `faith based': is there science behind Peer review? Faina Linkov et al

4. http://www.infotoday.com/it/apr03/peek.shtml -- Information Today, Vol. 20 No. 4 - April 2003, Could Peer Review Be Wrong?

................................... SNIPPETS from Linkov et al, www.jrsm.org article: [...snip...] Jefferson recently presented an outstanding review of peer review and could find only 19 studies on peer review that were scientifically sound.... As Jefferson has pointed out, there are almost no data suggesting that the existing peer review systems work and none to suggest that they are better than any other system. [...snip...] Why hasn't peer review, IMRaD, the editorial decision process and the overall journal process evolved into a new form of research communication? We would argue that the reason is that this has been due to the almost non-existent use of the scientific method to question and test the publication process itself.... [journals] are `faith based': we believe in them, we dare not question them. [...snip...] Isn't it strange that three features that are inherent to research communication have not been looked at scientifically? There are several possible reasons for this. The most likely is that we scientists have almost complete faith in the journal process as right and unassailable. We thus take a `faith based' approach to research communications. Faith is defined as a firm belief in something for which there is no proof. Many of us might view questioning of the journal process as an attack on science itself. Clearly, the scientific journal process is not a part of the scientific method. We are taught early in our training about the importance of learning to write articles (e.g. IMRaD), the power of peer review and a belief in the editorial system. We do not question the process, despite the fact that the essence of science is questioning. Questioning peer review is like questioning the Bible, Quran or Torah. One role of science is to help separate science from dogma, which we should now do with journals, and avoid a faith based approach. [...snip...] It is the scientific method that is central to science, not the scientific journal. The scientific method should be central to other research communication processes, but it is not and has not been used to continuously improve how we communicate research. Because of this, we are forced into a conundrum -- we cannot change the process if the process if based upon faith, not data.

-- Alan2012 14:59, 5 February 2007 (UTC)

I think the Cochrane study by Jefferson is important enough to be worth a section in the article, under a neutral heading. Remember that the conclusion was not that peer-review had been shown to be worthless. Any such discussion should include a discussion of subsequent criticism of their evaluation. DGG 02:53, 6 February 2007 (UTC)

No, of course it did not show peer-review to be worthless; that would be impossible. Please give links to the subsequent criticism, of which you are aware. Thanks. Alan2012 22:43, 19 March 2007 (UTC)