Szakértői bírálat (peer review) a tudományos kutatásban Válogatott tanulmányok a téma szakirodalmából U

U I N F O R M A T I K A

ÉS T U D O M Á N Y E L E M Z É S

BRAUN TIBOR . SCHUBERT ANDRÁS

SZERKESZTŐK

S z a k é r t ő i b í r á l a t ( p e e r r e v i e w )

a t u d o m á n y o s k u t a t á s b a n V á l o g a t o t t t a n u l m á n y o k

a t é m a s z a k i r o d a l m á b ó l

BUDAPEST. 1993

s z e r k e s z t ő k

SZAKÉRTŐI BÍRÁLAT ( P E E R REVIEW) A TUDOMÁNYOS KUTATÁSBAN

Válogatott tanulmányok

a téma szakirodalmából

AKADÉMIA

KÖNYVTÁRÁNAK INFORMATIKAI ÉS TUDOMÁNYELEMZÉSI SOROZATA

7 .

S o r o z a t s z e r k e s z t ő :

B r a u n Tibor

Előszó 3

EUGENE GARFIELD: Refereeing and Peer Review. Part 1. Opinion and

Conjecture on the Effectiveness of Refereeing 5

EUGENE GARFIELD: Refereeing and Peer Review. Part 2. The Research on

Refereeing and Alternatives in the Present System 15

EUGENE GARFIELD: Refereeing and Peer Review. Part 3. How the Peer Review of Research-Grant Proposals Works and What Scientists Say

About It 25

EUGENE GARFIELD: Refereeing and Peer Review. Part 4. Research on the

Peer Review of Grant Proposals and Suggestions for Improvement 31

DOMENIC V. CLCCHETTI: The Reliability of Peer Review for Manuscript and

Grant Submissions: A Cross-Disciplinary Investigation 39

IAN I. MITROFF and DARYL E . CHUBIN: Peer Review at the N S F :

A Dialectical Policy Analysis 107 RUSTUM ROY: Alternatives to Review by Peers: A Contribution to the Theory

of Scientific Choice 141

ALAN L. PORTER and FREDERICK A . ROSSINI: Peer Review of

Interdisciplinary Research Proposals 155

ANGELO S. DENISI, W . ALAN RANDOLPH a n d ALLYN G. BLENCOE: P o t e n t i a l

Problems with Peer Ratings 161

MARTIN RUDERFER: The Fallacy of Peer Review: Judgement without Science

and a Case History 169

(A cikkek reprint formában való közzététele a copyright tulajdonosok írásos engedélyével történt.)

s z e r k e s z t ő k

SZAKÉRTŐI BÍRÁLAT (PEER REVIEW) A TUDOMÁNYOS KUTATÁSBAN

Válogatott tanulmányok a téma szakirodalmából

Magyar Tudományos A k a d é m i a

Könyvtára

B u d a p e s t - 1 9 9 3

L i b r a r y o f t h e H u n g a r i a n A c a d e m y o f S c i e n c e s

I S S N 0 2 3 0 - 4 6 1 9 I S B N 9 6 3 7 3 0 2 8 6 7

Felelős kiadó: az MTA Könyvtára főigazgatója Alak: B / 5 - T e i j e d e l e m : 1 7 , 2 ( A / 5 ) ív

Megjelenés: 1993 — Példányszám: 500 Nyomdai előkésztés: W&T Consulting Kft.

Készült az MTA Könyvtára házi sokszorosító részlegében

Láng István, akadémikus

A Magyar Tudományos Akadémia vezetői a hetvenes évek második felében felismerték, hogy mielőbb szükséges megteremteni a szellemi és műszaki feltételeket ahhoz, hogy a magyar tudományos kutatás publikációs tevékenységét és annak ered- ményességét mennyiségi és minőségi mutatókkal lehessen jellemezni. Ennek érdeké- ben az MTA Könyvtára, és azon belül az Informatikai Igazgatóság felépítette a Magyar Természettudományos Alapkutás Publikációs Adatbankját, továbbá megteremtette a pénzügyi lehetőségeket az Institute for Scientific Information (USA, Philadelphia) által kiadott Science Citation Index gépi szakirodalom-figyelő szolgál- tatás hazai adaptációjához. Közép-Kelet-Európában Magyarországon jött létre első- nek ez az új, számítógépes rendszer, amely a szakirodalmi információigények magas szintű ellátása mellett, nagy hatással volt a tudományos kutatók publikációs stratégiá- jára. A neves nemzetközi folyóiratokban való publikálás igénye minőségi változást

hozott és ez a váltás nálunk előbb következett be, mint a többi közép- és kelet- európai országban. Ez jól tükröződik a Nature 1993. január 14-i számában (361.

kötet, 104. old., 1993) közölt ábrán, ahol ezen országok tudományos közleményeinek átlagos idézettségét mutatják be 1981 és 1990 között. A vizsgált időszakban a magyar publikációs mutatók meghaladják a többi országét. Ez az eredmény elsősorban a kellő időben felismert új publikációs stratégia megvalósításának következménye, de természetesen a jelentős és eredeti tudományos eredmények elérése is szükséges feltétel volt.

A nyolcvanas években a természettudományos kutatások értékelésénél, továbbá személyek és kutatócsoportok tevékenységének megítélésénél széles körben használták a publikációkra vonatkozó adatokat. Ez jelentősen hozzájárult az eredmé- nyesség objektív megítéléséhez. A módszer propagálói mindig hangsúlyozták, hogy a publikációs adat csupán egyike az értékelésnél használandó mutatóknak, mely számos egyéb adattal, információval kiegészítve adhat alapot a döntésekhez. Ennek ellenére túlzások, egyoldalú lekicsinylések, vagy ellenkezőleg, a publikációs adatok- nak vélt kizárólagossága egyaránt előfordult.

Az utóbbi években világszerte felerősödött az igény a szakértői bírálat (peer

review) metodikájának, legyen az egyéni vagy csoportos jellegű, jobb megismerésére

és szakavatottabb használatára az elbírálások és minősítések során. Kétségkívül

vannak fontos tényezők, amelyre a tudománymetria alig tud választ adni, a szakértői bírálat viszont képes lehet a válaszadásra. Ilyen pl. egy pályázatnál az eredetiség megítélése.

Szinte minden olyan jelentésben, amelyet amerikai vagy nyugat-európai tudósok írtak a magyar tudomány jelenlegi helyzetéről, megtalálható az az ajánlás, hogy fordítsunk nagyobb figyelmet a jövőben a peer review módszer szakszerű alkal- mazására és ahol lehet és indokolt, ott a tudománymetriai módszerekkel össze- kapcsolva hajtsuk végre az értékelési és elbírálási munkákat.

Ezt az igényt kívánja részben kielégíteni a jelen kiadvány, amelyben a peer review módszer alkalmazásáról, sajátos problémáiról találhatunk eredeti tanul- mányokat. Angol nyelven adjuk közre ezeket a cikkeket. A magyar nyelvre való lefordítás egyrészt jelentős többletkiadást igényelt volna, de ettől függetlenül is úgy érezzük, hogy a magyar tudományos kutatók döntő többsége ma már jól olvas angolul és megérti a kutatások értékelésével foglalkozó módszertani cikkeket.

Kívánom, hogy hasznosítsák mindazokat a gondolatokat, amelyek ezekben a cikkekben találhatók, és amelyek valóban újak és tényleg hasznosíthatók számunkra.

Nem a nulláról indulunk a szakértői bírálati módszer alkalmazásánál, de van még mit

tanulnunk és elsajátítanunk olyanoktól, akik valóban magas szintű módszertani

vizsgálatokat végeztek.

Refereeing and Peer Review. Part 1.

Opinion and Conjecture on the Effectiveness of Refereeing Current Contents, August 4,1986

Peer review is so much a part of the fabric of scholarly inquiry that it is often taken for granted. I have written many essays over the years that are directly or indirectly related to peer review. These include several on authorship

"

and editing,

faculty evaluation,

identifying Nobel-class science through citation analysis

—and even a few on various aspects of refereeing itself.

"

But I have never before discussed the intrica- cies of the system in detail. Since the subject is central to scholarly life, we have decided to devote a three-part es- say in Current Contents® to it.

The first two parts will cover referee- ing for publication. Part 1 examines how the refereeing system works and lists some of the common opinions about its advantages and disadvantages. Part 2 will cover scientific studies of refereeing and some proposed alternatives to the present system. Part 3 will follow later and will focus on the peer review of grant proposals. Note that I distinguish between a referee (one who evaluates an article before it is published) and a re- viewer (one who evaluates already pub- lished material or, in the case of grant re- views, research-grant proposals). I gen- erally use the term referee to mean one who advises editors on the publishability of a scholarly manuscript. The process by which this advice is solicited I usually call refereeing, but occasionally review-

ing or peer review seems appropriate.

The term peer review is also used to de- note the evaluation of research propos- als; more generally, it can refer to the professional review of patient records by special committees of physicians that many hospitals use to maintain high- quality patient care.

Refereeing: How It Came About and How It Works

Refereeing is meant to ensure that ar- ticles submitted for publication meet the accepted standards of their fields. Like editing, refereeing is a complex intellec- tual, political, and social process; it of- ten involves a spectrum of activities that blend into one another in complex ways, in a fashion similar to the range of prac- tices related to ghostwriting.

Among many who have expressed the idea, Peter Amiry, former editor, Journal of the Operational Research Society, wrote in an editorial that referees are an editor's insurance policy, providing a reservoir of knowledge that few editors could hope to match.

The practice of refereeing manu-

scripts prior to publication is now well

established, but it was not always so,

state sociologists Harriet Zuckerman

and Robert K. Merton, Columbia Uni-

versity, New York, in their classic 1971

study of patterns of evaluation in sei-

ence.

It evolved in response to the de- velopment of scholarly societies and the scientific journal. I summarized this and other work in an earlier essay on the changes in scientific communication over the past 300 years.

According to David A. Kronick, pro- fessor of medical bibliography, Universi- ty of Texas Health Science Center at San Antonio, "science in the seventeenth and eighteenth centuries...differed ir many ways socially, intellectually, am economically from the science of tin- twentieth century."

Although associ- ations and societies promoting scholarly activities had existed for hundreds of years,

(p. 46), the social role of "scien- tist," as well as conventions for doing research, had yet to emerge.

In fact, Kronick notes, "individuals did not begin to regard themselves as scientists rather than philosophers until the seven- teenth century."

(p. 34)

The learned journal as we know it to- day also traces its origins to the seven- teenth century, with the founding of the Philosophical Transactions of the Royal Society of London and the Journal des Sqavans, associated with the Académie des Sciences in Paris.

By the early eighteenth century, Kronick says, mem- bers of these and other scholarly soci- eties sponsoring official or, semiofficial publications began to realize that if scholars were to have confidence in the content of these journals, then material submitted for publication had to be critically evaluated before it was pub- lished.

Societies thus began to take measures to preserve their credibility. Some adopted strict regulations governing publication that members had to comply with to retain their membership. And by the mid-eighteenth century, according to Kronick, some—such as the Royal Society of Medicine of Edinburgh, Scotland—had developed techniques of evaluating and approving manuscripts before publication that are almost in-

distinguishable from today's system of refereeing.

Kronick, incidentally, is the author of a recent book on the literature of the life sciences that in- cludes a short section on the refereeing and the publication process in that branch of science.

The procedures involved in refereeing a manuscript vary from journal to jour- nal and from field to field, but there are certain general steps that virtually every paper has to go through before it is pub- lished. Among the first steps an editor takes, whether or not the journal is ref- ereed, is to evaluate a submission's com- patibility with the scope and style of the journal, according to Robert A. Day, consultant, ISI Pres^

, and former man- aging editor, American Society for Mi- crobiology (ASM) journals.

Once this is done, an editor must then choose ap- propriate referees for a given manu- script.

Donald Christiansen, editor, IEEE Spectrum, conducted a survey of referee selection practices among 26 of the IEEE Transactions editors. Common sources from which referees are recruit- ed include widely recognized experts, members of a journal's editorial board, professional acquaintapces, previous referees, and scientists cited in the au- thor's references.

Sometimes authors are asked to supply a list of suggested referees. A few journals are using manu- al and computei^assisted bibliographic retrieval methods to select referees. For example, Stevan Hamad, editor, Behav- ioral and Brain Sciences (BBS), reports that BBS staffers search a microcomput- er file of the journal's referees that has been coded by areas of expertise. They also search the current biobehavioral literature through the Science Citation Index® and the Social Sciences Citation Index® for additional referee candi- d a t e s .

^

Usually two referees are chosen, ac-

cording to Claude T. Bishop, director,

Division of Biological Sciences, National

Research Council of Canada (NRCC), and editor-in-chief, NRCC Research Journals. "The merits of this system," he writes, "are that it usually provides at least one solid [report], that the two [ref- erees] can be checked against each other, and that one referee may cover points that the other missed."

But Har- nad notes that, for many journals, the

"number of referees [selected for a manuscript] is an empirical matter re- quiring research."

BBS uses five to eight referees per paper. In Hamad's ex- perience, such a sample is more likely to produce a balanced review.

Along with the manuscript, referees generally receive a list of instructions and a form for comments and recom- mendations. Routinely, referees re- spond within a few weeks, recommend- ing either publication or rejection or re- questing modifications; they often in- clude specific comments for both the au- thor and the editor.

A paper is most likely to be accepted, according to Michael Gordon, research associate, Primary Communications Re- search Centre, University of Leicester, UK, when the referees agree that it meets three criteria.

(p. 6-8) First, it should be sound. The authorfs) should have employed reliable research tech- niques, drawn valid conclusions, and committed no flaws of logic. It should also be original, in the sense that ii.. End- ings have never before been published.

Finally, it should be significant, meaning that it should contain some new perspec- tive or observation of potential impor- tance.

(p. 6-8) Of course, published ar- ticles meet these criteria in varying de- grees.

Referees do not always agree with one another, and some authors take this as evidence that the system is unreliable or capricious. But disagreement is at the heart of scientific inquiry. Hamad says that "the current and vital ongoing as- pect of science consists of an active and often heated interaction of data, ideas,

and minds, in a process one might call 'creative disagreement.' "

Moreover, reviewer disagreements are not simply shrugged off; editors generally resolve each dispute on an individual basis. Gor- don described some of the options open to editors for dealing with these con- flicts.

(p. 20-5) When reviewer dis- agreements are mild, for example, edi- tors may rely on their own judgment to resolve them—with, perhaps, some communication with the author.

(p.

21) When differences are profound, edi- tors may reject the paper without further reviewing or they may send the manu- script out for review once again, togeth- er with the comments of the disputing referees. Editors may also ask the author to respond to the referees' observations.

After the "arbitrating" referee(s) and the author have reported, editors should be in a better position to make a final judg- ment. When authors take exception to referees' comments and provide editors with a point-by-point refutation, editors often follow a procedure similar to the one just outlined for adjudicating dis- putes between referees.

(p. 22-5) Research, Pseudo-Research, or Non-Research?

The results of our literature search for

this essay support the view that referee-

ing is an issue clouded with subjectivity

and emotionalism—at least for a vocal

minority. The dominant vehicle of dis-

cussion in the debate about the effec-

tiveness of refereeing has been editorials

and correspondence. Some contain inci-

sive discussions, but with little or no em-

pirical evidence to support what

amounts to a litany of opinion and anec-

dote. Indeed, in an endeavor such as sci-

ence, which depends on dispassionate

logic and systematic evidence for much

of its credibility, the dearth of rigorous

thinking and hard data in the corre-

spondence of many who are critical of

refereeing is remarkable. Of the relative-

ly few controlled studies that have been done, many suffer from such severe methodological shortcomings that their conclusions are questionable. More will be said about research on refereeing in Part 2.

Refereeing and other forms of peer re- view have been discussed at length, especially in the four decades since World War II, but discussion alone does not constitute science or scholarship.

Since we are all affected by peer review, it is not surprising that so many of us have opinions on the subject. Yet the lit- erature representing controlled studies of peer review is either pitifully small or disgracefully absent, while the body of anecdote and opinion is quite large. We carefully distinguish here between stud- ies, experiments, experience, and opin- ions.

In researching this essay, we also found that most published opinion on refereeing is negative. But we suspect that this is due, ironically, to the wide- spread acceptance of and satisfaction with the current system of peer review:

most scientists simply do not feel that refereeing needs defending, so positive opinions are relatively scarce. It should also be kept in mind that these opinions on refereeing are themselves unrefer- eed. Furthermore, the existence and ranking of hundreds of refereed journals is concrete evidence that they are the preferred medium of publication.

Fbws in the System?

In a note published in the New En- gland Journal of Medicine (NEJM), John C. Bailar III and Kay Patterson, Harvard School of Public Health, Boston, Massa- chusetts, speculate that current opinion on refereeing seems divided among one or more of four paradigms.

Based on their own informal observations, the au- thors assert that many scientists seem to

perceive the process as a sieve, sifting the wheat from the chaff. Many also liken the process to a smithy, in which

"papers are pounded into new and better shapes between the hammer of peer re- view and the anvil of editorial stan- dards." Some seem to view it as a switch, reflecting the widespread belief that a persistent author can eventually publish a manuscript somewhere (although ref- ereeing may determine exactly where).

Finally, some scholars seem to consider refereeing a capricious and essentially unpredictable process—a "shot in the dark."

Stephen Lock, editor, British Medical Journal, feels that refereeing "favours

unadventurous nibblings at the margin of truth rather than quantum leaps."

An example supporting his opinion is the reception given the early demonstration, via radioimmunoassay, of insulin-bind- ing antibody by the late Solomon A. Ber- son and Rosalyn S. Yalow, Veterans Ad- ministration, New York. This work was fundamental to the development of the radioimmunoassay into a "powerful tool for determination of virtually any sub- stance of biologic interest," according to Yalow.

Although Yalow would share the 197-7 Nobel Prize with Roger Guille- min, Salk Institute, San Diego, and Andrew Schally, Veterans Administra- tion Hospital, New Orleans, the initial research concerning radioiodine-labeled insulin was rejected both by Science and, at first, by the Journal of Clinical Investigation (JCI) as erroneous.

Nevertheless, when the paper was re-

vised to meet the objections of review-

ers, it was published in the JCI.

A com-

paratively recent poll of the authors of

manuscripts rejected by the JCI, con-

ducted by editor Jean D. Wilson, De-

partment of Internal Medicine, Univer-

sity of Texas Health Science Center at

Dallas, found that 85 percent of the re-

jected papers were subsequently pub-

lished elsewhere. And Wilson also re-

ported that "most of the authors of the [other] 15 percent...were convinced by the review process that [their papers]

were either unoriginal or wrong."

?

Delays in Publication

In addition to charges that referees make too many serious mistakes, com- plaints also focus on the delays in publication that many attribute to the refereeing process. While conceding the value of thorough, constructive reports by referees, Richard Shea, editor.

Transactions on Nuclear and Plasma Sciences, is nevertheless concerned about the time lost during the refereeing process; he is quoted by Christiansen as saying that "the ultimate referee is the reader."

And as noted by Kronick, the historical significance of papers ulti- mately depends on this reader evalu- ation and readers' willingness to cite what impresses them.

But one of the reasons for the existence of the referee- ing system is that readers of scientific ar- ticles have varying interests and back- grounds; they must be able to rely on a high degree of validity in what they read, especially if it is Somewhat outside their field.

Real or perceived, delays in publica- tion resulting from refereeing may be the most prevalent concern among scien- tists, who may have job security, promo- tions, or the need to establish priority for a discovery hanging in the balance. In a note in NEJM, Thomas P. Stossel, Mas- sachusetts General Hospital, Boston, voices his concern that the commercial potential of many new discoveries, espe- cially in biotechnology, is giving rise to new and particularly taxing demands for rapid publication.

In an editorial, Lawrence D. Grouse offers several explanations, based on his experience as senior editor of JAMA, for

the lag time between submission and publication: "Excellent manuscripts are often criticized by reviewers with vested interests or contrary views. Overcritical reviewers flay manuscripts for minor or supposed deficiencies.... Reviewers may also cynically delay the appearance of research competing with their own."

And in a 1979 editorial in the Journal of Clinical Psychiatry, associate editor Marc H. Hollender asked "why it takes three months or longer to review an arti- cle that takes three minutes to read and perhaps took less than three months to write.... Does it take the referee that long to come to a conclusion and to dic- tate comments? It is more likely that the article gathers dust among other low-pri- ority items."

In short, if I may use an old, informal phrase, referees should either fish or cut bait.

Bias and Unethical Behavior

Of all the complaints about referee- ing, however, some of the most bitter—

though not the most prevalent—concern the issue of referee bias (although little uncontested empirical evidence exists to indicate that authors' affiliations and the reputations of their institutions affect a referee's evaluation). Assuming that some bias exists, however, historian of science Donald deB. Beaver, Williams College, Williamstown, Massachusetts, suggests that a preconceived suspicion of scientific "have-nots" may be ex- plained in terms of the second part of the

"Matthew effect."

This concept, intro-

duced by Merton in 1968,

draws an

analogy between the misallocation of

scientific credit and a passage from the

gospel of St. Matthew: "Unto every one

that hath shall be given, and he shall

have abundance: but from him that hath

not shall be taken away even that which

he hath" (emphasis ours). Presumably,

contributions from unknown scholars

from unrecognized or little-known insti- tutions are less likely to be accepted for publication than occasionally compara- ble contributions by scholars of great repute.

Some cases of questionable referee ethics have been documented. Perhaps the most publicized example, according to a 1984 article by free-lance medical writer Barbara Fox in Medical Commu- nications, the journal of the American Medical Writers Association,

was one reported on by former Science staff writer William J. Broad.

It involved a paper submitted by Helena Wachslicht- Rodbard, NIH, Bethesda, Maryland, to NEJM. The paper was assigned to two referees, one of whom recommended acceptance, while the other—Vijay Soman of Yale University, who had similar research in progress—recom- mended rejection. Arnold Relman, editor, NEJM, informed Wachslicht- Rodbard that her paper had "engen- dered considerable differences of opin- ion among our referees"

and told her the manuscript was unacceptable unless revised.

But the matter was far from over.

Soman had photocopied Wachslicht- Rodbard's study and, without informing his coauthor, Philip Felig, vice chairman of the Department of Medicine at Yale, of what he had done, sent their article in- corporating the plagiarized data to the American Journal of Medicine, of which Felig was an associate editor. By coinci- dence, the journal sent the article out for review to Wachslicht-Rodbard's su- perior, who showed it to her. It con- tained more than a dozen passages, ver- batim, from her own manuscript; she wrote to Relman accusing Felig and Soman of plagiarism and conflict of in- terest in the refereeing of her paper. Rel- man agreed that it had been highly im- proper for Soman to agree to even read the paper, which was later published in the NEJM under Wachslicht-Rodbard's name.

The abuse of anonymity is a long- standing matter of concern. In an article appearing in New Scientist, biochemist Robert Jones, Royal College of Sur- geons, London, asserted that "the act of submission of a paper can place the author at the mercy of the malignant jealousy of an anonymous rival."

The belief seems to be that, from behind the walls of their fortress of anonymity, ref- erees are free to hurl at authors volleys of invective that cannot be effectively countered. "Anonymity tends to bring out the worst in people," according to Heinz Fraenkel-Conrat, Department of Molecular Biology and Virus Laborato- ry, University of California, Berkeley, in a letter to the editors of Nature A

"I was recently asked to review, and advocated rejection of, a paper for a virological journal on the basis of factual comments which I would have been quite willing to sign. The editor sent me, out of cour- tesy, copies of his rejection letter togeth- er with the other referee's sarcastic poison-pen comments, also rejecting the paper. There was no justification for one civilised person insulting another in such a manner.... That outburst was solely the joy of releasing adrenalin with anon- ymous impunity."

While Fraenkel- Conrat's analysis may be correct in this situation, there is little evidence, other than anecdotal, that this is a widespread phenomenon. But it suggests fertile ground for study: do ad hominem com- ments—those leveled at authors, as distinct from strong opinions about the authors' text—occur more frequently in signed or in unsigned reviews?

In a "Guest Comment" published in

Physics Today, F. Curtis Michel, pro-

fessor of space physics and astronomy,

Rice University, Houston, calls for ref-

erees to back up their comments. "Ac-

countability is now all directed back at

the author," he writes.

"If there is any

dispute, it is entirely the authors' fault

because they have 'failed to convince

their peers.' Here, the word 'peer' has a

nice ring of fairness to it.... However,...

when a group of colleagues is permitted to have [their] comments taken as some kind of gospel, [they] are no longer peers but quite definitely superiors insofar as power and influence go."

It is in answer to just this kind of criticism, Har- nad reports, that BBS is conducting an internal, statistical study of, among other things, the relationships among anonymity, referees' ratings of manu- scripts, and authors' ratings of the use- fulness of referee reports.

Another criticism of the system is of the "Newcomb variety." I have often re- ferred to the career of Simon Newcomb, who proved conclusively—just months before the Wright Brothers took off from the sands of Kitty Hawk—that a flying machine was impossible.

.

Sometimes this type of rejection is the result of referees who are hostile to inno- vative ideas or to those that clash with their own.

We don't know how often thoughtful, conscientious scientists—in good faith and in keeping with currently accepted theory—rendered an opinion concerning the implausibility of a given idea or theory, only to see that theory become the basis of a dramatic paradigm shift. Still, referees and journal editors should not consider such rejection ex- perience as sufficient reason for extend- ing some kind of "publication carte blanche" to would-be authors who want to prove, for example, that perpetual- motion machines are possible. I contin- ue to be in favor of refereeing that pre- vents the publication of intellectual atrocities, including papers with inade- quate documentation. For those articles straddling the border between science and speculation, there exist publications such as Speculations in Science and Technology, which was started specif- ically as a forum for the publication of ideas lacking support "in established theoretical and experimental work," ac- cording to an article by founder William M. Honig, senior lecturer in the physical

sciences and engineering, Western Aus- tralian Institute of Technology, Perth, in the Sciences.

Refereeing and Garfield's Uncertainty Principle

It is easy to "prove" on the basis of anecdotal evidence that the refereeing system doesn't work. From the hundreds of published Citation Classics® com- mentaries—such as those written by Os- car Buneman, Stanford University, Cali- fornia,

and Hans Lineweaver, US De- partment of Agriculture, Washington, DC

—or in correspondence with their authors, we know that dozens of signifi- cant papers have been rejected by some journals for various reasons. Some of these reasons might be described as

"N-I-H," that is, "not invented here."

Nevertheless, much scientific quackery is exposed by careful, insightful, con- structive refereeing, and this far out- weighs the ideas that have allegedly been suppressed because of referees who would not give them a chance to see the light of day.

A scientist's appreciation of the col- laborative, communal goal of referee- ing—protecting science and the public from errors and inferior work—varies according to a host of factors, including the scientist's age, status, and tempera- ment. Famous, tenured, or established researchers may be better able to weath- er the occasional rejection notice than scientists just starting their careers and trying to make their mark. No other ac- tivity is as fundamental to democratic scholarship as refereeing. From all this, I concluded that there is an Uncertainty Principle of Refereeing: The more we have of it, the less we like it—but the less we have of it, the more we miss it.

We sometimes trivialize what we take

for granted. Refereeing has been around

for so long that it's easy to forget that it

wasn't always there. The present stage of

its evolution will be affected by social and technological factors such as fund- ing and electronic publishing. But the public discourse of scholarship, both formal and informal, is essential to the very existence of science. In the modern era of big science—and by that I mean both large-scale projects and large num- bers of projects, whether small or large—we must find ways to inculcate new research practitioners with the pre- cepts and ideals that "naturally" were taught in the era of little science. We cannot allow squabbling over limited re- search funds to cloud the fundamental need to preserve the scientific process implied by refereeing. But we must rec- ognize that the very size of the scientific enterprise may make it necessary to modify rigid application of the Ingelfin- ger rule

[promulgated by the late Franz J. Ingelfinger, former editor, NEJM, which states that papers submitted to NEJM must "have been neither pub- lished nor submitted elsewhere (includ- ing news media and controlled-circula-

tion publications)"] or other precepts that may have been reasonable before the electronic revolution.

Indeed, the community of science may become even more relevant in the new communications age, and so we have to examine more carefully the con- sequences for intellectual property rights and methods of adjudicating disputes concerning priority of discov- ery. If much of this sounds Mertonian in tone it is no accident, since Robert K.

Merton is one of the few scholars who has devoted great effort to the definition of the problems involved in research on refereeing. In fact, the work of Zucker- man and Merton will form a significant part of the discussion in Part 2 of this essay.

My thanks to Stephen A. Bonaduce and Terri Freedman for their help in the preparation of this essay.

REFERENCES

1. Garfield E. From citation amnesia to bibliographic plagiarism. Essays of an information scientist.

Philadelphia: IS1 Press. 1981. Vol. 4. p. 503-7.

2. . More on the ethics of scientific publication: abuses of authorship attribution and citation amnesia undermine the reward system of science. Ibid.. 1983. Vol. 5. p. 621-6.

3 . — . Ghostwriting—the spectrum from ghostwriter to reviewer to editor to coauthor.

Current Contents (48):3-ll, 2 December 1985. (Reprinted in: Essays of an information scientist:

ghostwriting and other essays. Philadelphia: ISI Press, 1986. Vol. 8. p. 460-8.)

4. . Alternative forms of scientific publishing: keeping up with the evolving system of scientific communication. Op. cit., 1981. Vol. 4. p. 264-8.

5. . How to use citation analysis for faculty evaluations, and when is it relevant?

Parts 1 & 2. Ibid.. 1984. Vol. 6. p. 354-72.

6. - The 1984 Nobel Prize in medicine is awarded to Niels K. ferne, César Milstein, and Georges J.F. Köhler for their contributions to immunology. Current Contents (45):3-18, 11 November 1985. (Reprinted in: Essays of an information scientist: ghostwriting and other essays.

Philadelphia: ISI Press, 1986. Vol. 8. p. 416-31.)

7. . The 1984 Nobel Prize in physics goes to Carlo Rubbia and Simon van der Meer;

R. Bruce Menifield is awarded the chemistry prize. Current Contents (46):3-14, 18 November 1985. (Reprinted in: Essays of an information scientist: ghostwriting and other essays Philadelphia: ISI Press, 1986. Vol. 8. p. 432-43.)

8. — . The 1984 Nobel Prizes in economics and literature are awarded to Sir Richard Stone for pioneering systems of national accounting and to Jaroslav Seifert, the national poet of Czechoslovakia. Current Contents (49):3-l3, 9 December 1985. (Reprinted in: Essays of an infor- mation scientist: ghostwriting and other essays. Philadelphia: ISI Press, 1986. Vol. 8. p. 469-79.) 9. . Do Nobel Prize winners write Citation Classics? Current Contents (23):3-8, 9 June 1986.

10. . Publishing referees' names and comments could make a thankless and belated task a timely and rewarding activity. Op. cit., 1977. Vol. I. p. 435-7.

11. , Anonymity in refereeing? Maybe—but anonymity in authorship? No!

Ibid., 1977. Vol. 2. p. 438-40.

12. . Reducing the noise level in scientific communication: how services from ISI aid journal editors and publishers. Ibid., 1980. Vol. 3. p. 187-8.

13. Amlry P. Refereeing for JORS J. Oper. Res. Soc. 34:1025-6. 1983.

14. Znckennan H & Merton R K. Patterns of evaluation in science: institutionalisation, structure and functions of the referee system. Minerva 9:66-100, 1971. [Reprinted as: Institutionalized patterns of evaluation in science. (Merton R K.) The sociology of science. Chicago, EL: University of Chicago Press, 1973. p. 460-%.)

15. Garfield E. Has scientific communication changed in 300 years? Op. cit.. 1981. Vol. 4. p. 394-400.

16. Kronick D A. Authorship and authority in the scientific periodicals of the seventeenth and eighteenth centuries. Libr. Quart. 48:255-75, 1978.

17. —, A history of scientific & technical periodicals.

Metuchen, NJ: Scarecrow Press, 1976. 336 p.

18. — ——. Literature of the life sciences: reading, writing. research.

Philadelphia: ISI Press. 1985. 219 p.

19. Day R A. How to write and publish a scientific paper. Philadelphia: ISI Press. 1983. p. 82.

20. Christiansen D. Peer review reviewed. IEEE Spectrum 18:21. 1981.

21. Hamad S. Personal communication. 25 June 1986.

22. — . Commentary on "Computer-assisted referee selection as a means of reducing potential editorial bias" by H. Russell Bernard. Behav Brain Sei. 5:202, 1982.

23. Bishop C T. How to edit a scientific journal. Philadelphia: ISI Press, 1984. p. 53.

24. Hamad S. Commentary on "Peer review and the Current Anthropology experience" by C. Belshaw.

Behav. Brain Sei. 5:201, 1982.

25. Gordon M. Running a refereeing system. Leicester, UK: Primary Communications Research Centre.

University of Leicester. 1983. 56 p.

26. Hamad S. Creative disagreement. Sciences 19(7): 1820, 1979.

27. Baflar I C & Patterson K. journal peer review: the need for a research agenda.

N. Engl. J. Med. 312:654-7, 1985.

28. Lock S. U t t e r to P.B.S. Fowler. 4 December 1984. Brit. Med. J. 290:1560, 1985.

29. Yalow R S. Radioimmunoassay: a probe for the fine structure of biologic systems.

Science 200:1236-45, 1978.

30. Beison S A, Yalow R S, Bauman A, Rothschild M A & Newerty K. Insulin-I131 metabolism in human subjects: demonstration of insulin binding globulin in the circulation of insulin treated subjects. / Clin. Invest. 35:170-90, 1956.

31. Wilson I D. Peer review and publication. /. Clin Invest. 61:1697-701, 1978.

32. Kronick D A. Personal communication. 20 June 1986.

33. Stossel T P. Speed: an essay on biomedical communication. N. Engl. J. Med. 313:123-6, 1985.

34. Grouse L D. The Ingelfinger rule. JAMA—J Am. Med. Assn. 245:375-6, 1981.

35. Hollender M H. Authors, editors and referees. J. Clin. Psychiat. 40:331, 1979.

36. Beaver D D. On the failure to detect previously published research.

Behav. Brain Sei. 5:199-200, 1982.

37. Merlon R K. The Matthew effect in science. Science 159:56-63, 1968. (Reprinted in: The sociology of science. Chicago, IL: University of Chicago Press. 1973. p. 439-59.)

38. Foi B. Peer review and the public's right to know: a look at the Ingelfinger Rule.

Med. Commun. 12:33-7, 1984.

39. Broad W I. Imbroglio at Yale (I): emergence of a fraud. Science 210:38-41, 1980.

40. Wachsllcht-Rodbard H, Gross H A, Rodbard D, Ebert M H & Roth I. Increased insulin binding to erythrocytes in anorexia nervosa.

N. Engl. J Med. 300:882-7. 1979.

41. Jones R. Rights, wrongs and referees. New Sei. 61:758-9, 1974.

42. Frmenkel-Conrat H. U t t e r to editor. (Is anonymity necessary?) Nature 248:8, 1974.

43. Michel F C. Solving the problem of refereeing. Phys. Today 35(12):9; 82, 1982.

44. Garfield E. Negative science and "The outlook for the flying machine." Opt. cit., 1980. Vol. 3.

p. 155-72.

45. Newcomb S. The outlook for the flying machine. Independent 55:2508-12. 1903.

46. Honig W M. Science's Miss Lonelyhearts. Sciences 24<3):24-7, 1984.

47. Buaeman O. Citation Classic. Commentary on Phys. Rev. 115:503-17, 1959.

Current Contents/Engineering. Technology & Applied Sciences 15(37): 16, 10 September 1984.

48. Llneweaver H. Citation Classic. Commentary on / Amer. Chem. Soc. 56:658-66. 1934.

Current Contents/Life Sciences 28(11):19, 18 March 1985.

49. Definition of "sole contribution." N. Engl. J. Med. 281:676-7, 1%9.

Refereeing and Peer Review. Part 2.

The Research on Refereeing and Alternatives in the Present System Current Contents, August 11,1986

Continuing our discussion of referee- ing, which focused on complaints about the system in Part 1,' we now examine the empirical research on the subject, the anecdotal literature supporting the current system, and some of the sugges- tions for improving it. Part 3 will appear at a later date and will discuss the peer review of grant proposals. Again we will review the considerable literature of opinion and conjecture, but we will give special attention to the large-scale study by sociologists Stephen Cole, State Uni- versity of New York (SUNY), Stony Brook, and Jonathan R. Cole, Columbia University, New York,

-

as well as other papers

and special reports.

Editors: The Author's Guardians Each anecdote purporting to reveal some fault in the present system of refer- eeing seems to find a ready counterpart in the opinion of a supporter. For in- stance, many critics claim that some ref- erees do not review manuscripts dispas- sionately. But editors say that they usu- ally take great pains to ensure that refer- ees are fair. In Running a Refereeing System, Michael Gordon, research-asso- ciate, Primary Communications Re- search Centre, University of Leicester, UK, recommends the use of two or more referees to reduce the risk of an offhand, frivolous, or biased treatment of a manu- script.

(p. 13-5) When referees do cause excessive delays, return unsupported or capricious reports, or otherwise display

"questionable ethics," they tend to be

retired from the system, according to Patricia Dehmer, Argonne National Laboratory, Illinois, and member. Publi- cations Committee, American Physical Society (APS) in a "Guest Comment" in Physics TodayP Whether this is the case in other disciplines is not known.

Critics also suggest that referees some- times take advantage of the privileged information they are privy to in the manuscripts they review. But Dehmer asserts that many APS editors try to en- sure that referees are not working along lines precisely like those of the papers sent to them, to reduce the possibility of conflicts of interest. But this is contrary to the practice in biomedicine and else- where. Most editors try to match sub- missions with reviewers as closely as pos- sible, in an attempt to have the manu- script reviewed by those presumed to be most qualified to judge it. In either case, according to Claude T. Bishop, director.

Division of Biological Sciences, National Research Council of Canada (NRCC).

and editor-in-chief, NRCC Research

Journals, referees ought to disqualify

themselves when there is the possibility

of a conflict' of interest, or when they

feel they cannot be objective about the

paper or its author. In some instances,

however, they might propose simulta-

neous publication of their own paper

and the review paper, or even approach

the authors of the review paper and pro-

pose a collaboration.

(p. 50, 82) As a

parallel approach, many editors honor

author requests that a paper not be sent

to a given referee.

Authors Often Lack Knowledge of Publishing

Editors also point out that authors fre- quently do not understand the publica- tion process. For instance, many authors charge that referees make up a closed,

"elite" group. Yet the number of active referees for a journal can far exceed the number of active contributors.

Accord- ing to JAMA editor George D. Lund- berg, that journal's list of active referees contains over 3,000 names.

The Jour- nal of the Operational Research Society, a relatively small journal, used 285 refer- ees in 1982 alone.

And a careful study of nine years of materials from the ar- chives of Physical Review and Physical Review Letters by sociologists Harriet Zuckerman and Robert K. Merton, Co- lumbia University,

showed that au- thors of every rank participated in the refereeing process. Their main finding, which is based on referee reports for both published and rejected manu- scripts and which refutes another widely held belief, is that there is no consistent relationship between referee acceptance or rejection of manuscripts and the rela- tive standing of authors and referees.

In addition, informed authors know that it is not referees, but editors, who are ultimately responsible for rejecting a manuscript.

Bishop says that authors also show a lack of understanding when they point to differences of opinion among referees as evidence that the system is capricious and unreliable.

(p. 43-9) At the root of some of these reviewer disagreements, in Bishop's view, are differences in the algorithms and paradigms fundamental to every branch of science. For instance, referees less often disagree substantially in well-established fields. But in fields pressing at the frontiers of knowledge, significant differences of opinion among referees are bound to be more common.

When editors are confronted with a de- cision between two equally plausible ref- eree interpretations of a given manu- script, they often employ one of several options that range from publishing the paper without comment to publication

of the controversial paper along with comments by referees, invited critics, and rebuttals by the authors.

(p. 43-9)

Authors also seem to assume that their submissions are, in general, carefully written and based on substantial amounts of work. "Not so," asserts J. W.

Cornforth, Milstead Laboratory of Chemical Enzymology, Sittingbourne Research Centre, Kent, UK, who served as a referee for a dozen journals over a 30-year period.

"In my experience,"

Cornforth continues in his letter to the editors of New Scientist, "a regrettably high proportion [of manuscripts] show careless or misleading presentation and meager experimental work, and the ma- jority need some modification. Refer- ees—and, of course, editors—almost invariably improve a paper that passes through their hands; often, they are do- ing what the authors ought to have done."

The Many Faces of Rejection

Authors should also be aware that the scientific value of a paper is not neces- sarily the only factor that enters into edi- tors' decisions to publish or not; many manuscripts never make it past the screening process that eliminates papers that are incompatible with a journal's readership or have not been submitted in the required format.

Or a journal may reject a manuscript simply because it has recently published another, similar pa- per, or has one currently under consid- eration.

Rejection rates are also signif- icantly affected by the existence of page charges, which support publication and thus allow for much lower rejection rates. This practice is widespread in physics and chemistry but not unknown even in psychology.

It is also important to realize that re-

jection rates vary. In their study of pat-

terns of evaluation in science, Zucker-

man and Merton compiled a table of the

rejection rates for a sample of 83 jour-

nals in the sciences, the social sciences,

and the humanities.

Linguistics, geolo-

gy, and physics journals had the lowest

rate of rejection, turning down only 20

to 25 percent of the papers submitted to them. Biology journals rejected about 30 percent of the papers they received.

Journals in experimental and physiologi- cal psychology had a rejection rate of over 50 percent, while sociology jour- nals were over 80 percent and history journals hovered at 90 percent. Stephen Lock, editor, British Medical Journal (BMJ), made an observation that has also been noted by others who have read the study. He wrote that "the more hu- manistically oriented the journal, the higher the rate of [rejection]; the more experimentally and observationally ori- ented, with an emphasis on rigour of ob- servation and analysis, the lower the rate of rejection."

(p. 17)

Zuckerman and Merton also reported that the editorial staffs attitude con- cerning its own errors in judgment con- stitutes an often-overlooked factor influ- encing acceptance rates.

Although ed- itors and referees want to avoid errors in judgment altogether, they recognize that they cannot be infallible; thus, since they must make mistakes, they tend to have preferences for the kind of mis- takes they are willing to risk. The staffs of some journals—notably those presti- gious journals with high rejection rates—seem more willing to reject "un- orthodox" manuscripts that the wider community of scholars might eventually consider important, rather than to run the risk of publishing a substandard work. The staffs of low-rejection jour- nals, on the other hand, apparently prefer to publish the occasional work that doesn't measure up, rather than re- ject a paper that later turns out to be sig- nificant.

The Research

A research front consists of a group of current papers that, together, cite one or more of a cluster of older, core publica- tions. Since I referred earlier

to the paucity of empirical research on referee- ing and peer review and the abundance of anecdote and opinion on the subject, one may wonder how a research front of any size might be generated on this sub-

ject. But even a large anecdotal liter- ature, through repeated citations of pre- vious anecdotal literature, as well as rep- utable studies, can form a pseudo-re- search front. Only a careful analysis of the core and citing literature can deter- mine the nature and extent of the re- search front—even when very useful core review papers can be found. Since the literature on peer review and refer- eeing is vast, at the end of Part 2 of this essay I have added a selected bibliogra- phy of publications not mentioned in the text.

The 1983 ISI® research front entitled

"Objectivity of reviewers in peer review"

(#83-8291) consists of but 2 core papers and 12 citing papers. One core paper is the highly controversial 1982 study by Douglas P. Peters, University of North Dakota, Grand Forks, and Stephen J.

Ceci, Cornell University, Ithaca, New York.

The other core paper is a 1982 editorial by Lock, entitled "Peer review weighed in the balance."

In it Lock dis- cusses the conclusions drawn by Peters and Ceci and details some of the flaws in their study. In spite of these problems, however. Lock believes that Peters and Ceci have underscored some shortcom- ings within the system. Most of the rec- ommendations Lock makes for improv- ing refereeing—particularly double- blind review—are discussed in detail below.

Peters and Ceci

This controversial study involved the

resubmission of 12 psychology arti-

cles—published by authors from pres-

tigious and highly productive depart-

ments—to the journals that originally

published them.

Peters and Ceci be-

came interested in doing the study after

reading about an informal experiment

conducted by Los Angeles free-lance

writer Chuck Ross.

He reports having

submitted the untitled, untyped manu-

script of Polish-bom US literary author

Jerzy Kosinski's novel Steps

under a

pseudonym to publishers and literary

agents to see if "unknown" authors re-

ceive fair consideration. Although the book had won the 1969 US National Book Award, Ross claimed that 14 pub- lishers—including the book's original publisher—and 13 agents rejected it.

In the Peters and Ceci study, the pre- sentation of the data in the original pa- pers was slightly altered. Fictitious names and institutions were substituted for the real ones, but the content of the articles was unchanged. Three of the re- submissions were detected as such; of the other nine, eight were rejected. The authors concluded that the rejections re- sulted from a systematic bias against un- known authors and institutions. In the commentary section published along with Peters and Ceci's article, however, many commentators pointed out a num- ber of flaws in the study. For instance, according to anthropologist Sol Tax, University of Chicago, Illinois, and Robert A. Rubinstein, School of Public Health, University of Illinois Medical Center, Chicago, the names Peters and Ceci chose for their bogus institutions were far removed from the mainstream of psychology institutions. Thus, what the investigators really demonstrated, say Tax and Rubinstein, is a bias against materials originating outside appropri- ate institutions.

Nobel laureate Rosa- lyn S. Yalow, Veterans Administration, New York, commented, "How does one know that the data are not fabricated?...

Those of us who publish establish some kind of a track record. If our papers stand the test of time, it can be expected that we have acquired expertise in scien- tific methodology.... The work of estab- lished investigators in good institutions is more likely to have had prior review from competent peers and associates even before reaching the journal."

Garth J. Thomas, Center for Brain Research, University of Rochester, New York, suggests that referees and editors may have recognized the resubmitted ar- ticles as very like something they had seen before, but rather than raise the specter of plagiarism, they fell back on statistical criticisms to justify their negative comments.

Janice M. Beyer, School of Management, SUNY, Buffalo,

writes that the most likely fate of any submitted article is to be unanimously rejected, as 80 to 90 percent are in the social sciences.

In addition, psychologist Grover J.

Whitehurst, SUNY, Stony Brook, notes that Peters and Ceci had no control group.

Richard M. Perloff, Depart- ment of Communication, Cleveland State University, Ohio, and Robert Per- loff, Graduate School of Business, Uni- versity of Pittsburgh, suggest that, among other controls, Peters and Ceci's study should have included resubmitting articles by authors from low-status insti- tutions under by-lines with equally low- status affiliations, as well as resubmitting articles by high-status authors under equally high-status by-lines.

"Without such controls it is impossible to argue that the findings reflect the status bias [that Peters and Ceci] suggest," the Perloff s write.

But Is There Bias?

Still, Tax and Rubinstein feel that a bias preventing competent work from being published is much more damaging than one that lets mediocre work slip through.

And anecdotal evidence of bias is so widespread that the possibility should not be dismissed by researchers.

For instance, in another commentary on the Peters and Ceci article, Robert Ro- senthal, Department of Psychology, Harvard, said that as a young member of the psychology faculty at the University of North Dakota, he was unable to pub- lish 15 to 20 articles in mainstream jour- nals in the 1960s. Within a few years of his move to Harvard, however, he says that most of these articles were pub- lished in the same journals that had pre- viously rejected them.

He does not say, however, whether these were the identical articles, or if they had been substantially revised to meet the objec- tions of reviewers or changed in any other way.

In a 1970 investigation of how at-

titudes might influence referee judg-

ment, Leonard D. Goodstein and Karen

Lee Brazis, University of Cincinnati, Ohio, mailed abstracts of an empirical study on astrology to 282 members of the American Psychological Association.

They were asked to rate the design of the paper. Half were sent an abstract that re- flected a conclusion confirming com- monly held scientific attitudes toward astrology; the other half received an identical abstract, except that it includ- ed a conclusion that ran counter to scientific beliefs. The former was rated by most referees as better designed and having more significance for future research. The latter, which contradicted common wisdom, was rated as flawed.

When Zuckerman and Merton exam- ined the selection of articles for the Physical Review, they found that papers by physicists of great repute affiliated with prestigious institutions were more likely to be exempted entirely from the refereeing process. Their papers were accepted and published more quickly than papers by lesser known physi- cists.

And in a large-scale study of papers submitted to physics journals, Gordon reported a strong bias in refer- ees from major universities toward pa- pers by authors who were also from large, well-known universities.

Lock, however, found no evidence of referee bias in a study of 1,558 manu- scripts submitted to BMJ between Janu- ary and August 1979. The study was pub- lished in his book A Difficult Balance:

Editorial Peer Review in Medicine.

Of the 246 external referees who were sent manuscripts by BMJ, 143 held aca- demic positions, while the rest had non- academic affiliations; yet the proportion of papers recommended for acceptance did not differ from one group to the other.

(p. 56-71) Moreover, regardless of the affiliations of both referee and author, Lock said that referees judged manuscripts "to an equal standard."

(p. 61)

Suggestions for Improvement

A few years ago, Norton D. Zinder, Rockefeller University, New York City,

sent me the text of a talk he gave to the Society of Editors in 1969, when he was an associate editor of Virology.

Tongue partially in cheek, Zinder asked, "What would be so terrible if there were no refereeing of scientific pa- pers?... As we now operate, with [the]

restriction of publication by reviewing, the number of publications becomes a thing in itself.... If we were to cease ref- ereeing papers,...there'd be little bar [to publication, and] quality might reassert its role, since there'd be less pressure to have long lists of publications."

The Perloffs write that the "caveat emptor approach [of having no refereeing system at all] might be viewed as a nod to the free market of ideas. Let millions of flowers bloom."

Some may feel that the continued growth of the literature may lend support to these views. How- ever, others, including myself, believe that a few non-refereed publications can exist only because the refereed journals set the standards for all the others.

I believe that most scientists would agree that if something is indeed shown to be wrong with refereeing, an attempt should be made to repair the system, rather than to abandon it. Unfortunate- ly, with little or no solid, systematic evidence of refereeing's deficiencies, most suggestions for improvement are as conjectural as the ills they are meant to cure. Among the most discussed op- tions—one that is already prevalent among sociology journals—is that of double-blind refereeing, also called re- ciprocal anonymity, in which neither the authors nor referees are aware of the others' identities. There is precedent for author anonymity: David A. Kronick, professor, medical bibliography. Uni- versity of Texas Health Science Center at San Antonio, notes that "maintaining the anonymity of the author was a stan- dard practice in the prize essay competi- tions (a sort of early form of sponsored research) of eighteenth-century scien- tific societies, which had elaborate devices to maintain the anonymity of contributors."

The rationale behind double-blind

refereeing, as was pointed out in an ap-

propriately anonymous editorial in Nature, is that referees could still be frank about a manuscript's shortcomings without fear of ruining working relation- ships or being subjected to the anger of rejected authors.

Such a system would also, in the opinion of J. Scott Arm- strong, Wharton School, University of Pennsylvania, Philadelphia, "reduce the prejudice against unknown authors from low-status institutions."

Many justify the present system by cit- ing what Marcel C. La Follette, editor, Science, Technology, & Human Values, calls the "crackpot avoidance" theory.

According to this idea, an author's record of achievement and the stamp of legitimacy provided by the author's in- stitutional affiliation help referees evaluate manuscripts because they con- stitute presumptive "proof" that the research described was really done. La Follette says that accepting manuscripts without regard for the potential of misrepresentation or error is unwise, but she points out that a prestigious affili- ation is no guarantee against fraud—in fact, it may even help the perpetrator evade detection.

According to John Moossy, editor-in- chief, and Yvonne R. Moossy, managing editor, Journal of Neuropathology & Ex- perimental Neurology, a common ob- jection to double-blind refereeing is a widespread conviction that experienced referees can identify authors despite the removal of the authors' names from their manuscripts.

In a study conducted to test this contention, they removed the names of authors and their departmental and institutional affiliations from 33 papers sent out for refereeing from May 1983 through April 1984. Each of the 67 referees, who filed a total of 85 reports, was asked to identify the authors and their departments or disciplines; 34 per- cent were able to make correct identifi- cations. Eleven percent made incorrect identifications, and 55 percent would not even hazard a guess. Interestingly, only 9 referees objected to the double- blind procedure; a surprising number—

24—had "no opinion," while 33 favored it, citing such reasons as greater objec- tivity and less risk of being swayed,

either for good or ill, by the author's reputation.

Another frequently proposed reform is "open refereeing." It is the exact op- posite of double-blind refereeing: the referee's name is revealed to the author, who in turn is made known to the refer- ee. Proponents argue that open referee- ing might reduce the number of careless and superficial reports, on the presump- tion that referees will take more care with their reports if they have to sign their names to them. And in fact, I noted long ago that the time of the more quali- fied referees is of proportionately great- er value; thus, they may sometimes be less than enthusiastic over the prospect of a manuscript to evaluate.

Anonymi- ty is a dull spur to effort; "Aren't we all more likely to do something properly if our name is attached to it?" asks Ronald Mirman, Department of Physics, Long Island University, Brooklyn, New York, in a letter to the editor of the American Journal of Physics.

Armstrong proposes that referees might designate a portion of their report to be signed and published along with the manuscript. He believes this would provide useful information to scientists because few readers can devote the kind of attention to a paper that a referee gives to it.

However, a number of problems might be encountered were referee anonymity abolished. For in- stance, the late Franz J. Ingelfinger, former editor, the New England Journal of Medicine, believed that "the referee who is several steps below the author on the status ladder" might be put in an un- comfortably vulnerable position and might even be unwilling to criticize can- didly the manuscript in question.

Some reviewers might soften their ob- jections to manuscripts, rather than jeopardize working relationships with the authors.

(p. 16) Identifying referees would also enable authors to get in touch with them. This might foster a communi- cation process that excludes the editor, or even exposes referees to verbal at- tacks.

'

The Perloffs have another suggestion

for promoting a greater sensé of respon-

sibility among referees. They argue that