Use of peer reviews to improve editorial/funding decisions

3.1. The editor as final arbiter. A general concern is ex-pressed by Fletcher about how editors and granting officials use unreliable reviewer recommendations to arrive at publication or funding decisions. Strieker talks about t h e importance of t h e "active" editor who judi-ciously "weighs" the information, provided by reviewers, to arrive at a thoughtful publication decision. Similarly, R o u r k e speaks of "fair," "judicious," and "experienced"

editorial practices. The reader will also recall Kiesler's c o n c e p t of t h e "wise" editor. These descriptions, in turn, are similar to Bailar's informative notion of the editors' integration of their own knowledge with that provided by t h e additional "wisdom" of members of the editorial board, as well as "special consultants," as required.

Lock (see also Lock 1985) proposes a "hanging commit-t e e " commit-to examine and help resolve quescommit-tions aboucommit-t commit-those manuscripts receiving "gray area" or split-review recom-mendations. Both Bailar and Crandall speak of the need for editors to eschew a majority-vote-of-reviewers'

op-tion, by exercising their power to override the recom-mendations of reviewers, whenever required. Bailar states that a neglected area of the target article is the realization that an editor's decision is based not only on t h e overall scientific (or "technical") merit of a given submission but also on such manuscript attributes as

"originality," "importance to readership," or "succinct-ness." I am somewhat puzzled at how an editor accom-plishes this important goal, given the known unreliability of such attributes, that is, the data shown in Table 1.

Bailar's f u r t h e r elucidation of how he was able to accom-plish this objective for J NCI would make an important contribution to t h e field of peer review. T y r e r discusses t h e important role of t h e editor or chairman of a granting agency in which "executive decision" is used to resolve reviewer disagreements. In a slightly different context, Armstrong & H u b b a r d note that at least some innovative research is published by editors of the American Psycho-logical Association (APA) journals, despite t h e relatively low levels of r e v i e w e r agreement on submissions describ-ing such valuable research. Fuller expresses concern that such high quality research will frequently b e published in relatively few access journals.

In a more general sense, Bailar takes t h e position that t h e target article did not adequately "pound home" the relatively major role of t h e editor or granting official in the entire peer review process relative to what he feels is the more minor role played by the reviewers (merely the providers of "relevant information"). Unfortunately, in so doing, the views of Bailar (consistent with those of Kiesler) create a purely artificial distinction among authors, re-viewers, editors, and consumers of submitted and pub-lished papers. As noted earlier (Cicchetti 1982, p. 21),

"one of the most persistent problems we still face appears to be the false dichotomy we have t e n d e d to create between those w h o evaluate research and those who are being evaluated. Both derive from the same research species. "Theprovocativeandimaginativecommentary by K r a e m e r is consistent with this view. She speaks elo-quently of the conflicting roles each of us is called on to play ("submittors," "reviewers," "consumers" of scientific pa-pers) and the fundamentally different standards we might invoke, d e p e n d i n g on which peer role we are assuming at a given moment. H e r emphasis on an "objective," "dispas-sionate," and "quantitative" approach to t h e study of peer review as the only h o p e of identifying and correcting the many shortcomings in the peer review process is to be taken seriously. Toward this important goal, it is almost axiomatic that for science to continue to operate, it re-quires the imagination and talent of authors, dispassionate and sensitive editors and granting officials, and, finally, interested readers (or consumers) of the published re-search findings, so that t h e cycle can continue anew. In this basic and comprehensive sense I disagree with the more narrow-focus views of both Bailar and Kiesler.

Though Bailar seems to be unaware of it, data were provided in the target article, showing t h e positive rela-tionship between p e e r reviewer recommendations and t h e publication decisions made for more than 1,300 manuscripts submitted between 1973 and 1978 to the general focus Journal of Abnormal Psychology (JAP). It was further noted that the results were consistent with data deriving from reviews of both the Sociological Re-view and the Physical ReRe-view, which indicate that

re-viewer recommendations are t h e major factor in t h e editor's decision to accept or reject (Bakanic et al. 1987;

Hargens 1988).

Next is an attempt to broaden our knowledge base in this important area of peer r e v i e w research. The data is based on the total number of submissions to JAP b e t w e e n 1973 and 1977 (1,698 manuscripts). These could b e classi-fied as follows: 15 (0,9%) w e r e withdrawn before an editorial decision could be m a d e ; 14 (0.8%) were solic-ited; 384 (22.6%) were reviewed by the editor alone; 175 (10.3%) w e r e reviewed by a single referee (other than t h e editor); 996 (58.7%) were evaluated by two independent reviewers; 112 (6.6%) received t h r e e independent re-views; and 2 (0.1%) received f o u r independent reviews.

As r e p o r t e d in section 2.3 of this Response, 86.7% (333 of 384 manuscripts) were r e v i e w e d and summarily re-jected by t h e editor alone on t h e basis of very poor quality, inappropriateness for t h e readership of the j o u r nal, or on both accounts. All 14 of the solicited m a n u -scripts w e r e accepted for publication. Of the two manu-scripts receiving four reviews, one was accepted, t h e other rejected.

The fate of t h e remaining manuscripts, namely, those with a single review, two reviews, or three reviews follows.

3.2. The editor's use ot single reviews: "Go with the flow." Table 4 shows that of t h e 175 manuscripts sent to a single reviewer, 58 (33.1%) w e r e accepted by the editor and 117 (66.9%) were rejected.

The full set of data (Part A o f T a b l e 4) indicates that, as a general rule, the editor's final decision closely parallels single reviewer recommendations. Part B of the table indicates that when the reviewer recommended that t h e manuscript b e accepted ("as is" or "subject to revision") there was an 85% likelihood of acceptance. Analogously, when t h e reviewer r e c o m m e n d e d either resubmission or rejection, t h e r e was a 90% probability of rejection. An inspection of discrepancies b e t w e e n reviewer

recom-mendations and editorial decisions indicated that the editor was no m o r e likely to reject manuscripts receiving an "accept/as is" or "accept/revise" recommendation (8/54 = 14.8%) than to accept manuscripts receiving a

"resubmit" or "reject" recommendation (12/121 = 9.9%). Applying McNemar's (1947) statistic for correlated proportions p r o d u c e d a chi square(d) value of zero-order significance, that is, 0.45.

3.3. The editor's use of two reviews: "Go with the low."

Results are p r e s e n t e d in Table 5 for those 996 manu-scripts receiving two reviews d u r i n g t h e period 1973-1977.

The data can easily be understood if one considers that:

(1) Manuscripts receiving a joint r e v i e w e r recommenda-tion of " R e s u b m i t " had a 27% probability of being accept-ed for publication, which is indistinguishable from the base rate j o u r n a l acceptance rate of 28%; (2) those manu-scripts receiving two reviewer votes for acceptance or a split b e t w e e n acceptance and resubmission had a 72%

probability of being published, as compared to the 72%

baseline rejection rate of the journal; and (3) the remain-ing 635 m a n u s c r i p t s (65%) had only a 5.5% probability of being accepted for publication, a rate more than five times less than t h e journal baseline acceptance rate of 28%.

These findings also have cross-disciplinary implica-tions. Specifically, Lock (1985, pp. 20-21) presented analogous data for 282 articles, or 50% of the 564 articles that were s u b m i t t e d to the medical journal Thorax and evaluated independently by two referees. Thirty-eight percent (or 107) of the manuscripts received a unanimous reviewer recommendation for acceptance. All of them were accepted by the editor. T h e reviewers were in agreement that an additional 38% (or 107) manuscripts should b e r e j e c t e d . The editor r e j e c t e d all these submis-sions. T w e n t y - f o u r percent (or 68) of the submissions received a split-decision, with one reviewer recommend-ing "accept" and t h e other "reject." T h e editor accepted

Table 4. The fate of Journal of Abnormal Psychology submissions receiving a single editorial review (1973-1977) A. Considering each reviewer recommendation

Reviewer Number of Editorial decision Percentage

Recommendation Manuscripts Accept Reject Accepted

1 = Accept/As Is 24 22 2 91.7

2 = Accept/Revise 30 24 6 80.0

3 = Resubmit 26 11 15 42.3

4 = Reject 95 1 94 01.1

Total 175 58 117 33.1

B. Considering 1 - 2 — Accept; 3 - 4 = Reject

Reviewer Number of Editorial decision Percentage

Recommendation Manuscripts Accept Reject Accepted

(1-2) = Accept 54 46 8 85.2

(3-4) = Reject 121 12 109 90.1

Total 175 58 117

Table 5. The fate of Journal of Abnormal Psychology submissions receiving two editorial reviews (1973-1977) A. Considering "Accept/As Is" and "Accept/Revise" as "Accept"

Reviewer Number of Editorial Decision Percentage

Recommendation Manuscripts Accept Reject Accepted

Accept-Accept 181 159 22 87.8

Accept-Resubmit 150 79 71 52.7

Resubmit-Resubmit 30 8 22 26.7

Accept-Reject 195 28 167 14.4

Resubmit-Reject 162 6 156 03.7

Reject-Reject 278 1 277 00.4

Total 996 281 715 28.2

B. Considering 1 — Accept; 2-3 = Reject

Reviewer Number of Editorial decision Percentage Recommendation Manuscripts Accept Reject Agree Disagree Accept 353 212 141 60.1% 39.9%

Reject 643 69 574 89.3% 10.7%

Total 996 281 715

2% (or 6) such manuscripts and rejected the r e m a i n i n g 22% (or 62) manuscripts. Thus, t h e r e was a 62/6 or m o r e than tenfold probability that an editor would reject r a t h e r than accept a manuscript receiving mixed reviews.

In s u m m a r y , at least for t h e general focus j o u r n a l s examined thus far, when editors a r e faced with split-decisions, they tend overwhelmingly to go with the lower of the two reviewer recommendations.

3.4. The editor's use ot three reviews; "Go with the mode." Of the 1698 (6.6%) submissions to JAP ( 1 9 7 3 -1977), 112 received three reviews. As shown in Table 6, the disposition of these manuscripts is again closely relat-ed to reviewer recommendations.

Thus, all 5 manuscripts receiving unanimous a c c e p -tance votes w e r e accepted for publication. Analogously, the 9 submissions receiving unanimous rejection v o t e s were r e j e c t e d . The disposition of t h e remaining 98 (or 87.5%) manuscripts is best understood by the editor's general adoption of majority rule or applying the formula

"go with t h e m o d e . " Thus, 24 of 33 (or 73%) of t h o s e submissions receiving 2 "accept" recommendations w e r e accepted, whereas 80% (or 52/65) submissions receiving two " r e j e c t " votes were rejected. Application of t h e M c N e m a r test of correlated proportions indicated n o significant difference favoring e i t h e r the editor's accep-tance of t h e s e articles (20%) with majority rejection votes or his rejection of those articles (17%) with majority acceptance votes (McNemar's chi square(d), corrected, 1 df = .41, or of zero order significance).

I am unaware of comparable studies on manuscripts submitted to medical journals. Bailar noted that it was not unusual, in his role as editor of JNCI (1974-1980), however, e i t h e r to reject manuscripts receiving t h r e e positive reviews or to publish submissions receiving three negative reviews. Given t h e importance of this p h e n o m e n o n , I would invite Bailar to publish these data,

because they contrast so sharply with what I have pre-sented here for a prestigious behavioral science journal.

It would be i m p o r t a n t to know; (a) precisely how fre-quently the p h e n o m e n o n occurred; and (b) in what important r e s p e c t s the targeted manuscripts were dis-similar from t h o s e that were less problematic. Perhaps Bailar could p r o v i d e this information in a BBS Continuing Commentary.

In the field of physics, recall that journal editors in the more specific focus areas use the "single initial reviewer system"(e.g., H a r g e n s & Herting 1990; Lock 1985, p.20) and so would t e n d not to have much data on the fate of manuscripts receiving three reviews. Although it is equal-ly clear that t h e m o r e general focus areas of physics often receive t h r e e reviews (or more), t h e extent to which an editor uses this information to make specific publication decisions is u n k n o w n . Despite this lack of specific informa-tion, there are s o m e sparse data, deriving from the field of physics, that b e a r on the broader issue of how editors use information gained from referees to i m p r o v e the quality of the editorial decision-making process. These data ap-peared in the Physical Review (PR) and Physical Review Letters (PRL) (1987, p. 7) Annual Report for the previous year, 1986. T h e statements pertain to a change in editorial policy for manuscripts submitted from t h e community of particle theorists to the subfield section, Elementary Particles, which represents one of 10 P R L areas of spe-cialization. (The remaining nine subfields, orPRLcontent areas, are: G e n e r a l Physics, Cross-Disciplinary Physics, Astrophysics and Geophysics, Condensed Matter (CM), Electricity, C M Mechanics, Plasma Physics, Optics and Fluids, Nuclear Physics, and Atoms and Molecules.) The statement of j o u r n a l policy change bears quoting;

In March, 1985, a new system of handling papers in the theory of particles and fields was introduced. The divisional Associate Editors w e r e enlisted to work closely on t h e processing of these papers, with the

Table 6. The fate of 112 Journal of Abnormal Psychology manuscripts receiving three reviews (1973-1977)