Author's Response
4. Improving the reliability and validity of peer review
4.8. Developing peer review appeals systems. Whereas an argument for a more formal appeals system for rejected
manuscripts is made by Zentall, Cole, citing the finding by Stinchcombe and Ofshe (1969) that many acceptable articles are falsely rejected, opts for editors gradually to increase publication rates for submitted articles even at the risk of levying page charges on authors. Although this is an interesting suggestion, I am not sure what specific criteria editors would apply to justify increasing their acceptance rates. Commentators were in agreement, however, that t h e unfair disapproval of a grant submission is far m o r e serious in its consequences than the unfair rejection of a journal article (e.g., Adams, Cole, Kiesler, Mahoney, Salzinger, Zentall). T h e problem is especially serious for f u n d i n g in the social and behavioral sciences.
Thus, as Mahoney notes, the National Research Council (1988) emphasized the need for a 30% increase in funding in these areas where funding has dropped 25% b e t w e e n 1972 and 1987, whereas it has increased by 36% in other areas of science during t h e same funding period.
G r e e n e advocates strongly the need for an appeals system for'any funding agency, because the peer review system is an imperfect one. H e notes that the DVA has had an "effective" system for m o r e than a decade. H e also admits that appeal is a "sensitive" and "complex" phe-nomenon, so t h e ground rules on which it is based require periodic assessment. I agree with Greene's position.
Rather than a formal appeal process per se, Cole recom-mends that granting foundations admit publicly that many of their rejected proposals are as fundable as many that are approved. He advocates specifically that the approval of such previously declined proposals should be undertaken even at the expense of reducing funding levels for t h e ensuing round of new grant proposals.
My concern with Cole's recommendation is that once a grant proposal receives the official federal stamp of "dis-approval," it becomes more and more difficult to con-vince such lay persons as m e m b e r s of Congress that the submission should really have been funded in t h e first place. My solution would b e to assign high priorities (no n u m b e r attached) to the best considered proposals quite independently of whether t h e r e is funding available to support t h e m . One could then request from Congress whatever additional funds may be required to support all the high priority grants. I believe that the way the system works today - assigning arbitrary funding cutoffs based on arbitrary n u m b e r s - creates the dilemma of funding a proposal with a priority score of, say, 112 and declining one with a score of 113 when in fact no reasonable p e e r
reviewer can be expected to make a reliable differentia-tion of this minute degree of magnitude. To paraphrase Delcomyn's analogy, the task that grant reviewers face is one of being asked to measure t h e dimensions of a nerve cell with a yardstick. My recommendation is intended to help obviate that measurement problem.
4.9. Training reviewers. The important issue of training reviewers was mentioned, in varying degree, by several commentators (i.e., Adams, Crandall, Delcomyn, Kiesler, R o u r k e , and Zentall)
Adams describes the typical "haphazard" and "uncer-tain" m a n n e r in which reviewers eventually learn to b e c o m e "constructive" evaluators. Adam's previously m e n t i o n e d support of reviewers disclosing their identity to authors is one way of producing such constructive reviewer reports. With a somewhat similar purpose in mind, Zentall proposes that editors send to reviewers a list of recommended guidelines for avoiding potential biases in t h e evaluation of a given submission. T h e same general strategy can be used with grant proposals.
C r a n d a l l , Delcomyn, and Rourke write of t h e impor-tance of reviewers sharing others' reviews of t h e same manuscripts. Unfortunately, some granting agencies (e.g., N S F ) have policy forbidding such a learning experi-ence. T h e DVN, on t h e other hand, does provide this valuable service to its reviewers.
C r a n d a l l and Delcomyn note that the ability to write a useful review improves with experience. Crandall la-ments t h e fact that this experience is often gained at authors' expense. To help r e m e d y the situation, De-lcomyn provides a useful set of guidelines for reviewers that, though it derives from physiology, is pitched at a level general enough to be of cross-disciplinary use. The advantage of his guidelines over many others I have examined is that they contain within them the message that it is neither the task of reviewers nor editors to settle differing points of view in a given area of inquiry. Thus, if important questions raised in the introduction are an-swered through carefully controlled, well executed ex-periments, and the conclusions spring from the data, then the article should be accepted quite apart from whose particular theory or hypothesis is or is not being sup-ported.
C r a n d a l l addresses more formally the notion of train-ing reviewers by introductrain-ing the provocative idea of using prototype "ideal" reviews as guides. Filling in some of the required details, I would imagine that editors could locate in their files appropriate prototypie reviews that could b e reliably rated as evidence for: "Accept/as is";
"Accept/Revise"; "Reject/Revise/Resubmit"; and "Re-ject/Unconditionally." With the necessary identifying information removed and t h e content disguised, these can b e sent to authors to use in the same general manner that, for example, prototypie stages of cataract have been used to train ophthalmologists to classify cataract stages (i.e., Cicchetti et al. 1982).
T h e n e e d for such formal training of reviewers may have b e e n implicit though it appeared via a different route to Nelson. In a thoughtful commentary, she raises the issue of the specific process by which reviewers use information to arrive at publication or funding recom-mendations. She is right that very little is known about this process in peer review. Some findings reported a few
y e a r s back (Cicchetti & E r o n 1979) were d e s c r i b e d in m o r e detail in a s u b s e q u e n t BBS commentary (Cicchetti 1982). W e found that although there were high correla-tions between what reviewers perceived as " i m p o r t a n t "
a n d "well-designed" s t u d i e s and their t e n d e n c y to rec-o m m e n d publicatirec-on (rs b e t w e e n .62 [research design]
a n d .73 [importance]), t h e reliability of these ratings was appreciably lower (.19 a n d .28, respectively, as given in T a b l e 1 of the target article). Although r e v i e w e r s were
Regardless of what specific interpretations may be a p p r o p r i a t e , the more formal training of r e v i e w e r s would p r o b a b l y enable them t o use the same set of specific evaluative criteria m o r e consistently. Then t h e process t h a t reviewers use to arrive at a recommendation will h a v e become standard, reliable, and if applied appropri-a t e l y (e.g., using prototypic reviews appropri-as stappropri-andappropri-ards), vappropri-alid.
5. Concluding comments
A n u m b e r of commentators have suggested f u r t h e r inves-tigations to place the area of peer review on an e v e r more solid scientific foundation. Given the interdisciplinary n a t u r e of science - my strongest appeal is that t h e cross-disciplinary approach taken in this target article should be f u r t h e r encouraged in f u t u r e investigations. I w o u l d simply r e f e r the interested r e a d e r to the specific c o m m e n -taries of Bornstein, C o h e n , Gorman, H a r g e n s , Lock, M a r s h & Ball, Nelson, and Salzinger.
In my opinion, the training of reviewers, as well as e d i t o r s , authors, and c o n s u m e r s of research, is pivotal in increasing both the reliability and the validity of peer r e v i e w .
I have recently come across the first study of which I am a w a r e that broaches this topic directly. Oxman e t al. (in press) w e r e able to train successfully t h r e e classes of r e f e r e e s ("experts in research methodology," " M D s with r e s e a r c h training," and "research assistants" - t h r e e in e a c h group) to assess t h e overall scientific quality and o t h e r evaluation attributes of 36 review articles published in a w i d e range of journals in medicine (e.g., New En-gland Journal of Medicine), psychiatry (e.g., American J o u r n a l of Psychiatry), and psychology (e.g., Psychologi-cal Bulletin):
Following specific training (or practice) on r e v i e w arti-cles and an additional o n e h o u r training session, t h e 36 articles were evaluated independently by t h e n i n e re-v i e w e r s . For lere-vel of "ore-verall scientific q u a l i t y , " the intraclass R, across the n i n e examiners, was .71; Rf values for each of the three g r o u p s of reviewers, separately, w e r e , as follows: "Experts in research methodology" - R,
= . 7 7 (EXCELLENT); " M D s with research training" -R, = .74 (GOOD); and "Research Assistants" - -R, = .62 ( G O O D ) .
N i n e additional evaluative attributes, w e r e m e a s u r e d on 7-point ordinal scales with four anchorage points
provided for the scoring of each attribute (e.g., see Cicchetti et al. 1987). T h e nine attributes and their average R, values, across t h e nine judges, concerned the extent to which: (1) search methods were reported (R, >
.8, or E X C E L L E N T ) ; (2) a comprehensive search of the literature was conducted (R, > .6 or GOOD); (3) inclu-sion criteria were r e p o r t e d (Rä > .8 or EXCELLENT);
(4) selection biases w e r e avoided (R, > .6, or GOOD); (5) validity criteria w e r e reported (R, > .6, or GOOD);
(6) validity data w e r e reported (R, > .6, or G O O D ) ; (7) findings w e r e combined appropriately (Rj = .5 or FAIR;
(8) m e t h o d s for combining t h e data were reported (R, >
.6, or G O O D ) ; and (9) conclusions were supported by the data (R, = .40, or FAIR). Although t h e r e were somewhat lower levels of agreement among t h e research assistants than within t h e other two groups of reviewers, for eight of the 10 evaluations (items 1 - 7 and t h e overall evaluation of scientific quality) the differences in R, val-ues w e r e small. The average lower R, valval-ues of the remaining two attributes, however, were d u e to the very low Rj levels achieved by "research assistants" relative to the o t h e r two groups of evaluators. For rating t h e extent to which "the findings w e r e combined appropriately,"
the R, for "experts" was at .6 (GOOD) and for "MDs with research training," it was > .9 (EXCELLENT).
The corresponding R, for "research assistants," however, was in t h e very " P O O R " range, at > .2. Similarly, the extent to which "conclusions were supported by the data," t h e R / s for " e x p e r t s " and "MDs with research training" were beyond .6 (GOOD), whereas t h e corre-sponding R, for "research assistants" was again in the very P O O R range, or barely beyond the . 1 level.
These results to my knowledge are the first to demon-strate that reviewers of different levels of experience, can be taught to evaluate reliably the same scientific docu-ments. It is hoped that additional investigations of this kind will b e undertaken across a broad range of research topics both within and across disciplines. Following the lead of commentator Lock, I would also h o p e that the important issue of training p e e r reviewers will be dis-cussed at t h e 1992 Second World Conference on Peer Review.
Finally, in the open forum of "creative disagreement,"
I would extend a special invitation to those two commen-tators (Bailar and Kiesler), who were the most dubious about t h e need to study f u r t h e r the reliability and validity of t h e p e e r review process. I hope the panorama of ideas expressed by commentators across disciplines will con-vince t h e m of the n e e d to t u r n some of their own anec-dotal experiences into f u r t h e r valuable research in this area. As editors of prestigious journals in behavioral science and medicine, their f u t u r e insights and empirical investigations can make major contributions to the fur-ther understanding of t h e vicissitudes of p e e r review.
N O T E
1. Author is also affiliated with Yale University.
References
Abelzon. P. H . (1960) Scientific c o m m u n i c a t i o n . Scicnce 2 0 9 : 6 0 - 6 2 . (aDVC) Abramowitz, S. I . , Gomez, B. A A b m m o w i t z . C. V. (1975) Publish or politic:
Keferec bias in manuscript r e v i e w . Journal of Applied Social Psychology 5 : 1 8 7 - 2 0 0 . l a D V C )
Abt, H. A. (1988) W h a t h a p p e n s to r e j e c t e d astronomical papers? Publications of the Astronomical Society of the Pacific 100:506-08. ( a D V C ] Ad H o c W o r k i n g C r o u p for Critical A p p r a i s a l of the Medical L i t e r a t u r e (1987) Review Letters be c h a n g e d ? Physical Review Letters 43:1969-74. ( a D V C )
Allen, E . M. (1960) W h y are r e s e a r c h g r a n t applications d i s a p p r o v e d ? Science 132:1532-34. (aDVC)
Amabile, T . M. (1983) Brilliant b u t c r u e l : Perceptions of negative e v a l u a t o r s . Journal of Experimental Social Psychology 19:146-56. [RC]
American Psychological Association (1983) Publication manual, 3 r d ed. ( a D V C )
American Psychological Association (1985) Standards for educational and psychological testing. (RFB) Experimental Psychology: Human Perception and Performance 6 : 2 7 7 -89 ( D L )
Bakanic, V. McPhail, C. Ac Simon, R. J. (1987) The manuscript r e v i e w and decisionmaking process. American Sociological Report 5 2 : 6 3 1 -42. ( a D V C , LJS)
Bartko, J. J. (1966) T h e intraclass c o r r e l a t i o n coefficient as a m e a s u r e of reliability. Psychological Reports 1 9 : 3 - 1 1 . (aDVC) (1974) C o r r e c t i v e note to: " T h e I n t r a c l a s s Correlation Coefficient as a
M e a s u r e of Reliability." Psychological Reports 34:418. ( a D V C ) (1976) O n various intraclass c o r r e l a t i o n reliability coefficients. Psychological
Bulletin 83:762-65. (aDVC)
Bartko, J. J. Ac C a r p e n t e r , W. T. (1976) O n the methods and t h e o r y of reliability. Journal of Nervous and Mental Disease 163:307-17. ( a D V C ) Beck, A. T . (1976) Cognitive therapy and the emotional disorders.
I n t e r n a t i o n a l Universities P r e s s . ( a D V C )
Ben well, R. (1979) Authors a n o n y m o u s ? Physics Bulletin 30:288. ( a D V C ) Berelson, B. (1960) Graduate education in the United States. M c G r a w
-Hill. ( a D V C )
B e r n s t e i n , G . S. (1984) Scientific rigor, scientific integrity: A c o m m e n t on S o m m e r Ac S o m m e r . American Psychologist 39:1316. ( a D V C ] Beyer, J. M. (1978) Editorial policies a n d practices among leading j o u r n a l s in
f o u r scientific fields. The Sociological Quarterly 19:68-88. ( a D V C ) Blashfield, R. K. (1976) Mixture m o d e l tests of cluster analysis: A c c u r a c y of
technology, and society: A cross-disciplinary perspective, e d . I. Spiegel-Rosing Ac D . d e Solla Price. S a g e . ( a D V C )
B o e h m e , C . , van d e r Daele, W . Ac K r o h n , W . (1976) Finalization of science.
Social Science Information 1 5 : 3 0 6 - 3 0 . (aDVC)
Boor, M. (1986) Suggestions to i m p r o v e manuscripts submitted t o professional j o u r n a l s . American Psychologist. 4 1 : 7 2 1 - 2 2 . (RC)
Bornstein, R. F. (1990) M a n u s c r i p t r e v i e w in psychology: An a l t e r n a t i v e m o d e l . American Psychologist 45:67fc-73. (RFB)
(in p r e s s ) Publication politics, e x p e r i m e n t e r bias, and the r e p l i c a t i o n
Bradley, J. V. (1981) Pernicious publication practices. Bulletin of the Psychonomic Society 18:31-34. ( a D V C )
Braida, L. D . Ac Durlach, N. T. (1972) I n t e n s i t y perception. U. R e s o l u t i o n in o n e i n t e r v a l paradigms. Journal of the Acoustical Society of America 5 1 : 4 8 3 - 5 0 2 . (DL)
B r e n n a n , R. L. Ac Light, R. J. (1974) M e a s u r i n g a g r e e m e n t w h e n t w o o b s e r v e r s classify people into categories not defined in advance. British Journal of Mathematical and Statistical Psychology 27:154-63. ( r D V C ) Broad, W . J. (1988) Science can't k e e p u p with t h e flood of n e w j o u r n a l s . The of the American Medical Association 263:1405-08. (SPL) C h a l m e r s , T. C . , Frank, C. S. Ac R e i t m a n , D . (1990) Minimizing t h e t h r e e
stages of publication bias. Journal of the American Medical Association 2 6 3 : 1 3 9 2 - 9 5 . (SPL)
C h a s e , J. M. (1970) Normative criteria f o r scientific publication. American Sociologist 5:262-65. (aDVC) Journal of Clinical and Experimental Neuropsychology 1 0 : 6 0 5
-22. ( a D V C )
C i c c h e t t i . D . V. Ac C o n n , H. O . (1976) A statistical analysis of r e v i e w e r a g r e e m e n t and bias in evaluating medical abstracts. Yale Journal of Biology and Medicine 4 5 : 3 7 3 - 8 3 ( a D V C )
(1978) R e v i e w e r evaluation of m a n u s c r i p t s s u b m i t t e d to medical j o u r n a l s . P a p e r p r e s e n t e d to the American Statistical Association M e e t i n g s , San D i e g o , CA. (also abstracted in Biometrics (1978) 34:728) ( a D V C ) C i c c h e t t i , D . V. Ac E r o n , L. D. (1979) T h e reliability of manuscript r e v i e w i n g
for t h e Journal of Abnormal Psychology. Proceedings of the American Statistical Association (Social Statistics Section) 22:596-600. [ a D V C ] C i c c h e t t i . D . V. Ac Feinstein, A. R. (1990) High agreement but low k a p p a : II.
R e s o l v i n g t h e paradoxes. Journal of Clinical Epidemiology 4 3 : 5 5 1 -68. ( a r D V C )
C i c c h e t t i , D . V. Ac Fleiss, J. L (1977) C o m p a r i s o n of t h e null d i s t r i b u t i o n s of w e i g h t e d kappa and the C ordinal statistic. Applied Psychological Measurement 1:195-201. ( a D V C ) a s s e s s m e n t . In: Personality disorders: Diagnosis, management and course, e d . P. L. Tyrer. B u t t e r w o r t h Scientific Ltd. [ r D V C ]
C i c c h e t t i , D . V., Aivano, S. L. At Vitale, J. (1976) A c o m p u t e r program for q u a l i t a t i v e data. Educational and Psychological Measurement 38:805-13.
( a D V C )
(1988) Statistical power analysis for the behavioral sciences. 2 n d e d . L a w r e n c e E r l b a u m . ( r D V C , R R )
C o l e , J. Ac C o l e , S. (1973) Social stratification in science. U n i v e r s i t y of C h i c a g o Press. (aDVC)
(1981) Peer review in the National Science Foundation: Phase II of a study.
N a t i o n a l Academy of S c i e n c e ; . [ a D V C ]
(1985) E x p e r t s ' "consensus" a n d decision-making at t h e National Science F o u n d a t i o n . In: Selectivity in information systems: Sunyival of the fittest, e d . K. S. Warren. Praeger. ( a D V C )
C o l e , S. (1978) Scientific reward s y s t e m s : A comparative analysis. In:
Research in sociology of knowledge, sciences, and art, ed. R. A. Jones.
JAI Press. (aDVC)
(1972) Invisible colleges. University of Chicago Press. [ D L E ]
C r o n b a c h , L. J. (1981) C o m m e n t on " C h a n c e and consensus in p e e r review."
Science 214:1293. [ L L H ]
C u l l i t o n , B. J. (1984) F i n e t u n i n g p e e r review. Science 2 2 6 : 1 4 0 1 -02. [aDVC, RC]
D a r l e y , J. M. At Latane, B. (1968) B y s t a n d e r intervention in e m e r g e n c i e s : Diffusion of responsibility. Journal of Personality and Social Psychology 8 : 3 3 7 - 8 3 . [AMC]
Journal of Educational Psychology 21:48-52. [ D L ]
E i c h o r n , D . H . At VandenBos, C . R. (1985) Dissemination of scientific and
E s t e s , W . K. (1975)-Some targets for mathematical psychology. Journal of Mathematical Psychology 1 2 : 2 6 3 - 8 2 . [PHS]
Educational and Psychological Measurement 30:71-76. [ a D V C ] F i s h e r , A. (1989) Seeing atoms. Popular Science: 102-07. [JSA]
(1981) Statistical methods for rates and proportions, 2nd ed.
W i l e y . [aDVC, RR]
Fleiss, J. L. At Cicchetti, D. V. (1978) I n f e r e n c e about weighted kappa in the n o n - n u l l case. Applied Psychological Measurement 2 : 1 1 3 - 1 7 . [ r D V C ] Fleiss, J. L. Ar C o h e n , J. (1973) T h e e q u i v a l e n c e of weighted k a p p a and the
intraclass correlation coefficient as m e a s u r e s of reliability. Educational and Psychological Measurement 3 3 : 6 1 3 - 1 9 . [aDVC] beginner's guide. Royal C o l l e g e of Psychiatrists/Gaskell Books. [PT]
F u l l e r . S. (1989). Philosophy of science and its discontents. W e s t v i e w
Garcia, C . , Rosenfield, N. S., M a r k o w i t z , R. K., Seashore, J. H., Touloukian,
Garfield, E. (1972) Citation analysis as a tool in journal evaluation. Sciencc 178:471-79. [RFB]
C á r i u n k é i , J. M . . Ulshen. R. H . H a m r i c k , H. J őr Lawson, E E . (1990) P r o b l e m s identified by s e c o n d a r y r e v i e w of accepted manuscripts.
Journal of the American Medical Association 263:1369-71. [ r D V C . S P L ] G a m e r . W. R. (1962) Uncertainty and structure as psychological concepts.
Wiley. ( D L )
C a m e r , W . R. őr McGill, W. J. (1956) T h e relation between information and variance analyses. Psychometrika 2 1 : 2 1 9 - 2 8 . [arDVC], J B C Garvey, W . D . , Lin, N. őr Nelson, C . E . (1970) Some comparisons of
c o m m u n i c a t i o n activities in t h e p h y s i c a l and social sciences. In:
Communication among scientists and engineers, ed. C. E. Nelson őt D. K. Pollock. Health. [ S C ] British Journal of Mathematical and Statistical Psychology 3 8 : 5 8 -66. [ r D V C ]
G i l m o r e , J. B. (1979) Illusory reliability in journal reviewing. Canadian Psychological Review 2 0 : 1 5 7 - 5 8 . [ a r D V C , JBG]
G l e n n , N. D. (1976) T h e journal a r t i c l e r e v i e w process. Some proposals for c h a n g e . American Sociologist 1 1 : 1 7 9 - 8 5 . [aDVC]
G o o d m a n , L. A. & Kniskal, W. H. (1954) Measures of association for cross classifications. Journal of the American Statistical Association 4 9 : 7 3 2 -64. [aDVC]
G o o d r i c h , D. W. (1945) An analysis of m a n u s c r i p t s received by t h e e d i t o r s of the American Sociological Review f r o m May 1, 1944, to S e p t e m b e r 1, 1945. American Sociological Review 10:716-25. [aDVC]
G o o d s t e i n , L. D. (1982) W h e n will t h e e d i t o r s start to edit? Behavioral and Brain Sciences 5:212-13. [ L J S ]
G o o d s t e i n , L. D. őc Brazis, K. L. (1970) Credibility of psychologists: An empirical study. Psychological Reports 27:835-38. [aDVC, JSA]
G o r d o n . M. D . (1977) Evaluating t h e evaluators. New Scientist 7 3 : 3 4 2 investigation. Quarterly Journal of Experimental Psychology, 41A, 3 8 5 -412. [ M E C ]
G o r m a n , Michael E. őt G o r m a n , M a r g a r e t E. (1984) A comparison of disconfirmatory, confirmatory a n d a control strategy on Wason's 2, 4, 6 task. Quarterly Journal of Experimental Psychology 12:129-40. [ M E C ] G o t t f r e d s o n , S. D. (1978) E v a l u a t i n g psychology research reports: (1976) An editorial. Journal of Personality and Social Psychology 3 3 : 1
-7. [ a D V C ] Guilford, J P (1954) Psychometric methods, 2nd ed McCraw-Hill (RR Culliksen, H . O (1950) Theory of mental tests Wiley [DL. LJSi G u y a t t . G. H . T o w n s e n d , M őr Berman. L (1987) A comparison of Liker*
and visual analogue scales for measuring c h a n g e in function Journal of Chronic Diseases 40:1129-33 [rDVC]
Hall, J. A. (1979) Author review of reviewers. American Psychologist 34 798. [ a D V C ]
H a r g e n s , L. L. (1988) Scholarly consensus and journal rejection rates American Sociological Review 53:139-51. [aDVC, SC]
(1990) Variation in journal peer-review systems Possible causes and c o n s e q u e n c e s . Journal of the American Medical Association 263 1 3 4 8
e d . (1983) Peer commentary on peer review: A case study in scientific quality control. C a m b r i d g e University Press (reprinted from Behavioral and Brain Sciences, vol. 5). [aDVC]
(1985) Rational disagreement in p e e r review. Science, Technology O Human Values 10(3):55-62. [aDVC, LJS) i n p u t . Proceedings of the American Statistical Association (Statistical C o m p u t i n g Section) 21.366-70. [aDVC] American Psychological Association (APA) Monitor 16:18. [aDVC]
H o r r o b i n , D. F. (1990) T h e philosophical basis of p e e r review and the s u p p r e s s i o n of innovation. Journal of the American Medical Association 2 6 3 : 1 4 3 8 - 4 1 . [JSA] H u n t . E . (1971) Psychological publications. American Psychologist
26:311. [ a D V C ]
Ison, J. R. (1985) T h e granting system and healthy research. Science 230:376. [ a D V C ]
Iyengar, S. őr C r e e n h o u s e , J. B. (1988) Selection model and the file d r a w e r h y p o t h e s i s . Statistical Science 3 3 : 1 0 9 - 3 5 . | P H S )
J e s t e a d t , W . , L u c e , R. D. őt G r e e n , D. M. (1977) Sequential effects in j u d g m e n t of loudness. Journal of Experimental Psychology: Human Perception and Performance 3 : 9 2 - 1 0 4 . [ D L ]
J e s t e a d t , W . , W i e r , C . C. őr G r e e n , D. M. (1977) Intensity discrimination as a f u n c t i o n of f r e q u e n c y and sensation level. Journal of the Acoustical Society of America 61:169-77. | D L ]
J o n c k h e e r e , A. R. (1970) T e c h n i q u e s for o r d e r e d contingency tables. I n : Proceedings of the NUFFIC International Summer Session in Science, Het Oude Hof, ed. J. B. Riemcrsma őr H. C. van der Meer. T h e H a g u e . [ a D V C ]
J o n e s , R. (1974) Rights, wrongs, and r e f e r e e s . Neu: Scientist 61:758-59 l a D V C ]
K a h n e m a n , D . . Slovic, P. Ac Tversky. A., eds. (1982) Judgment under uncertainty: Heuristics and biases. C a m b r i d g e University Press. [ H L R ] Kamin. L. J. (1981) The intelligence controversy, ed. H. J. Eysenck.
Wiley IPHSJ
Kazdin, A. E. (1982) Single-case research designs: Methods for clinical and applied settings. Oxford University Press. [aDVC]
Kerr, S.. Tolliver, J. At P e t r e e , D. (1977) Manuscript characteristics which i n f l u e n c e a c c e p t a n c e for m a n a g e m e n t and social science journals.
Academy of Management Journal 2 0 : 1 3 2 - 4 1 . [aDVC]
Klayman, J. & Ha. Y.-W. (1987) C o n f i r m a t i o n , disconfirmation and information in hypothesis testing. Psychological Review 94:211-28. | M E C )
(1988) A s s e s s m e n t of 2x2 associations: Generalization or signal-detection m e t h o d o l o g y . The American Statistician 42:37-49. [rDVC]
precision of absolute j u d g m e n t s . Quarterly Journal of Experimental Psychology 42A:239-54 [ D L ] scientific debate. University of California Press. [aDVC]
Lawlis, G . F. At Lu, E. (1972) J u d g m e n t of counseling process: Reliability.
a g r e e m e n t , and error. Psychological Bulletin 7 8 : 1 7 - 2 0 . [aDVC]
Lazarus, D. (1982) I n t c r r e f e r e e a g r e e m e n t and a c c e p t a n c e rates in physics.
Behavioral and Brain Sciences 5:219. [aDVC]
L e L e w i s . D. At B u r k e , C. J. (1949) T h e use and misuse of the chi s q u a r e t e s t . Psychological Bulletin 4 6 : 4 3 3 - 8 9 . [ r D V C ]
L e a c h , C. (1979) Introduction to statistics: A nonparametric approach for the social sciences. John Wiley. [ a D V C ]
L i n d s e y , D. (1977) Participation and influence in publication review p r o c e e d i n g s . American Psychologist 3 2 : 3 7 9 - 8 6 . [RFB]
(1978) The scientific publication system in social science. Jossey-Bass. [ a D V C , R F B ]
(1988) Assessing precision in t h e manuscript review process: A little b e t t e r than a dice roll. Scientometrics 14:75-82. [ L L H ]
Lock, (1985) A difficult balance: Editorial peer review in medicine. 1SI Pre«s. [ a D V C , JF)
Lodahl, J. B. (1970) Paradigm d e v e l o p m e n t as a source of consensus in scientific fields ( u n p u b l i s h e d m a s t e r ' s thesis). [aDVC]
L o r d , F. N. At Novick, M. R. (1968) Statistical theories of mental test scores.
A d d i s o n - W e s l e y . [ L L H ]
L u c e , R. D . (1989) A history of psychology in autobiography, vol. 8, e d . C . Lindzey. Stanford University Press. [ P H S ]
L u c e . R. D. At C r e e n , D . M. (1978) T w o tests of a neural attention hypothesis for auditory psychophysics. Perception O Psychophysics 2 3 : 3 6 3 - 7 1 . [ D L ]
Luce. R. D., Nosofsky, R. M . , C r e e n , D. M. Ac Smith, A. F . (1982) T h e bow and sequential e f f e c t s in absolute identification. Perception 6 Psychophysics 3 2 : 3 9 7 - 4 0 8 . | D L ]
Machol, R. (1981) L e t t e r to t h e editor. The Sciences 21:xxi. [ a D V C ] M ä h e r . B. A. (1978) A r e a d e r ' s , writer's, and reviewer's g u i d e to assessing
research r e p o r t s in clinical psychology. Journal of Consulting and Clinical Psychology 4 6 : 8 3 5 - 3 8 . [aDVC]
Mahoney, M. J. (1976) Scientist as subject: The psychological imperative.
Ballinger [ L L H ]
(1977) Publication p r e j u d i c e s : An experimental study of c o n f i r m a t o r y bias in the p e e r review s y s t e m . Cognitive Therapy Research 1 : 1 6 1 - 7 5 . |aDVC, L D N . P H S , S P L , JSAJ
(1978) Publish and p e r i s h . Human Behavior 7 : 3 8 - 4 1 . [ a D V C ] (1982) Publication, politics, and scientific progress. Behavioral and Brain
Sciences 5 : 2 2 0 - 2 1 . [ A M C ]
Marsh. H . W. Ac Ball, S. (1981) Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology 73:872-80. [ a D V C ] , H W M Observation scales to identify serious illness in febrile c h i l d r e n . Pediatrics 70:802-09. [ r D V C ]
(unpublished doctoral dissertation). State University of N e w York at Stony Brook. [ S C ]
Mezzich, J. E., K r a e m e r , H . C . , Worthington, D. R. L. Ac C o f f m a n , G. A.
(1981) Assessment of a g r e e m e n t among several r a t e r s f o r m u l a t i n g multiple diagnoses. Journal of Psychiatric Research 1 6 : 2 9 - 3 9 , [rDVC]
Mitroff, 1. I. At C h u b i n , D. E . (1979) Peer review at t h e N S F : A dialectical policy and analysis. Social Studies of Science 9 : 1 9 9 - 2 3 2 . Sage [aDVC]
Mulkay, M. (1977) Sociology of t h e scientific research c o m m u n i t y . In: Science, technology, and society, ed. I. Spiegel-Rosing Ac D . d e Sofia Price.
Sage. [aDVC]
Mulkay, M. At William, A. T . (1971) A sociological study of a physics d e p a r t m e n t . British Journal of Sociology 22:68-80. [ S C ] M u r p h y . R. J. L. (1978) Reliability of marking in eight G C E examinations.
British Journal of Educational Psychology 48:196-200. [ D L ] (1982) A f u r t h e r r e p o r t of investigations into the reliability of marking of
C C E examinations. British Journal of Educational Psychology 52:58-63. [ D L ]
National Research C o u n c i l (1988) T h e behavioral and social sciences:
Achievements a n d o p p o r t u n i t i e s . National Academy Press. [MJM]
Nelson, L., Satz, P . , M i t n i s h i n a , M., Van C o r p , W . , C i c c h e t t i , D . , Lewis, R.
Ac Van Lancker, D . (1989) D e v e l o p m e n t and validation of t h e Neuropsychology B e h a v i o r and Affect Profile. Psychological Assessment: A Journal of Consulting and Clinical Psychology 1:266-72.
[ r D V C ]
N e w m a n , H., F r e e m a n , F. Ac Hoizinger, K. (1937) Twins. A study of heredity and environment. University of Chicago Press. [ P H S ]
Newman, S. H. (1966) I m p r o v i n g t h e evaluation of submitted m a n u s c r i p t s . American Psychologist 2 1 : 9 8 0 - 8 1 . ( a D V C )
Nisbett, R., Ót Ross, L. (1980) Human inference: Strategies and shortcomings in human judgments. P r e n t i c e - H a l l . [ H L R ]
Nunnally, J. C. (1978) Psychometric theory, 2nd ed. McGraw-Hill. [ a D V C ] Orlinsky D. ót H o w a r d , K. (1978) T h e relation of process to o u t c o m e in
Selectivity in information systems: Survival of the fittest, e d . K. S.
W a r r e n . P r a e g e r Scientific. ( a D V C ) particularism: Journal publication in t h r e e scientific disciplines. Social Forces 5 5 : 9 3 8 - 5 1 . [aDVC]
Price, D. d e Sollt (1963) Little science, big science. Columbia University Press. ( a D V C )
Reid, L. N., Soley, L. C. i t W i m m e r , R. D . (1981) Replications in advertising r e s e a r c h : 1977, 1978, 1979. Journal of Advertising 1 0 : 3 -13. ( a D V C )
Relman, A. S. (1978) Are journals really quality filters? Rockefeller Foundation working papers ( c o n f e r e n c e , May 22-23). C o p i n g with t h e review. Journal of the American Medical Association 2 5 6 : 2 3 9 1 -92. ( M J M )
In: Scientific excellence: Origins and assessment, ed. D. N. Jackson and J. P. R u s h t o n . Sage. ( L L H , H L R ] Accuracy of t h e barium e n e m a examination. Radiology 150:393-400. [ r D V C ]
Rosenthal, R. (1979) T h e "file d r a w e r p r o b l e m " and tolerance for null results.
Psychological Bulletin 8 6 : 6 3 8 - 4 1 . [ P H S ]
(1984) Meta-analytic procedures for social research. Sage. | RH]
(1987) Judgment studies: Design, analysis, and meta-analysis. C a m b r i d g e U n i v e r s i t y Press. [RR]
Rosenthal, R. Ót Rosnow, R. L. (1984) Essentials of behavioral research Methods and data analysis. McGraw-Hill. [RR]
(1985) Contrast analysis: Focused comparisons in the analysis of vanance.
C a m b r i d g e University Press. [RR]
Rubin, D . B. (1982) Rejection, rebuttal, revision: S o m e flexible features of p e e r r e v i e w . Behavioral and Brain Sciences 2 : 2 3 6 - 3 7 . [PHS]
Scarr, S. (1982) Anosmic peer review: A rose by a n o t h e r name is e v i d e n t l y not a rose. Behavioral and Brain Sciences 5 : 2 3 7 - 3 8 . [aDVC]
Scarr, S. ót W e b e r , B. L. R. (1978) T h e reliability of reviews for t h e A m e r i c a n Psychologist. American Psychologist 33:935. [aDVC, LJS]
S c h ö n e m a n n , P. H . (1971) T h e m i n i m u m a v e r a g e correlation between bias? Journal of the American Medical Association 263:1390-91. [ S P L ] Shrout, P. E . , Spitzer, R. L. ót Fleiss, J. L. (1987) Quantification of
S c h ö n e m a n n , P. H . (1971) T h e m i n i m u m a v e r a g e correlation between bias? Journal of the American Medical Association 263:1390-91. [ S P L ] Shrout, P. E . , Spitzer, R. L. ót Fleiss, J. L. (1987) Quantification of