• NOTES
A presentation based o n an early version o f this paper was m a d e at the Second A n -nual Meeting o f the Society for Social Studies of Science, held in B o s t o n , Massachusetts, and Harvard University, 14-16 October 1977. T h e incisive comments o f P. T h o m a s Carroll on that version were most helpful in rethinking and rewriting.
t. See T. Gustafson, 'The Controversy Over Peer Review', Science, Vol. 190 (2 December 1975), 1060-66; D. Shapley, ' H o u s e Votes Veto Power on All N S F Research Orants', ibid., Vol. 188 (25 April 1975), 338-41; Shapley, ' N S F Violations o f Personnel C o d e Alleged', ibid. (31 May 1975), 915; J. Walsh, ' N S F and Its Critics in Congress: New Pressures on Peer Review', ibid. (6 June 1975), 999-1001; Walsh, ' N S F H o u s e Appropriations Panel Gives Warning Tug on Purse Strings', ibid., V o l . 189 (4 July 1975), 26-28; Walsh, "NSF Peer Review Hearings: H o u s e Panel Starts with Critics', ibid. (8 August 1975), 435-37; Walsh, 'Peer Review: NSF Faces C h a n g e s , the Question is H o w Extensive', ibid., Vol. 190(17 October 1975), 253-56.
2. S. MacLane, 'Peer Review and the Structure of Science,' Science, Vol. 190 (14 N o v e m b e r 1975), 617.
3. G. M . Lyons, The Uneasy Partnership: Social Science and the Federal Government in the Twentieth Century ( N e w York: Russell Sage, 1969). See also J.
Haberer, Politics and the Community of Science (New York: Van Nostrand R e i n h o l d , 1969).
4. Our mention o f this caveat stems f r o m two sets o f experiences: firstly, in response t o the first public draft of this paper, where colleagues rhetorically asked, ' Y o u ' r e not against peer review, are y o u ? ' ; and secondly, our inability to convince N S F programme managers that despite the recent studies o f their system, the 'crucial experiment' had yet to be performed. We claimed that the present system had been legitimated, but not accurately appraised because (1) many of the right questions had not been asked, or if they h a d , the answers were not carefully weighed (for example, could not be quantified), or ( 2 ) the crucial data were not in the public d o m a i n , but with special consent could be m a d e available for analysis. Consequent-ly, the study whose findings could be used t o alter policy and strengthen the opera-tion o f the system never materialized. Our interpretaopera-tion may smack of sour grapes, but w e think the opportunity for internal self-scrutiny was lost (although congres-sional critics were pacified — for the m o m e n t ) . W e do not think our argument fell o n deaf ears; rather, n o o n e was willing to handle a hot potato which had temporari-ly c o o l e d .
5. For a review, see H . Zuckerman and R. K. Merton, ' A g e , Aging, and A g e Structure in Science', in M. W. Riley, M. Johnson and A. Foner (eds), A Theory of Age Stratification, Vol. 3, Aging and Society (New York: Russell Sage, 1972), 292-356.
6. In general, referees and advisors are older, m o r e eminent and published, and located at m o r e prestigious institutions than the 'average' member o f the scientific community. S e c N. C. Mullins, ' T h e Structure o f an Elite: The Advisory Structure of the Public Health Service', Science Studies, Vol. 2 (1972), 3-29; L. Groeneveld, N.
Koiier and N. Mullins, 'The Advisers of the U S National Science Foundation', Social Studies of Science, Vol. 5 (August 1975), 343-54; M. J. M u l k a y , 'The Mediating R o l e o f the Scientific Elite', Social Studies of Science, Vol. 6 (1976), 445-70.
7. Studies o f NIH peer review include G. M. Carter, 'Peer Review, Citations, and Biomedical Research Policy: N I H Grants to Medical School Faculty' (Santa Monica, C a l i f . : Rand Corporation Report R - 1 5 8 3 - H E W , 1974); C. H e n l e y , 'Peer Review of Research Grant Applications at the National Institutes of Health', Federa-tion Proceedings, Vol. 36 (July 1977), 2066-68, 2 1 8 6 - 9 0 , 2335-38; N I H Grants Peer Review Study T e a m , 'Grants Peer Review: Report t o the Director, N I H , Phase I, Vols. I-IIF (Bethesda, Md.: National Institutes o f Health, December 1976).
8. A provocative formation o f this relationship c a n be found in J.-J. Salomon, 'The Mating o f Knowledge and Power', Impact of Science on Society, Vol. 22 (January-June 1972), 123-32.
9. By 'dialectic' we mean m o r e than a mere polar opposition or c o n f l i c t bet-ween viewpoints. We mean that viewpoints are intensely opposed to o n e another, though the meaning of one is dependent on the other; that is, either viewpoint is completely self-contained but is d e f i n e d , if only in part, through the other. Be this as it may, we are more interested at present in the operational use of the dialectic as a unique m e t h o d o l o g y for analyzing issues, rather than in quibbling about the various historical meanings of the term.
10. C. West Churchman, The Design of Inquiring Systems (New York: Basic Books, 1971); P . Feyerabend, Against Method ( L o n d o n : New Left B o o k s , 1976); G.
Holton, Thematic Origins of Scientific Thought (Cambridge, Mass.: Harvard University Press, 1973); A. Kantrowitz et al., 'The Science Court Experiment, an In-terim Report', Science, Vol. 193 (20 August 1976), 653-56; M. Levine, 'Scientific Method and the Adversary M o d e l : Some Preliminary Thoughts', American Psychologist, Vol. 29 (September 1974), 661-77; R. O . Mason, 'A Dialectical Ap-proach to Strategic Plarfning', Management Science, V o l . 15 (1969), B-403-14; R. K.
Merton, Sociological Ambivalence (New York: T h e Free Press, 1976); D . Nelkin, 'Thoughts on the Proposed Science Court', Newsletter on Science, Technology, and Human Values, N o . 18 (1977), 2 0 - 3 1 .
11. C h u r c h m a n , Feyerabend and Mason, ops. cit. note 10.
12. ^ l a s o n , o p . cit. note 10.
13. ' N a t i o n a l Science F o u n d a t i o n Peer Review, V o l u m e I', A Report o f the Sub-committee o n Science, Research and Technology o f the Committee o n Science and Technology, U S House of Representatives, Ninety-Fourth Congress, Second Session (January 1976).
14. For example, Gustafson, o p . cit. note 1.
15. Verbatim quotes have been purposefully included in Table 1 t o indicate the depth and sincerity with which the respective parties hold their views. In all cases, the assumptions are direct or abridged quotes excerpted from the d o c u m e n t cited in note 13.
16. C h u r c h m a n and Mason, o p s . cit. note 10.
17. Gustafson, o p . cit. note I, 1060 (our italics).
18. Subcommittee, op. cit. note 13, 571 (our italics).
19. Ibid., 579-80.
20. Ibid., 2.
21. Ibid., 25.
22. Ibid.s'27.
23. Ibid., 41.
24. Ibid., 43.
25. D . Hensler, 'Perceptions of the National Science Foundation Peer Review Process: A Report on a Survey of N S F Reviewers and A p p l i c a n t s ' , Prepared for the C o m m i t t e e on Peer Review, National Science Board, and t h e Committee on S c i e n c e and Technology, U S H o u s e of Representatives (Washington, DC: NSF 7 7 - 3 3 , December 1976); R. A b e l , 'Applicants' and Reviewers' A s s e s s m e n t s of the N S F Peer Review Process', International NSF d r a f t paper (Washington, DC: NSF, N o v e m b e r 1976).
26. S. Cole, L. Rubin and J. R. C o l e , 'Peer Review and the Support of Science', Scientific American, Vol. 237 (October 1977), 34-41. This article is an interim report o n the project, which is still in progress (see note 41, b e l o w ) .
27. One o f us ( I . I . M . ) was asked t o undertake such a review for the O f f i c e o f Planning and Policy Analysis, National Science Foundation, under Contract N o . O M , Order N o . 77-SP-0370.
28. Hensler, o p . cit. note 25, 15-18, quote on 17.
29. Ibid., 23.
30. Ibid., iv. Furthermore, as Hensler explains (considering both successful and unsuccessful applicants), principal investigators'
evaluations o f the appropriateness o f the review procedures used are related t o disposition o f the proposal. But e v e n a m o n g those w h o s e proposals were declin-ed, half feel the procedures were appropriate. A majority o f unsuccessful ap-plicants feel that the decision to d e c l i n e was unfair but a substantial proportion
— forty-three percent — feel that [even this] decision w a s fair. About 84 percent o f declinees w h o thought the decision w a s unfair, say t h e y would have appealed the decision if a formal appeals process had existed. Assessments o f ap-propriateness o f procedures and fairness o f the funding decision do not a p p e a r t o be related to academic generation, institutional affiliation or region.
However, those w h o have served as N S F reviewers or w h o have received N S F grants in the past are more likely to evaluate their m o s t recent experience positively — even if they were turned d o w n — than t h o s e with less successful ex-perience dealing with NSF . . . A b o u t 73 percent o f the P i s , including b o t h grantees and declinees, would favor N S F adopting a f o r m a l appeals system. T h e reason for supporting such a system which is volunteered most frequently is that it would provide a remedy for mistakes and misjudgments; the leading reason f o r o p p o s i n g it is that it will further bureaucratize and b u r d e n the review process, (ibid., v-vi).
31. A s many writers have indicated, what is perceived as an innovative idea is relative to time and place in any research community. The line separating innovation f r o m charlatanism or, in the lexicon o f the exemplary 'science studies' literature, the d i f f e r e n c e between transgressions o f c o g n i t i v e norms and true anomalies, is fine in-deed. Our view is that 'excessively' innovative ideas will so challenge the paradigmatic f o u n d a t i o n s o f a research area that the i n n o v a t o r s and their ideas
will neither gain ready access to the literature nor approval of proposals t o pursue their research programme. Mainstream thought, in short, can sustain only moderate innovation. T h e issue, in the c o n t e x t o f peer review, is whether the agencies which administer the system and its resources are the guardians of the mainstream or a refuge for innovators. Surely they are a little of both; hence, the issue o f bias and evidence are elusive at best. For related discussion, see below, plus T. S. K u h n , 'Se-cond Thoughts on Paradigms', in F. Suppe (ed.), The Structure of Scientific Theories ( U r b a n a , III.: University o f Illinois Press, 1974), 459-82; M. J. Mulkay, The Social Process of Innovation (London: Macmillan, 1972); H. M. Collins, 'The TEA Set: Tacit Knowledge and Scientific Networks', Science Studies, Vol. 4 (1974), 165-86; D . E . Chubin, 'The Conceptualization o f Scientific Specialties', Sociological Quarterly, Vol. 17 ( A u t u m n 1976), 448-76, esp. 459-70.
32. While Hensler does not s p e c i f y these 20 institutions, the following ten US universities have been identified as receiving more than a third of all federal expen-ditures in universities, producing about a third of all t h e doctorates, and providing 37 percent o f the members of federal review panels in the 1960s: California, Caltech, Chicago, C o l u m b i a , Cornell, Harvard, Illinois, MIT, Michigan and Minnesota. See W. Hirsch, Scientists in American Society (New York: Random House, 1968), 106.
33. These findings accord with previous perceptions, as Hensler (op. cit. note 25, 50) observes that 'reviewers in general, and applicants who have also served as reviewers, are significantly less likely to perceive bias in the process than other ap-plicants . . . . Apap-plicants who h a v e not been successful in obtaining N S F grants recently or in the past are most likely to think that process is biased.'
34. Hensler, o p . cit. note 25, 84.
35. Ibid., 1.
36. Ibid.
37. It could well be the case that those who have either experienced the most 'bias' (no matter h o w it is defined) or w h o attribute bias t o the system may b e those who have either (a) 'dropped o u t ' o f the system prior to the sampling period or (b) never 'dropped in' in the first place. Without sampling this group, such conjec-tures simply c a n n o t be evaluated; yet they cannot be dismissed out of hand. In this respect, Hensler can be criticized for a uniform lack o f conjecture; she apparently feels no c o m p u l s i o n to explain why her survey generated the responses it did.
38. This q u o t a t i o n appears in an earlier version of Hensler's report (op. cit. note 25), dated September 1976, on p. 80.
39. G. G o r d o n and E. V. M o r s e , 'Creative Potential and Organizational Struc-ture', Academy of Management Journal, Vol. 12 (1969), 37-49; I. I. M i t r o f f , The Subjective Side of Science, A Philosophical Inquiry into the Psychology of the Apollo Moon Scientists (New York: Elsevier, 1974); E . V. Morse and G. G o r d o n , 'Cognitive Skills: A Determinant o f Scientists' Local-Cosmopolitan Orientation', Academy of Management Journal, Vol. 17 (1974), 7 0 9 - 2 3 .
40. A prior question (underlying all of these) c o n c e r n s the distribution o f in-novativeness in the community at large, and whether the purposive sampling of scientists for the role of peer reviewer proportionately captures this characteristic.
Given the profile o f reviewers d e v e l o p e d in the works cited in note 6, o n e would think so. Indeed, one would think that innovators are overrepresented a m o n g reviewers; likewise, one would hope that innovators are overrepresented a m o n g the recipients of research funds. (As far as we can tell, there are no questions in the
Hensler study dealing directly with the topic of 'grantsmanship', although it is im-plied by a few of her questions.)
41. Cole et al., o p . cit. note 26. The longer version of this report had yet to ap-pear when we were preparing the final revision of this paper. It was published while the paper was in press: see S. Cole, L. Rubin and J. R. Cole, Peer Review in the Na-tional Science Foundation: Phase One of a Study (Washington, DC: NaNa-tional A c a d e m y of Sciences, 1978). We have been unable to incorporate and respond to details o f the published report. An earlier draft of our paper benefitted from an un-published preliminary version o f the Cole report forwarded to I.I.M. as part of his review (see note 27). However, since it was the only public version available at the time, we have taken special pains to confine our comments here to the published in-terim Scientific American report.
42. Ibid., 36.
43. Ibid., 37-39.
44. Ibid., 37.
45. Ibid., 38.
46. Ibid., 40 (our italics).
47. Ibid.
48. Ibid., 41.
49. Ibid.
50. Ibid., 38.
51. The use of citations as a measure or indicator of the perceived importance or standing o f a scientist within his or her domain of research seems non-problematic to C o l e , despite the reservations expressed by some investigators regarding the validity o f the measure. See D. E. Chubin and S. D. Moitra, 'Content Analysis of References: Adjunct or Alternative to Citation Counting?' Social Studies of Science, Vol. 5 (1975), 432-41; N . Kaplan, 'The Norms of Citation Behavior: Prologomena to the Footnote', American Documentation, Vol. 16 (1965), 179-84. Do numerous citations to the work o f a scientist truly reflect the long-term importance and significance of the work, or merely its short-term popularity? Does one cite a work to support one's o w n , or for more critical and rhetorical reasons? Even if it is presumed that the reviewer has knowledge of citation performance, without taking into account the reasons why scientists cite others, and why variations in citation behaviour exist across disciplines and research areas, can one confidently use cita-tion counts as a ' c o m m o n denominator' predictor variable? On this score, Cole finds — interestingly enough — that past citations contribute little to the explained variance o f the funding decision, suggesting to us that citation may be an irrelevant criterion of performance because most reviewers are ignorant o f an applicant's cita-tion 'performance'.
52. Cole et al., o p . cit. note 26, 39.
53. This is really an expression of our dismay over Cole's decision to report only the results o f quantitative analyses. Such analyses can mask individual differences manifested in anecdotal accounts, such as interviews. For example, it would be in-structive, if not indispensable, to know how NSF personnel view the system of science, and the rationality o f the enterprise in which they are engaged. In the study of eminent scientists w h o investigated the moon rocks returned by the Apollo mis-sions (see note 39), it was found that nearly the entire sample of forty-two scoffed at, antf in the most derisive of terms, the stereotypical view of the scientist and science itself as the 'open, free, unbiased exchange of pure ideas' that is so
c o m m o n l y portrayed in college texts and in popular accounts of science. Surely it is important to know whether a respondent holds a conventional or a radical view o f the workings o f science before one can properly evaluate the respondent's attitudes towards the peer review system. Surely programme directors differ in discharging their duties. Are those harbouring a less conventional view of science m o r e critical and sceptical o f the operation o f N S F peer review?
54. Gordon and Morse, op. cit. note 39.
55. Ibid., 42. In the typical situation used by Gordon to measure differentiation, a person is asked to rate ten o f his most immediate colleagues, friends, associates, and so on, on a ten-point scale with respect to (a) their productivity, (b) their creativity, and (c) h o w easy it is to get along with the individual being rated. Low differentiaiors tend to rate all ten persons identically; in other words, low differen-tiators make use of only a small portion of the total ten-point scale, whereas high differentiators tend to make significantly more use o f the whole scale. High dif-ferentiators tend to view people as different and unique; low difdif-ferentiators view them as the same.
56. Cole ct al., o p . cit. note 26, 34.
57. To this end, w e are surprised that the Cole study was not couched in terms o f the Ortega hypothesis which the Coles (J. R. Cole and S. Cole, 'The Ortega Hypothesis', Science, Vol. 178 [27 October 1972], 368-75) investigated a few years a g o . The Coles' rejection of this hypothesis (that all scientists contribute through their modest research efforts to the incremental progress o f science) raises questions as to the concentration of funding support among a small portion of the research community. S. J. Turner and D. E. Chubin, in 'Another Appraisal of Ortega, the C o l e s and Science Policy: The Ecclesiastcs Hypothesis', Social Science Information, Vol. 15 (1976), 657-62, argue that to equate the distributions o f scientific talent, pro-ductivity, and reward is little justification for a science policy that deliberately con-centrates resources a m o n g the elite that populates one tail of those distributions.
Rather, they question the efficiency o f a policy that would waste the talents o f train-ed personnel without modifying the organizations that train and employ them (though we realize this is far easier said than done). T o sustain the research o f more scientists could calculably enhance their contributions. Yet no experiments in the democratization o f research allocation have been carried out. Thus the proposition remains untested, and for us at least, the Ortega hypothesis, like the 'old boy' and 'rich get richer' hypotheses to which it is intimately related, has been gratuitously laid to rest by the Coles.
58. An 'unbiased' system would obviously not be o n e that randomly f u n d s pro-posals; rather, it would fund primarily according to merit, which might be defined as innovative, feasible, relevant, or s o m e combination thereof. In the discussion that f o l l o w s we assume that a lack of bias is both desirable and attainable.
59. We purposefully use the term 'admittedly oversimplified' because the actual situation may be t o o complex to admit o f the two exclusive categories, 'biased' or 'unbiased'. The actual state of the system may be neither biased nor unbiased, or it may be a condition o f both — that is, a complicated mixture of partially 'biased' and 'unbiased' elements. Nevertheless, for the purpose o f this analysis, it s u f f i c e s to consider the 'idealized' cases in Table 2.
60. Notice that w e d o not say that Cases II and IV necessarily represent 'incor-rect cases', since the difficulty in knowing the 'true' state of the actual system also
makes it difficult to know or assess 'error'; the term 'problematic' is more ap-propriate than such decisive terms as 'truth' or 'error', since complex social systems may not admit o f such rigid or precise determinations.
61. P. Feyerabend, op. cit. n o t e 10; A. H. Maslow, The Psychology of Science (New York: Harper and R o w , 1966); D. C. McClelland, 'On the Dynamics of Creative Physical Scientists', in L. Hudson (ed.), The Ecology of Human In-telligence (Harmondsworth, M i d d x . : Penguin B o o k s , 1970); Mitroff, o p . cit. note 39.
62. McClelland, op. cit. n o t e 61.
63. On this issue, Kuhn and L a k a t o s appear t o agree. See S. S. Blume, Toward a Political Sociology of Science ( N e w York: Free Press, 1974); T. S. Kuhn, 'The Essential Tension: Tradition and Innovation in Scientific Research', in C. W.
Taylor and F. Barron (eds). Scientific Creativity, Its Recognition and Development (New York: Wiley, 1963), 341-54.
64. M i t r o f f , o p . cit. note 39.
65. M u l k a y , o p . cit. note 6.
66. Scientists are not only aware o f their relative position (for example, depart-ment or institution rank), but they tend to aggrandize their position relative to their perception o f other departments and institutions. Sec T. Caplow and R. J. McGce, The Academic Marketplace ( N e w York: Basic B o o k s , 1958).
67. For a review, see M u l k a y , o p . cit. note 6; M. J. Mulkay, 'The Sociology of the Scientific Research C o m m u n i t y ' , in I. Spiegel-Rösing ánd D. de S. Price (eds), Science, Technology and Society: A Cross-Disciplinary Perspective (Beverly Hills:
Sage, 1977), 93-148; and P. B o f f e y , The Brain Bank of America ( N e w York:
McGraw-Hill, 1975). For a discussion of the institutional 'halo e f f e c t ' which blurs the empirical distinction between prestige of institution and scientist's reputation (as a proxy for performance, quality o f research, and so on), see H. Zuckerman, 'Stratification in American Science', in E. O. Laumann (ed.), Social Stratification:
Theory and Research for the 1970s (Indianapolis: Bobbs-Merrill, 1970), 235-57.
68. M i t r o f f , o p . cit. note 39.
69. See, for instance, C h u r c h m a n and Feyerabend, ops. cit. note 10.
70. R. K. Merton, 'The M a t t h e w Effect in Science', Science, Vol. 159 (5 January 1968), 56-63.
71. M u l k a y , o p . cit. note 67; B o f f e y , op. cit. note 67; Salomon, o p . cit. note 8.
See also D . K. Price, The Scientific Estate (Cambridge, Mass.: The Belknap Press, 1965; and D . S. Greenberg, The Politics of Pure Science (Washington, DC: New American Library, 1967).
72. J. R. Cole and S. C o l e , Social Stratification in Science (Chicago: The University o f Chicago Press, 1973); J. Gaston, Originality and Competition in Science (Chicago: The University o f Chicago Press, 1973); Gaston, The Reward System in British and American Science (New York: Wiley-Interscience, 1978).
73. T o q u o t e an a n o n y m o u s referee (for an earlier version o f this paper) on decision-making behaviour:
My hunch is that, as the uncertainty of peer evaluation increases, more and more of the elements of the dialectic are brought to bear so that in s o m e cases, after 'objective' criteria have been used and are f o u n d not to distinguish between pairs of proposals, other more subjective and politically, controversial premises are used . . . and I think for quite defensible reasons.
74 We thank an anonymous referee lor reminding us to call attention to these ques-tions.
75. This is the spirit oi the proposed Science Court — the involvement of various publics in scientific decisions which arc 'too important to be left to the scientists'. But another anonymous referee observed:
Basic t o the Science Court concept is the idea o f scientific judgment. Its purpose is not simply 'presentation and review' of the issues, but indeed, a verdict that reflects the assessment of a 'scientific judge' . . . I have argued this out with Kan-trowitz suggesting that he maintain the 'presentation and review' procedure and minimize the importance o f the verdict, but he claims that would change the in-tention in a fundamental way.
See D. Nelkin, 'The Political Impact of Technical Expertise', Social Studies of Science, Vol. 5 (February 1975), 35-54, and op. cit. note 10.
76. As Dorothy Nelkin, reflecting on the recombtnant-DNA debate, has recently argued: see her 'Threats and Promises: Negotiating the Control of Research',
76. As Dorothy Nelkin, reflecting on the recombtnant-DNA debate, has recently argued: see her 'Threats and Promises: Negotiating the Control of Research',