Organization Science informs

Vol. 16, No. 2, MarchApril 2005, pp. 180200 doi 10.1287/orsc.1040.0107

issn 1047-7039
eissn 1526-5455
05
1602
0180 2005 INFORMS

How Much Better Are the Most-Prestigious Journals?

The Statistics of Academic Publication
William H. Starbuck
Department of Management, Stern School of Business, New York University, 40 West Fourth Street, Tisch 7-22,
New York, New York 10012, wstarbuc@stern.nyu.edu

A rticles in high-prestige journals receive more citations and more applause than articles in less-prestigious journals, but
how much more do these articles contribute to knowledge?
This article uses a statistical theory of review processes to draw inferences about differences value between articles in
more-prestigious versus less-prestigious journals. This analysis indicates that there is much overlap in articles in different
prestige strata. Indeed, theory implies that about half of the articles published are not among the best ones submitted to
those journals, and some of the manuscripts that belong in the highest-value 20% have the misfortune to elicit rejections
from as many as ve journals.
Some social science departments and business schools strongly emphasize publication in prestigious journals. Although
one can draw inferences about an authors average manuscript from the percentage in top-tier journals, the condence limits
for such inferences are wide. A focus on prestigious journals may benet the most prestigious departments or schools but
add randomness to the decisions of departments or schools that are not at the very top. Such a focus may also impede the
development of knowledge when mediocre research receives the endorsement of high visibility.
Key words: citations; journals; knowledge; peer review; personnel evaluation; research; reviewing

Introduction economists what they thought about review processes,

A few years ago, some deans of business schools an-
nounced new criteria for tenure and promotion that
place extreme emphasis on publications in so-called A
journals. The deans statements implied that articles in
A journals are essential and those in B or C journals are
insignicant. To gain tenure or promotion to full profes-
sor, they said, a professor must have published at least
N publications in A journals.
Even though many social science departments and
business schools use such an approach to faculty eval-
uation, its apparent oversimplication and narrow focus
is disturbing. Journals with higher prestige usually pub-
lish more high-value articles, but lower-prestige journals
also publish excellent articles and high-prestige journals
publish pedestrian articles. One can see substantial value
overlap among journals with very different prestige. The
most inuential writings have included books and chap-
ters in books, and distinguished social scientists have
told me they refuse to submit their manuscripts to jour-
nals. When Gans and Shepherd (1994) asked 140 leading
180
60% of them took time to reply, some wrote several
blistering pages (p. 165), and many told stories about
very inuential articles that journals had rejected.
This situation has contradictory aspects for me. Per-
sonal experience tells me that it makes no sense to judge
articles solely on the journals in which they appear. I
have served on editorial boards of 15 journals in four
elds, including journals with great prestige and journals
with none; I have reviewed manuscripts for many other
journals; and I have read debates about peer review.
However, emphasis on A journals is widespread, and
widespread practices almost always benet someone. To
move beyond philosophy and ad hoc opinion, I decided
to review evidence about peer review and to analyze sta-
tistically the articles in different prestige strata.
One needs to analyze these issues in terms of fun-
damental statistics rather than only in terms of data
because reputations and citations are strongly inuenced
by social construction. Journals citation rates and repu-
tations reect other factors as well as the average value
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS 181

of articles. Citation rates of journals correlate with such Figure 1 Average Impact Factors of Articles in Economics
factors as elds citation practices, journals circulations, Journals
journals languages, and the nations where journals are 5 Top-tier 20 Top quintile
published. Journals reputations have been rather iner- 3 11 Second-tier 41 Second-third quintiles
tial despite shifting editorial policies. The statistics help 5 Third-tier 41 Fourth-fifth quintiles

to distinguish the processes themselves from their social

context.
This article draws on data about review processes to

Average impact factor

frame a statistical analysis of differences between the 2

top 20%, the middle 40%, and the bottom 40% of jour-
nals. Although the theory generalizes to other elds, this
article focuses on economics, psychology, sociology, and
their business-school cousin, management. The next sec- 1
tion reviews trends in citations to journals in these four
elds. This review suggests that increasing emphasis on
top-tier journals results from administrative choice rather
than from adaptation to widespread trends in social sci-
0
ences. Final sections of this article discuss how emphasis

81

83

85

87

89

91

93

95

97

99

01
on top-tier publications affects personnel evaluations, the

19

19

19

19

19

19

19

19

19

19

20
standings of departments and schools, and the develop- Year

ment of knowledge. Because examination of these issues

needs a theoretical framework, the middle sections of
this article develop a statistical theory that describes how
editorial acceptances and rejections vary with reviewers
abilities to assess manuscripts.
Some ideas in this article, especially those in the mid-
dle section, will be familiar to readers of the many arti-
cles about peer review. What this article contributes is
a more integrated and systematic analysis rooted in a
statistical theory. This theory helps to relate properties
of review processes to citation practices, and it allows
inferences about the usefulness for personnel evaluations
of counts of publications in prestigious journals.
For Whom Have Very Prestigious Journals
Become More Important?
One of my initial conjectures was that the deans pol-
icy change might be part of a widespread trend. I read
an article in which Ellison (2002, p. 978) remarked
that articles in ve top-tier economics journals had
been attracting more and more citations, whereas articles
in second- and third-tier economics journals had been
attracting fewer and fewer citations. Ellisons observa-
tion raised the possibility that the new policies at some
business schools might reect trends in social sciences
generally. The solid lines in Figure 1 graph impact fac-
tors of 21 economics journals Ellison studied. It uses his
classications of these journals as top tier, second tier,
or third tier, but uses my own data about impact factors
from 19812002.
Impact factors are published by the Institute for Scien-
tic Information (ISI), which surveys citations in thou-
sands of academic journals (Amin and Mabe 2000). An
impact factor is the average number of citations received
in one year by articles published during two previous
years:
Citations in year t to articles published in t 1 or t 2

Articles published in years t 1 or t 2
ISI has said that two years are long enough to obtain
high percentages of citations for nearly all articles, so
impact factors accurately reect visibility of journals
even though they may not accurately reect visibility of
specic articles. Because this article aggregates impact
data across many journals, it places no reliance on the
representativeness of impact factors for single articles or
even single journals.
Ellisons observation suggested two possibilities. First,
trends in economics might be widespread across social
sciences. Second, deans who are economists might have
adopted a policy that reected conditions peculiar to
their own discipline.
To nd out whether many business schools were
increasing their emphasis on top-tier publications, I
queried colleagues at 16 top North American business
schools. Three respondents pointed out that because
rankings by Business Week and Financial Times now
take account of publications in journals that these mag-
azines deem inuential, such publications have become
more salient. Colleagues at six schools replied that their
schools have been placing strong emphasis on publi-
cations in prestigious journals for many years; these 6
schools received an average rank of 6.6 from Business
Week in 2002, much higher than the average rank of 11.8
for the other 10 schools. Another six colleagues stated
that their schools use broader criteria than publications
in prestigious journals, and that they deem their own
faculties to be more capable of evaluating research than
journal reviewers are; Business Week assigned these six

182
schools heterogeneous ranks in 2002. Colleagues at only
four schools said they had noticed an increased empha-
sis on publishing in prestigious journals during recent
years; these 4 schools received an average rank of 15.8
from Business Week in 2002, much lower than the aver-
age rank of 7.6 for the other 12 schools.
To nd out whether economics is distinctive among
social sciences, I examined data about impact factors
of journals related to business. These data include only
journals that received at least six citations during 2001
from some combination of 150 business journals, and I
have not retained data about journals that have ceased
publication. Both biases imply an underrepresentation
of journals having very low citation rates. I have data
about 102 economics journals, 95 management journals
(including industrial relations), 90 psychology journals,
and 47 sociology journals.
My rst discovery was that Ellisons nding is spe-
cic to economics, and especially to the 21 journals
he examined and how he classied them. If one clas-
sies journals into strata based on their citation rates
during recent years, then top-strata journals do appear
to have gained over lower-strata journals. Gains appear
to have occurred in all elds. However, these gains are
illusions created by retrospective sense making: Those
that did better appear to have done better in retrospect.
When I reclassied journals each year based on cita-
tion rates in that year, different trends appeared. The
dashed lines in Figure 1 show the impact factors for
102 economics journals when these are divided into
three broad categoriesthe most-cited quintile, the sec-
ond and third quintiles, and the two least-cited quintiles.
The average impact factor of the most-cited quintile of
economics journals actually decreased about 10% over
the two decades, and the increase Ellison observed was
specic to the ve journals on which he focused.
Figure 2 graphs two ratios for each of four elds:
Three-year average impact factor for the most-cited
quintile of journals
Three-year average impact factor for the two
least-cited quintiles of journals
and
Three-year average impact factor for the second and
third quintiles of journals
Three-year average impact factor for the two
least-cited quintiles of journals.
For example, the heavy solid line in Figure 2 is the ratio
of citations to the most-cited quintile versus citations to
the two least-cited quintiles for management journals,
and the light solid line is the ratio of citations to the
second and third quintiles versus citations to the two
least-cited quintiles for management journals. I divide
journals into three large categories partly because rank-
ings of specic journals are unstable over time and partly
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
Figure 2 Ratios of Impact Factors: Ratio of Top Quintile to
Fourth-Fifth Quintiles and Ratio of Second-Third
Quintiles to Fourth-Fifth Quintiles
14
Top management, Mean = 6.2
Top economics, Mean = 5.6
12
Top psychology, Mean = 8.6
Top sociology, Mean = 4.2
Second-third management, Mean = 2.7
Second-third economics, Mean = 2.1
Ratio of average impact factors
10 Second-third psychology, Mean = 3.0
Second-third sociology, Mean = 2.1
8
6
4
2
0
81
83
85
87
89
91
93
95
97
99
01
19
19
19
19
19
19
19
19
19
19
20
Year
because I want to discuss the overall stratication of
academic journals and do not want to focus on a small
number of highly prestigious journals.
Figure 2 shows that, in economics, there has been an
increase in citation rates of higher-status journals rel-
ative to lower-status journals. The ratio of citations to
the most cited versus the least cited rose 27%, and the
ratio of citations to the second and third quintiles versus
the least-cited rose 33% over the two decades. However,
these changes are entirely due to decreasing citations
of the least-cited journals. As Figure 2 shows, cita-
tion rates decreased in all three categories over the two
decades, but the percentage decrease was larger for the
least-cited journals.
The most striking trends in Figure 2 are decreases
in the ratios for management and psychology journals,
especially during the early 1980s. The ratio of top-to-
bottom declined 12% in sociology, 35% in management,
and 40% in psychology. These decreases are largely
due to increases in citations of the least-cited journals.
The average impact factor of the least-cited journals
increased 4% in sociology, 36% in psychology, and 81%
in management. Thus, stratication decreased in all three
elds, but it decreased less in sociology, which was ini-
tially more egalitarian.
Figure 2 does not support the notion that social sci-
ence researchers have been citing top-tier journals more
often and bottom-tier journals less often. On the con-
trary, with the exception of economics, citation data
mainly indicate that researchers have been citing bottom-
tier journals more often. These data do suggest, how-
ever, that economists might have perceived an increased
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
emphasis on their top-tier journals, whereas psycholo-
gists, sociologists, and management researchers might
have perceived a decreased emphasis on their top-tier
journals.
Thus, an increasing emphasis on publishing in top-
tier journals seems to be a consequence of administra-
tive choice rather than adaptation to trends in social
sciences in general. Some business schools that are very
highly rated by business magazines have been empha-
sizing top-tier publications for a long time. Because
those business schools that have only recently begun to
place more emphasis on publishing in top-tier journals
are not among the most highly rated schools, they may
be pursuing greater legitimacy by imitating their more-
prestigious competitors (DiMaggio and Powell 1983).
One issue this situation poses is whether an emphasis
on top-tier publications may produce the changes in fac-
ulty that departments and schools seek. A second issue
is whether overemphasis on top-tier publications may
degrade research and slow development of knowledge.
I can address these issues more intelligently after devel-
oping a theory about the review process and then reex-
amining data about impact factors. Although a theory
has the disadvantage of requiring explicit assumptions
about the review process, it claries issues regarding the
dependability of editorial processes.
A Statistical Theory
Rousseeuw (1991, p. 41) commented, It is commonly
known and a constant course of frustration that even
well-known refereed journals contain a large fraction
of bad articles which are boring, repetitive, incorrect,
redundant, and harmful to science in general. What
is perhaps even worse, the same journals also stub-
bornly reject some brilliant and insightful articles (i.e.,
your own) for no good reason. Rousseeuw explained
that because editorial decision makers are imperfect,
errors occur; because bad papers are submitted in such
vast quantities    the small fraction of them that gets
accepted may outnumber the good ones (1991, p. 43).
Unfortunately, most academics may not appreciate the
degree to which even well-known refereed journals
contain a large fraction of bad articles and reject some
brilliant and insightful articles. As Rousseeuw implied,
we nd it easier to believe that errors occurred when
journals rejected our own manuscripts than when the
same journals rejected other peoples manuscripts (an
example of the self-serving bias; Heider 1958).
Lindsey (1978) and Stinchcombe and Ofshe (1969)
presented arguments similar to Rousseeuws, and they
and Rousseeuw supported their observations with sta-
tistical calculations that made hypothetical assumptions.
The analysis below improves on their analyses by allow-
ing for more complex review processes, by anchoring
key assumptions in data about real reviewers behavior,
183
and by showing how inferences do or do not change with
different assumptions. Although the analysis is algebraic
at root, this article states no formulas because its formu-
las appear in every textbook on mathematical statistics.
Instead of formulas, this article presents graphs show-
ing the sensitivity of inferences to different assumptions.
Some of these graphs are surprising in that they indicate
review processes would act similarly for a wide range of
assumptions about these processes or about reviewers
abilities.
What Is the Value of a Manuscript? This article talks
about three kinds of values. One of these, discussed
near the end of this article, is citation valuethe
value that researchers see when they choose published
works to cite. Citation value is partly a self-fullling
prophecy because articles in higher-status journals
attract more attention and more citations (Gottfredson
1978, Gottfredson et al. 1977, Lindsey 1978, Palacios-
Huerta and Volij 2004). This creates circular causality:
Articles receive more citations because they appear in
higher-prestige journals, and journals gain prestige
because they publish articles that receive more citations.
However, noncircular factors are also at work. Citations
correlate strongly with each journals circulations, pub-
lication by a professional association, and nationality
and language. Fields have different norms about citation
practices. Citations of individual articles correlate with
articles lengths, numbers of references they cite, num-
bers of coauthors, and nonuse of mathematics. Theoret-
ical discussions attract more citations than do empirical
studies, and general-interest articles attract more cita-
tions than articles on specialized topics (Stigler 1994).
Undoubtedly, a published article ends up with a retro-
spective value that reects what journal published it,
who cited it, and so forth.
A second value is the manuscripts shared valueits
value in the personal value systems of reviewers. Shared
value may differ from citation value because review-
ers differ from those who cite published articles. Bowen
et al. (1972) suggested that shared value might some-
times increase agreement between reviewers for unde-
sirable reasons, as when reviewers favor manuscripts
written by members of their journals editorial board.
The third value is the manuscripts inherent true
valueits ability to serve as a basis for widespread
social consensus (Calhoun and Starbuck 2003). True
value is the hypothetical quality on which nearly all
reviewers ought to be able to agree. Because the main
purpose of social science is to produce beliefs with
which nearly everyone can agree, even two review-
ers who disagree strongly with each other about many
issues should be trying to agree with each other about
manuscripts true values. Indeed, one advantage of edi-
tors choosing reviewers who might disagree would be
that intersections of their judgments would approach

184
true values (Bailar 1991, Cicchetti 2003, Hargens and
Herting 1990, Kiesler 1991). However, editors and ref-
erees may agree about properties that constitute true
value but be unable to perceive true value accurately.
Gottfredson (1978), Gottfredson and Gottfredson (1982),
and Wolff (1970) found that reviewers for psychologi-
cal journals agree rather strongly about properties that
manuscripts ought to possess, but their judgments about
whether specic manuscripts possess various properties
agree much less strongly; correlations ranged from 0.16
to 0.50.
Assumptions About Perceptions. Because perception
involves error, analysis of how reviewers perceive manu-
scripts needs to begin from assumptions about errors in
perceptions. Psychological research suggests that human
perceptions of differences between stimuli depend
mainly on percentage differences between stimuli rather
than on absolute differences, and Luce and Galanter
(1963) showed that a logarithmic transformation of
stimuli approximates the idea that people react sim-
ilarly to equal percentage differences. That is, as a
rough approximation, reviewers perceive the logarithms
of manuscripts properties. Psychological research also
suggests that perceptions are less discriminating for
extreme stimuli, partly because perceivers have less
experience with extreme stimuli. Thus, reviewers prob-
ably make less sensitive distinctions among extremely
good manuscripts or extremely poor ones than among
manuscripts they see frequently. However, the analysis
below does not rely strongly on the accuracy of extreme
perceptions because it only divides manuscripts into
three large categoriesthe most-cited quintile, the sec-
ond and third quintiles, and the fourth and fth quintiles.
The analysis uses gross categories so that inferences
will not depend on the precise accuracy of assumptions,
which are approximate.
It also seems reasonable to suppose that manuscripts
true values have skewed distributions that generally
resemble lognormalsmany mediocre manuscripts and
then fewer and fewer manuscripts as value increases.
Many phenomena associated with publishing have
skewed distributions that resemble lognormals, including
impact factors, lengths of books and articles, circula-
tions of journals, and numbers of authors who pro-
duce various numbers of articles (West and Shlesinger
1990). One rationale for such distributions is that intel-
lectual contributions result from multiple elements. Few
manuscripts incorporate almost no elements that consti-
tute intellectual contributions, whereas most manuscripts
incorporate several of these elements, but fewer and
fewer manuscripts incorporate additional elements as
the numbers of elements increase. I conjecture that
the lognormal pattern of intellectual contributions may
derive from humans abilities to perceive. Because
authors abilities to recognize their own intellectual
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
contributions depend on percentage differences, and
readers abilities to recognize intellectual contributions
depend on percentage differences, the numbers of ele-
ments in articles and books may have distributions that
reect these perceptual tendencies. Thus, the analy-
sis below assumes that the logarithms of true values
in the population of manuscripts have a Normal dis-
tribution with a mean at zero and a standard devia-
tion of one. Although the lognormal seems the best
choice on balance, it is only one of several similar
distributions.
Assumptions About Review Processes. Many social-
science journals report that they publish around 15%
to 25% of submissions, so the analysis assumes that
editors publish 20% of the manuscripts they receive.
The analysis also assumes that editors send manuscripts
to two reviewers who determine the disposition of
many manuscripts. If two reviewers cannot agree on
acceptance or rejection, even after two revisions and
three reviews, editors themselves intervene and accept
a fraction of the manuscripts about which two review-
ers disagreed. The analysis assumes that editors and
reviewers have approximately equal abilities to assess
manuscripts and that editors make independent judg-
ments about manuscripts without being inuenced by
reviewers. However, based on reviews of editorial deci-
sions, Cicchetti (2003) inferred that when faced with
judgments from two reviewers, editors typically adopt
the more negative judgment, and when faced with judg-
ments from three reviewers, editors typically adopt the
modal judgment.
These assumptions fall near the middle of actual edi-
torial practices (Campanario 1998a, Schminke 2002).
In economics and nance, editors commonly send a
manuscript to only a single reviewer, who makes a
nal decision. A survey by Seidl et al. (2001) revealed
that more-prestigious economics journals use fewer
reviewers than do less-prestigious journals. At the other
extreme, one prestigious psychology journal sometimes
sends manuscripts to ve reviewers. Lindsey (1978) doc-
umented substantial variance in qualications of review-
ers and editors. Some editors seek highly qualied
reviewers, others choose reviewers who are likely to dis-
agree with each other, and still others use inexperienced
reviewers such as doctoral students. Some editors inject
their personal evaluations, whereas others mainly try to
implement reviewers decisions. Although these varia-
tions would alter the analysis, they do not affect the basic
logic and they would shift the numbers little: Adding
more reviewers has weak effects because correlations
between reviewers are low.
My experience says that journals that use several
reviewers, including editors who offer personal opin-
ions, place more emphasis on authors conforming to
social norms. More reviewers generate more advice, so
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS 185

authors have to confront more demands for compliance.
One reviewer explained without embarrassment: The
roles of the reviewers and editors are both gatekeepers
and co-developers. Facing such attitudes, authors tend
to believe that they must follow reviewers advice even
when they deem the advice ill-founded (Bedeian 2003,
2004; Frey 2003; Starbuck 2003).
Most journals ask reviewers to recommend accep-
tance, revision, or rejection. The analysis below assumes
that each reviewer recommends rejecting 58% of the
manuscripts, accepting 17%, and seeking revisions of
25%. These frequencies resemble ones that editors have
reported, as listed in Table 1. Although the assumed fre-
quencies of acceptance and rejection do affect the distri-
butions discussed below, they affect actions taken after a
rst review more than they affect actions taken after two
or three reviews.
The Correlation Between Reviewers Judgments and
Manuscripts True Value
How accurately do reviewers appraise the true value of
manuscripts? This is a key issue. Peters and Ceci (1982)
conducted what may be the most discussed and contro-
versial study of reviewing by journals. They resubmit-
ted 12 articles to the very journals that had published
them just 18 to 32 months earlier. All 12 were top-tier
journals. Whereas authors from prestigious psychology
departments originally had written the articles, Peters
and Ceci gave the resubmissions ctitious authors with
return addresses at obscure institutions. In all, 38 edi-
tors and reviewers saw the resubmissions. Three of these
38 recognized that submissions had already been pub-
lished, which cut the sample to 9 articles that received
18 reviews. Out of 18 reviewers, 16 recommended rejec-
tion, and the 9 editors rejected 8 of the 9 articles.
Peters and Cecis article drew 50 pages of commen-
tary by others, and Cicchettis (1991) report about low
interrater agreement in 10 psychological and sociologi-
cal journals drew 33 pages of commentary. Peer review
elicits strong feelings and diverse attitudes (Baumeis-
ter 1990; Bedeian 1996a, b; Holbrook 1986). Gossip
about inconsistent reviews and biased reviewers per-
vades academe, and studies have documented various
biases of reviewers (Armstrong 1997; Bedeian 2003;
Campanario 1996, 1998b; Ellison 2002; Hargens 1990;
Horrobin 1990; Mahoney et al. 1978; Nylenna et al.
1994). One interesting nding is that reviewers criticize
the methodology of studies that cast doubt on theories
that the reviewers like and they applaud the methodology
of studies that support theories that they like (Mahoney
1977, 1979).
There is no way to directly measure reviewers abili-
ties to discern manuscripts true values, but two kinds of
observations offer tangential evidence about these abili-
ties. First, Gottfredson (1977, 1978) found that review-
ers forecasts of manuscripts impacts correlated only
0.37 with later citations and their ratings of manuscript
quality correlated only 0.24 with later citations. If true
values are less visible and more difcult to discern
than citation values, correlations of reviewers judg-
ments with manuscripts true values are lower than these
correlations.
Second, as detailed in Table 2, various studies have
reported measures of agreement between reviewers that
fall between 0.08 and 0.54 (Starbuck 2003). These
measures present three issues, however. First, two of
the measures are product-moment correlations, two are
Kappas, and most are intraclass correlation coefcients.2
These statistics have slightly different properties. Sec-
ond, these measures were computed from data about
reviewers judgments that utilized three, four, or ve
categoriessuch as accept, revise, and rejectwhereas
the statistical models below assume continuous evalua-
tion variables. Third, the measures in Table 2 indicate
agreement about shared values rather than agreement
about true values.
The next subsection of this article draws on the num-
bers in Tables 1 and 2 to infer condence limits for
reviewers ability to assess a manuscripts true value.
This inference process is rather complicated. However,
I need to explain why my assumptions about these con-
dence limits are plausible, and I want to do this in a
technically defensible way.
The Information Value of Agreement Between Re-
viewers. What might measures like those in Table 2
say about relations between reviewers opinions and

Table 1 Frequencies of Accept, Revise, or Reject Recommendations by Reviewers

Personality
Journal of and Social American Journal of American
Administrative Academy of Personality Psychology Psychologist Abnormal Sociological
Science Management and Social Bulletin (Cicchetti 1980, Psychology Review
Reviewers Quarterly Journal and Psychology (Hendrick Scarr and (Cicchetti and (Hargens and
judgments (Starbuck 2003) Review1 (Scott 1974) 1976, 1977) Weber 1978) Eron 1979) Herting 1990)

Accept (%) 25 13 9 14 10 11 12
Minor revise (%) 28 9 9
Revise (%) 25 34 25 15 18 44 21
Drastic revise (%) 26 13
Reject (%) 50 53 66 18 50 46 58
 Starbuck: The Statistics of Academic Publication
186 Organization Science 16(2), pp. 180200, 2005 INFORMS

Table 2 Reported Measures of Agreement Between Reviewers

Manuscripts
Agreement reviewed Categories

Reviewers judgments, Administrative Science Quarterly pm = 012 500 3

(Starbuck 2003)
Reviewers judgments, American Psychologist (Cicchetti 1980, R = 050 to 0.54 and 87 2, 3, and 5
Scarr and Weber 1978) K = 052 to 0.53
Reviewers judgments, American Psychologist (Cicchetti 1991) R = 038 72 5
Reviewers judgments, American Sociological Review R = 029 322 4
(Cicchetti 1991, Hargens and Herting 1990)
Reviewers judgments, Developmental Review (Whitehurst 1984) R = 027 73 4
Reviewers judgments, Journal of Abnormal Psychology, 1973 R = 008, 0.17, and 0.16 215 2, 3, and 4
(Cicchetti and Eron 1979)
Reviewers judgments, Journal of Abnormal Psychology, 1974 R = 029, 0.34, and 0.30 213 2, 3, and 4
(Cicchetti and Eron 1979)
Reviewers judgments, Journal of Abnormal Psychology, 1975 R = 019, 0.16, and 0.15 191 2, 3, and 4
(Cicchetti and Eron 1979)
Reviewers judgments, Journal of Abnormal Psychology, 1976 R = 007, 0.10, and 0.08 216 2, 3, and 4
(Cicchetti and Eron 1979)
Reviewers judgments, Journal of Abnormal Psychology, 1977 R = 005, 0.21, and 0.20 232 2, 3, and 4
(Cicchetti and Eron 1979)
Reviewers judgments, Journal of Educational Psychology R = 034 325 5
(Marsh and Ball 1981)
Reviewers judgments, Journal of Personality and Social Psychology R = 026 286 3
(Scott 1974)
Reviewers judgments, Law and Society Review (Cicchetti 1991) R = 023 251
Reviewers judgments, Personality and Social Psychology Bulletin R = 022 177 5
(Hendrick 1976, 1977)
Reviewers judgments, Social Problems (Smigel and Ross 1970) K = 040 193 4
Reviewers judgments, Sociometry (Hendrick 1976, 1977) pm = 021 140 5
Best paper, Consumer Psychology Division, American Psychological
Association (Bowen et al. 1972) W = 011 10 judges ranked 8 papers 8

Notes. pm = product-moment correlation; R = intraclass correlation; K = Kappa; W = coefcient of concordance.

manuscripts true values? If two reviewers opinions cor-
relate very strongly with manuscripts true values, the
reviewers opinions would correlate very strongly with
each other. However, reviewers opinions could correlate
strongly with each other even if they correlate weakly
with true values. For their opinions to correlate strongly,
reviewers must share values, but these values might
involve elements other than manuscripts true value. For
example, reviewers might value a manuscripts relevance
to a faddish topic that will drift into obscurity, they
might value manuscripts conformity to social norms
that do not support generation of valid knowledge, or
they might have high regard for authors from prestigious
schools and low regard for authors from little-known
schools (Bowen et al. 1972, Lindsey 1978, Pfeffer et al.
1977). Although specialized journals might promote dis-
tinctive values, Gottfredson (1978) found no such differ-
ences among 299 editors and consulting editors for nine
psychological journals.
Going from the measures in Table 2 to statements
about reviewers abilities to assess true value involves
four steps. As would any measures of selection processes
that are subject to error, the measures in Table 2 entail
some uncertainty. Thus, the rst step involves estimating
standard deviations for the reported measures. The mea-
sures in Table 2 are of three different typesKappas,
product-moment correlations, and intraclass correlations.
I found trivial differences between product-moment cor-
relations and intraclass correlations, but Kappas have
smaller absolute values than the other two measures. The
second step transforms Kappas into numbers comparable
with the other measures. The measures in Table 2 indi-
cate correlations between reviewers judgments, which
fall into just three to ve categories. The third step trans-
lates statements about correlations between reviewers
judgments into statements about theoretical correlations
between reviewers opinions, which the theory assumes
to be nely measured evaluations symbolized by num-
bers accurate to decimal places. Both small numbers of
judgment categories and random errors of classication
cause judgment measures to differ from theoretical cor-
relations between opinions. Thus, the third step draws
inferences about correlation B in Figure 3 from mea-
sures of correlation A. The fourth step translates state-
ments about correlations between reviewers opinions
into statements about correlations between reviewers
opinions and manuscripts true values. That is, the fourth
step attempts to draw inferences about correlation D
from correlation B in Figure 3. Because conjectures are
necessary at this point, I consider a range of possibilities
that show how correlation D depends on assumptions.
All four steps utilize simulations.3 The analyses could be
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS 187

Figure 3 Relations Discussed in this Subsection
D ment between two short lists of random numbers could
Reviewer 1s
C
opinions
True Shared
values values
B A
Reviewer 2s
opinions
algebraic, but simulation seemed easier, and it produces
scatter diagrams that I intended to use in this article.
Table 3 shows implications of the rst three analytic
steps. The third and fourth columns report 95% con-
dence limits for correlations between reviewers judg-
ments, results of the rst two analytic steps. The three
right-hand columns report 95% condence limits for
correlations between reviewers opinions, results of the
third analytic step.
The term opinion denotes a nely measured assess-
ment, but reviewers do not report such opinions, so one
must infer correlations between opinions from corre-
lations between categorized judgments such as accept,
revise, and reject. Potential distortions from reviewers
classifying manuscripts into small numbers of categories
are not widely appreciated. For example, one reader of
a draft of this article pointed out that reviewers often
agree about whether manuscripts ought to be published.
Possibly this reader, and many editors, overlooked the
possibility that totally random judgments may agree
strongly when the sample is small and high percent-
ages of the judgments fall into one or two categories
(reject, or drastic revise and reject). Cicchetti (1985)
showed that if utterly random numbers are converted to
the binary categories reject-or-not-reject, apparent agree-
Reviewer 1s range as high as 80%. Indeed, because rejects outnum-
judgments ber not-rejects, such random data would exhibit the oft-
reported property that reviewers are more likely to agree
about which articles to reject than about which articles to
accept.
Reviewer 2s The fourth analytic step translates correlations be-
judgments
tween reviewers opinions into statements about correla-
tions between reviewers opinions and manuscripts true
values. That is, the fourth step attempts to draw infer-
ences about correlation D from correlation B in Figure 3.
Because conjectures are necessary at this point, I con-
sider a range of possibilities.
Reviewers shared values intervene between review-
ers opinions and manuscripts true values, so correla-
tion D relates more directly to correlation C than to
correlation B. Correlations between reviewers opinions
(as estimated in Table 3) do not arise solely from review-
ers mutual appreciation of manuscripts true value. True
value should be a quality on which almost all review-
ers agree. If reviewers agree strongly on shared values
that differ from the values of most other people, one
can observe a strong correlation between reviewers that
has only a weak component coming from a manuscripts
true value. When data encompass hundreds of diverse
reviewers, it seems very unlikely that the correlations
between shared values and true values go as low as zero
or as high as one. Would it be reasonable to assume that
shared values correlate 0.5 with true values? Or 0.25? Or
even 0.75? Editors who choose reviewers who tend to
disagree with each other might be able to raise this cor-
relation to higher values, but no journal receives numer-

Table 3 Condence Limits for Observed Correlations Between Reviewers Judgments and Opinions

95% Condence limits for correlation 95% Condence limits for correlation
between judgments between reviewers opinions

Below Above Below Expected Above

Social Problems 0.38 0.59 048 0.63 0.78

American Psychologist 0.49 0.76 055 0.75 0.94
Administrative Science Quarterly 0.05 0.19 005 0.17 0.29
Sociometry 0.08 0.34 007 0.25 0.42
Journal of Abnormal Psychology, 1976 0.02 0.18 005 0.10 0.26
Journal of Abnormal Psychology, 1975 0.04 0.26 003 0.19 0.36
Journal of Abnormal Psychology, 1973 0.06 0.26 005 0.21 0.36
Journal of Abnormal Psychology, 1977 0.10 0.30 011 0.26 0.41
Personality and Social Psychology Bulletin 0.11 0.33 011 0.26 0.41
Journal of Personality and Social Psychology 0.16 0.36 022 0.36 0.51
Developmental Review 0.09 0.45 007 0.35 0.63
American Sociological Review 0.20 0.38 024 0.37 0.51
Journal of Abnormal Psychology, 1974 0.19 0.41 023 0.39 0.54
Journal of Educational Psychology 0.25 0.43 029 0.40 0.52
American Psychologist 0.20 0.56 021 0.45 0.69
American Psychologist 0.38 0.70 042 0.64 0.86

Notes. pm = product-moment correlation; R = intraclass correlation; K = Kappa.

 Starbuck: The Statistics of Academic Publication
188 Organization Science 16(2), pp. 180200, 2005 INFORMS

ous citations to more than two thirds of its articles, and Figure 4 Ratios of the Correlations Between Reviewers
only a tiny fraction (0.6%) of journals receive numerous Opinions and Manuscripts True Values vs. the
citations to more than 30% of their articles.4 The many Correlations Between Reviewers Opinions,
200 Manuscripts in Each Sample
reasons why people share values that are uncorrelated
or weakly correlated with scientic principles suggest 5
that true values ordinarily correlate less than 0.5 with 4
shared values. However, there would be variation among
pairs of reviewers and possibly variation among elds of 3
study, so I have made computations for correlations of 2
0.25, 0.5, and 0.75.
Figure 4 shows scatter plots for ratios of correlations 1

Ratios
between individual reviewers opinions and manuscripts 0
true value as functions of correlations between review- 0.4 0.2 0.2 0.4 0.6 0.8 1.0
ers opinions. The three distinct groups of data corre- 1

spond to alternative assumptions about the correlations, 2 R = 0.75

denoted R, between shared values and true values; the Expected when R = 0.75
R = 0.50
horizontal lines describe R = 025, 0.5, or 0.75. Vari- 3 Expected when R = 0.50
ances are similar for all three assumptions and vari- 4
R = 0.25
Expected when R = 0.25
ances are small only when correlations between review-
ers opinions are above 0.4. When correlations between 5

reviewers opinions fall below 0.3, variances become
very large and the three assumptions about R become
nearly equivalent. Variances are also smaller when there
are more manuscripts in each sample and larger when
there are fewer manuscripts in each sample.
Table 4 estimates correlations between reviewers
opinions and manuscripts true values for the measures
in Table 2. In the upper portion of the table, the six
right-hand columns give 95% condence limits for cor-
relations of reviewers opinions with manuscripts true
values. The condence limits are very wide. According
to the lower limits, many reported measures are consis-
tent with the possibility that reviewers opinions might
Correlations Between Reviewers Opinions
have correlated not at all or slightly negatively with
manuscripts true value. The upper limits indicate that
reviewers opinions might have correlated between 0.2
and 0.5 with manuscripts true value.
The three bottom rows of Table 4 give statistics for
expected values across all 16 reported measures. For
example, if one assumes that shared values correlate 0.5
with true values, the average across all 16 measures
would be a correlation of 0.18 between reviewers opin-

Table 4 95% Condence Limits and Expected Values for Correlations of Reviewers Opinions with True Values

If Correl(TV, SV) = 025 If Correl(TV, SV) = 05 If Correl(TV, SV) = 075

Lower Upper Lower Upper Lower Upper

Social Problems 004 0.36 011 0.52 027 0.67

American Psychologist 006 0.44 012 0.62 031 0.81
Administrative Science Quarterly 011 0.19 007 0.24 003 0.28
Sociometry 018 0.30 011 0.36 005 0.42
Journal of Abnormal Psychology, 1976 019 0.24 016 0.27 014 0.29
Journal of Abnormal Psychology, 1975 017 0.27 013 0.32 008 0.37
Journal of Abnormal Psychology, 1973 016 0.26 011 0.32 006 0.37
Journal of Abnormal Psychology, 1977 014 0.27 008 0.33 001 0.40
Personality and Social Psychology Bulletin 015 0.28 008 0.34 002 0.41
Journal of Personality and Social Psychology 010 0.28 000 0.37 009 0.46
Developmental Review 024 0.42 016 0.51 007 0.59
American Sociological Review 008 0.27 001 0.36 010 0.46
Journal of Abnormal Psychology, 1974 011 0.30 001 0.40 008 0.50
Journal of Educational Psychology 006 0.26 004 0.36 014 0.46
American Psychologist 019 0.41 007 0.52 004 0.64
American Psychologist 011 0.43 005 0.59 021 0.75
Minimum expected value among all reports 0.03 0.05 0.08
Average expected value over all reports 0.09 0.18 0.27
Maximum expected value among all reports 0.19 0.37 0.56

Notes. TV denotes true value, and SV denotes shared value.

Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
ions and manuscripts true values. Also, if shared values
correlate 0.5 with true values, the smallest reported mea-
sure implies a correlation of 0.05 between reviewers
opinions and manuscripts true values, and the largest
reported measure implies a correlation of 0.37.
One of my earlier reactions to the large variances
in the foregoing relationships was to propose that jour-
nals should use more categories for reviewers judg-
ments and that they should compel reviewers to use
categories rather equally. Simulation experiments have
subsequently convinced me that these are not useful
ideas. Five categories do yield more reliable data about
agreement between reviewers than do three categories,
and seven equal categories yield slightly more reliable
data than ve somewhat unequal categories. However,
increasing the categories to seven does not reduce the
variances associated with inferences about the agreement
of reviewers opinions with true values, which are dom-
inated by uncertain relationships between shared values
and true values. Three categories, ve categories, and
seven categories all have very similar implications for
inferences about true values. If reviewers are poor judges
of manuscript quality, more-rened measures of review-
ers opinions have insignicant utility, whereas small
numbers of categories may remind editors that review-
ers judgments are crude.
All considered, it seems reasonable to infer that cor-
relations between reviewers opinions and manuscripts
true values are almost always less than 0.5 and usu-
ally below 0.3. Indeed, these correlations could fall
below 0.1 in some journals or with some pairs of review-
ers. Therefore, the computations to follow consider 0.15,
0.3, and 0.45, and implicitly the range between them.
To reduce verbosity, I call reviewers average correlation
with true value Rho.
The next two sections of this article show how edito-
rial reviews vary as functions of Rho. The next section
looks at a single review cycle in which two reviewers
render independent judgments about each manuscript,
and the ensuing section looks at the disposition of
manuscripts after successive revisions and reevaluations
by reviewers and a terminating judgment by an editor.
Both parts of this process are graphed for both Rho =
015 and Rho = 045. If Rho is sometimes below 0.15
or above 0.45, these are very unusual events. Because
0.15 and 0.45 are supposed to be extremes, the discus-
sion also reports implications of Rho = 03, which may
approximate expected values.
Outcomes of First Reviews
Saying that a reviewers evaluations correlate Rho with
logarithms of true value implies that the evaluations
have a distribution with a mean at Rho (true value).
For example, if a reviewers evaluations correlate 0.2
with logarithms of true value, manuscripts with a true
189
value of 2 would have a mean evaluation of 02 03 =
006, because log2 = 03. Likewise, if a reviewers
evaluations correlate 0.3 with logarithms of true value,
manuscripts with a true value of 5 would have a mean
evaluation of 03 07 = 021 because log5 = 07.5
The gray area in Figure 5 shows the population
of submissions. The gray lines show distributions of
manuscripts that would receive rejects from both of two
reviewers after rst review, and the black lines show dis-
tributions for two accepts. The solid gray line assumes
that Rho = 015; 12% of these rejected manuscripts rep-
resent editorial errors because they actually belong in
the highest-value 20% of manuscripts. The dashed gray
line assumes that Rho = 045; 6% of these rejected
manuscripts represent editorial errors because they come
from the highest-value 20%. For the in-between case
where Rho = 03, 10% of the rejected manuscripts
represent editorial errors. All distributions for Rho
between 0.15 and 0.45 overlap the population distri-
bution to considerable degrees, so rejected manuscripts
have a distribution much like the population of submis-
sions.
Figure 5 also graphs distributions of manuscripts that
would receive accepts from both reviewers after the rst
review. The solid black line assumes that, on average,
Rho = 015, and the dashed black line assumes that,
on average, Rho = 045. When Rho = 015, only one-
fourth of the accepted manuscripts actually belong in
Figure 5 Percentages of Manuscripts that Receive Two
Accepts or Two Rejects After First Review by Two
Reviewers
Population
Rho = 0.45, two rejects, 6% belong in
highest-value 20% of manuscripts
Rho = 0.15, two rejects, 12% belong in
highest-value 20% of manuscripts
1.2 Rho = 0.45, two accepts, 78% belong
in highest-value 20% of manuscripts
Rho = 0.15, two accepts, 26% belong
1.0
in highest-value 20% of manuscripts
Percent of manuscripts
0.8
0.6
0.4
0.2
0.0
4 3 2 1 0 1 2 3 4
Log of true value of manuscript

190
the highest-value 20% of submissions and three quar-
ters represent errors because they should not have been
accepted. When Rho = 045, these percentages reverse:
78% of the accepted manuscripts come from the highest-
value 20% and 22% represent errors. When Rho = 03,
half of the accepted manuscripts belong in the highest-
value 20% and half represent errors. Acceptance errors
occur often because four-fths of the submissions, and
hence four-fths of the opportunities to make errors,
are manuscripts that should not be accepted (Rousseeuw
1991, Stinchcombe and Ofshe 1969). However, the error
rate is much lower when Rho is above 0.4.
Figure 6 shows what happens to the best 20%
of manuscripts when Rho varies from 0.15 to 0.45.
Manuscripts classied as mixed evaluation are any
that did not receive accepts from both reviewers or
rejects from both reviewers. The percentage of excel-
lent manuscripts receiving two rejects after rst review
declines from 20% when Rho = 015 to 11% when
Rho = 045. The percentage of excellent manuscripts
receiving two accepts rises from 4% when Rho = 015
to 22% when Rho = 045. Unfortunately, it seems that
Rho rarely goes as high as 0.45. For the in-between case
when Rho = 03, just 9% of the excellent manuscripts
receive two accepts.
The curvature in Figure 6 nearly disappears when cal-
culations take all submissions into account. The per-
centage of all manuscripts receiving two rejects after
the rst review rises only slightly from 34% to 37%
as Rho rises from 0.15 to 0.45, whereas the percentage
of all manuscripts receiving two accepts rises from 3%
to 6%. Obviously, for Rho in this range, the disposi-
tion of manuscripts depends very weakly on reviewers
abilities to discern true values. This selection process
discriminates so poorly that the phrase blind review is
apt in an unintended way.
Figure 6 After First Review by Two Reviewers, Distributions of
Actions on Manuscripts that Belong in the Highest-
Value 20% of Manuscripts
100
80
Percent of manuscripts
60
40
20
0 0
0.15 0.25
Rho, average correlation of reviewers with log of true value
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
How Editorial Decisions After Three
Reviews Differ from Those After
First Reviews
Many of my colleagues believe that reviewers will be
more likely to accept their manuscripts if they comply
with reviewers suggestions and more likely to reject
their manuscripts if they reject reviewers suggestions
(Bedeian 2003, Frey 2003). On the premise that review-
ers do reward compliance, these calculations assume that
reviewers become 50% more likely to issue two accepts
and only 70% as likely to issue two rejects during the
second and third reviews. Experiments disclose, how-
ever, that these moderate assumptions about reviewers
reactions have trivial effects.
Figure 7 shows the dispositions of all submissions
after authors have revised some of them once or even
twice. This calculation assumes authors revise and
resubmit half of the mixed evaluations from the rst
review and 80% of the mixed evaluations from the sec-
ond review. The percentage of manuscripts receiving two
rejects is quite independent of Rho.
One can now estimate how often editors must be inter-
vening. For Rho between 0.15 and 0.45, dual accep-
tances fall between 6% and 13%, being 8% when
Rho = 03. Yet journals accept 15% to 25% of sub-
missions. One explanation for this difference would be
that editors are making nal decisions about manuscripts
with mixed evaluations.
Editorial intervention eliminates the blank area labeled
mixed evaluations in Figure 7; all manuscripts are
nally accepted or rejected. Indeed, because there is
no reason to believe that Rho affects the percentage of
manuscripts that journals publish, editorial actions turn
Figure 7 into two regions separated by a horizontal line:
75% to 85% rejections and 15% to 25% acceptances. For
lower values of Rho, manuscripts accepted through edi-
tors interventions outnumber those accepted by review-
Figure 7 After Third Review by Two Reviewers and Two
Revisions, Distributions of Actions on All
Manuscripts
100
90
Two accepts
80
Mixed evaluations
Two accepts
Percent of manuscripts
70 Two rejects
Mixed evaluations
Two rejects
60
50
40
30
20
10
0.35 0.45 0.15 0.25 0.35 0.45
Rho, average correlation of reviewers with log of true value
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
Figure 8 Selection of Manuscripts After Two Revisions, Third
Review by Two Reviewers, and Evaluation by
the Editor
1.2
1.0
Percent of manuscripts
0.8
0.6
0.4
0.2
0.0
4 3 2 1
Log of true value of manuscript
ers, and for higher values of Rho, manuscripts accepted
by reviewers outnumber those accepted through editors
interventions.
Figure 8 compares the distributions of manuscripts
that ultimately receive rejects with those that receive
accepts. The two solid lines assume that Rho = 015,
and the two dashed lines assume that Rho = 045. The
distributions of rejected manuscripts (gray lines) are
very similar: About one-seventh of the manuscripts that
receive rejects belong in the highest-value 20%, and
these comprise more than half of the manuscripts in
the highest-value 20%. Because journals reject four to
seven times as many manuscripts as they accept, there
are many more opportunities for erroneous judgments
among rejected manuscripts than among accepted ones.
The largest effect of higher correlations between
reviewers judgments and true value is better selec-
tion of manuscripts for acceptance. For manuscripts
in the highest-value 20%, accepts rise to 71% when
Rho = 045, although only 43% receive accepts when
Rho = 03. These acceptance percentages are slightly
lower than after a single review because under the
assumptions of this model, editors must agree only with
themselves, whereas reviewers must agree with each
other, so editors are more likely to accept the wrong
manuscripts.
Figures 5 and 8 are very similar. Similarity results
in part from assuming that authors are equally likely to
revise all mixed-evaluation manuscripts and that review-
ers evaluate revised manuscripts afresh. Obviously, edi-
tors and reviewers inuence authors decisions about
Rho = 0.45, rejects, 15% in
highest-value 20% of manuscripts
Rho = 0.15, rejects, 14% in
highest-value 20% of manuscripts
Rho = 0.45, accepts, 71% in
highest-value 20% of manuscripts
Rho = 0.15, accepts, 23% in
highest-value 20% of manuscripts
0 1 2 3
191
whether to revise, and reviewers usually remember (or
journals remind them) what they said earlier. However,
encouragement or discouragement by editors or review-
ers cannot have effects beyond those supported by the
underlying Rho, because people who misperceive true
value are very likely to offer advice or to demand
changes that are uncorrelated with true value.
A few studies have found that reviewers agree more
strongly about whether to publish a manuscript than
about what is wrong with the manuscript. For instance,
Mahoney (1977, 1979) obtained interrater correlations
around 0.3 for recommendations about acceptance and
for ratings of scientic contribution, but he found inter-
rater correlations close to zero for ratings of methodol-
ogy, relevance, and quality of discussion.
How Much Better Can the Most Prestigious
Journals Be?
We now return to the topic that initiated this study:
the stratication of journals. This section estimates the
maximum advantages of high status by analyzing a
review process that gives all of the advantages to the
4 highest-status journals. Thus, the inferences drawn likely
overstate quality differences among strata. This section
also compares inferred differences in the value of arti-
cles published by different strata with reported citation
rates.
The analysis assumes authors initially submit all of
their manuscripts to journals in the rst quintile, which
choose 20% to publish. Disappointed authors then sub-
mit the remaining manuscripts to journals in the second
quintile, which also choose 20% to publish. However,
because the second-quintile journals receive only 80%
as many manuscripts as rst-quintile journals, they pub-
lish only 16% of the original submissions. Authors then
submit the remaining manuscripts to journals in the
third quintile, and so on through all ve quintiles. If
the highest-quintile journals were actually able to select
the best 20% of the manuscripts, none of the best 20%
would remain for second-quintile journals, and the arti-
cles appearing in the higher-ranking journals would have
distinctly higher true value. However, if the highest-
quintile journals are able to select only one third of the
manuscripts in the highest-value 20%, two thirds remain
for lower-status journals; in this case, a sequence of ve
journals would publish 67% of the best 20% of submis-
sions, leaving 33% unpublished.
Figure 9 compares distributions of manuscripts that
ultimately receive acceptances from journals in various
strata when Rho = 015. The gray area shows the popula-
tion of manuscripts when initially produced. The heaviest
black line shows acceptances by journals in the rst
quintile to receive them; 23% of these manuscripts
belong in the highest-value 20% of all manuscripts, and
the average accepted article is one-fourth of a standard
 Starbuck: The Statistics of Academic Publication
192 Organization Science 16(2), pp. 180200, 2005 INFORMS

Figure 9 Manuscripts Accepted After Reviews by Journals in Figure 10 Manuscripts Accepted After Reviews by Journals in
Different Strata when Rho = 0
15 Different Strata when Rho = 0
45

Population Population
First quintile, 23% in highest-value First quintile, 71% in highest-value
20% of manuscripts 20% of manuscripts
Second-third quintiles, 20% in Second-third quintiles, 38% in
1.2 highest-value 20% of manuscripts highest-value 20% of manuscripts
1.8
Fourth-fifth quintiles, 15% in highest- Fourth-fifth quintiles, 2% in highest-
value 20% of manuscripts value 20% of manuscripts
1.6
1.0

1.4
Percent of manuscripts

0.8

Percent of manuscripts
1.2

1.0
0.6

0.8

0.4 0.6

0.4
0.2
0.2

0.0 0.0
4 3 2 1 0 1 2 3 4
4 3 2 1 0 1 2 3 4
Log of true value of manuscript Log of true value of manuscript

deviation above the population mean. The medium black gure probably represents a realistic average across the
line shows acceptances by journals in the second and social sciences.
third quintiles; 20% of these manuscripts belong in the Table 5 summarizes the frequencies of articles with
highest-value 20% of all manuscripts. The lightest black different true values in each quintile of journals. Each
line shows acceptances by journals in the fourth and column sums to 1.0.
fth quintiles; 15% of these manuscripts belong in the
highest-value 20% of all manuscripts, and the average
accepted article has a value near the population mean. Of Figure 11 Manuscripts Accepted After Reviews by Journals in
Different Strata when Rho = 0.30
the manuscripts in the highest-value 20%, a depressing
30% are rejected by ve journals in sequence. Population
Figure 10 compares distributions of manuscripts that First quintile, 43% in highest-value
20% of manuscripts
receive two accepts when Rho = 045. Again, the gray Second-third quintiles, 29% in
area shows the population. The heaviest black line 1.2
highest-value 20% of manuscripts
Fourth-fifth quintiles, 13% in
shows acceptances by the rst quintile; 71% of these highest-value 20% of manuscripts
manuscripts belong in the highest-value 20%, and the
average accepted article is more than one standard devi- 1.0

ation above the population mean. The medium black

line shows acceptances by the second and third quin-
Percent of manuscripts

0.8
tiles; 38% of these manuscripts belong in the highest-
value 20%. The lightest black line shows acceptances
by the fourth and fth quintiles; only 2% of these 0.6
manuscripts belong in the highest-value 20% of all
manuscripts, and the average accepted article has a value
slightly below the population mean. All manuscripts 0.4

in the highest-value 20% receive acceptances from one

journal or another. 0.2
Figure 11 shows the in-between case when Rho =
03: 43% of acceptances by the rst quintile belong in
the highest-value 20% of all manuscripts, as do 29% of 0.0
-4 -3 -2 -1 0 1 2 3 4
acceptances by the second and third quintiles, and 13%
Log of true value of manuscript
of acceptances by the fourth and fth quintiles. This
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS 193

Table 5 Distributions of Manuscripts True Values for Journals in Different Quintiles

Rho = 015 Rho = 03 Rho = 045

Journal Journal in Journal in Journal Journal in Journal in Journal Journal in Journal in

in rst second-third fourth-fth in rst second-third fourth-fth in rst second-third fourth-fth
quintile quintiles quintiles quintile quintiles quintiles quintile quintiles quintiles

Value in lowest 20% 0.09 0.11 0.14 0.05 0.08 0.14 0.01 0.03 0.10
Value in fourth 20% 0.18 0.20 0.23 0.10 0.15 0.23 0.03 0.09 0.28
Value in middle 20% 0.23 0.24 0.25 0.16 0.21 0.26 0.07 0.18 0.38
Value in second 20% 0.26 0.25 0.24 0.25 0.27 0.25 0.17 0.32 0.23
Value in highest 20% 0.23 0.20 0.15 0.43 0.29 0.13 0.71 0.38 0.02

This portrayal of academic publishing does not allow
for specialization by journals or for authors select-
ing journals that focus on specic subtopics or specic
methodologies. The analysis assumes that all journals
are equally interested in all manuscripts and that all
authors send their manuscripts to the highest-quintile
journals rst. Specialization enables less-prestigious
journals to attract higher-value manuscripts, which could
decrease differentiation between lower-status and higher-
status journals. Less-prestigious journals have probably
made gains over the last two decades, as graphed in
Figure 2, by developing specialized niches. Many less-
prestigious journals also take more risks when selecting
manuscripts for publication, which increases the vari-
ance in what they publish but also makes them more
welcoming to innovative topics or methodologies.
Of course, not all authors send all of their manuscripts
to rst-quintile journals initially, but it is debatable
how much these deviations from the assumptions may
affect overall patterns. Authors make misjudgments
about their manuscripts at least as often as reviewers do,
and because an authors decision is very categorical
whether to submit this manuscript to a journal in that
quintileit would be a noisy decision. Furthermore,
authors choices of where to submit manuscripts reect
factors other than their judgments about manuscripts
true values. Many authors submit to lower-status jour-
nals not because they deem their manuscripts to lack
value, but because they are hoping for friendlier editors
and less-intrusive reviewers.
How Citers Judgments Correlate with Articles Cita-
tion Values. Table 6 reinterprets average true values
implicit in Figures 9 and 10 to consider the stratication
of average citation values. The three center columns give
the ratios when the correlation between citers judg-
ments and citation values is 0.15, 0.3, or 0.45. The right-
hand column gives averages of the ratios graphed in
Figure 2. The table shows that citation patterns of social
scientists have a steeper status hierarchy than would be
consistent with the belief that citers judgments corre-
late 0.15 with articles citation values, but citation pat-
terns have a much atter status hierarchy than would be
consistent with the belief that citers judgments corre-
late 0.45 with articles citation values. Citation patterns
resemble the belief that citers judgments correlate 0.31
with articles citation values.
The numbers in Table 6 cast doubt on the idea that
Rho could be as high as 0.45. If Rho were that high, cita-
tions would be much more stratied than they actually
are, because readers of journals would have higher con-
dence that the best articles appear in the most prestigious
journals. Although there are other reasons to cite articles
other than their true values, it seems unlikely that these
other reasons are sufcient to cut the ratios of citation
rates for the rst-quintile journals from 32.9 to 5.8.
Table 7 takes this notion of citation value further to
examine differences among social science elds. Cita-
tions of psychology journals are consistent with belief
that citers judgments correlate between 0.33 and 0.37
with citation values. By contrast, citations of sociology
journals are consistent with the belief that this correla-
tion falls between 0.28 and 0.31.
Although the numbers in Table 7 generally resemble
those assumed in the analyses of editorial review, the

Table 6 Stratication of the First Quintile, Second and Third Quintiles, and Fourth and Fifth Quintiles of Journals

Ratios of average Ratios of average Ratios of average Observed average ratios

citation values if citation values if citation values if of impact factors in
correlation = 015 correlation = 030 correlation = 045 social sciences

Ratio of rst quintile to average of fourth 1.7 4.9 329 5.8

and fth quintiles
Ratio of rst quintile to average of second 1.2 2.1 58 2.4
and third quintiles
Ratio of average of second and third 1.3 2.4 56 2.4
quintiles to average of fourth and
fth quintiles
 Starbuck: The Statistics of Academic Publication
194 Organization Science 16(2), pp. 180200, 2005 INFORMS

Table 7 Correlations of Citers Judgments with Articles Citation Values that Are
Consistent with the Impact Factors of Journals in Different Fields

Ratios of rst Ratios of rst Ratios of second-

quintile to fourth- quintile to second- third quintiles to
fth quintiles third quintiles fourth-fth quintiles

Sociology journals 0.30 0.31 0.28

Economics journals 0.33 0.35 0.28
Management journals 0.33 0.34 0.33
Psychology journals 0.37 0.33 0.35

two processes are not equivalent. Citation values dif-
fer from true values, and the criteria that citers apply
differ from the criteria that reviewers apply. However,
citation values undoubtedly correlate positively with true
value, both citing and reviewing rely on similar percep-
tion capabilities, and both processes operate on distribu-
tions that are approximately lognormal.
Implications for Faculty Evaluation
Expectations about true values should depend on Rho:
The more condence one has in reviewers judgments,
the higher value one should assign to publishing in pres-
tigious journals. Using average true value of articles in
the fourth and fth quintiles of journals as a norm, the
upper half of Table 8 lists implied average true values
of articles in the rst quintile and second and third quin-
tiles of journals. Although one could use these averages
as metrics with which to evaluate articles by individ-
ual faculty, every category of journals publishes articles
with a very wide range of true values, so average true
values for journals actually provide no useful informa-
tion about any single article. The lower half of Table 8
demonstrates this dispersion by listing ranges of true val-
ues that one can expect to nd in different quintiles of
journals.
Departments and schools rarely base faculty eval-
uations on a single article, and the ambiguity indi-
cated by Table 8 decreases as researchers produce more
manuscripts. Figures 12 and 13 describe expected dis-
tributions of articles by a single author as a function
of the average true value of that authors manuscripts.
For example, when Rho = 015, authors whose average
manuscripts are at the 90% percentile (1.28 standard
deviations above the mean) would publish 32% of their
manuscripts in the rst quintile of journals, 46% in
the second and third quintiles, and 23% in the fourth
and fth quintiles, and none of their manuscripts would
remain unpublished. At the opposite extreme, when
Rho = 045, authors at the 60% percentile would publish
25% of their manuscripts in the rst quintile of journals,
34% in the second and third quintiles, and 20% in the
fourth and fth quintiles, and 20% would be rejected
by ve journals in sequence. These distributions assume
that the standard deviation of a hypothetical authors
manuscripts is one-fourth of the population standard
deviation. Because Figures 12 and 13 describe expected
percentages, which would be approximated only by very
large samples, small samples of an authors work may
depart noticeably from these expectations.
Higher values of Rho give an advantage to authors
who consistently produce higher-value manuscripts and
give a disadvantage to authors who produce lower-value
manuscripts. When Rho is low, journals make erro-
neous decisions more often, and poorer manuscripts have
higher probabilities of acceptance by higher-prestige
journals. However, according to Figure 13, even when
Rho = 045, an author whose manuscripts fall into the
90th percentile would publish only 77% in the rst quin-
tile of journals, and the other 23% would appear in the
second and third quintiles. Under the more plausible
assumption that Rho = 03, an author whose manuscripts
fall into the 90th percentile would publish only half in
the rst quintile of journals, and would publish the other
half in the second and third quintiles. Figures 12 and 13
imply that many authors meet repeated failure. In fact,

Table 8 Means and Probable Ranges of the Mean True Values of Journals as Functions of Rho

Rho = 015 Rho = 02 Rho = 025 Rho = 03 Rho = 035 Rho = 04 Rho = 045

Mean true values

First quintile 1.7 2.2 3.1 4.9 8.7 16.8 32.9
Second-third quintiles 1.3 1.5 1.9 2.4 3.1 4.2 5.6
Fourth-fth quintiles 1.0 1.0 1.0 1.0 1.0 1.0 1.0
Ranges that encompass 80% of the true values
First quintile 0.1 to 19 0.2 to 27 0.2 to 41 0.3 to 68 0.4 to 121 0.7 to 228 1.3 to 446
Second-third 40% 0.1 to 14 0.1 to 18 0.1 to 23 0.2 to 29 0.2 to 36 0.3 to 45 0.5 to 56
Fourth-fth quintiles 0.1 to 10 0.1 to 11 0.1 to 11 0.1 to 10 0.1 to 8.5 0.1 to 6.7 0.2 to 5.6
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS 195

Figure 12 Expected Distributions of Numerous Manuscripts Looking Only at Counts of Articles in A Journals.
by an Author Whose SD = 0
25 when Rho = 0
15 It is possible to draw some inferences about the aver-
100 age value of an authors manuscripts based solely on
90
Unpublished information about the percentage of publications in top-
Two bottom quintiles
tier journals, and these inferences have some surprising
Second-third quintiles
80
Top quintile characteristics. We will consider rst inferences possible
70 from very large numbers of articles and then inferences
Percent of manuscripts

60 possible from just 20 articles.

Figure 14 shows how the percentiles of authors aver-
age manuscripts vary with the percentage that appeared
50

40
in the highest-quintile journals. The steepest lines made
30 of small dashes represent the assumption that Rho =
20
015, the least-steep lines made of alternating short and
long dashes represent the assumption that Rho = 045,
10
and the solid lines represent the assumption that Rho =
0 03. Heavy black lines describe expected average val-
0 10 20 30 40 50 60 70 80 90
Percentile of an authors average manuscript compared to all submissions
100
ues, and thinner gray lines describe 80% condence
by all authors intervals that assume the author has produced numerous
manuscripts. For example, if Rho = 015, authors with
40% of their publications in the rst quintile of jour-
Trieschmann et al. (2000) reported that authors from the
nals have published manuscripts with an average value
top 50 business schools account for 70% of the articles
in the 86th to 90th percentiles. However, if Rho = 045,
in 20 prestigious business journals.
authors with 40% of their publications in the rst quin-
I have been unable to think of a rationale other than
tile journals have published manuscripts with an aver-
laziness or simplicity that would support actually ignor-
age value in the 44th to 62nd percentiles. The curves
ing publications that are not in top-tier journals. Any
for Rho = 015 terminate near the center of the chart
decision rule that discards potential information cannot
because the randomness of review processes makes it
perform better than an equally reasonable rule that uti-
impossible to place more than 60% of a large sample in
lizes all information. Furthermore, because even articles
the highest-quintile journals.
in the lowest quintile of journals may actually belong
Figure 14 shows an interesting effect of Rho. If one
among the best 20% written, it makes no sense to dis-
believes Rho = 0.3 or 0.45, one should have a higher
miss these articles as valueless merely based on where
opinion of authors who have published less than 20% in
they appeared. For those who insist on basing assess-
the highest-quintile journals than if one believes Rho =
ments solely on the journals in which articles appear,
0.15. If Rho = 0.15, placing a very small percentage of
a more rational evaluation scheme would be to use the
articles in the highest-quintile journals is a strong sign of
average true values associated with different categories
of journals, as listed in Table 8.
Figure 14 Percentile of an Authors Average Manuscript
as a Function of the Percentage Published by
Figure 13 Expected Distributions of Numerous Manuscripts Top-Quintile Journals, Numerous Manuscripts
by an Author Whose SD = 0
25 when Rho = 0
45 in Sample
100 100
Percentile of authors average manuscript among all

Unpublished
90 Two bottom quintiles 90
Second-third quintiles
80 Top quintile 80
Submissions by all authors

70
Percent of manuscripts

70

60 60

50 50
10% limit, Rho=0.15
40 40 Mean, Rho =0.15
90% limit, Rho=0.15
30 30 10% limit, Rho=0.3
Mean, Rho =0.3
20 20 90% limit, Rho=0.3
10% limit, Rho=0.45
10 10 Mean, Rho =0.45
90% limit, Rho=0.45
0 0
0 10 20 30 40 50 60 70 80 90 99 10 20 30 40 50 60 70 80 90 100
Percentile of the authors average manuscript compared to all submissions Percentage of numerous manuscripts published by top-quintile
by all authors journals

196
Figure 15
100
Percentile of authors average manuscript among all
90
80
70
submissions by all authors
60
50
40
30
20
10
0
2 4
Number of manuscripts published by top-quintile journals, out of 20
low average value because review processes are so ran-
dom that mediocre articles get into the highest-quintile
journals by accident. But conversely, if one believes
Rho = 0.3 or 0.45, one should have a lower opinion
of authors who have published more than 30% in the
highest-quintile journals than if one believes Rho =
0.15. If Rho = 0.15, placing a high percentage of arti-
cles in the highest-quintile journals is a strong sign of
high average value because work must have very high
value to overcome the randomness of review processes.
For instance, an author who places 40% to 60% of a
very large sample in the rst-quintile journals should be
deemed in the 86th to 93rd percentile if Rho = 0.15, in
the 59th to 84th percentile if Rho = 0.3, or in the 44th
to 69th percentile if Rho = 0.45.
Its assumption of a very large sample limits the use-
fulness of Figure 14. A practically useful analysis must
allow for small to moderately large samples. Actual pub-
lications are binomial processes with each author sub-
mitting a limited number of manuscripts and with proba-
bilities of acceptance set by review processes. Figure 15
shows what happens to Figure 14 when authors have
produced 20 articles. Again, the steepest lines made of
small dashes represent the assumption that Rho = 015,
the attest lines made of alternating small and large
dashes represent the assumption that Rho = 045, and the
solid lines represent the assumption that Rho = 03. The
heavy black lines describe expected average values, and
the thinner gray lines describe 80% condence intervals.
Although the small sample widens the condence
intervals, two regions have rather clear implications. The
blank area at the upper left shows that an author who has
published 20 articles with 3 or fewer in the rst-quintile
journals is very likely to be producing research that aver-
ages below the 60th percentile. Likewise, the blank area
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
Percentile of an Authors Average Manuscript as at the lower right shows that an author who has pub-
a Function of the Percentage Published by Top- lished 20 articles with 4 or more in the rst-quintile
Quintile Journals, 20 Manuscripts in Sample journals is very unlikely to be producing research in the
lowest 30% of submissions. The different values of Rho
have similar implications when authors have published
Mean, Rho = 0.15 Mean, Rho = 0.3
Mean, Rho = 0.45
3 to 7 articles in the rst quintile out of 20. Authors
who have published 5 articles out of 20 in the rst quin-
tile may have been producing manuscripts that average
value as high as the 80th percentile or as low as the 44th
percentile if Rho = 015, as high as the 71st percentile
or as low as the 35th percentile if Rho = 03, and as high
10% limit, Rho=0.15
Mea n, Rho=0.15
as the 65th percentile or as low as the 30th percentile if
90% limit, Rho=0.15
10% limit, Rho=0.3
Rho = 045.
Mea n, Rho=0.3
90% limit, Rho=0.3
Because articles published in high-prestige journals
10% limit, Rho=0.45 receive signicantly more citations, departments and
Mea n, Rho=0.45
90% limit, Rho=0.45 schools that strongly emphasize publication in the most-
6 8 10 12 14 16 18
prestigious journals may be pursuing visibility, which
attracts students and faculty. However, actual citation
manuscripts behavior does not support an exclusive focus on high-
prestige journals. On average, 2.4 articles in the second
and third quintiles of journals draw as many citations
as 1 article in the rst quintile of journals, and 5.8 arti-
cles in the fourth and fth quintiles of journals draw as
many citations as 1 article in the rst quintile of jour-
nals. These ratios vary, however, across academic elds.
Top-tier publications attract higher percentages of cita-
tions in psychology and lower percentages in sociology.
Moreover, if departments and schools want to measure
individuals visibility, in addition to using journals pres-
tige as a proxy, they can use citations of specic arti-
cles, appearances at conferences, and even solicited data
about reputations.
Conclusion
Although higher-prestige journals publish more high-
value articles, editorial selection involves considerable
randomness. Highly prestigious journals publish quite
a few low-value articles, low-prestige journals publish
some excellent articles, and excellent manuscripts may
receive successive rejections from several journals. Eval-
uating articles based primarily on which journals pub-
lished them is more likely than not to yield incorrect
assessments of articles values. Yet personnel evalua-
tions by many departments and schools seem to underes-
timate or even to ignore this randomness, and in extreme
cases these evaluations focus on one myopic measure.
For most departments and schools, extreme empha-
sis on publication in top-tier journals has a signicant
probability of introducing randomness because the con-
dence intervals associated with such publications are
very wide. Furthermore, an evaluation process that dis-
cards potentially useful information is less sensible than
a process that takes advantage of all available infor-
mation. However, departments and schools could use
publication in top-tier journals as a reliable criterion
for personnel who have published many articles (say,
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
more than 50) with high percentages of these articles
in top journals. Indeed, widespread use of this criterion
would help elite departments and schools to retain their
statuses by enabling them to select and retain excel-
lent researchers while adding randomness into the per-
sonnel decisions of most departments and schools. If
lower-status departments and schools are adopting this
criterion to imitate their more successful competitors,
imitation is placing the imitators at a disadvantage rel-
ative to those they are imitating. Might it be possible
to nd evidence about whether departments and schools
fare differently over time if they use different criteria in
personnel decisions?
People who are dealing with unreliable data should
attempt to cross-check their data by triangulating differ-
ent kinds of data and data from different sources. Even
departments and schools that could use publication in
top-tier journals as a reliable criterion could give their
decisions more reliable foundations by taking additional
evidence into account and by analyzing all available data
in some depth. Of course, it is debatable whether bet-
ter data would produce better personnel decisions, but
that is a topic for another article. Personnel decisions
entail even more uncertainty than does peer review. For
one thing, universities are making commitments for long-
term future periods whereas data concern past activi-
ties. For another thing, faculty migration gives individ-
uals opportunities to overcome adverse decisions and it
undercuts universities efforts to overtake higher-status
competitors.
The simulations behind Figure 4 and Table 4 sug-
gest some practical implications for editors. If the cor-
relation between reviewers is below 0.4, which is very
likely, editors should assume that the reviewers judg-
ments correlate weakly with manuscripts true values.
Only when there is a fairly high correlation between
reviewers should editors consider reviewers judgments
to yield useful information about manuscripts true val-
ues. Furthermore, editors are more likely to obtain useful
information about manuscripts true values if they choose
reviewers who hold different values from each other.
The simulations and algebraic analyses in this arti-
cle assume that reviewers opinions have the same
variance as the population of submissions. I also exper-
imented with an alternative assumption that reviewers
opinions have only half the population variance, which
amounts to saying that reviewers judge manuscripts as
being more similar than they really are. This assump-
tion greatly reduces agreement between reviewers and
greatly reduces correlations between reviewers opinions
and manuscripts shared values. Thus, editors receive
less-useful information when they use reviewers whose
judgments have restricted ranges.
Authors of research manuscripts can draw similar
inferences from reviewers judgments. That is, if review-
ers disagree with each other, authors should infer that
197
reviewers judgments say little about the true values
of manuscripts, but if reviewers agree fairly strongly,
authors should infer that reviewers judgments reect the
true values of manuscripts to some degree. When they
agree with each other, reviewers who seem to espouse
different values are probably offering especially infor-
mative judgments.
Of course, reviewers who agree strongly are not preva-
lent, and reviewers are especially prone to disagree
about a specic manuscriptwhat does it do right;
what does it need? In particular, reviewers comments
about research methodology may be cloaking agreement
or disagreement of their personal values with studies
ndings. Authors should expect to receive inconsistent
reviews, and they dare not rely on editors and review-
ers to tell them what to do. Because academic research
elicits mainly negative judgments, authors dare not let
negative reviews undermine their condence in their
work. Although it is useful to listen to what reviewers
say (Starbuck 2003), authors need to base their ultimate
decisions on their own expertise and their own values.
The belief that high-status journals publish excellent
articles whereas low-status journals publish poor arti-
cles may be impeding the development of knowledge.
The analysis in this article indicates that 29% to 77%
of the articles in the rst quintile of journals do not
belong among the highest-value 20% of manuscripts, the
in-between estimate being 57%. If publication in high-
status journals leads social scientists to adopt less valu-
able articles as exemplars, mediocre articles are exerting
as much or more inuence on scientic values as are
excellent articles. Like people making personnel deci-
sions, researchers should triangulate different kinds of
evidence about scientic contributions and draw evidence
from diverse sources. Before one can deal effectively
with randomness, one must acknowledge its existence.
Acknowledgments
This article has beneted from contributions from Linda
Argote, Joel Baum, Tom Cummings, Joan Dunbar, Ned Elton,
Moshe Farjoun, Bruno Frey, Ari Ginsberg, Paul Hirsch,
Marshall Meyer, Wanda Orlikowski, Jeff Pfeffer, Karlene
Roberts, Sandra Robinson, Josh Ronen, Ehsan Soo, and Mike
Tushman, and especially from extensive comments by Art
Bedeian, Roger Dunbar, Joanne Martin, Bill McKelvey, Joe
Porac, and Sim Sitkin.
Endnotes
1
Statistics for Academy of Management Journal and Academy
of Management Review are averages of data reported by
Angelo DeNisi and Susan Jackson to the Academys Board of
Governors in 1996.
2
Both intraclass correlation coefcients and Kappas charac-
terize the consistency with which nominal variables were
classied on multiple occasions or by multiple classiers.
Both measures are supposed to equal zero when there is no
consistency, and both can take negative and positive values.

198
However, it may be impossible for either intraclass correla-
tion coefcients or Kappas to have absolute values as large
as 1.0. One can use different weighing schemes when calculat-
ing Kappas, and calculations in this article use linear weights.
3
The rst set of simulations explored properties of the mea-
sures of agreementintraclass correlations, Kappas, and
product-moment correlations. The computer generated a rst
reviewers opinion as a normally distributed variable with a
mean of zero and variance of 1, and then generated a sec-
ond reviewers opinion as a normally distributed variable with
a mean of R times the rst reviewers opinionR being an results were recorded. All of the above was repeated for sam-
assumed correlation between reviewers. Because this corre-
lation transmits some variance from the rst reviewer to the
second reviewer, if the rst reviewer has a variance of 1, the
generated opinions of the second reviewer must have a vari-
ance of 1 R-squared in order that both opinions will have
the same variance. Both reviewers opinions were then con-
verted to the categories accept, revise, and reject or the cate-
gories accept, minor revise, revise, drastic revise, and reject.
Categorization used values of the normal distribution that
would hypothetically produce frequencies of 17%, 25%, and
58% or 17%, 12.5%, 12.5%, 29%, and 29%. For example, a
reviewer who recommends acceptance if the normal deviate
is above 0.954 would accept 17% of the manuscripts. After
this process was repeated for a sample of 50 manuscripts, the
classied judgments were used to compute an intraclass corre-
lation, a Kappa, and a product-moment correlation. All of the
foregoing was repeated 50 times for R = 095, 50 times for
R = 090, and so forth up to 50 times for R = 095. Results
were recorded, then all of the above was repeated for samples
of 100, 200, 500, 1,500, and 15,000 manuscripts. These cal-
culations allowed estimates of the standard deviations of the
measures in Table 2, and they showed that intraclass correla-
tions and product-moment correlations give very similar num-
bers whereas Kappas generally have smaller absolute values.
A second set of simulations looked at relations between
reviewers judgments, which place manuscripts in a few
discrete categories, and their nely graded opinions of
manuscripts. The computer generated a normally distributed
shared value for each manuscript, and then two opinions
that correlated with this shared value. Both reviewers opinions
were normally distributed variables with means of R (shared
value) and with variances of 1 R-squared. Opinions were
classied in three or ve categories with the expected frequen-
cies of 17%, 25%, and 58% or 17%, 12.5%, 12.5%, 29%, and
29%. A sample of 50 manuscripts was generated and product-
moment correlations were computed between opinions and
between judgments. This process was repeated 10 times for
R = 0, R = 001, R = 002, and so on up to R = 099, then
results were recorded. All of the above was repeated for sam-
ple sizes of 50, 100, 200, 500, 5,000, and 15,000. These cal-
culations supported estimates of the ratios of the correlations
between opinions versus the correlations between judgments
as well as standard deviations for these ratios.
A third set of simulations investigated relations of review-
ers opinions and judgments with manuscripts true values. It
had a logical ow similar to Figure 3. The computer gener-
ated a normally distributed true value for each manuscript,
then generated a normally distributed shared value that cor-
related with this true value, and then generated two opinions
that correlated with this shared value. The shared value was
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
normally distributed with a mean of R (true value) and with
a variance of 1 R-squared, and the two reviewers opinions
were normally distributed variables with means of P (shared
value) and with variances of 1 P -squared. Opinions were
classied in three or ve categories with the expected frequen-
cies of 17%, 25%, and 58% or 17%, 12.5%, 12.5%, 29%, and
29%. A sample of 50 manuscripts was generated and product-
moment correlations were computed between opinions and
between judgments. This process was repeated 10 times for
P = 0, P = 005, P = 01, and so on up to P = 095, then
ple sizes of 50, 100, 200, 500, and 1,000 and for R = 025,
0.5, 0.75, and 0.95. These calculations veried the conjecture
that the assumed value of R affords a good basis for estimat-
ing the correlation of each reviewers opinions with true value
as well as regression estimates of the standard deviations of
this relationship (Figure 4).
4
Citations are skewed and approximately lognormal. ISI cat-
aloged citations to 1,705 social science journals that received
citations during 1999. During 1999, these journals published
over 70,000 articles, which received nearly 2 million citations,
almost all of these citations to articles published before 1999.
In fact, the articles published during 1997, 1998, and 1999
account for only about 300,000 of the 2 million citations, so
the bulk of citations refer to articles that are more than two
years old. On average, articles published by these journals
during 1997 and 1998 received 0.5 citations during 1999; the
most-cited quintile had an average impact of 1.93, the sec-
ond and third quintiles had an average impact of 0.67, and the
fourth and fth quintiles had an average impact of 0.18.
In mid-2004, ISIs listing of most-cited articles included
600 articles published during 1999; these had received at least
10 citations from 1999 to mid-2004, and as many as 193 cita-
tions. For these 600 articles, the logarithm of the number of
times they were cited correlated 0.44 with the logarithm of
the 1999 impact ratings of the journals that published them.
Among the 1,705 journals, 1,536 journals (90%) had no arti-
cles among the most-cited 600 and 169 had one or more arti-
cles among the most-cited 600. Only two journals (Behavioral
and Brain Sciences and Journal of Economic Literature) had
more than 50% of the articles they published among the most-
cited 600, and just eight other journals had over 30% of the
articles they published among the most-cited 600. The fore-
going statistics only describe citations by journals to journals;
citations to books or by books are not included.
5
The calculations assume that the editor and the two review-
ers have slightly different correlations with true value: One
reviewer has a correlation of 12 Rho, the other has a correla-
tion of 08 Rho, and the editor has a correlation of Rho. For
example, if the editors correlation is 0.2, one reviewers corre-
lation is 0.24 and the other reviewers correlation is 0.16. This
assumption seems more realistic than three identical reviewers,
but it has a negligible effect on the inferences.
References
Amin, M., M. Mabe. 2000. Impact factors: Use and abuse. Perspec-
tives in Publishing. Elsevier Science, http://www.elsevier.com/
framework_editors/pdfs/Perspectives1.pdf.
Armstrong, J. S. 1997. Peer review for journals: Evidence on quality
control, fairness, and innovation. Sci. Engrg. Ethics 3 6384.
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
Bailar, J. C. 1991. Reliability, fairness, objectivity and other inappro-
priate goals in peer review. Behavioral Brain Sci. 14 137138.
Baumeister, R. E. 1990. Dear journal editor, its me again: Sample
cover letter for journal manuscript resubmissions. Dialogue
5(Fall) 16.
Bedeian, A. G. 1996a. Thoughts on the making and remaking of the
management discipline. J. Management Inquiry 5 311318.
Bedeian, A. G. 1996b. Improving the journal review process: The
question of ghostwriting. Amer. Psychologist 51 1189.
Bedeian, A. G. 2003. The manuscript review process: The proper
roles of authors, referees, and editors. J. Management Inquiry
12 331338.
Bedeian, A. G. 2004. Peer review and the social construction of
knowledge in the management discipline. Acad. Management
Learn. Ed. 3 198216.
Bowen, D. D., R. Perloff, J. Jacoby. 1972. Improving manuscript eval-
uation procedures. Amer. Psychologist 27 221225.
Calhoun, M. A., W. H. Starbuck. 2003. Barriers to creating knowl-
edge. M. Mark Easterby-Smith, M. A. Lyles, eds. Handbook of
Organizational Learning and Knowledge Management. Black-
well, Oxford, UK, 473492.
Campanario, J. M. 1996. Have referees rejected some of the most-
cited papers of all times? J. Amer. Soc. Inform. Sci. 47 302310.
Campanario, J. M. 1998a. Peer review for journals as it stands today
Part 1. Sci. Comm. 19 181211.
Campanario, J. M. 1998b. Peer review for journals as it stands
todayPart 2. Sci. Comm. 19 277306.
Cicchetti, D. V. 1980. Reliability of reviews for the American Psy-
chologist. Amer. Psychologist 35 300303.
Cicchetti, D. V. 1985. A critique of Whitehursts Interrater agreement
for journal manuscript reviews: De omnibus, disputandem est.
Amer. Psychologist 40 563568.
Cicchetti, D. V. 1991. The reliability of peer review for manuscript
and grant submissions: A cross-disciplinary investigation.
Behavioral Brain Sci. 14 119186.
Cicchetti, D. V. 2003. The peer review of scientic documents: Sug-
gestions for improvements. Presentation to the Committee on
Research in Education, National Research Council, Washington,
D.C. (February 26).
Cicchetti, D. V., L. D. Eron. 1979. The reliability of manuscript
reviewing for the Journal of Abnormal Psychology. Proc. Amer.
Statist. Assoc. 22 596600.
DiMaggio, P. J., W. W. Powell. 1983. The iron cage revisited: Insti-
tutional isomorphism and collective rationality in organizational
elds. Amer. Sociological Rev. 48 147160.
Ellison, G. 2002. The slowdown of the economics publishing process.
J. Political Econom. 110 947993.
Frey, B. S. 2003. Publishing as prostitution? Choosing between ones
own ideas and academic success. Public Choice 116 205223.
Gans, J. S., G. B. Shepherd. 1994. How are the mighty fallen:
Rejected classic articles by leading economists. J. Econom.
Perspect. 8 165179.
Gottfredson, S. D. 1977. Scientic quality and peer-group consen-
sus. Dissertation Abstracts International 38 1950B, University
Microlms No. 7719, Johns Hopkins University, Baltimore,
MD, 588.
Gottfredson, S. D. 1978. Evaluating psychological research reports:
Dimensions, reliability, and correlates of quality judgments.
Amer. Psychologist 33(10) 920934.
199
Gottfredson, D. M., S. D. Gottfredson. 1982. Criminal justice and
(reviewer) behavior: How to get papers published. Criminal
Justice Behavior 9(3) 259272.
Gottfredson, S. D., W. D. Garvey, J. E. Goodnow. 1977. Quality indi-
cators in the scientic journal article publication process. JSAS
Catalog Selected Doc. Psych. 7 74.
Hargens, L. L. 1990. Variation in journal peer review systems:
Possible causes and consequences. J. Amer. Medical Assoc. 263
13481352.
Hargens, L. L., J. R. Herting. 1990. Neglected considerations in the
analysis of agreement among journal referees. Scientometrics 19
91106.
Heider, F. 1958. The Psychology of Interpersonal Relations. Wiley,
New York.
Hendrick, C. 1976. Editorial comment. Personality Soc. Psych. Bull.
2 207208.
Hendrick, C. 1977. Editorial comment. Personality Soc. Psych. Bull.
3 12.
Holbrook, M. B. 1986. A note on sadomasochism in the review pro-
cess: I hate when that happens. J. Marketing 50 104106.
Horrobin, D. F. 1990. The philosophical basis of peer review and
the suppression of innovation. J. Amer. Medical Assoc. 263(10)
14381441.
Kiesler, C. A. 1991. Confusion between reviewer reliability and
wise editorial and funding decisions. Behavioral Brain Sci. 14
151152.
Lindsey, D. 1978. The Scientic Publication System in Social Sci-
ence: A Study of the Operation of Leading Professional Jour-
nals in Psychology, Sociology, and Social Work. Jossey-Bass,
San Francisco, CA.
Luce, D. R., E. Galanter 1963. Discrimination. D. R. Luce,
R. R. Bush, E. Galanter, eds. Handbook of Mathematical
Psychology. John Wiley & Sons, New York, 191243.
Mahoney, M. J. 1977. Publication prejudices: An experimental study
of conrmatory bias in the peer review system. Cognitive
Therapy Res. 1 161175.
Mahoney, M. J. 1979. Psychology of the scientist: An evaluative
review. Soc. Stud. Sci. 9(3) 349375.
Mahoney, M. J., A. E. Kazdin, M. Kenigsberg. 1978. Getting pub-
lished. Cognitive Therapy Res. 2 6970.
Marsh, H. W., S. Ball. 1981. Interjudgmental reliability of review
for the Journal of Educational Psychology. J. Ed. Psych. 73
872880.
Nylenna, M., P. Riis, Y. Karlsson. 1994. Multiple blinded reviews of
the same two manuscripts: Effects of referee characteristics and
publication language. J. Amer. Medical Assoc. 272 149151.
Palacios-Huerta, I., O. Volij. 2004. The measurement of intellectual
inuence. Econometrica 72 963978.
Peters, D. P., S. J. Ceci. 1982. Peer-review practices of psychological
journals: The fate of published articles, submitted again. Behav-
ioral Brain Sci. 5 187255.
Pfeffer, J., A. Leong, K. Strehl. 1977. Paradigm development and
particularism: Journal publication in three scientic disciplines.
Soc. Forces 55 938951.
Rousseeuw, P. J. 1991. Why the wrong papers get published. Chance:
New Directions Statist. Comput. 4(1) 4143.
Scarr, S., B. L. R. Weber. 1978. The reliability of reviews for the
American Psychologist. Amer. Psych. 33 935.
Schminke, M. 2002. From the editors. Acad. Management J. 45
487490.

200
Scott, W. A. 1974. Interreferee agreement on some characteristics of
manuscripts submitted to the Journal of Personality and Social
Psychology. Amer. Psychologist 29 698702.
Seidl, C., U. Schmidt, P. Grsche. 2001. A beauty contest of referee
processes of economics journals. Manuscript, University of Kiel,
Kiel.
Smigel, E. D., H. L. Ross, H. L. 1970. Factors in the editorial deci-
sion. Amer. Sociologist 5(February) 1921.
Starbuck, W. H. 2003. Turning lemons into lemonade: Where is the
value in peer reviews? J. Management Inquiry 12 344351.
Stigler, S. M. 1994. Citation patterns in the journals of statistics and
probability. Statist. Sci. 9 94108.
Starbuck: The Statistics of Academic Publication
Organization Science 16(2), pp. 180200, 2005 INFORMS
Stinchcombe, A. L., R. Ofshe. 1969. On journal editing as a proba-
bilistic process. Amer. Sociologist 4 116117.
Trieschmann, J. S., A. R. Dennis, G. B. Northcraft, A. W. Niemi, Jr.
2000. Serving multiple constituencies in business schools: MBA
program versus research performance. Acad. Management J. 43
11301141.
West, B., M. Shlesinger. 1990. The noise in natural phenomena. Amer.
Scientist 78(JanuaryFebruary) 4045.
Whitehurst, G. J. 1984. Interrater agreement for journal manuscript
reviews. Amer. Psychologist 39 2228.
Wolff, W. M. 1970. A study of criteria for journal manuscripts. Amer.
Psychologist 25 636639.