I l l e p r i n t e d l l o m R ' \ l ) I O L O ( l \ ' . V o l . l - 1 ; ] .N o . l . l ) a g e s 2 ! ) : l i i .A p r i l . 1 9 8 2 .

]
( ' o p v r i g h t 1 ! ) 8 2b r ' 1 h e I l a d i o i o g i c a S
l o t ' i e t vo l N o r l l r A n r e l i t a . I n ( o r l ) o n l c ( i

J a m e sA . H a n l e y , P h . D .
Barbara|. McNeil, M.D., Ph.D.

A representationand interpretation of
the area under a receiver operating characteristic (ROC) curve obtained by the
"rating" method, or by mathematical predictions basedon patient characteristics,
is presented.It is shown that in such a
setting the arearepresentsthe probability
that a randomly chosendiseasedsubject
is (correctly) rated or ranked with greater
suspicion than a randomly chosennondiseasedsubject.Moreover, this probability of a correct ranking is the same quantity that is estimated by the already wellstudied nonparametric Wilcoxon statistic.
These two relationships are exploited to
(a) provide rapid closed-form expressions
for the approximate magnitude of the
sampling variability, i.e., standard error
that one usesto accompanythe area
u n d e r a s m o o t h e dR O C c u r v e ,( b ) g u i d e
in determining the size of the sample required to provide a sufficiently reliable
e s t i m a t eo f t h i s a r e a /a n d ( c ) d e t e r m i n e
how large sample sizesshould be to ensure that one can statistically detect differencesin the accuracyof diagnostic
techniques.
Index terms:

Receivt'r operating ch.r.rcteristic curv('

(noc)

The Meaningand Use of the Area

undera ReceiverOperating
(ROC)Curvel
Characteristic

T) ECEIvERoperating characteristic (ROC) curves (1-3) have become

l\
i.,.."uringly popular over the past few years as the radiologic
community has become concerned with the measurement of the information content of a variety of imaging svstems (4, 5). In addition,
ROC curves are being used to judge the discrimination ability of
various statistical methods that combine various clues, test results,
etc. for predictive purposes (e.9.,to determine whether a patient will
need hospitalization or will benefit from treatment). In most cases,
however, ROC curves have been plotted and evaluated qualitatively
with relatively little attention paid to their statistical characteristics.
This has occurred for a number of reasons.
First, the most common quantitative index describing an ROC curve
is thc'area under it, and there has not been a full description in the
radiologic literature of an intuitive meaning of this area. Second, most
quantitative measures used to describe an ROC curve are derived
seen
assuming that the varying degrees of normality/abnormality
in the images can be represented by twc'rseparate but usually overIapping Gaussian digributions, one for the diseased group and one
for the nondiseased group. This has led a number of investigators to
question the validity of the ROC approach altogether. Third, iterative
numerical methods rather than closed-form expressions are required
to estimate the standard error of these quantitative indices; operationally, these methods are cumbersome and require a specialized
computer program. Fourth, there has been no method shown to date
to indicate the sample size necessarv to ensure a specified degree of
statistical precision for a particular quantitative index.
In this paper, we wili elaborate on the meaning of the area under
an ROC curve and, using the links between it and other, better known
statistical concepts, we will develop analytic techniques for eliciting
its statistical properties.

R a d i o l o g y 7 4 3 : 2 9 - 3 6 ,A p r i l 1 9 8 2

METHODS
IndicesUsedto SummarizeROC Curves
I F r o n r t h t L ) e p . r r i r n r . n to f E p i c l e m i r r l o g l a n t l
H e a l t h , M c ( l i l I U r r i v e r s i t r ' ,N I o n t r t ' a l ,C a n a d a ( J . , \ . H . ) ,
ancl the L)epartrnent o1 llatliolog\', Harvartl Nletlical
School ancl Brigham trrcl l\itrmen's Hosirit.rl, Boston.
M A ( B . J . N { . )R. ec e i v e c lM a r c h 1 8 , l 9 8 l ; a c c e p t e eal n c l
l c v i s i o r r r ( ' q u o s t e c lJ u l r ' 2 1 ; r c v i s i o n r r c c i v e t { l ) e c . l 5
Supported in part lrv the Ilartforcl Ijountl.rtion rntl
the National Ctntr-r for Hcalth Carc Technologr".
See .rlso the articles bv Hessel ci a/ (pp. I 29- l 33)
.rntl Abrams rf n/. (pp. 121 128) in this issue.
cc1

A large number of theoretically base.d measures has been proposed to reduce an entire ROC curve to a singlc quantitative inclex
of diagnostic accuracy; all of these measures have been rooted in the
assumption that the functional form of the ROC curve is the same as
that implie'd bv supposing that the undc'rlyine distributions for
normal and abnormal groups are Caussian (4). When an llOC curve,
plotted on double probabilitl, paper, is fitted by eye to a straight iine
or when the RC)C points are submitted to an iterative maximum
likelihood c.stimation program, two parameters, one a difference of

29

TABLE I:

Rating of 109 CT Images

Rating

True
Disease
Status
Normal
Abnormal
Totais

Probably
Abnormal
(4)

Definitely
Abnormal
(5)

Probably
Normal
(2)

Questionable
(3)

6
2

6
2

11
11

JJ

llr = 58
t1n= 51

36

22

.1f

109

Definitely
Normal
(1)
JJ

Total

the Wilcoxon statisticor by the trapc'zoidal rule will be virtually identical tcr
any smoothedarea.)Second,and more
important.we show how the statistical
propertiesof the Wilcoxon statisticcan
be used to predict the statisticalproperties of the area under an ROC
curve.
RESULTS

means and the other a ratio of variances, are obtained. From these, a
number of indices can be calculated,
the most popular being an estimate of
the area under the fitted smooth curve
(4). This index, denoted A(z) to symbolize its Gaussian underpinnings,
varies from 0.5 (no aPParent accuracy)
to 1.0 (perfect accuracy) as the ROC
curve moves towards the left and top
boundaries of the ROC graph.2 When
one fits the two parameters by maximum likelihood rather than by eYe,
one also obtains their standard errors,
thereby allowing the area derived from
the two parameters to be also accompanied by a standard error. This can be
used to construct confidence intervals
and to perform statistical tests of significance.
The Meaning of the Area under
an ROC Curve
A precise meaning of the area
under an ROC curve in terms of the
result of a signal detection exPeriment
employing the two-alternative forced
choice (2AFC) technique has been
known for some time. In this system,
Green and Swets (6) showed that the
area under the curve corresponds to the
probability of correctly identifying
"noise"
and
which of the two stimuli is
"signal
plus noise." In medical
which is
imaging studies, the more economical
rating method is generally used: images from diseased and nondiseased
subjects are thoroughly mixed, then
presented in this random order to a
reader who is asked to rate each on a
discrete ordinal scale ranging from
definitely normal to definitely abnormal. Very often a five-category scale is

2 An area can also be calculated by thc trapezoidal rule; the;rrea obtainecl in this u';rv has been
clesignated I'(A). As is seen in Figure 1, /'(,4)is
smaller than the area under anv smooth curve,
ant-lis someu'hat more sensitive to the loc.rtion
anti spread of the points tiefining the curve than
i s t h e a r e a / ( : ) c a l c u L a t e d . r st h e s m o o t h ( l . r u s s i . r n
cstrmil te.

used. Although the points required to

produce the ROC curve are obtained in
a more indirect way, i.e., by successively considering broader and broader
categories of abnormal (e.9., category 5
alone, categories 5 plus 4, categories 5
plus 4 plus 3), the important point is
that on a conceptuallevel, and thus from
a statistical viewpoint, the area under
the curve obtained from a rating experiment has the same meaning as it has
when it is derived from a 2AFC experiment. As we will explain below, the
ROC area obtained from a rating experiment can be viewed, at least conceptually, in the same way as the area
obtained from a 2AFC experiment.
Basically, when an investigator calculates the area under the ROC curve directly from a rating experiment, he is
in fact, or at least in mathematical fact,
reconstructing random pairs of images,
one from a diseased subject and one
from a normal subject, and using the
reader's separate ratings of these two
images to simulate what the reader
would have decided if these two images had in fact been presented together
as a pair in a 2AFC experiment. Indeed,
this mathematical equivalence (equivalent in the sense that the two areas are
measuringthe same quantity, even if the
two curves are constructed differently)
has also been verified empirically in a
recognition memory experiment by
Green and Moses (7).
More important, however, was the
more recent recognition by Bamber (8)
"probability
of correctly
that this
ranking a (normal, abnormal) pair" is
intimately connected with the quantity
calculated in the Wilcoxon or MannWhitney statistical test. We now elaborate on this relationship in two ways.
First, we show empirically that if one
performs a Wilcoxon test on the ratings
given to the images from the normal
and diseased subjects, one obtains the
same quantity as that obtained by calculating the area under the corresponding ROC curve using the trapezoidal rule. (If in fact the ratings are on
a continuous scale,the area obtained by

A p r i l 1 9 8 2 V o l u m e 1 4 3 ,N u m b e r 1

I.

A Three-Way

Equivalence

To amplify the three-way equivaIence between the area under an ROC

curve, the probability of a correct
ranking of a (normal, abnormal) pair,
and the Wilcoxon statistic, we present
it as two pairwise relationships:
A. The area under the ROC curve
measures the probability, denoted bv
d, that in randomly paired normal and
abnormal images, the perceivecl abnormality of the twc.rimages will allow
them to be correctly identified.
B. The Wilcoxon statistic also
measures this probability Flth.lt r.rndomlv chosen normal and abnormal
images will be correctly ranked.
We now deal with A and B in turn.
IL
Mathematical
of Relationship A

Restatement

We make an .impiicit assumption

that the sensory information convevecl
by a radiographic image can be quantified by and ordered on a one-climensional scale represented by,r, with lon'
values of .r favoring the decision to call
the image normal and high values iavoring the decision to call it abnormal.
The distributions of -t values for randomly selectedabnormal images, clenoted by "r,1, and those for notm;rl images, denotc'd by -tr, will overlap; tl-re
,tr distribution will be centered to the
r i g h t o f t h c . r ' \ ( ) n e .I n a r . r t i n ge r . p t ' r i ment, the degree of suspicion, r, n'ill
actually be reported on an orclererl
c a t e g o r i c a l s c a l e . T A R L I ,I p r e s e n t s i i lustrative data showing hon' a single
reader rated the computc'cl tomographic (CT) imagesobtained in.r s.ulplq'
of 109 patients with neurological
problems. As expectecl,the' r.1 ancl r'
d i s t r i b u t i o n so v e r l a p ( i . c . .r o m e t t , ' r r diseased patients had abnormal re.rtlings and some diseased patients hatl
n o r m a l r e a d i n g s )T. h u s , i f t h e i m . r g t ' from a randomly chosen notmal anrl a
randomlv chosen abnormal case \\'orc
paired. there woulcl be' le-ssthan a 100'i
probability that the. sensory informa-

Hanley an<-lMcNeil

A9OIOICVU

JIISONCVIC

a^rnf IOU r.rpr.rpun parv aql Jo asll pue burueal

-qloddq (llnu) ar{t tsal ot pasn .,(11ensn ol rasolJ L{Jnw aq ol ser{ Alrlrqeqorcl LIJns'a^rnf, fou aql Jo slradse Jerllo uJ
s r L l l' J o l p u i l . I I u : s r pp o o S p d q ( ) l I J o j
prlsaralur uauo sr JoleSrJsaAuIaLIl (-r)
sr M 'a^oqe polels sP lllSnoqllv
'9 = ('tr
'JUnOJJe
aroru aJe J JnJ
pue luodn paar8e .,(1rsee
OlUr SUOS
< v x ) q t t t d : 1 ,t e q t r o ' l I u e r { l
0
-rredruor palpiarralur asar{l sa)el M JO JalParb aq ol sr 1l sP N ruoJJ iPnpr^rp
qloorus e 3ur11r1 roJ errallrr alil (./)
-urue urorJ anle^ .r ue ueql rallpurs aq
1a^JnJ IOU Llloours e dlr1ea,rur sI lPt{.ry\
roJJa pJepuels aql pue 'aJuarua^uoJ e
JaPun PaJeaLIl ale_wrlsaJopunol puol
ol Ale)rl se lsnt sr H tuoJJ Ienpl^rpuI uE
^{laratu sr IIe urar{} Surpnlrui lluapuad
-apur alp suosrredluor Nl o HrrII lou
uroJj anle^ r ue leql ''J'l 'JoleurrulJJ>^lp IIrM alnJ Ieprozaderl aql lPt{l sueatu
Sr,rrsn1o alqeder sr Jepear p serJogelpJ
,{lsnor.Lqg's,Nr aql 1o uorlrodo,rd 1eq.u l n J . r s np l o u s r r ' l e L l ls r s l q r q l o d ^ q l l n u
8ur1er xrs Jo aArJ Jo urnrurxeuJ aql
aql suorlelndod o.u.1er{l ul palnqrrl
ueql ralearS aJe s,sr aql Jo uorlrod
-srp are s,r ar{l .r.toqSurrunsse .'i11en1rr: (r) :suorldrunssp Ieuorlnqlrlslp Jo asn
-ord 1uq.tl /plnoLIS lr se 'slJalJar pue
'(;r-) uorlelndod puoJas P uI
aql roJ suoseal IeJII)eJo ale alar{J'JaAa
I pue 0 uea.nalaqsoII M alors a8era,,re lnolllr^4.
-.MoH 'parJrluenb .{1l1rt1dxaaq paau
uuqt rolearSaq ot puot (y) uollelndod
ar{l '0 ro 'zfr'I se paJOJSsr uosrrudurol
's-rz
euo ur r slqPrJe^ a^rlPlrluPnD auros Jc) (erur-ueB :-r2lurluauodxa ertrlegau
qrea aJuts (Nr < t'.r)qo;4 = 6SurqrDs
-ap uarI.l,t pauorluauJ suosrreduor
uerssneg) ruroJ Ieuorlnqlrlslp lrexa aql
sla^rl aql raqlaLI.^ lsal ol palndruor ^l
' c , 2p . r ; r e 1
-lensn sr 'M 'Jrlsrlels uoxoJlrM aLiI
Jou (piltPl lJal
lqBryqr)JIJlrtu
lo pul) arll saTeur d11en1relr aJurs
i
-ru,,\s) arnleu aql Jrqlrau :rred (1eru
M Jo uortelnturol .{ra-t ar{l uroJJ snorl
sBurIueU
-Jouqe 'lerurou) e Jo SurlueJ JJaJJoJe
-qo
g
.trou
plnor{s
aq
drqsuor1e1a11
asrMJredlJaJJoJ Jo dlrlrqPqoJd
'sBurluer rraql
;o ,{lrlrqeqord aq1 Surlnsearu se .,(ldrurs
ar{l pue JllsllPls uoxorlrM
go lq8noqi aq uef, alJnr aql rapun
aql:gdrqsuorlelal'III
Par or{l /snLIJ sllolirlqr)lstpNx pttu r!x JtlJ
Jo uuo| J\l ltloEJ suotldunssu oll sa)ru
uo ,,{1uc'r
lnq s,-r aLI} Jo sanlel lenlJe
ar{} uo puadap lou saop 1,4'suosrred
-uror ou/sa.{ uo paseq sr lsal aLIl aJurs
'oslv 'AI uorl)as ur Molaq paqrrJsop aq
ol poqlaru ralseJ Ll)nur e.,{q pau,roJJad
aq ueJ uorlelndruor aql 'aJrlJeJd uI
I

N 1 1 1 H 1

(nr'Yr)s33

=t

''r'l'suosuPdruoJ
Nu c vu aql Ja o s,S eLIl Sur8e,rarrepue

(.,{1uoelep
alarrstp)

'.{ressalauaq lou .l,lqeqord

11r.u lr leql sl /snlnJIeJ leJgalur uo spuadap
Sulug aarnr xaldruo) pue 'Jallloours qllr{,!\ ';oo.rd aql J-o arntpal lueuod
qrnru aq oslP IIr,^^a^rnJ )ou IeJIJIo -urr lsoru aLIl 'lulooMer^ Jno uloJc
-rua aql leql suParualeJs lllrqeqoJo

JOaJOfSaql Jo aJnleu snonulluof, aJoiu

aLIl 'uorlenlls srlll ul 'raAa.MoIJ suolJ
-nqrrlsrp uerssneS ol iuJoJuoJ Alrres
- s J ) J U l o u l r , l , , ts J r l r l r q p q o l d r o s i J t l J s
asaql Jo suorlnqrrlsrp aLIl '.{Sololpluol
-duis Surluasard s,luerlpd e uo paseq
suorlenba ad.{1-uorssar8ar ruor; alers
snonurluoJ e uo palJnJlsuoJ sarllllq
-eqo,rd ro soJors r{ll.lt pa^ali{Je uolleu
-rLurJJSrpor{1 aqIJJSaPO1 so^rnJ JOU
Sursn ur palsorolul sJoJegrlsalur JoC
'suorlnq

t,:",,-ol
=,u,,,-,,,
^n*

T..?il;l

olnr aql
o1 Surprome uosrredruoc qrea Surrors
's,Nr aldtues Nl aLIl pue s, vr aldures F'lr
aql uaa.^.\laqsuosrredruor alqrssod Nrt .
vr IIe Sulletu Jo slsrsuor 'hlpnldatuttt
'N urorJ Nu pue
$aal la'arnparorcl aLI]
y ruor1 vu az\s 1o aldrues p qllM 'f,]lun

-rJJSrpuerssneC o.trl .{q parldrur r.uro;

a^rnJ JOU aql qllrvr IIa,v\ lqeuosear
aar8e o1 pual sarpnls Sur8er,urI)ipalu
pup Ip)rs{qdoqr,,(sd ruorJ elp a^rs
- u J l y J e q l ' l , r A o a J o 4 1' , { ; r r r 3 r ; a d sp u r :
dlr,rrlrsuas uaamJoq sJJo-aperl arll se

(Nr < t'r)qo;d =

p - a^Jnr JOU rapun eal ,,anrJ.,,

ro'1enba a,re8ur1uel l)aJJoJ e

1o ,{lrliquqord aql pue alrnf, or{l Jopun
eare er{l 's8urler .,{ro8aler Jo raqwnu
alrurJ e ,{1uo ueql JaqleJ runnurluo)
r arrlua aql Sursn Surlroder yo alqed
-e) sr rapear e leLIl pue sluaried Jo ald
-rues aJrurJuruP aAPq a.4^lPql lueruoru
d L l l J o J J r . ! n s s PJ , M J t l e q l s , { e s 1 1 n s o . r
'(Nr
l vx)qot4
,slo.^ s puP uaarC u_or{I
= // se lrlrqeqoro srlll ol raJal a.^^
'puequor{s
Ie)rlsrlels e sy aSerur leru
-Jou Jr{l uroJJpaurelqo Nr Surpuodsar
-roJ aLIl ueql ralearS aq lreJ ur plnoM
a8erur Ierurouqe aqt ruoJJ surplqo
auo LIITLIM'i,r'uorlrdsns yo aar8ap aql
'urJa] asrJJJd aJour e
'Jo uorl
]o lue,^4JoJ

= (d)3S

'kZt=ZtO0:
9899261 + rgltEj 0 + 1996600

7 , t 6 8 = e 6 80 = ( i 9 . 8 9 ) + Z V 9 ' Z: ( / 4 . N i i ) + ( 9 ) 1 e 1 o= 1H = M
Z 8 I 8 0 = ( [ r r . N a l + ( t ) 1 e 1 o g= r O
I8'0 = (f rr. vxl + (9) Ielol =zO

gZL

tlzZtt'Zl

tlz9ll'91 Z9I'Z\

l z Z I 9 ' Z n I gZZ'tU lz9r'82 ,9'

6 8 0 : ( Y i l . N r r )+ ( 9 ) I e 1 o 1 = , 1 4

Zr9'Z

0 ol su()ll
-rppe alrssa)f,ns ,{q 1 tuor; paure}qo

99
vu = 19

I c'lsvJ. LUorJPauIlqO
I g = i l r rr u o J Js u o r l J e J l s
- q n s a l r s s a J J n s, { q g u o r ; p a u l e l q o

rl.l :

I 3-ISVJ tuorJ paulelqo

sj r P u a N

89

zheZl

CI?

II

II

0lZ
6l
Z

bh
9

=x

IPIOf

ssz'r
86s'z
Z8Z
g

t88'08 [z()x /I+ ()x (z)+ z(z)]x (I)

680'I [.(t) x /r+ (I) x (]) + z(t)lx (s)
'ltE9'l

r< paler i^leuuouqe Jo raqtunN

8t

r paler sleulou Jo raqunN

t
9

( ) x ( r ) x z ^ + ( z ) x ( Is)
r > p r l r s l P r u r o uJ o r a q u n N

r p a l P Js l l ? l u J o u q eJ o J J q r u n N

91
g

Z
I

slualuo)

z=x

( 3 u r 1 e 1 )u t u n l o J

rorrg pJepuels sll pue 14 1o uorielndtuo3

:II aravl

esis that variable r cannot be used to

discriminate between A and N (i.e.,that
d equals 0.5), its behavior when d exceeds 0.5 (i.e., when r is actually of
discriminatory value) is also well established. In that regard, for our purposes, the most important characteristic
is its standard error, since our main
interest is in quantifying how variabie
W (or its new alias, the area under the
ROC curve) will be in different simiIar-sized samples. When d ) 0.5, W is
no longer nonparametric; its standard
error, SE(W), depends on two distribution-specific quantities, Q1 and Q2,
which have the following interpretation:
Qt = Prob (two randomly chosen
abnormal images will both be ranked
with greater suspicion than a randomly
chosen normal image)
Qz = Prob (one randomly chosen
abnormal image will be ranked with
greater suspicion than two randomly
chosen normal images)
If we assume for the moment (as Green
and Swets do in their proof regarding
fl and the area under the ROC curve)
that the ratings are on a scale that is
sufficiently continuous that it does not
"ties,"
produce
then SE(W), or equivalently SE(area underneath empirical
ROC curve), can be shown (8) to be

=
sE(r,v)

(1)
The quantity W can be thought of as an
"true"
estimate of d, the
area under the
curve, l.e., the area one would obtain
with an infinite sample and a continuous rating scale. In the rating category
situation, of course, W will tend to
underestimate d, but Formula 1 will be
useful nevertheless.
We now go on in the following section to calculate the Wilcoxon statistic
for the data in TanrE I and to show that
it does indeed correspond to an area
under the ROC curve, albeit the area
found by the trapezoidal rule. We also
carry out the computations required to
estimate directly from the data (i.c.,
without any distributional assumptions) the two quantities Q1 and Q2.
With these, and using W as an estimate
of 0, we then use Formula 1 to estimate
a standard error for what is in this case
a somewhat biased area under the
curve. As will become evident in subsequent sections, there is a second
method, requiring fewer computations,
for estimating Qr and Q2 for use in

32

Formula 1. We give both methods for

the sake of completeness.

IV. W andSE(lY)-Calculated
without Distributional
Assumptions
We illustrate the calculations using
the data from TasI-E I. Since the Wilcoxon statistic is based on pairwise
comparisons, the specific values 1
through 5 that we have applied to the
five rating categories are to be thought
of simply as rankings. The computations can be conveniently carried out
according to the scheme shown in
Tanle II.3 Rows 3 and 1 are taken directly from Tenrr I, while rows 2 and
4 are d.erived from 3 and 1 by successive deletion and cumulation respectively. The quantity W can be computed in row 5 by using the entries in rows
I,2, and 3; the SE requires calculation
of the two intermediate probabilities
Q 1 and Q2 (see rows 6 and 7 for details),
which are then used to compute an est i m a t e o f S E (W ) f r o m F o r m u l a | .
Tast-r II shows the detailed calculatiop of W and its standard error. The W
= 0 = 0.893 = 89.3%derived in this way
agrees exactly with the area under the
ROC curve calculated bv the trapezoidal rate. By way of comparison, the area
under the smooth Caussian-based ROC
curve fitted by the maximum iikelihood technique of Dorfman and Alf (9)
is 0.911 or 9I.I(/o; the area under the
smooth ROC curve derived from the
parameters of a straight-line fit to the
ROC plotted on double probability
paper (see Swets [4], pp. 114-115) is
0.905 or 90.57,. The slightiv lower estimate provided by W, or equivalentlv
by the trapezoidal rule, merelv reflects
the fact that the rating scale does not
"grain."
have infinitely fine
In another
context, where the ratings might have
been expressed on a more continuous
scale (i.c., without ties), the two would
agree even better. What is morc. im,
portant is that TABLEII, using 89.37 as
its estimate of d, produces a standard
error of 3.2%,,compared with the SE of
2.96% predicted by the maximum
likelihood parametric technique. Although this 3.2% appears to be a little
high, it is not greatly so; moreover it is
on the conservative side, and guards
against the possibility that the distributional assumptions that producecl

l'l'he ration.rle for this schemc is conceptuallr

s i m p l e b u t l e n g t h v . D c t a i l s c a n b e o b t . r i n e c lf r o n r
the authors.

A p r i l 1 9 8 2 V o l u m e1 4 3 ,N u m b e r1

the SE of 2.96c,4are not entirely justified.

The Wilcoxon statistic now provides
a useful tool for the researcher who
does not have access to the computer
p r o g r . r m d e s c r i b e da b o v e , b u t w h o s t i l l
wishe's to use as an inclex of discrimination the area under a smooth ROC
curve and to accompany it by an approximate standard error. He can use
lhe parameters of the straight-line fit
to produce the smooth ROC curve and
the areaunder it and he can useSE(W)
as a slightly conservative estimate of
the SE of this smoothed area. In our
example, simplv by plotting the clata
on double probability paper, estimating a slope and intercept from a
straight line fit, calculating from these
a quantity that Swets (4) calls z(,4). and
Iooking z(A ) up in Gaussian probability tables, one obtains a smoothed area
of 90.5%.,which is only 0.6% (in absolute terms) or 0.667c(in relative terms)
different from the 91.17,obtained by a
full maximum likelihood fit. By an
equally straightforward approach, one
can use the calculations in Tast.r II to
come reasonablv close (3.2% compared
with 2.96'/c)to the standard error procluced by the maximum likelihood approach. As we will see later in section
V, we will be able to improve even
further on predicting the SE Produced
by this method.
A11 of the discussion thus far has
centered on computing an are.aancl its
SE from observed data; we now turn to
"How
the commonlv askeci question,
big a sample do I need?"
V.

Planning Sample Sizes

Perhaps the more important use of

the three-way equivalence between the
area under the curve, the probability of
a corre'ct ranking, ancl the Wilcoxon
statistic is in pre-experiment calculat i o n s . A t t h i s s t a g e ,o n e i s o f t e n a s k e d :
"We
wish to use (i, the area under the
ROC curve, to describe the performance of an imaging system, and we
would like to have this index accompanied by a measure of its uncertaintv,
i.c., of the fluctuations in the index
caused by the random sampling of
cases.How many casesmust be studied
to ensure an acceptable level of precision?" This is equivalent to asking how
large rt.1and n1, must be so that the resulting SE is of a reasonably small
magnitude, and the resulting confidence interval is correspondingly
narrow.
In addition to the quantities 1.1and

H a n l e v a n c lM c N e i l

ADO-IOICVU

f,IISONSVI(I

-u;dra 8ur1eru iuorJ pa^rrap (fi)gs .,n

asn o] MoLlSuraasur dlaq op spoqlau

a^Jnl -)ou up Japun paJV aq] JclJsIl prJpsurupatri

iq ildurs d atptullsapInt)ia.
3ur1e1nr1er
r u o / t J t l u l a q l u l ' t u a t u r r a d x . Jr J V Z

o,\\l JLII uJaMJJq SJr] IeJrlsrlPls IeurJC),I lufil.)t1uu UJOJJpautelqo uJJq JAPr{
alduura luaurlodra
J L I I ' T J J . t , r o H ' s l L r . l u r r a d r o3 u r . 9 e u r plnom Iell.lr r.{lr.&\
flriyl; pauorluau Alsnor,rard rno r-uor,y
plrurptqo 7,9ilalerurxoldde yo (d)ESaql
crBduor ol
yo sr 1r '3urss'edu1
lsaJalur
'l'r/sJsPJ
Ieurouqe Jo raqLunu aql
lenba 1ou saop Ni/ seseJ IPuJJouJo Jaq
-urnu rq] uJr{.ry\s,1s alelnf,ler o} z pue
't
I selnruJo-I ol uos.lJ ]snru auo
'(Z 3lC)
Irpour uerssnP) rr{}
rJpun paurelqo asoql uer{J a^rle^Jas
-rror arour {1lq8r1s are s,fs aLII

'I o] asolr asoll] ' r'l 's,1,LISILI

'Z
.{ra,r.ro1 lsalleurs als s,fs aq1
'ES ar{1a^leq ol azls aldules
aql aldnrpenb lsnLu auo 'alduera
LIIIM Alasra^ut .{le.l
roJ 'leqt os 'lll
'I
Aoql ' s,1s LIIrM sl)adxa auo sv
:patou aq plnoqs
slurod;o rrqrunu y s,p palednlJup snol
- J e Ap u e s a z r sa l d u e s s n o r r e l J o J ( d ) E S
sa,rr8 E arngr{ '%9g E o} ES er{l aJnpar
'hl'V p
ASue
III,vr09 - Nil - vr allqm
slrrperd I plnruroC'97 = Nrr = f'lr qlll,t
uaql IISl'0 = 98'I +.(gg'O)Z= zO puP
' o / , 9 8=
l6El'0 = 9I I + S8 0 : t8 uaql
1iJo poor{roqq8reu aq} uI aq ol polrad
-ra sr ,irernrre rrlsouBerp aql eJJq.{\
'alduera
JuarurJadxa ue Japrsuor
roc IIetus .{1lua1rg;;ns sl (d)ES Ulun

Ir?)rparu ur pol{laru 8ur1er aql oJ Jolr

-1.:rdruorsn()rJase lou sr polllatu fllvz
Jr{l'suosPJJ JaLIlo pue asJql Joc
.^]
IJIJ
- 1 ; ; 6 l s p u c . i l r n r l r s u a s' . i l . , u e u ' , r r e . r
-nrre rrisou8erp Jo sluirrodruor ;1ere
-d.-rso,u1 rL{l uo ulpp e{qenlp^ sp1ar,{1r
'(squarleclro.uo1 Suurnbar ' ; i)
Ie)rurou
-oJa aJoru Suraq o1 uorlrppp ul a)rol{J
Jo Lroqlau aql oq plnom poqletu Surlur
aqt'rualsis Sur8eur Ierrprru e iq paryrs
-sep .{11rarroJaq plno,l,r leLIl sluarlpd
rql SLrrlpurrlsrur
to JBeluaJJrd 11ut,t,to
,i1uclpalsaralLrrJJa,\ auo Jr rrJAJ 'snqJ
'60[ rno yo
luale,rinba ierr]srle]s rr.ll
se (srred rrr)sa8eurr rlz aql Jo {urllt uer
e.n'p Sur.rnseJurur (AS ,uot) uorsrrerd
'luJrutJadra Surier
Jo sai-odJnd JoC
aql ul pasn 60I - vlt + Nll aLIl UPLIj
(002 ro ut7) stliuuu Jo raqrunu rrterr8
ilqeraprsuor e'sagetur (leurouqe'1uul
-rou)
s.irarlgg1 - lr paau p{noM auo
1o
' '/.06 to
6 0 Jq ol lno pauJnl luaturradxo
JCVZ l{l ur 1;ltreJuI }pt{} Surunsse pue
',/, ro
90 0 ;d rorra pJppupls e aAJrL{Je
oI'tu lq) - I)0t Jo 'uoltnqrrtslP lelluou
- l . l . r q l u ( r P J q t ' q ' J ( ) J J rp J t ' p u t ' l st ' t { l r M
d Jo alerurlsa sn{l ,{uedruoJre plno.,',t
auo pue /parJrluapl f llrarror ara,n
a8er.urleurouqe pup leurJou Jql eJaLIM
se8uur Jo srred u p 0 uorllerJ aLIl

-ocira.r,r.11e8au
= rlrrl prlr)s1tuop.)rr1
,f :t:1;
's.)stlJ
IeuJJOuJo (,\rr) raqtunu Ienb.l
(p) a,rrnr JC)U J.rpLIn
up JunssP suorlplnJle]
P J I P J n J l p U P ( S J S [ ? JI C T U J O U q p J ( ) J i l q u I n U
= f'rr);zrs aldues o1 uorlplar ur (r) a^rnJ.)C)tl
rapun pJJp prlprxrtsa loj (:1S)roJrJ LrJplruptS

-ilp sn()rJpALIll,u uuuES = sa:enbs lE 6:1 or1e.r

JJuuIJllA tJlll\ LIeIssneD = saliutrtJl 19 I: I OIIPJ
,)lurlrJP,\ r{Jr,\,ruL'rssnPS = salJJrJ suorlnqrr}srp
Bur,il.rapun turJaJJrp ruoJJ palpraur8 .r,r.rn:
eaJe JoJ JoJJa lrrepue]s palec{rrrguy
.)(JU rJpun

a^rn) lou

raPun oarv,,anrI,,

vu

'uorlPur)sa poor{rl
-alrl urniurxpu ,iq parnporcl L96 Z Jt.ltot JasolJ
ua^a'li,/g1 ro l000 JO:JS rrp sart3 1 uotlenb"1
olur sanlpA JJJqI asrrqt Srrr;n]r1sqn5 66gg g
- z D p u p S g Z g 0 = l O u r t ? l q oa m ' 7 u o r l e n b t u r g
r o l s r q l H u r s l , 1 , 90 6 r o 9 0 6 0 l o e r r e u e p a : n p o r d
roq
Al uorlJas ur pJssnJsrp elpp arll'alcluer:
plep l2lJsrJo JOJ35 ue iJurlelnf,leJ JO porllJu puo
- : a s e a p r r o r d o l ' l l l u o r l f , a sJ o p u a a q t t p p J U o I l
- u a u s p ' o s l p ) n q f S ) t n l i l l u 1 - t , t l t u lo 1 , { 1 u o 1 o u
pJsn aq upr qreordde prspq-uoltnqlrlslp slVJ I

s,r aql .'{re,ruBr pue'1; oJupruroyrad 1t-r

Ir^al paledrJrlue .,{ue1PipalJadxa aq ol
:1Srr{1 urelqo a,4^'I elnurroc olul paln}
-rlsqns ale suorssardxa JSar{JuaL{M

ft)

Q l + t ) + 7 0 e= z O
Qt-z)+d=tO

''J't'0lo suoil)unJ
a l d u r s s e p a s s a r d x aa q u e J z | p u e t f
sarlrluenb ar{l 'laporu slr{l rapun eJUIS
rlr?unlroJ .{11enadsa sl slt{I paraprs
-uor
aql uelll aAIleAJas
slapotu
JaLIlo
-uor rJour {11t13r1s
are lelll s,1g sp1ar.{
Irporu Ierluauodxo aarle8au alil'(arotu
Jo %0g Jo seare) lsaralur 1o aSuer aql
ur leq] sruaas JI'Ja^oarol/{ suollnqul
-srp ierluauodra anrleSau ro 'euue8
'uprssne.J
salelnlsod auo se
3ur,{1,iapun
sa8ueqry pue (d):IS uaa.{ laq drqsuorl
".4^oq
-elar
ernSl{
s1t{or{s
orlt
alllll
z
'oldr.uexa up sV 'suorlnqrJlsrp aql
Jo
sJelaruered rrqlrnJ .{ue .{q paluanlJul
pue'B dq paurruralap
.{ltq8r1s ^,{.ra.L,{1uo
.4.1arr1ua
lsotule sr I elnwJoC suollnq
-rrlsrp Jo rred par;neds ^,{uero1 ',,{1a1eu
-nlloC Nr pue t'-r JoJ suollnqrJlslp
3ur,{lrapun ar{l Jo suorlf,uny xalduor
are z0 pue I0 serlrluenb aql 'p Pare
anJl arll Jo sanle^ paledrrrlue asn UPJ
ouo alrr{M zO pue'10'(/-sralaurerd
'Nll
Jaqlo aarql suleluo) I elnruroC

'uollPullullf,slp ou =
a u r l l e u o B e r pp a l o p 1 r ^ r n l ( p a i ^ e q - u e r s s n e 3 )
p;r{}oorus = .rrirl prlos 1o,t:n: 1e:rrrclura
= a u r l p r q s P c J : n { v . L u r e l P pr o J . r ^ r n r J o } l
f

ooL

oso

ooJ")

'ooo

a.D

ooo

o
m
?.
f
o 6
o

l::
]"",
1 arn81g

TasrE III:

0r
0.700

o.725

0 . 75 0

or 95% of Detecting
Number of Normal and Abnormal SubjectsRequired to Provide a Probability of B0%,90'/<,,
Various Differences between the Areas 0y and 02 under Two ROC Curves (Using a One-Sided Test of
Significance with P = 0.05)
0.750
652
897
I131

0.775
286
392
493

0.800
158
216
271

0.825
100
135
t69

0z
0.8s0
68
92
115

0.875
49
82

0.900
37
49
61

610
839
1057

267
366
459

148
201
252

93
126
157

63
85
106

61
75

565
776
976

,).)/

136
185

85
115

423

/.1 |

l + l

516
707
889

221
306
383

r23

0.775

463
634
797

0.800

0.825

Ot)

9f,

.1+

55

0.975
18
23
29
20
27
33

41
55
68

3l
41
50

3l
38

77
104

az
69
86

37
49
60

27
36
44

6fl
92
113

46
6I

33
43

342

110
149
185

75

tr.1

408
557
699

176
739
2q8

96

r29

40
52

160

59
79
97

r50
203

8i

50

r08

oo

za1

134

8i

290
393
'l9l

123
165
205

66
87
107

960
13 1 4
16 4 8

228
30tt
3E3

96
727
156

710
966
I209

165
220
272

167
209
20'l

l?q

350
477
597

0.875

0.900

0.925

457
615
765

0.950

80? probability = top number;90'rl probability = middle numbt'r;95?

34

46

0 9s0
22
29
36

58
77
96

0.850

me-nt to construct a confidence interval

o n 0 . A s o n e m i g h t e x p e c t ,a n d i n d e e d
as we htrvefound by repeatcdly simulating data from two overlapping
Gaussian clistributions, constructing an
ROC curve, and deriving tht- area t/
gnder the curve, the distribution of the
d's we obtainecl is not entirell, symmetric, but is^ instead somewhat
skewec1towards li = 0.5. This skewne'ss
"true"
/i apis more' marked as the
proaches 1, and as the expected numbe'r
"misclassified
p a i r s " [ r r ( t * 1 , 1f)a] l l s
of
below 5; this is iclentical to wht.rtoccurs
rt'ith the binomial distribution ancl a
s u c c e s sp r o b a b i l i t y c l o s e t o u n i t v . I n
s u c h c a s e s ,o n e u s u a l l v r e s o r t s t o a n
exact (asymrnetric) conficlence intervarI
for 0, rather than using the approlim a t e ( s l m m c t r i c ) o n e o f + 1 . 6 4 5S E ( l i) ,
1.96 SE(//),. . . , provided bv the normal
distribution. In the examplc here rrr(1
- 0 ) = l 0 i s c o n s i d c - r a b l vg r e a t e r t h a n

0.925
28
38

probabilitv : bottom number

company an estimate of f/,we'can calthe rule of thumb of 5; thus, the symculate how many casesmust be stuclied
9
5
%
i
n
t
e
r
v
a
l
o
f
metric
confidence
so that a comParison of two imaging
9 0 . 5 ' / ,I 1 . 9 6 ( 3 . 0 7 ) o r ( 8 4 . 5 7 ,9 6 . 5 % )w i l l
be reasonablv correct, cttmpared rt'itl-r s y s t e m s w i l i h a v e a n v g i v e n d e g r e e o f
'istastatisticalPower. This power or
the exact, slightly asvmmetric interval
o f ( 8 2 . 4 ' 7 , , 9 5 . 1 1 ,o)b t a i n e c l b v c o n s u l t - tistical sensitivitv" depends on how
small the probabilities a and P of
ing chartecl conficlence- lirni^ts for bin o m i a l s a m p l i n g ( 1 0 ) w i t h 1 / = 0 . 9 0 5 committing a tvpe-I or tyPe II crror are.
T v p i c a l l y , o n e s e e k sa p o w e r ( f 0 0 - / l )
and lr = 100.
dif
of 85% or 90'7 so that if t sTtt'cificd
o
f
t
h
e
q
u
c
s
t
i
o
n
w
e
c
o
n
s
i
d
e
r
Finallv,
o b t a i n i n g s u f f i c i e n t l y I a r g e s a m p l e f crurce r'risfs,it is 85% or 907 ce'rtain to
r . t n B e i w h e l l ( ) n e w i r l t e : t ( r ( ' \ , t n l i l l e be reflectecl in samples that will be
"statistically
different." Trathe tlifference be-tween twt'r areas, str d e c l a r e d
that lf nl irtryorlnrt diif crurcc itt yt'rior'' d i t i o n a l l v , o n e u s e s a t y P e I e r r o r
n r t t r t L 'ec x i s ! s ,i t w i l l b e u n l i k e l y t o g c r p r o b a b i l i t v o r o o f 0 . 0 5 ( 5 ? ) a s t h e c r i terion for a significant difference-.
u n d e t e c t e c -i ln a t c s t o f s i g n i f i c a n c e .
'Ianll
III gives the numbers of norm a l a n d a b n o r m a l c a s e sr e q u i r e ' d f o r
VI.
DetectingDifferences
each ROC curve to have an 801r,90(Ic,ctt
between Areas under Two ltOC
9 5 7 a s s u r a n c et h a t v a r i o u s r e a l d i f f e r Curves
errces 6 between two areas, 0y and H2,
A g a i n , k n o w i n g i n a c l v a n c et h e a p - l v i l i i n d e e d r e s u l t i n s a m p l e c u r v e s
p r o x i m a t e S E ' s t h a t a r e l i k e l v t o a c - s h o w i n g ; r s t a t i s t i c a l l ys i g n i f i c a r r t c l i f -

A p r i l 1 9 8 2 V o l u m c 1 4 3 ,N u m b r - r 1

Hanley and McNe:il

ADOTOIOVU ]IISONCVI(I

J^lnl

-)ou ue Japun eaJV rrl]

IO asll pue iuruPJtrAl

o l s a q s r ma u o J I I e u o 8 e r p( a , L r l e u r o 1 'a^rnJ .)ou aLIl rapun /, urrP ,,rnrl,, rlll

-Lrruou)Jr{} Ol .r^rlelaJaAJnJlC)U ue EurlelurlsJ Jo poL[]JuJaJJJ-sPrqJJoLUe
aq .{\ou IIr,MJrlsrlels uoxoJlrA\ Jr{l 'ra^a
l o l t o t l u ; t 0IlI e J a ^ o a L I l a n l P I u l r J J u r n u
-.!roH slrolldunssp Irurourq JLII Lro
a18urs e ur ,{;rluenb ol pasn JJlarupr
-ed p sr aAJnJ JOU up Japun ,.)Jrl aLIl
prsLrqsas^IL?ue
fQu r()J riqetins rq lou
'puoJas
'laslno
,\Pul
SA)rpUr
alrsocluJoJ
Jsaql Jo suorlnq
alil
le pauorluaur se
'r,ualqord
'a^oqP
sri{l
Jo aldr.uera -rJlsrp aql
PaqrJJsaP .]APrJJl\4
sairoBalel 3u11eraq1 uerll ,,snonurJurlJ,,
JerlnueJe sr slsoll parredun pue parred
u , ) . l . u l a q r r u r J r J J r p a L t I s a l d r , u n s JJOLUl.lJnuI sI lutll JIPJSP apl^oJcl Jb^Jtil
q B n o q l l V u o r s s a r B a ;r r l s r B o l p u p s r s , {
l u c p u a d a p u r r o J s l s o l 1 e r r 1 s r 1 e 1{sq
-]eue
polPJJl aJe suorlAJasqo parrecl uaq^^
q
J
n
s
s
e
s
JnDruqJaJ
l
u
e
u
r
r
u
r
J
l
s
r
P
p J u r p l q o s r u p L l l l s J l I n J J J , t , r ( ) d . r l , r L uu r r u J l ) . l A u d P s a r 1 r ; r q t ' . 1 o l P
d L r r s. 'J l ( r ) {
qJnuJ e saprnord sas.,ileue IerrlsrJels Sursn suorlrrpard lerrparu ;o i1rre1
-ndod
parred Jo asn aql /alqelrele ale suorlel
S
u
r
s
e
a
r
J
u
r
u
u
s
r
a J a L I l' p J I r { I
'aJrJrrJPs
-Jrsqo parred uaLI.r\ler{l aldrrurrd rql
IIPUS e SuraJslr
' l ) l ' \ ; u r . i l t q B r 1 s . r .qi e u l ( ) l r . r p r p p
uo paseq are .{aql ',{11errseg(Zt) atrp
lnq
p
-;n
-uelr- a{1 ',{1pa11rupy ralndruor o8rel
J
q
]
a
J
p
u
(
)
r
l
l
)
J
[
q
n
q
l
u
a
n
b
a
*
q
n
s
J
o
-enlrs rallpi slql qll^{
o1 spoq}al\l
e ol asJnolJJ IPnlJp lnoLIlr,Mpoor{rla)rl
Ieap
'(lserluor
urnrurxPur
)rJlarueJPcl e Jo lrJauaq
Jo sla^al 3ur.,{re,Llnoqlr,4r pup
q1r.u, '{rotslq lnoqtl,4t pue qll-{{ ' rl ,r)
l l n J r L l l l s o L U l Pu r P l q ( ) u r ' ) r u l P b t l i i n
-ur aql lPrll suearu .{4ouJoJJapJepuels
suorlrpuo) luararJrp rapun saseJJo las
s l r J ( ) Jp l n u J o J l u u o r g e l n d u o )n p u u
aues aql 3ur1en1e.La
sr Japear auo uaq^^
l o s J s e JJ o s l ; s ; r u p s J r . l lS u r u r r u u v . r. r . r t , eJJe paqloorus e Bururelqo JoJ poqlaur
sarlrlepouI o,&tlaql uaq,^^ suorlenlrs JoJ
lqder8 p Jo uorleurqruol aLIl 'pu.oJas
d I q p l r n sl o u J J Es J d q t u n ua s a q l ' p a . r u d
( sanssr
-ruor Suraq suor;rpuoJ
uorsrJap ruoJJ alqrssod se qJnr.u
leluaurrJadxa
o M l a q l J o L l r p J. r o . s; l r a f q n s l p t u r o u q p
se ,{1r1eporuSurBetur aql Jo sarlrlenb
pue IeruJou ;o 1asaleredas e 8ur1ra1as ,{loleurturlrsrp JrsurJlur aletedas ol luel
-Jodur sr lr 'sassaJoJd1errs,'{qdoqr,,(sd
uo pase,qare III Etgvl ur suollPlnrler
azrs rldures aql lelll Palou aq plnolls
a.4-{OH SJOIJA JTJSOU
JaLIIO r{}r,\{ sP'JO
'.{pn1s slrll
-3erp ;o sad^,{lorr1 aql Jo slsoJ alllelar
11
Jo stlnsar aql Surlard
-Jalur ur .{ressarau aJe slealeJ o.nlrl
aql pup sguarled pasasrpuou pue pasea
'paurelqo
-slp
uorleruJo,
Jo [1:y ra8uol ou] arnlnu lurt aqt
-ur
qlro.tt
aql
sr
elep
aql
xululelqo
apnlJur aJupruJo;Jad rrlsouBerp arua
lou
-nl;ur leql srolret Iuorsrrap] ieura]xa
Jo lsoJ aql lPr{] palrnDaJ aJe suorlnlrJS
-ur .{ueru os lJeJ ur Jar{lar{,tr ro 'parrnb
rarlto 1.{llenba paiqBre .na ,{1nr;rrads
-aJ
alorllnl.u Jarllaq-^,{ pue .,{1r,r11rsuas
qllM uor}eururrJJsrp
aJe
suorlnlrlsur
'euo1e
qf,reasar
,{laTrl
^,{pn1s
e
ruroJrad
roJ
Ierlualod rrsurrlur aql sia,ruor
ueJ uorlnJrJsur auo Iar{laLIM noul
.{1uo Surluer lrarro) e ;o ,{lrlrqeqord
srole8rlsa.aul lal III,lt sraqrunu esaqJ. s r L l l l c L l , u a q a z r s u q d L U .r)M ) , ) l p l s
'.{1ure1rar
J S e d s r pl r d r . { l o l s p p d l u p r . { 1 1 - l ; . r l o :
;o saar8ap rrJrrads qll,vr
(sarlrlepou SurBerur rraql aruaq pue)
aq IIlr!1 sa8Prrr IeurJouqe puP IPruJou Jo
rred_uropue.r e tpql {tlllqeqord aqt stuos
seale o^/!tJuaa.{}aq saruaJJJJrp }Jalap
-alclaJ
o1 ,{ressarau sazrs aldures alerurxordde
a^JnJ
Jou aLIl Japun paJe arll
'qreordde
r l l t a l p l n J l e r o 1 . r l q r s s o dl r a l e t u p ) p p
IOU rr{1 Sursn sarlrlepotu
SurSerur 3urz,{1eue ro1 pa,{o1dua
asar{I 'palerur+sa aq uer JII E-trrvJ ur
ulvror{sseoJe aLIl 'suorldrunsse 3ur,,{;r1d , { 1 1 e u o r l u a , r u o J ' p o q l a u 3 u r 1 e r a q l u 1
-rurs auros LIIIM 'sallllepotu Hurlstxa o1 lpr{} MoLIs sllnsar aq1 '.{11errseglnydlaq
ere radBd srql ul u.^.{oqs
drqsuorlela.r ur flnrynads pue ,{1rrrrlrs alrnb ^,{11en1re
-uas
aruos
a^eq
alerurxordde
slr
aleturlsa
uaaq
sllnsal
a^rlrnlur
aqJ
lPql
Jo
'plarJ slr{t ur
apr,rord 11rm dpnls 1o1rd e /elqelre^e
{ro.4^ roj srseq
sauroJaq .,{111epourSurBerur rurau e
Ierrlsrlels JrurrrJ e epr.Lord ol .{{lueuod
-uJr arour lnq eJJe srql ul uorlrnlur rno
uaqzur'dlleraua3'tueuodur fllerrlrrr
pre o1 lred ur'uorle8rlsalur srr{l Toouap
sr sazrsaldrues Jo uorlerurlsg 'suorlelnJ
-1er
-un
pue
ra.trod
sazrs
aldures
uorlerx
J
,
t
.
r
'
s
n
q
l
'
s
J
S P qe l e p s n o ; u a 3 o ; o
to
-1aq '{11ensnpup
-rlsa JoJ Jrlsrlels uoxoJlrM or{l Jo asn
IIeus palerJosserraql
pue srualqoJd lecrpatu o1 sanbruqral
aql sl Jaded slql Jo uorlnqrrluol rofeu
asar{l Jo uorlerrldde aql qll^,\ palelJ
aql 'tuelJodurr lsoru pue qunoC
'uerssneJ
AIqeJaprs -osse suralqord lenads are araq] 'oslv
uoJJ
-uor
suorlnqrJlsrp
JaJJrp
3ur,,(1"rapun '3ur>pe1 sr saAJnr f,OU ,{lt^ sarlrlppour
rql
urla 'ruole
wor; pal:rpa.rd Surreduror JoJ pesodord sanbru
J
r
B
- q J a l l p J r l s r l e l sJ q l ; o S u r p u e l s . r o p u n
aq .,{1a8re1uer }r ler{l 1sa33ns 7 a.rn8rg
ur u/vlor.{ssuorlPInJleJ Jno 'JJAoJJol
a^rlrnlur ue lelll realJ auroJaq se! ll
'(lI
'JalalraoH'srs.{1eue JraLIl ul saAJnJ
srgvJ ur se) elep aql ruorJ .{lt)or
JOU
-rp
pasn
aleq suosrreduror asaql yo ^,(ueyq
palelnJleJ
aq
uer
aJurs'prle^
aq
lr
'sanbruqral SurBetur asar{l ruoJJpeurel
oJ anurluoJ JlrlvrtoJJa pJepuels slr pue

s e , u . i 1 1 u r :: , u : . r 1 1r c { . i 1e J o I i . \ l r i r q P q ( u d , ) t l l
1 1 3 r 1.1[ o 1 1 l n ( ) p t I r J ( ) l ! ] 1 J o J I u d r l P n b : l . ) , \ l ( ) \ L l f - l
! ) L l ( /)p i r p n l s J r . r . \ \l r r l l s , r /J L { }I L I t , \ \ o L I l ' l t t t J L I I L

-qo
aq+
sttostredwo.l
u
o
r
l
e
u
r
r
o
J
u
l
1
o
.3urn1o,rur s.rrpnls iueu
1o JJLIT?LII
-rolrad aql ol pel seq ualqord rrlsou
-3erp aurr:s aql JoJ (aurrrpar.u JeJIJnLI
'punoserlln'1.3) sarlrlepou Sur8eurr
a.Lrlrladtuor .ltau Jo lualpp JLIJ
NOISSNf,SICI
'/r0g pue rr0l uarMlaq rruaraJjrp e
sr ll Jr uelll %06 puP 7108uaa.{.{]rqaJua
-rrJllp e sl ll JI patrrlrp iltsea aror.u sr
,1,)0I
Jo aruaralJrp e lelll s^{orqsIII:rtfrvJ,
'i11eurg
lsrxa lreJ ur lou op saJuoJrJ
-Jrp palPls rLIl lelll apnl)uor ,{lqeuos
-EaJLreJJuo'III E-tgVJ uI u^{Ot{s Jsor{l
se qJns sazrs aldues elenbape ;o alrds
ur sgsrsrad,,a)uara;Jrp ou,, aql JI 'puell
ralllo aql uo c'rorra 11ad.fi e;o .'{1r1rqrs
'aJuaraJJrp
-sod
lue)rJIu
lear e sr araql
-8rs ,{11err1sr1els
e .{ ot{s o1 pa1re1.{pn1s
'ra.r.e.rtc)11
IIerus e Jr tpr{l ,Mor{sop .{aql
'saJuaraJJrp
lueirodurr lsetu o1 1de
pue,,purjun,, aq IIIM Surldrues ar{} teLIt
rpnlrlle rrlsrrurssad aql qll.4^ peuSrs
-ap are ieql '11eratJe lSuIpurq ,{1rres
-sa)3u Jou aJe suorlelaplsuoJ asaql
'ra,ltod
]Ptrlsllels ssJI
a.lr8 seut'rrallews alrLI.tt'aluaJaJJIp azls
aues Jql 3ur1ra1apJo JJueJnsse JalJJB
e a^Pq Io saJuaraJJlp Jsllerus lJalap
ol auo ,\^olle sezrs aldues raSJP'I 'aJua
-rarJlp luerryru8rs .,{11err1sr1e1s
e p1ar.{
plnom saldures aql uo aruelrltu8ts
''.t r)
Jo tsal ar{t }eql (.{trlrqeqord 469 ue
aJueJnsse qBIq e aAeL{ ol aAJnJ qJea
ro1 slralqns lPrurouqe 9 LI pue slrafqns
Ierurou 9lI lo eldwes e uo uuld o1
paau plno^L auo'%0'06 pue 7,9'28 araM
(suorlelndod alrurJur r{llm pa^alqle aq
plno.\4.lelll asoql '';r'l) seare ,,onrl,, aLIl
paapur JI ler{} smotls 1r 'aidruexa ue sy
(7 uorlenbE s1 zp Sursn paurPl

-qoz0 pue r0) 9gZ- zO + rO = zA

(7 uorleirbl g1 Ig Sursn paulel

-qo zO puP I0)
- rO + tO - I/
!.gZ
t0_20=Q
ra,ryrodtbg6 to
'
'
'
'rB' : r,
"1,06
,
hjg tol grg' I to gz' I
0
aruerryruBrs
'9v9'I = ilZ
Jo lsal paprs-auo %9 e rcl
aseJ Jno uI aJaqM

(s)

-lof {q u_a,rr8elnuroJ azrs aldues aql

Jo uorleldepP u uo pasPq aJe suorlPln)
-[p) rr.{l (prpls-auo 's0 0 > ,/; arue.r.l;

test statistically whether two curlcs ate

different (they could still subtend the
same area but cross each other), then
one must resort to a bivariate statistical
test (13). This test simultaneously
compares the two parameter values-a
the difference between the r.1 and ,t,y
distributions and b the ratio of their
variances, which describe one ROC
curve with the corresponding values
for the second curve. The test requires
that one supPly eitimates of n and &, as
w e l l a s e s t i m a t e so f t h e i r v a r i a t t c e sa n d
covariance. These estimates are provided by the maximum likelihood estimation technique clescribecl by
Dorfman and Alf (9).
Wt' u'ish kr thank Colin
Acknowledgments:
Bt'gg, Charlt's Metz, ancl Stanlev Shapiro for
helpful suggestions ancl lrt'ne McC.rmmon for
preparing tho mrnuscript.

J a n r c sA . H a n l e r ' , P h . D .
D e p ; r r t m c n t o f I i p i c l e m i o l o g v a n c 1H t ' a I t h
McCill University
3775 Universitv Strer't
Montrcal, Quebec
C a n a c i aH 3 A 2 U . 1

36

A p r i l 1 9 8 2 V o l u m e1 4 3 ,N u m b c r I

References
I.

2.

3.
4.

5.

6.

7.

8.

Lustccl LIl. l)ccisiott-making studics in

patient managcment. N Engi I lv{etl1971;
6-'12'1.
28'1:.11
Coocicnough DJ, Rossmanrr K, Lusted LB.
Radiographic applications of receiver ope r a t i n S c h a r a c t e r i s t i c( R O C ) c u r v e s . R a d i o l o g v 1 9 7 4 ;I l 0 : 8 9 - 9 5 .
M e t z C F . . B a s i c p 1 i 1 1 6 i p 1 eosf I I O C , r n . r l v s i s .
S e m i n N u c l l v { e t l 1 c ) 7 8 ;8 : 2 8 3 - 2 9 8
Swets JA. ROC analr-sis applietl to tht'
e v a I u a t i r ) n o f m e c l i c a I i m a g i n g t e c h t r i c 1u t ' s .
l n y c s t R a t l i g l 1 9 7 c . l) ,; l : 1 0 9 1 2 1 .
Snets lA. I'ickett RM, IVhitehcad Sli,cl .rl.
Assessnlert of diagnostic it'chnologit's
S c i r n c e 19 7 9 ; 2 0 5 : 7 5 3 - 7 5 9 .
(lrt'en Il, Swets J. Signal cletection theorv
anci psvchophvsics. Nerv York: Johrl Wilev

t3

Hanlt'r'lA, McNr'il BJ. Conrparing thc ar('as

uncler tu'o IIOC curves tleriveti from thc
s am e s . r mp i c ( ) f p J t i e n t s . R a c li o l o g r ' ( l o r t h c ( ) mi n g ) .
M r t z C I : , K r o n n r , r n H I l . S t . t j s t i c . t l sj g r t i f i c . r n c c t e s t s f o r b i r r o r m . r l I I O C Ic u n ' e s . I M a t h
I'svch I 980; 22:218 743.

a n c lS o n s , I 9 6 6 : , 1 5 - 4 9 .
C , r e e n l ) M , l v l o s e sF L . ( ) n t h t ' e q u i v . r l e t t c c
of two recogniliotr ntt'asutes of slrllrt'tcrm
nlcm()r]'. Psvchol llull 1966;66:228 23'1.
The rre,r above the ortlill.rl
Barrbt'r Il.
c l o n r i n . r n c tg r a p h a n c l t h c . r r e . r b e l o x ' t h e
receiver operating graph. I Math I'svclr I 975;
l2:387 ,115.

9.

L)orfrnan IlD, Alf [. N{.rrintum likclihoot]

e s t i n l d t i ( ) no f p . t r a m e t e r st l f s i g t t a l t l c t e c t i o l l
tlreorr' .rnt1 dett'rntinatiol of cotllitlt'nct'
interv.rls rating-nrethorl tlat,r. I Nl.rth
f ' s v c h 19 6 9 ; 6 : 1 8 7- 1 9 6 .

10.

Pe;rrsou ES, H.rrtlev I IO. ecls. llitrnletrika

t a h l e s f o r s t a t i s t i c i . t t t s\.' o l . 1 . 3 t 1c r l . , L o t t t l o t r :
I l i o m c t r i k . r T r r r s t , 1c ) 7 6 : 2 2 8 .
C o l t o n T . S t , r t i s t i c si n m e r i i c i r t c . I J o s t o n ;
l - i t t l e ,B r t r t v nn n d C o m p a n \ ' , 1 9 7 . 1 : 1 6 E .

11.

L2

H a n l e r . ' a n t lM c N e i l