Critical Appraisal of Articles About Diagnostic Tests

Rani Gereige, M.D., MPH

Associate Professor University of South Florida

Learning Objectives

Test validity What is it? and how to ensure it? Putting diagnostic tests to the test: Critical appraisal of studies about a diagnostic test Be familiar with the concepts of

Treatment and Test thresholds Pre-test and post-test probabilities and odds Likelihood Ratios (LRs) The 2x2 table Calculations and interpretation of Sensitivity, Specificity, PPV, NPV, pre and post test probabilities and odds, LR(+) and LR(-)

Evaluate a VALID evidence using

Apply the evidence

Questions

A patient presents to us with a chief complaint

Why

do we order tests? What tests to order? Based on what? What do we hope to achieve as we get the result of the test? What if there are multiple tests that are related to this complaint? What if we are considering 6 or 7 possible diagnoses that might explain this chief complaint?

Hurricane Fever

What ifthere is a new test that claims to accurately pinpoint hurricanes location of the attack??
What

would you want to know about this test? Anything you want to know about the hurricane itself? What about if the hurricane is right at your doorstep?

What if this new test can only be done at an elevation of 15000 ft above sea level and you live in Florida?

Diagnostic Test
Validity Definition

The validity of a diagnostic test is the extent to which the results represent an unbiased estimate of the underlying truth (it is testing what it supposed to test)

Validity versus Relevance

Validity

Relevance

Does it matter to my Is it true? patients/population? Was the study appropriately Is the test feasible for my patients? randomized, blinded? Did they use appropriate gold Will it affect something that MATTERS to patients? (Morbidity, standard? mortality, quality of life) Was the population similar Will it change my practice? enough to your own? (generalizable) Was follow-up adequate? Most clinicians look at

Recommended first by the text

relevance first Easy and quick to find (abstract) If does not matter do not bother with validity testing

How to Use an Article about a Diagnostic Test??

Three main questions: Is this evidence about the accuracy of a diagnostic test valid? Does this (valid) evidence demonstrate an important ability of this test to accurately distinguish patients who do or do not have a specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test to a specific patient?
If answer is NO Stop

I- Is this evidence about the accuracy of a diagnostic test VALID?

Measurement Representative Ascertainment Reproducibility

I- Is this evidence about the accuracy of a diagnostic test VALID?

Measurement: Was there an independent, blind comparison with a reference Gold Standard?

Patients should have undergone both the test and the reference (gold standard) Confirmation by surgical findings, biopsy, autopsy, long term follow-up Investigator should not know the result of gold standard

Limitations:

Gold standard is not always possible (psych disorders) Bias cannot always be avoided Some gold standards are for research only

I- Is this evidence about the accuracy of a diagnostic test VALID?

Representative: Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in who we would use it in practice)?
Was the spectrum of disease appropriate? Florid cases vs asymptomatic volunteers Disease severity: Mild, severe, early or late cases, both treated and untreated Apply the test to patients with different disorder commonly confused with the target disorder Was the spectrum of patients appropriate? Similar ages, sexes, co-morbidities Enough patient numbers?

I- Is this evidence about the accuracy of a diagnostic test VALID?

Ascertainment: Was the reference standard ascertained regardless of the diagnostic test result?
Important

to confirm or refute the test on ALL patients, not just those with abnormal test May use different method of gold standard to avoid harming patients (Surgery vs close follow-up)

I- Is this evidence about the accuracy of a diagnostic test VALID?

Reproducibility: Was the test (or cluster of tests) validated in a second, independent group of patients?
Diagnostic

tests are predictors, not explainers of the diagnosis Same results of the test in a separate independent group will reassure us on the accuracy Not always done initially

Eligible Patients n= Excluded Patients Reasons n = Index test n=

Positive n=

Negative n=

Inconclusive

Missing
n= Reference Standard

Missing
n= Reference Standard

Missing
n= Reference Standard

How to Use an Article about a Diagnostic Test??

Three main questions: Is this evidence about the accuracy of a diagnostic test valid? Does this (valid) evidence demonstrate an important ability of this test to accurately distinguish patients who do or do not have a specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test to a specific patient?
If answer is NO Stop

Probability versus Odds

Odds of disease does not equal probability of disease Example: 10 patients 3 have anemia and 7 do not

Probability

of having anemia is 3/10 = 30% = Prevalence = Pretest probability Odds of having anemia are 3:7 or 0.43

Probabilities and Odds

Can be calculated back and forth

Odds of a:b will give probability of a/(a+b) Probability of X% will give Odds of X/(100-X)

You start with

Pre-test probability (Prevalence)
Pre-test Odds

Post-test Odds
Post-test probability

Pre-test Probability and Odds

Pre-test probability = Prevalence = The proportion of people with the target disorder in the population at risk at a specific time Pre-test Odds = The odds that the patient has the target disorder before the test is carried = Pre-test Probability/[1- Pre-test Probability]

Post-test Probability and Odds

Post-test probability = The proportion of patients with that particular test result who have the target disorder = Post-test odds /[1+ Post-test Odds] Post-test Odds = The odds that the patient has the target disorder after the test is carried out

Sources of Pre-test Probability

Clinical experience
Biased

by frequency of diagnosis, last patient, fear of missing a diagnosis Easily and quickly accessed Use with caution Not for young careers

Regional/ national prevalence statistics on disorder frequency (as good as accuracy of diagnosis) Local/ regional/ national practice databases Calculated from the study being appraised Track research report of a study devoted to pretest probability documentation (Best - if well done)

Inputs Into Pre-Test Probability

The frequency of the disease in populations or groups of individuals similar to a particular patient Risk factor exposure of the individual Pattern of symptoms presented by the patient Use of a prior screening test

Test and Treatment Thresholds

Test Threshold probabilities below which a clinician would dismiss a diagnosis and order no further tests Treatment Threshold probabilities above which a clinician would consider the diagnosis confirmed, and would stop testing

Test accuracy
Ability of a test to change our minds from what we thought before the test (pre-test probability) to what we think afterwards (post-test probability) Diagnostic tests that produce big changes from pre-test to post-test probability, are important and likely to useful in practice

Clinical Implications
When the probability of the target disorder falls between the test and treatment thresholds, further testing is mandated Once test and treatment thresholds are determined, the post-test probabilities have direct treatment implications

Evaluating the Evidence Diagnostic Test

Construct the 2x2 table

Sensitivity

= a /a+c = P(+/D)= P(TP among diseased)=PID Specificity = d /b+d = P(-/D)= P(TN among non diseased) PPV = a /a+b = P(D/+)= P(TP among all Positives) NPV= d /c+d = P(D/-)= P(TN among all Negatives) SnNout = Result of a test with high Sensitivity, when Negative, rules out the diagnosis SpPin = Result of a test with high Specificity, when Positive, rules in the diagnosis

More Definitions
Sensitivity = PID = Positive in Disease Specificity = NIH = Negative in Health Discriminant ability of a test = (sensitivity+specificity)/2

Likelihood Ratios

LR(+) = Probability of person WITH disease having positive test/probability of person WITHOUT disease having a positive test

LR(+) = Sens/(1-spec) Corresponds to clinically ruling in disease

LR(-) = Probability of person WITH disease having negative test/probability of person WITHOUT disease having negative test

LR(-) = (1-sens)/spec Corresponds to clinically ruling out disease

Depending on article or source, a single LR might be presented without reference to positive or negative

Likelihood Ratios

Indicate by how much a given diagnostic test result will raise or lower the pretest probability of the target disorder
LR

= 1 means the post-test probability is exactly the same as the pre-test probability LR > 1 increases the probability that the target disorder is present LR < 1 decreases the probability that the target disorder is present

Guide to the Significance of LRs

LR > 10 or < 0.1 generate large and often conclusive changes from pre-test to post-test probability LR = 5 - 10 or 0.1 - 0.2 generate moderate shifts pre-test to posttest LR = 2 5 or 0.5 0.2 generate small, but sometimes important changes in probability LR = 1 2 or 0.5 1 are rarely important shifts

How to Use LRs?? Bayes Theorem

Used to calculate Post-test probability

LR

Pre-test probability (Prevalence) - X

Nomogram (Textbook card 3A) Table LR calculator PDA Med Calc

Pre-test Odds: Y = X/(100-X)

Post-test Odds: Z = Y x LR

Post-test probability = Z/(Z+1)

LR calculator:

http://araw.mede.uic.edu/cgi-alansz/testcalc.pl

LR Nomogram & Table

Sensitivity = 731/809 = 90%

The Ferritin Systematic Review

Disease Disease Totals Present Absent Ferritin < 65 mmol/L Ferritin > 65 mmol/L Totals 731 270 (a) (b) 78 1500 (c) (d) 1770 (b+d) 1001 (a+b) 1578 (c+d) 2579
(a+b+c+d)

Specificity = 1500/1770 = 85%

PPV = 731/1001 = 73% NPV = 1500/1578 = 95% LR(+) = Sens/(1-Spec) = 90%/15% = 6

LR(-) = (1-Sens)/Spec = 10%/85% = 0.12

Pre-test Probability = Prevalence = (a+c)/(a+b+c+d) = 809/2579 = 31%

809 (a+c)

What do you think of these LRs??

Calculating Post-Test Probability

(-)
(+)

Pre-test Odds = Prevalence/(1-Prevalence) = 31%/69% = 0.45 Post-test odds=pre-test odds x LR

For (+) = 0.45x6=2.7 For (-) = 0.45x0.12=0.054

Post-test probability= Posttest odds/(post-test odds+1)

For (+) = 2.7/3.7=72.99% For (-) = 0.054/1.054=5.12%

What Conclusions Can You Draw About Ferritin?

High sensitivity and specificity LR(+) of 6 means that using Ferritin level allows to increase the probability of iron deficiency anemia from 31% (or 50%) to 73% (or 86%) for levels < 65 mmol/L LR(-) of 0.12 means that using Ferritin level allows to decrease the probability of iron deficiency anemia from 31% (or 50%) to 5% (or 11%) for levels > 65 mmol/L This is known as sensitivity analysis

Multilevel Likelihood Ratios

Provide superior information to sensitivity and Specificity Allow detailed and specific clinical conclusions

809

1770

How to Use an Article about a Diagnostic Test??

Three main questions: Is this evidence about the accuracy of a diagnostic test valid? Does this (valid) evidence demonstrate an important ability of this test to accurately distinguish patients who do or do not have a specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test to a specific patient?
If answer is NO Stop

Applying the Valid Evidence to YOUR Patient

Three Questions Q1. Is the diagnostic test available, affordable, accurate and precise in our setting?
Requires

knowledge of your clinical

capacity
Do you have the test? Personnel? Cost? Insurance coverage? What is your patient population? And status of disease presentation?

Applying the Valid Evidence to YOUR Patient

Three Questions Q2. Can we generate a clinically sensible estimate of our patients pre-test probability?
From

personal experience (H&P, memory), prevalence statistics (texts, government), practice databases (Hospitals, med records), primary studies (validity testing) Are the study patients similar to our own? Is it unlikely that the disease possibilities or probabilities have changed since this evidence was gathered? (e.g. study from 1996 on AIDS diagnosis)

Applying the Valid Evidence to YOUR Patient

Three Questions Q3. Will the resulting post-test probabilities affect our management and help our patient?
Could it move us across a test-treatment threshold? Depends on the disease and the relative risks of treatment vs no treatment May move us from one test to another Post-test odds after one test become the pre-test odds for the next Would

our patient be a willing partner in carrying it out? (This is critical) Would the consequences of the test help our patient reach his or her goals in all this?

Look at things CRITICALLY