Introduction to PARSCALE

American Board of Internal Medicine

Item Response Theory Course
 Overview
• Using PARSCALE for polytomous
analyses.
– Fitting the Graded Response Model
Model.
– Fitting the Generalized Partial Credit Model.
 PARSCALE
• The PARSCALE p program
g estimates the
parameters of several IRT models for
polytomous data:
– Graded Response Model
– Partial Credit Models

• Lik
Like BILOG
BILOG-MG,MG so PARSCALE was written itt
specifically for IRT analyses, so estimation is
very fast and efficient.

• Much of the code for PARSCALE is similar in

syntax to that of BILOG
BILOG-MG.
MG
Fitting the Graded Response
Model in PARSCALE
 PARSCALE Example
• To demonstrate how to use PARSCALE to
produce estimates of the GRM, we will use
data from:
– A 20-item test (simulated for demonstration
purposes).
purposes)
• Each item had four score categories.
– 1,000 examinees took the test.
 First Step: Data File Formats
• Prior to using
g PARSCALE we must set our data
into a useable form for PARSCALE.

• For our example data, we will use a file

containing the item response option the
examinee selected
selected.
– This file is named pexample1.dat

• Unlike BILOG-MG, an answer key is not needed.

– Just the score each examinee received for each item.
Data File Format
 No Other Additional Files
• PARSCALE can also accept data where some
responses have been omitted.

• The omitted key for each file can be placed into

a file jjust as in BILOG-MG.
– The specification of the file comes on the FILE
command line, using the OFNAME = syntax.

• For our example, we will not include such a file.

 Running PARSCALE
• PARSCALE is a script-p
based program that
features a graphical
system to help set up the
script files.

• To get started, open

PARSCALE.
– Start
Start…All
All
Programs…PARSCALE
4.1…PARSCALE 4.1
 Creating PARSCALE Script
• To begin, we will run the GRM on the
example data.

• Click on File…New.
– This will create a new *.psl
psl file containing the
script that will run the analysis.
– Be sure to pplace the script
p in a folder that has
fewer than 80 characters total (perhaps on the
root drive directly).
 PARSCALE Script
EXAMPL01.PSL - ARTIFICIAL EXAMPLE (MONTE CARLO DATA)
GRADED MODEL - NORMAL REPONSE FUNCTION: EAP SCALE SCORES
>COMMENTS

Example Graded Response Model Analysis

>FILE DFNAME='EXAMPL01.DAT', SAVE;

>SAVE PARM='EXAMPL01.PAR', SCORE='EXAMPL01.SCO';
>INPUT NIDW=14, NTOTAL=20, NTEST=1, LENGTH=(20), NFMT=1;
(14A1,20A1)
>TEST1 TNAME=SCALE1,, ITEM=(1(1)20),
( ( ) ), NBLOCK=20;;
>BLOCK1 BNAME=SBLOCK1, NITEMS=1, NCAT=4, CADJUST=0.0;
>BLOCK2 BNAME=SBLOCK2, NITEMS=1, NCAT=4, CADJUST=0.0;
>BLOCK3 BNAME=SBLOCK3, NITEMS=1, NCAT=4, CADJUST=0.0;
.
.
.
>CALIB GRADED, LOGISTIC, SCALE=1.7, NQPTS=30, CYCLES=(2500,2,2,2,2),
NEWTON=5, CRIT=0.005, ITEMFIT=10;
>SCORE EAP, NQPTS=30, SMEAN=0.0, SSD=1.0, NAME=EAP, PFQ=5;
 PARSCALE Script,
Script Annotated
• The initial p
portion of the script
p contains the name
of the analysis

EXAMPL01.PSL - ARTIFICIAL EXAMPLE (MONTE CARLO DATA)

GRADED MODEL - NORMAL REPONSE FUNCTION: EAP SCALE SCORES
>COMMENTS

E
Example
l GGraded
d d R
Response M
Model
d l A
Analysis
l i

• This is followed by a comment section where

comments about the analysis can be included
included.
– The comments section (and all other command
sections) must start with a >
 FILE Command Line
>FILE DFNAME='PEXAMPLE1.DAT', SAVE;

• The FILE command supplies the general information

used byy the PARSCALE p
program
g ((like the GLOBAL line
in BILOG-MG).

– DFNAME p provides the name of the file containing g the data

(which must be in the same folder as the script file).

– SAVE instructs BILOG-MG to look for a Save command later in

syntax (for saving item or person parameter estimates).
estimates)

– The GLOBAL command (and all others except for comment)

must conclude with a semicolon.
 SAVE Command Line
>SAVE PARM='EXAMPL01.PAR', SCORE='EXAMPL01.SCO';

• The SAVE command provides PARSCALE with

th names off the
the th files
fil th
thatt will
ill contain
t i
information from the analysis.
– The PARM option will save the item parameter
estimates in a file with the name given.
– The SCORE option
p will save the examinee p
parameter
estimates in a file with the name given.
 INPUT Command Line
>INPUT NIDW=14, NTOTAL=20, NTEST=1, LENGTH=(20), NFMT=1;

• The INPUT command line provides the

details of the input data set:
– NIDW gives the number of examinee ID
characters listed on the input
p file.
– NTOTAL gives the total number of items.
– NTEST gives the number of tests.
– NFMT gives the number of lines in the format
statements.
 Variable Format Statement
(14A1,20A1)

• The variable format statement is a required statement that lists the

way the data are stored in the data file.

• This statement uses FORTRAN-like syntax for reading data from

files.

• The first statement, 14A1, lets BILOG know there is a column of

data for the examinee id files.
– 14A1 gives the information that the column has a width of 14
characters.

• The 20A1 states the data are contained in the next 20 columns,
each with zero breaks in between.
 TEST Command Line
>TEST1 TNAME=SCALE1, ITEM=(1(1)20), NBLOCK=20;

• The TEST1 command line provides the number of items

to be analyzed along with the name for the test.
– The TNAME option allows the user to name the test (here
(here, we
call it ‘SCALE1’).

– The ITEM option specifies which items are part of the test.
• The user can choose not to analyze some items.
• The syntax shown instructs PARSCALE to start with the first item
and increment up until the 20th item.

– The NBLOCK option specifies the number of blocks of items that

share common categorical parameters.
• We will have one block per item (so each item has differing category
parameters estimated)
estimated).
 BLOCK Command Lines
>BLOCK1 BNAME=SBLOCK1, NITEMS=20, NCAT=4, CADJUST=0.0;...

• The BLOCK command line specifies the details

for each block of common items
items.
– The BNAME option provides the block name.
– The NITEMS option provides the number of items in
the block.
– The NCAT option provides the number of score
categories
t i for
f items
it in
i the
th block.
bl k
– The CADJUST option provides the location
adjustment for the mean of the block parameters.
 CALIB Command Line
>CALIB GRADED, LOGISTIC, SCALE=1.7, NQPTS=30, CYCLES=(2500,2,2,2,2),
NEWTON=5,
NEWTON 5, CRIT=0
CRIT 0.005,
005, ITEMFIT=10;
ITEMFIT 10;
• The CALIB command line provides the estimation
routine details for PARSCALE.
– GRADED
G iss the
e option
op o too spec
specify
y to
o es
estimate
a e the
eGGRM
parameters.
• PARTIAL is used for the GPCM.
– LOGISTIC is the option to specify the scale of the analysis
(compared to NORMAL)
NORMAL).
– SCALE is the scaling constant (here 1.7 is approximately
normal).
– NQPTS is the number of quadrature points.
– CYCLES is the maximum number of EM cycles to complete.
– NEWTON is the maximum number of Gauss-Newton iterations
that follow each EM cycle.
– CRIT is the convergence criterion for EM and GN iterations
iterations.
 SCORE Command Line
>SCORE EAP, NQPTS=30, SMEAN=0.0, SSD=1.0, NAME=EAP;

• The SCORE command provides the details of

the scoring procedure that will provide examinee
estimates.
– EAP is the method of estimation (Expected A
P t i i)
Posteriori).
– NQPTS is the number of quadrature points.
– SMEAN is the scale mean.
– SSD is the scale standard deviation.
– NAME is the name of the estimates.
 Running PARSCALE
• Now that the syntax has been created, the PARSCALE program
must be run.
run
– PARSCALE runs in a set of four phases (much like the three phases of
BILOG-MG).

• To run the analysis:

– Save the input script file first.
– Click on Run…Run All

• What should happen is a set of satellite windows will appear that will
run the PARSCALE analysis.
– After a few minutes, you will hopefully see a set of text files pop-up in
th PARSCALE window.
the i d
– If a pop-up box appears, there was an error somewhere in the analysis
(either in syntax or in estimation).
 PARSCALE Program Flow
• The PARSCALE program runs in four phases:

0. Phase 0 inputs the data and prepares the analysis.

1. Phase 1 inputs the data and estimates classical
item statistics.
2. Phase 2 estimates the item parameters of the
model.
3 Phase 3 estimates the examinee parameters
3. parameters.
 PARSCALE Output
• Each pphase of BILOG contains output
p specific
p to
the phase.
– The name of the output file will be the name of the
script
p file ((in our example,
p , “example1”).
p )
– The output files will have extensions indicating which
phase they come from:
• .ph0
p contains output
p from p phase 0.
• .ph1 contains output from phase 1.
• .ph2 contains output from phase 2.
• .ph3 contains output from phase 3.

• All files will be visible from within PARSCALE

upon
p successful convergence g of the algorithm.
g
 Phase 0 Output
• Phase 0 output contains information which
may not be very valuable to most users:

– Quadrature points

– Analysis specifics

– Input from the first two examinees

 Phase 1 Output
• Phase 1 output contains several useful
tables of information:
– Item summaryy statistics – the p
proportion
p of
examinees with responses for each category.

– Mean and standard deviation for each item.

– Pearson and polyserial correlations between

each item and the total scores.
 Phase 2 Output
• Phase 2 output
p contains details about the estimation
and, if the algorithm converged, the final estimates of the
item parameters.
– The item p
parameter estimates can be found in the middle of the
file.
– The category parameters can be found in the middle of the
output
– For each item, the first row of output are the estimated
parameters, the second row contains the estimated standard
errors.
• The SLOPE column contains the “a”
a parameter (the discrimination).
discrimination)
• The INTERCEPT column contains the “b” parameter (the base
difficulty).
• Also included are the item fit statistics for each item
item.
 PARSCALE’s
PARSCALE s Graded Response Model

• PARSCALE fits a slightly reparameterized

version of the GRM.
– The resulting GRM estimates can be obtained
by parsing the output.
• The PARSCALE version has:
– An overall intercept parameter regardless of
response level (category)
(category).
– An adjustment parameter that is subtracted
from the intercept for all but one category
category.
 Graded Response Model
• Samejima
Samejima’s
s Graded Response Model:

P (X ij = xij | θ i ) = Px*ij (θ i ) − Px*ij +1 (θ i )

• Where:
(
Da j θ i −bxij )
P (θ i ) = P(X ij ≥ xij | θ i ) =
e
( )
*
xij Da j θ i −bxij
1+ e
 PARSCALE’s Graded
Response Model
• Samejima
Samejima’s
s Graded Response Model:

P (X ij = xij | θ i ) = Px*ij (θ i ) − Px*ij +1 (θ i )

• Where:
(
Da j θ i − b j − c xij )
P (θ i ) = P (X ij ≥ xij | θ i ) =
e
( )
*
xij Da j θ i −b j − c xij
1+ e
 New Parameterization
• PARSCALE fits M-1M 1 category parameters
with the following constraints:
– c1 < c2 < ... < cM −1
M −1

– ∑ cm = 0
m =1

• The sum
sum-to-zero
to zero constraint can be modified in
options.
 Parameter Output Example
• First
First, PARSCALE presents the category
parameters:
ITEM BLOCK 1 SBLOCK1

CATEGORY PARAMETER : 1.022 0.010 -1.033

S.E. : 0.041 0.033 0.041
ITEM BLOCK 2 SBLOCK2

• Next, PARSCALE presents the a and b

parameters:
t
+------+-----+---------+---------+---------+---------+---------+---------+
| ITEM |BLOCK| SLOPE | S.E. |LOCATION | S.E. |GUESSING | S.E. |
+======+=====+=========+=========+=========+=========+=========+=========+
| 0001 | 1 | 1.505 | 0.064 | 0.008 | 0.042 | 0.000 | 0.000 |
+------+-----+---------+---------+---------+---------+---------+---------+
 Recovering GRM Parameters
• To move from PARSCALE
PARSCALE’s s
parameterization to that of the GRM,
subtract each category parameter (c) from
the item’s b parameter.
• Example item:
– Category 1: 0.008 - 1.022 = -1.014
– Category
C t 2
2: 0 008 – 0
0.008 0.010
010 = -0.002
0 002
– Category 3: 0.008 - -1.033 = 1.041
 Viewing Item Parameters
• PARSCALE makes viewing the item parameters
a bit easier with the inclusion of IRT Graphics, a
package for plotting the estimated IRFs.

• To view some of the item results:

– Close the output window (the lower “x” at the top
right).
– Go to Run…Plot.
– The IRT Graphics program should open.
 IRT Graphics
• To view the item parameter results, click
on the ICC button (at the top right).

• For each item, the ICC will be plotted,

along with the estimated item parameters.

• The error bars represent the number of

places where the fit of the model is
checked (to be discussed this afternoon).
IRT Graphics Example
Item Characteristic Curve: 0001
Graded Response Model (Normal Metric)
1.0 1

0.8

0.6
Probability

0.4

0.2

2
3 4
0
-3 -2 -1 0 1 2 3
Ability

Category legends Item: 1

Solid Lines: 1= Black 2= Blue 3= Magenta 4= Green
 Phase 3 Output
• Phase 3 output contains the scoring
information for the test:

• Examinee estimate and standard errors

are contained
t i d within
ithi thi
this fil
file.
 Examinee Estimates
SUBJECT IDENTIFICATION WEIGHT/FREQUENCY
SCORE NAME GROUP WEIGHT MEAN CATEGORY ATTEMPTS ABILITY S.E.

--------------------------------------------------------------------------------
0001 .44739 | 1 GROUP 01 1.00
1 EAP 1 | 1.00 3.00 1.00 0.6435 0.2193
--------------------------------------------------------------------------------

• The examinee estimates portion of

the output contain:
– Ability (latent trait estimate)
– Standard Error of the Ability Estimate
Fitting the Generalized Partial
Credit Model in PARSCALE
Fitting the GPCM in PARSCALE
• To fit the GPCM in PARSCALE, all the previous
code still applies with the exception of one word.
• On the CALIB line,, the word PARTIAL must
appear.
– In the previous example the word GRADED
appeared.
• Code for fitting an example GPCM model with
th same d
the datat sett can be
b ffound
d iin th
the E
Examples
l
folder.
 Conclusion
• This afternoon’s
afternoon s introduction to
PARSCALE scratched the surface of the
things the program can accomplish.
accomplish

• Th
The G
Graded
d dRResponse M
Model
d l can b
be fit
using PARSCALE.
– The Nominal Response Model can be fit using
MULTILOG.
 Next…
Next

• Evaluating Model Fit

–How
o well
e do the
e da
data
a fit the
e
model we’ve estimated?
–Are
A we meeting
i theh
assumptions
p of the IRT model?