http://informahealthcare.

com/dre
ISSN 0963-8288 print/ISSN 1464-5165 online

Disabil Rehabil, 2015; 37(3): 231–238

! 2015 Informa UK Ltd. DOI: 10.3109/09638288.2014.913702

ASSESSMENT PROCEDURE

The performance quality rating scale (PQRS): reliability, convergent

validity, and internal responsiveness for two scoring systems
Rose Martini1, Jorge Rios2, Helene Polatajko3, Timothy Wolf4, and Sara McEwen2,5
1
School of Rehabilitation Sciences, University of Ottawa, Ottawa, Ontario, Canada, 2Sunnybrook Research Institute, St. John’s Rehab Research
Program, Toronto, Ontario, Canada, 3Department of Occupational Science and Occupational Therapy, University of Toronto, Toronto, Ontario,
Canada, 4Program in Occupational Therapy, Washington University School of Medicine in St. Louis, MO, USA, and 5Department of Physical Therapy,
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15

University of Toronto, Toronto, Ontario, Canada

Abstract Keywords
Purpose: The performance quality rating scale (PQRS) is an observational measure of Development coordination disorder, outcome
performance quality of client-selected, personally meaningful activities. It has been used measures, performance quality rating scale,
inconsistently with different scoring systems, and there have been no formal publications on its PQRS, psychometric properties, stroke
psychometric properties. The purpose of this study was to test and compare the psychometric
properties of two PQRS scoring systems in two populations. Methods: A secondary analysis of History
video recorded participant-selected activities from previous studies involving either adults
living with stroke or children diagnosed with developmental coordination disorder (DCD) was Received 1 May 2013
For personal use only.

conducted. Three pairs of raters scored the video recorded performances with PQRS Revised 2 April 2014
operational definitions (PQRS-OD) and a generic rating system (PQRS-G). Results: For inter- Accepted 7 April 2014
rater reliability, PQRS-OD ICCs were substantial, ranging from 0.83 to 0.93; while the PQRS-G Published online 28 April 2014
ICCs were moderate, ranging from 0.71 to 0.77. Test–retest reliability was substantial, 40.80
(ICC), for both rating systems across all rater pairs. Internal responsiveness was high for both
rating systems. Convergent validity with the Canadian Occupational Performance Measure
(COPM) was inconsistent, with scores ranging from low to moderate. Conclusion: Both scoring
systems have demonstrated they are reliable and have good internal responsiveness. The PQRS-
OD demonstrated greater consistency across raters and is more sensitive to clinically important
change than the PQRS-G and should be used when greater accuracy is required. Further
exploration of validity with actual rather than perceived performance measures is required.

ä Implications for Rehabilitation

 The PQRS provides clinicians and researchers with a means of measuring actual performance
of self-selected activities.
 Rehabilitation scientists are encouraged to improve our understanding of the relationship
between perceived and actual performance as a means to improve interventions and
outcomes.

Introduction The performance quality rating scale (PQRS) is an observa-

tional, video-based tool intended to measure the actual perform-
Combining task-specific training with a client-centred approach,
ance of individual client-selected activities. We distinguish that
wherein the client self-selects the tasks or goals to be the focus of
the PQRS measures actual performance rather than capacity
an intervention, has demonstrated efficacy in several populations,
using definitions provided by Lemmens et al. [7]; they suggest
including people with stroke, brain injury, and developmental
that actual performance is the objectively detectable performance
coordination disorder (DCD) [1–4]. However, measuring changes
in the current environment, whereas capacity is the highest
in the ability to perform client-selected goals has proven
possible level of functioning in a standardized environment.
challenging, usually relying on client self-report, such as using
Martini and Polatajko first described PQRS in 1998, when the
the Canadian occupational performance measure (COPM) [5], or
tool was developed to evaluate children’s performance in self-
combining self-report with standards for the specific goal, such as
selected activities in a single case experimental series testing an
using goal attainment scaling (GAS) [6].
intervention for children with DCD [8]. It was described as a
6-point (0–5) scale used to examine both the activity steps and the
performance quality, with scoring based on objective, operatio-
Address for correspondence: Sara McEwen, PT, PhD, Sunnybrook
Research Institute, St. John’s Rehab Research Program, 285 Cummer nalized criteria developed for each activity. In 2001, Miller et al.
Avenue, Toronto, Ontario, Canada M2M 2G1. Tel: 416-226-6780, ext. modified the PQRS, making it a 10-point performance rating
7957. E-mail: sara.mcewen@utoronto.ca scale to better complement the 10-point client-rated COPM [4].
 232 R. Martini et al.
Subsequently, the PQRS has been used in other intervention
studies [2,3,9–11], however, it has been used inconsistently, with
different scoring systems, and a formal evaluation of its psycho-
metric properties has not been conducted. In fact, in Lemmens
and colleagues’ recent review of upper extremity assessments for
stroke and cerebral palsy, PQRS was excluded specifically
because of lack of published psychometric properties [7].
The PQRS has been used with either generic scoring, or with
specific operational definitions for different rating levels for each
participant-selected skill. With generic scoring, such as used by
Miller et al. [4], a rating of ‘‘1’’ indicates that the skill is not done
at all, and ‘‘10’’ indicates that the skill is performed very well.
Alternatively, Martini and Polatajko [8] and McEwen et al. [2,3]
developed operational definitions for PQRS ratings for specific
skills. For example, if the skill was tying shoelaces, an operational
definition for a rating of ‘‘2’’ may be ‘‘crosses laces but cannot
make a knot’’; descriptions of progressing skill level are given for
at least every other rating level [2,3,8].
The PQRS has generally been used in conjunction with the
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15
COPM. In the COPM, the therapist administers a semi-structured
interview to identify and prioritize individualized, meaningful,
real-life performance issues subsequently developed into the
goals/activities that are the focus of treatment. The COPM also
measures clients’ self-perceived changes in performance and
performance satisfaction of identified activities [5]. The PQRS
complements the COPM by assessing actual rather than perceived
performances of the activities that were identified. Each chosen
activity is executed without verbal or physical guidance in a non-
standardized environment, i.e. any appropriate space within a
treatment facility, community, or home. An independent observer,
For personal use only.
using a 10-point scale, then rates video recordings of the activity.
PQRS inter-rater reliability has previously been reported, but
within intervention studies, and with no reporting of the methods
used for evaluation [3,4]. No direct comparisons between generic
ratings and specific operational definitions have been made using
the same population with the same goals. Developing operational
definitions for PQRS ratings for individual, client-selected
activities for all participants in a research study or all clients in
a therapist’s practice takes time and is therefore costly, but may
deliver more measurement precision than using a 10-point scale.
Furthermore, using the PQRS as a tool for research in interven-
tions with participant self-selected activities necessitates the
availability of clearly standardized administration procedures and
good psychometric properties. Therefore, the objective of this
study was to estimate and compare the reliability, convergent
validity, and internal responsiveness of two PQRS rating systems
in two populations, adults with stroke and children with DCD.
Methods
To meet the objective above, three pairs of raters were asked to
watch video-recorded activity performances from the two patient
populations, and to score performances using both a generic rating
system (PQRS-G) and a system using specific operational
definitions (PQRS-OD). This was a secondary analysis of
video-recorded task performances from participants from previ-
ous intervention studies evaluating the efficacy of the Cognitive
Orientation to daily Occupational Performance treatment
approach with either adults living with stroke or children
diagnosed with DCD. The stroke participants, described in two
previously published studies [2,3], comprised six community-
dwelling adults between the ages of 38 and 56, four men and two
women, all more than 1-year post-stroke, and with pre-interven-
tion impairment levels ranging from 2/7 to 6/7 on the Chedoke
McMaster Stroke Assessment Impairment Inventory [12]. The
children were participants in a study on the use of CO-OP in a
Disabil Rehabil, 2015; 37(3): 231–238
group format described in [13] and in Canadian Association of
Occupational Therapists Conference presentations 2011 and 2012.
They were between 7 and 11 years, six boys and two girls, and all
performed below the 15th percentile on the Movement
Assessment Battery for Children – 2 [14].
The investigators selected videos for inclusion in this study
that had associated COPM scores and that depicted an activity
that was amenable to video analysis. For example, videos of a
stroke patient doing a yoga breathing activity that was perceived
as too subtle to be assessed with video, was not included.
Three pairs of blinded raters were used: two trained research
assistants without a health-care professional background (RA),
two registered occupational therapists (OT), and two occupational
therapy professional masters students (ST). None of the raters had
ever been involved in the treatment of the patients. Within each
pair, one rater was randomly chosen to be the A Rater, and the
other automatically became the B Rater.
Instruments
The COPM is an outcome measure that, through a semi-structured
interview, is designed to (a) identify and prioritize occupational
performance issues or goals, and (b) measure clients’ self-
perceived changes in occupational performance and satisfaction
with their performance over time, of identified goals [5]. The
COPM has demonstrated test–retest reliability of 0.89 in people
with stroke [15], and has been used with children to identify
goals [4,8,16]. The COPM has been shown to be sensitive to
changes over time in a client’s perceived occupational perform-
ance and satisfaction in client-identified activities [17]. A change
of two points or more on the COPM is considered clinically
significant [18].
The PQRS rates performance on a 10-point scale, with a score
of 1 indicating ‘‘can’t do the skill at all’’ and 10 indicating ‘‘does
the skill very well’’ [4]. An independent observer rates perform-
ances from video recorded trials of each skill at all assessment
points. In this study, we compared two versions of the PQRS
scoring – PQRS-OD, in which operational definitions were
developed for at least every other numerical rating for each goal
selected by participants and PQRS-G, in which a generic rating
system was used. Table 1 shows both systems. In this instance, all
operational definitions were developed or edited by either RM
(child definitions) or SM (adult definitions).
Procedure
Two rating periods were established, with a 2-week washout
period in between. During the first rating period, A Raters used
the PQRS-G system for the first half of the videos, and PQRS-OD
for the second half of the videos. During the second rating period,
2 weeks later, this was reversed, and A Raters used the PQRS-OD
system for the first half of the videos and the PQRS-G system for
the second half. B Raters did the opposite (Figure 1). At the end of
both rating periods, all raters had watched each video twice, had
used both rating systems, and had used each rating system first in
one rating period or the other. This last procedure was enacted in
case there was an effect of having used one rating system or
the other first.
One set of videos was used to explore inter-rater reliability,
convergent validity, and responsiveness, and a second set of
videos was used to explore test–retest reliability. For inter-rater
reliability, convergent validity, and responsiveness, 20 pairs of
videos, 40 videos total, were viewed. Ten pairs, 20 videos, were of
adults and 10 pairs, 20 videos, were of children. Each pair
consisted of a pre-intervention video recording of a self-selected
activity and a post-intervention video recording of the same self-
selected activity, performed by the same research participant.
 DOI: 10.3109/09638288.2014.913702 The performance quality rating scale (PQRS) 233
Table 1. Comparison of PQRS-OD and PQRS-G rating systems for tying a tie.

Example of PQRS-OD rating scale and definitions

Rating Operational definitions for tying a neck tie
1 Hands do not grasp tie and attempt to tie it.
2 Knot is partially completed.
4 Knot is completed, but loosely. Uses teeth for part of the process, and/or bottom tail of tie is longer than top tail, and/or tie bottom
is not sitting at belt. (3/3 or 2/3)
6 Knot is completed loosely. Uses teeth for part of the process OR top tail of tie is longer than bottom tail OR tie bottom is NOT sitting
at belt. (1/3)
8 Knot is completed loosely. Top tail of tie is longer than bottom tail, and tie sits roughly at belt.
10 Tie is tied quickly, with no errors or false starts. Knot is taut, top tail of tie is longer than bottom tail, and tie bottom sits roughly
at belt.

PQRS-G rating scale

Instructions: Watch video. Rate the performance on a scale of 1 to 10, with 1 being ‘‘cannot perform the task at all’’ and 10 ‘‘can perform the
task well’’. Use the table below as a guideline to assist you in rating the performance. Please note that only ONE score is assigned. The video may
be watched a second time, but not more than twice.
Quality takes into account, as applicable to the task, timeliness, accuracy, safety, and overall quality of performance or product.
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15

Rating is based on average of completeness and quality, and is rounded to the nearest whole number with X.5 being rounded up. For example, an
observed performance was scored as 75% task completion or 6 for Completeness and poor or 4 for Quality of Performance Product. The average
rating for that performance is 5 [(6 + 4)/2].
Scores of 3, 5, 7, and 9 are to be used at the rater’s discretion for observed performance that are marginally better or worse than something previously
observed, but don’t quite meet the criteria for the next defined category.

Rating Completeness Quality of performance or product

1 0% task completion n/a

2 25 % task completion Very poor
3
4 50% task completion Poor
For personal use only.

5
6 75% task completion Moderate
7
8 100 % of task completion Good
9
10 100 % of task completion Excellent

Figure 1. Visual representation of the study A Rater B Rater

design.
1. PQRS - G
• inter-rater reliability: 10 pairs
of videos (videos 1 - 20)
• test-retest reliability: 5
Time 1 individual videos (21 -25)
2. PQRS-OD
• inter-rater reliability: 10 pairs
of videos (videos 26 - 45)
• test-retest reliability: 5
individual videos (46 -50)
1. PQRS-OD
• inter-rater reliability: 10 pairs
of videos (videos 1 - 20)
• test-retest reliability: 5
individual videos (21 - 25)
2. PQRS-G
• inter-rater reliability: 10 pairs
of videos (videos 26 - 45)
• test-retest reliability: 5
individual videos (46 - 50)

2-Week Washout Period

A Rater
1. PQRS-OD
• inter-rater reliability: 10 pairs
of videos (videos 1 - 20)
• test-retest reliability: 5
Time 2 individual videos (46 -50)
2. PQRS-G
• inter-rater reliability: 10 pairs
of videos (videos 26 - 45)
• test-retest reliability: 5
individual videos (21 -25)
B Rater
1. PQRS-G
• inter-rater reliability: 10 pairs
of videos (videos 1 - 20)
• test-retest reliability: 5
individual videos (46 -50)
2. PQRS-OD
• inter-rater reliability: 10 pairs
of videos (videos 26 - 45)
• test-retest reliability: 5
individual videos (21 -25)
 234 R. Martini et al.
As part of previously conducted intervention studies, the research
participants had been interviewed by a therapist using the
standardized COPM interview to select personally meaningful
activities for assessment and intervention. The order of the pre
and post intervention videos was randomized within each pair,
and the videos within a pair were always viewed in sequence. In
other words, if the self-selected activity was skipping rope, both
the pre-intervention and post-intervention videos of skipping rope
would be viewed back-to-back, but the rater may or may not see
the pre-intervention video first. Each video recording had a
corresponding COPM score that had been previously assigned by
the research participant during the respective intervention study.
For test–retest reliability, an additional 10 individual videos of
self-selected activities were viewed, five of adults and five of
children. These videos were not paired, as pre- and post-
intervention pairs were not required for assessment of test–retest
reliability. At each rating period, A Raters scored five videos with
the PQRS-OD and the other five videos with the PQRS-G,
repeating the procedure at the second rating period. The B Raters
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15
used the same videos and procedures, but employed the alternate
PQRS used by A Raters.
The file names of the video recordings were designed to
indicate the order that the raters should view them in and the
video files’ metadata was altered to ensure the raters could not
determine the chronological order of the video pairs. Videos were
transferred to memory sticks encrypted using TrueCrypt1. Each
memory stick contained videos for a specific rating period and
rating scale. To accommodate the schedules of the raters across
two study sites, the memory sticks were given to the raters in
advance and when a rater was ready to begin a rating period, they
For personal use only.
were e-mailed the passwords to the appropriate memory sticks.
Raters were instructed to rate each video to the best of their ability
using the assigned system, but not to review each video more than
twice to decide on the score. This instruction was given to
minimize practice effect.
Ethics approval for the present study was obtained from the
Research Ethics Boards of the participating institutions.
Analysis
Data were cleaned and examined for discrepancies. A decision to
remove some videos was made based on feedback from raters. If a
video was identified by raters as problematic, it was removed only
if there were between-rater differences of 4 points or more across
more than one pair of raters and with both rating systems. This
resulted in five videos being removed, leaving 35 videos for inter-
rater reliability analysis. Since pairs of videos (pre-test and post-
test) were required for validity and responsiveness analysis, an
additional three videos were removed for those analyses, leaving
32 videos (16 pairs or tasks: 9 by children and 7 by adults). For
test–retest reliability one video was removed, leaving nine videos
of different skills, four of adults and five of children. The decision
to remove these videos from the analysis was made to reduce
measurement error based on the quality of the videos rather than
on issues with the scoring systems. Table 2 lists all skills that
were viewed.
Test–retest and inter-rater reliability were calculated using the
intra-class correlation coefficient (ICC). To determine inter-rater
reliability, two-way mixed model ICCs with absolute agreement
were calculated for pairs of raters (OTs, RAs and STs) for both the
PQRS-OD and the PQRS-G scoring systems. While the PQRS is
an ordinal measure Normal and Streiner (2008) have argued that
using a weighted kappa for ordinal scales is identical to a two-way
1
Copyright ß 2003-2013 TrueCrypt Developers Association; http://
www.truecrypt.org
Disabil Rehabil, 2015; 37(3): 231–238
Table 2. Skills viewed on videos.
Children Adults
Inter-rater reliability and responsiveness videos
Putting on and tying a Carrying objects with
hockey helmet affected arm
Tying shoe laces Getting in and out of a chair
(4 different children)
Jumping rope Buttoning a shirt
Stopping with rollerblades Using a computer mouse
Riding a bicycle Putting on a jacket
Swinging on a swing Sewing
Gardening
Climbing stairs
Cutting fruit
Riding a bicycle
Test–retest reliability videos
Swinging on a swing Clipping nails
Jumping rope alone Walking
Brushing teeth Tying a tie
Lacing skates Photography
Running
mixed, ICC, and the two may be substituted interchangeably [19].
The following benchmarks proposed by Shrout (1998) were used
for the interpretation of inter-rater reliability coefficients: 0.00 to
0.10 (virtually none); 0.11 to 0.40 (slight); 0.41 to 0.60 (fair); 0.61
to 0.80 (moderate); and 0.81 to 1.0 (substantial) [20].
Intra-class correlation coefficients were calculated to evaluate
convergent validity between PQRS and COPM performance and
satisfaction scores at pre-test and post-test. Convergent validity of
items is the degree to which theoretically similar concepts are
related [21].
Internal responsiveness is the ability of a measure to detect
change over a specified time frame [22]. The PQRS is an ordinal
scale, so a significant change finding was determined using the
non-parametric Wilcoxon signed rank test. It should be noted that
in their comparison of the Wilcoxon signed rank test with the t-
test, Meek, Ozgur and Dunning (2007) concluded that the t-test
may be preferred over the Wilcoxon signed rank test as the t-test
seemed to possess higher power than the Wilcoxon signed-rank
test when the sample sizes are small, even though its assumptions
were violated [23]. As such, a paired t-test between pre and post
scores was used to confirm significant change.
The effect size statistic provides direct information on
magnitude of change in a measure, as such it is widely
recommended for use as an indicator of responsiveness [22].
The effect size for an ordinal scale was calculated using ¼Z/ˇN.
Relative efficiency (RE) is another estimate of change in a
measure. It allows one to determine whether one measure is more
or less efficient for measuring change than another. It can be
computer for any pair of instruments. An RE51 means that the
instrument was a less efficient tool for measuring change; while
an RE41 means that the instrument was a more efficient tool for
measuring change.
Husted et al. (2000) recommended that this be complemented
by another measure. The standardized response mean (SRM) was
therefore used to confirm the paired t-test and RE index findings.
The SRM is the ratio of observed change and the standard
deviation reflecting the variability of the change scores [22].
Norman (2010) argues that parametric statistics are robust with
respect to violations of assumptions regarding sample size and
normal distribution and hence can be applied to ordinal ratings
[19]. As such the SRM, based on the paired t-test calculations was
applied to ordinal PQRS ratings. Calculated SRM effect sizes
were compared to Hopkins’ more conservative Likert-scale
approach to predetermined values of effect size representing
 DOI: 10.3109/09638288.2014.913702 The performance quality rating scale (PQRS) 235
responsiveness as: trivial (0–0.20), small (0.20–0.60), moderate Convergent validity
(0.60–1.20), large (1.20–2.0), very large (2.0–4.0) or ‘‘nearly
The correlations between both versions of the PQRS and the
perfect’’ (4.0 or greater).
COPM Performance Score are noted in Table 4. The negative ICC
The ability of scale to detect clinically relevant changes over
scores should be interpreted as indicating a low intra-class
time, or responsiveness, is influences by its test–retest reprodu-
correlation whereby the correlated pair of scores vary as much as
cibility [24]. It is important to calculate changes in the means
any two randomly selected pair of scores [27]. It seems that little
from the measurements obtained in test–retest in order to
convergent validity exists between the PQRS and the COPM’s
determine what change arises from typical variation. A change,
performance component. Examining the adult and child popula-
following an intervention, that is smaller than typical variation is
tions separately, correlations were similar at pre-test. At post-test,
usually not clinically important [25]. Minimal detectable change,
correlations varied from slight to moderate for children and from
or smallest real difference (SRD), is the smallest change in score
virtually none to slight for adults. Correlations of change scores
that can be interpreted as a real change in a client [24]. SRD can
varied from virtually none to fair for children and from slight to
be used as a threshold to determine whether changed scores for a
moderate for adults. Findings for both children and adults
particular client illustrate real improvement, rather than improve-
confirmed the general lack of convergent validity between the
ment due to measurement error or chance variation [26].
COPM performance scores and the PQRS scores.
A change, such as in task performance between pre-test and
The ICC’s for the COPM satisfaction component varied from
post-test, is based on the change score and its error size As such,
virtually none to slight for both versions of the PQRS for both pre-
SRD was calculated using the standard error of measurement
test and post-test correlations. Slightly higher correlations, slight
(SEM) with the following formula 1.96  ˇ(2  ‘‘SEM’’) [24].
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15

to fair, are noted for the correlations of change scores. A fair level
SEM was calculated by taking the square root of the mean square
error term from the analysis of variance [25].
Table 4. Intra-class correlation coefficients between PQRS scores and
Results COPM scores.
Reliability
COPM performance COPM satisfaction
Table 3 displays ICC correlations between each pair of raters, as
Rater PQRS-G PQRS-OD PQRS-G PQRS-OD
well as an overall correlation of all raters for both inter-rater and
test–retest reliability. Based on Shrout’s (1998) classification, all Pre-test
inter-rater correlations for the PQRS-OD displayed substantial Research assistants 0.08 0.37 0.12 0.23
inter-rater correlations with ICCs varying from 0.83 to 0.93; while
For personal use only.

OT students 0.10c 0.14 0.13 0.10

the PQRS-G displayed moderate inter-rater correlations with Occupational therapists 0.07 0.18 0.07 0.21
ICCs varying from 0.71 to 0.77. To determine whether inter-rater Post-test
reliabilities were influenced by the population performing the Research assistants 0.37b 0.53a 0.14b 0.24b
tasks, inter-rater correlations were also calculated per population OT students 0.44a 0.33b 0.16b 0.25b
(Table 3). For both children and adults, the ICC coefficients Occupational therapists 0.20b 0.49a 0.04c 0.29b
indicated substantial inter-rater correlations for the PQRS-OD, Change scores
whereas most correlations for the PQRS-G were in the moderate Research assistants 0.35 0.14 0.57 0.58
range. OT students 0.20 0.36 0.47 0.44
While test–retest reliability is slightly higher for the PQRS-G Occupational therapists 0.17 0.12 0.17 0.21
for the RAs and OTs, test–retest was higher for the PQRS-OD for a
the students. Regardless, test–retest reliability is substantial for Fair; bSlight; cVirtually none; negative ICC scores should be interpreted
as indicating a low intra-class correlation.
both the PQRS-G and the PQRS-OD for all categories of raters.

Table 3. Intra-class correlation coefficients between rater pairs and SRDs.

Children Adult Combined

Rater pairs PQRS-G PQRS-OD PQRS-G PQRS-OD PQRS-G PQRS-OD
Inter-rater reliabilities
Research assistants 0.87a 0.93a 0.66b 0.92a 0.77b 0.93a
OT students 0.75b 0.90a 0.80 a
0.96a 0.76b 0.92a
Occupational therapists 0.68b 0.84a 0.77b 0.98a 0.71b 0.88a
Total 0.76b 0.88a 0.74b 0.94a 0.74b 0.91a
Test–rest reliabilities
Research assistants 0.92a 0.94a 0.52c 0.39d 0.86a 0.81a
OT students 0.94a 0.99a 0.78b 0.99a 0.88a 0.99a
Occupational therapists 0.95a 1.0a 0.90a 0.68b 0.99a 0.88a
Total 0.93a 0.99a 0.77b 0.67b 0.88a 0.88a
SRD
Research assistants 2.91 0.90 2.96 5.10 6.23 8.17
OT students 2.63 0.90 3.66 0.89 2.83 1.61
e
Occupational therapists 2.13 2.23 2.91 3.35 1.36
Total 2.55 0.70 2.85 4.24 2.69 2.99
a
Substantial agreement; bModerate agreement; cFair agreement; dSlight agreement; eNot possible to calculate as
standard error of the difference was 0; SRD ¼ smallest real difference ¼ 1.96  ˇ(2  ‘‘SEM’’) .
 236 R. Martini et al. Disabil Rehabil, 2015; 37(3): 231–238

Table 5. Testing for pre-post intervention differences: paired t-tests and Wilcoxon signed ranks tests (z).

Time 1 Time 2
Median (range) IQR Mean SD Median (range) IQR Mean SD z t p
PQRS-OD: Pre-Post (n ¼ 33)
Research assistants 3 (5) 2 3.06 1.52 8 (6) 3 7.42 2.0 4.7 9.53 50.001
OT students 3 (5) 2 3.06 1.50 8 (8) 3.5 7.64 2.16 4.8 9.64 50.001
Occupational therapists 2 (5) 2 2.79 1.47 8 (6) 2.5 7.15 1.89 4.7 10.1 50.001
PQRS-G: Pre-Post (n ¼ 31)
Research assistants 3 (9) 3 3.72 2.36 8 (7) 2 7.47 1.98 4.1 6.58 50.001
OT students 4 (8) 4.8 4.03 2.62 8 (6) 3.5 8.06 1.66 4.6 7.50 50.001
Occupational therapists 2.5(6) 1.8 2.91 2.91 7 (8) 3 6.63 2.15 4.7 8.49 50.001

IQR, interquartile range; p, probability value for both z and t values.

of convergent validity is noted between the change scores rated by Table 6. Relative efficiencies, parametric effect sizes (SRM), and non-
the RAs and STs and the participant’s satisfaction change scores. parametric effect sizes (responsiveness).
At post-test, correlations for children were virtually none for both
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15

versions of the PQRS, while for adults correlations varied from SRM Responsiveness
virtually none to fair with the PQRS-G and slight to fair for the Rater pairs RE PQRS-G PQRS-OD PQRS-G PQRS-OD
PQRS-OD. For change scores, correlations varied from virtually
Research assistants 0.69 1.11 1.59 0.73 0.82
none to moderate for children and virtually none to fair for adults.
OT students 0.79 1.30 1.62 0.83 0.82
While adults demonstrated a greater correlation between satis- Occupational 0.83 1.44 1.70 0.81 0.84
faction scores and PQRS scores than the children, there does not therapists
appear to be one version of the PQRS where great convergent All raters 0.76 1.28 1.68 0.79 0.82
validity was observed consistently. t
RE, relative efficiency ¼ tPQRSOD
PQRSG
; SRM, standardized response mean ¼
x
D
; responsiveness ¼ pZffiffiffi.
Internal responsiveness SDðD Þ
x N
For personal use only.

All paired variances for all paired raters for both versions of the
PQRS were found to be not significant, as verified using the
Pitman-Morgan test, a test of variance for paired samples [28,29].
The results of the paired t-tests, confirmed by the Wilcoxon
signed ranks test, are presented in Table 5. All paired t-tests, and
Wilcoxon signed rank tests, for both PQRS-G and PQRS-OD
were significant, indicating that the task performance change from
pre-test to post-test was significant. Effect size (Z/ˇN) and
relative efficiencies are presented in Table 6. Large effect sizes
were obtained for both the PQRS-OD and the PQRS-G (R40.72).
With respect to relative efficiencies, all scores are less than 1,
indicating that the PQRS-OD is the more efficient tool for
measuring change than the PQRS-G. This is confirmed by the
SRM in Table 6. For both the PQRS-G and the PQRS-OD the
various effect sizes are large (greater than 1) which suggests that
the observed changes in task performance are clinically mean-
ingful. Middel and van Sodersen (2002) report that ‘‘to give
clinically relevant meaning to change scores gained on two
different points in time’’ (p. 13) individual client’s perceptions
should be considered more explicitly [30]. In keeping with this
suggestion, the RE and SRM were again calculated after having
removed the only item that did not have a clinically meaningful
change of two points on the COPM (data not shown), and the
PRSQ-OD was still found to be the more responsive tool.
SRD for different raters and PQRS versions are presented in
Table 3. When a change score (pre-post) is greater than the SRD,
a true change can be ascertained with a 95% confidence interval
[31]. Generally, smaller SRD values are obtained using the PQRS-
OD than the PQRS-G. For children, the SRDs for the PQRS-OD
varied from 0.69 to 0.89. This indicates that with a PQRS-OD a
change in score of 1 can indicate a true change in children. With
the PQRS-G, SRDs with children varied from 2.13 to 2.91,
indicating that a change in score of 3 is needed to be 95% sure that
the change is real and not due to a measurement error. With
adults, it seems that a greater change score is required to ensure
that it is a true change. For the PQRS-OD, adult SRDs varied
from 0.89 for OT students to 5.10 for research assistants; while for
the PQRS-G, SRDs varied from 2.23 for occupational therapists
to 3.66 for OT students. In general, for both the PQRS-OD and the
PQRS-G, greater SRDs are noted for research assistants than for
occupational therapy students and OTs.
Discussion
This is the first study to formally estimate the psychometric
properties of the PQRS, an observational system for rating
performance quality of client-selected activities. Two different
PQRS scoring systems were compared using a variety of client-
selected activities performed by both adults and children.
These results (conducted with two distinct populations and a
wide variety of self-selected activities derived from COPM
interviews) indicate that using PQRS-OD provides better inter-
rater reliability than the PQRS-G, overall, as well as per
population. Reliability data indicated that both versions of the
PQRS are reliable, a requisite characteristic for a measure to be
responsive. Both the PQRS-G and the PQRS-OD demonstrated the
ability to detect change in general. When comparing the two
versions, the RE coefficients point to the PQRS-OD as being more
sensitive to general and clinically important changes over time than
the PQRS-G. The PQRS-OD generally obtained smaller SRDs,
confirming that smaller changes detected with the PQRS-OD may
be interpreted as a real change with 95% confidence, thus
corroborating that it is a more sensitive instrument than the
PQRS-G.
These data suggest that PQRS-OD is a precise tool that permits
cross-comparison of a wide range of client-chosen goals and
activities. On the other hand, the PQRS-G provides moderate
inter-rater reliability, substantial test–retest reliability, and large
effect sizes demonstrating good internal responsiveness, indicat-
ing that it is an adequate measure for use in clinical settings, or in
research projects when taking the time and resources required to
write operational definitions is not feasible.
 DOI: 10.3109/09638288.2014.913702 The performance quality rating scale (PQRS) 237
The PQRS was designed to loosely mirror the COPM, to  If the goal/activity cannot easily be observed on video, PQRS
provide an additional measure of self-selected activity perform- ratings can be done with a physical product, such as
ance using ratings of actual observed performances rather than handwriting samples, or by superimposing another objective
self-perceived performance. The COPM is used to elicit the rating system onto PQRS scores, such as the time it takes for
activities that will subsequently be performed, videotaped, and task completion.
rated. We hypothesized that PQRS and COPM scores would be
highly correlated because the PQRS activities were derived from Study limitations
the COPM interview and were thus matched. This did not prove to
As we conducted a secondary analysis of existing video data, the
be true. While the COPM has demonstrated low convergent
video clips were not always ideal, as described above. While it is
validity with other measures in the past, this was presumed to be
likely that the reliability scores would have been higher with
because the COPM items are defined by the client whereas the
perfect videos, and lower if the lowest quality videos had not been
items in other measures are predetermined and can differ greatly
removed from the analysis, we applied strict criteria for removing
from what the COPM is measuring [32]. That issue does not help
certain videos designed to strike a balance between ideal videos
to explain low convergent validity between the COPM and PQRS
and videos that mirror the realities of clinical research. While we
in this study, as the PQRS items came directly from the COPM.
did ensure a wide range of activities, some activities such as bike
The low convergent validity in this current study is better
riding and shoelace tying were self-selected by more than one
explained by the idea that actual performance and perceived
participant, increasing raters’ familiarity with a particular task.
performance may be different constructs. Chan and Lee (1997)
Also, raters were required to view the same video on two
found virtually zero correlation between perceived performance
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15

occasions, thus increasing familiarity with that particular video,

(COPM) and actual performance measures, providing support for
which may have influenced the rating on the second occasion.
this idea [33]. Thus, to explore the convergent validity of the
Nevertheless, it is expected that the 2-week washout period was
PQRS, other measures of actual performance should be used. For
sufficient to guard against this potential practice effect. Another
example, the Functional Arm Activity Behavioral Observation
limitation was that only 10 videos were used for the test–retest
System (FAABOS) makes use of random samples of video
reliability. While this is a small number, correlations were
recordings from stroke and brain injury survivors’ homes to
substantial. As such, it is unlikely that more samples would result
evaluate actual upper extremity performance, and could perhaps
in a substantive change. As all videos came from a small sample
be used as a starting point to assess PQRS validity [34].
of six adults and eight children, the results may not be
Previous use of the PQRS has identified certain usability
generalizable to a broader population. Additional testing is
issues, generally related to whether or not the activity of interest
recommended.
can be seen adequately on video [3]. This issue resurfaced in this
For personal use only.

The PQRS-OD requires that someone, usually the clinical or

study, for example in the ‘‘brushing teeth’’ goal, some raters
reported that video did not show the inside of the mouth, making
performance quality difficult to rate. Some additional issues were
also described by raters. Lack of clarity about the specifics of
some goals made using PQRS-G ratings difficult. For example,
with the ‘‘sewing’’ goal, it was not clear if that meant to sew a
single stitch or to sew two pieces of fabric together, making it
difficult for the raters to judge performance completeness.
Another issue reported by some raters, with both rating systems,
arose when the video showed more than one attempt of the same
task, particularly if the attempts had markedly different perform-
ance quality. For instance, in the ‘‘skipping rope’’ goal, the video
showed the child missing the first attempt at jumping, but then
shows the child readjusting herself and completing 10 consecutive
jumps. It was not clear to raters how to manage the observation of
different trials within one video clip. A third issue was raised
using the PQRS-OD. At times, raters described occurrences when
the video did not match the criteria in the operational definition.
For example, for bike riding, one video clip began when the
participant was already on the bike, but the PQRS-OD criteria
specified the amount of assistance required to mount the bike.
To mitigate these problems the following recommendations
regarding PQRS administration are made:
 When using PQRS-G, the goal/activity should be well-
specified and should have a degree of measurability to it. For
example, recording the goal as ‘‘skipping’’ is too vague,
whereas recording ‘‘skipping a minimum of 10 jumps’’ may
provide the rater with sufficient additional information to
improve rating accuracy.
 For both rating systems, care should be taken while video
recording and making video clips that show: only one
performance attempt, the entire performance of the task, and
close-ups of important task details where possible. In a
clinical setting, performances can be rated during a live
observation of the activity rather than using a video
recording.
research therapist, take the time to write clear operational
definitions in line with the client’s self-selected COPM goal. It
is possible that definitions written by different individuals would
have resulted in different psychometric properties. However, in
this study the definitions were prepared by two different research
therapists (R.M. and S.M.) with different styles, suggesting that
reliability remains high even with varying definition development
styles.
Future validation studies should use other actual performance
measures, and should be expanded to include other properties
such as predictive validity. In addition, the relationship between
the PQRS and an individual’s overall level of functioning should
be examined.
Conclusions
Overall, PQRS showed substantial reliability and internal respon-
siveness in two different populations and across raters with very
different backgrounds. PQRS-OD had better results than the
PQRS-G for inter-rater reliability, internal responsiveness, and
SRD, and is therefore recommended when greater precision is
desired. PQRS-G demonstrated acceptable inter-rater reliability,
substantial test–retest reliability, and good internal responsive-
ness, and therefore is acceptable to use when preparing oper-
ational definitions is not feasible. Further validation of the PQRS
against actual performance and overall functional measures is
required.
Acknowledgements
The authors would like to thank Dr. Carolyn Baum for her
contributions to the development of the PQRS-G scoring system
and for her helpful comments on the manuscript, and Dianne
Cirone, Tammy Craig, Alisa Grigorovich, Mindy Simon, Elise
Pomerleau, and Christina Lamontagne for conducting the
ratings.
 238 R. Martini et al.
Declaration of interest
This study was partially funded by the Canadian Institutes of
Health Research (FRN #111200). The authors report no conflict
of interest.
References
1. Dawson DR, Gaya A, Hunt A, et al. Using the cognitive orientation
to daily occupational performance approach with adults with
traumatic brain injury. Can J Occup Ther 2009;76:115–27.
2. McEwen SE, Polatajko HJ, Huijbregts MPJ, Ryan JD. Inter-task
transfer of meaningful, functional skills following a cognitive-based
treatment: results of three multiple baseline design experiments in
adults with chronic stroke. Neuropsychol Rehabil 2010;20:541–61.
3. McEwen SE, Polatajko HJ, Huijbregts MP, Ryan JD. Exploring a
cognitive-based treatment approach to improve motor-based skill
performance in chronic stroke: results of three single case experi-
ments. Brain Inj 2009;23:1041–53.
4. Miller L, Polatajko HJ, Missiuna C, et al. A pilot of a cognitive
treatment for children with developmental coordination disorder.
Human Mov Sci 2001;20:183–210.
Disabil Rehabil Downloaded from informahealthcare.com by Nyu Medical Center on 04/21/15
5. Law M, Baptiste S, McColl MA, et al. The Canadian occupational
performance measure: an outcome measure for occupational therapy.
Can J Occup Ther 1990;57:82–7.
6. Forbes DA. Goal attainment scaling. A responsive measure of client
outcomes. J Gerontol Nurs 1998;24:34–40.
7. Lemmens RJ, Timmermans AA, Janssen-Potten YJ, et al. Valid and
reliable instruments for arm-hand assessment at ICF activity level in
persons with hemiplegia: a systematic review. BMC Neurol 2012;
12:21.
8. Martini R, Polatajko H. Verbal self-guidance as a treatment
approach for children with developmental coordination disorder: a
systematic replication study. Occupat Therap J Res 1998;18:157–81.
For personal use only.
9. Phelan S, Steinke L, Mandich A. Exploring a cognitive intervention
for children with pervasive developmental disorder. Can J Occup
Ther 2009;76:23–8.
10. Polatajko HJ, McEwen SE, Ryan JD, Baum CM. Pilot randomized
controlled trial investigating cognitive strategy use to improve goal
performance after stroke. Am J Occup Ther 2012;66:104–9.
11. Rodger S, Ireland S, Vun M. Can cognitive orientation to daily
occupational performance (CO-OP) help children with Asperger’s
syndrome to master social and organisational goals? Br J Occup
Ther 2008;71:23–32.
12. Gowland C, VanHullenaar S, Torresin W, et al. Chedoke-McMaster
stroke assessment. Development, validation, and administration
manual. Hamilton, ON: Chedoke-McMaster Hospitals and
McMaster University; 1995.
13. Martini R, Mandich A, Green D. (2014) Implementing a modified
cognitive orientation to daily occupational performance approach for
use in a group format. Br J Occup Ther 2014;77:214–19.
14. Henderson SE, Sugden DA, Barnett AL. Movement assessment
battery for children – second edition (movement ABC-2). London,
UK: The Psychological Corporation; 2007.
15. Phipps S, Richardson P. Occupational therapy outcomes for clients
with traumatic brain injury and stroke using the Canadian occupa-
tional performance measure. Am J Occup Ther 2007;61:328–34.
Disabil Rehabil, 2015; 37(3): 231–238
16. Law M, Darrah J, Pollock N, et al. Focus on function – a randomized
controlled trial comparing two rehabilitation interventions for young
children with cerebral palsy. BMC Pediatr 2007;7:31.
17. Eyssen IC, Beelen A, Dedding C, et al. The reproducibility of the
Canadian Occupational Performance Measure. Clin Rehabil 2005;
19:888–94.
18. Law M, Babtiste S, Carswell-Opzoomer A, et al. Canadian
Occupational Performance Measure, 3th ed. Ottawa, ON: CAOT
Publications ACE; 1998.
19. Norman GR, Streiner DL. Biostatistics: the bare essentials, 3rd ed.
Toronto, Canada: BC Decker; 2007.
20. Shrout PE. Measurement reliability and agreement in psychiatry.
Stat Methods Med Res 1998;7:301–17.
21. Bragante KC, Nascimento DM, Motta NW. Evaluation of acute
radiation effects on mandibular movements of patients with head
and neck cancer. Rev Bras Fisioter 2012;16:141–7.
22. Husted JA, Cook RJ, Farewell VT, Gladman DD. Methods for
assessing responsiveness: a critical review and recommendations.
J Clin Epidemiol 2000;53:459–68.
23. Meek GE, Ozgur C, Dunning K. Comparison of the t vs. Wilcoxon
signed-rank test for Likert scale data and small samples. J Modern
Appl Statist Meth [Internet]. Revised 2007;6:91–106.
24. Beckerman H, Roebroeck ME, Lankhorst GJ, et al. Smallest real
difference, a link between reproducibility and responsiveness. Qual
Life Res 2001;10:571–8.
25. Lexell JE, Downham DY. How to assess the reliability of
measurements in rehabilitation. Am J Phys Med Rehabil 2005;84:
719–23.
26. Lu WS, Wang CH, Lin JH, et al. The minimal detectable change of
the simplified stroke rehabilitation assessment of movement meas-
ure. J Rehabil Med 2008;40:615–19.
27. Kenny DA, Kashy DA, Cook WL. Dyadic data analysis. New York:
Guilford Publications; 2006.
28. Jones G, Noble ADL, Schauer B, Cogger N. Measuring the
attenuation in a subject-specific random effect with paired data.
J Data Sci 2009;7:179–88.
29. Pitman-Morgan Test: test the difference between correlated
variances. Available from: http://how2stats.blogspot.ca/2011/06/
testing-difference-between-correlated.html [last accessed 14 Oct
2013].
30. Middel B, van Sonderen E. Statistical significant change versus
relevant or important change in (quasi) experimental design: some
conceptual and methodological problems in estimating magnitude of
intervention-related change in health services research. Int J Integr
Care 2002;2:e15 (2–18).
31. Wang CY, Sheu CF, Protas EJ. Test-retest reliability and measure-
ment errors of six mobility tests in the community-dwelling elderly.
Asian J Gerontol Geriatr 2009;4:8–13.
32. Cup EH, Scholte op Reimer WJ, Thijssen MC, van Kuyk-Minis MA.
Reliability and validity of the Canadian occupational performance
measure in stroke patients. Clin Rehabil 2003;17:402–9.
33. Chan CCH, Lee TMC. Validity of the Canadian occupational
performance measure. Occupat Therap Int 1997;4:231–49.
34. Uswatte G, Hobbs Qadri L. A behavioral observation system for
quantifying arm activity in daily life after stroke. Rehabil Psychol
2009;54:398–403.