See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/263466770

Inter-rater reliability and clinical utility of the

Prevocational Screening Test

Article November 2011

CITATIONS READS

0 41

5 authors, including:

Edward James R Gorgon

University of the Philippines Manila
19 PUBLICATIONS 41 CITATIONS

SEE PROFILE

All content following this page was uploaded by Edward James R Gorgon on 13 May 2016.

The user has requested enhancement of the downloaded file. All in-text references underlined in blue are added to the original document
and are linked to publications on ResearchGate, letting you access and read them immediately.
Inter-rater Reliability and
Clinical Utility of the
Prevocational Screening
Test
Irene Carolina A. Sarmiento
Edward James R. Gorgon
Carina Marie L. Castillo
Deanne Mea R. Esguerra
John Stanley D. Catacutan
Keywords: video-based assessment adolescents
developmental disabilities test development
transition planning
Abstract
OBJECTIVES. This study examined the inter-rater reliability and
clinical utility of the Prevocational Screening Test (PST).
METHOD. Twenty-eight occupational therapists, physical
therapists, speech-language pathologists, and fieldwork students rated
two videos of clients engaged in sandwich making and packaging
using the PST. They also rated the clinical utility of the PST. The
ratings for both videos were analyzed separately using average
deviations.
RESULTS. Overall, the PST had acceptable inter-rater reliability
(r=75% to 78%) and clinical utility as assessed by occupational
therapists, physical therapists, speech-language pathologists, and
fieldwork students. Several items, particularly those pertaining to
work behaviors, demonstrated poor reliability.
CONCLUSION. The PST has the potential for use by
occupational therapists, physical therapists, speech-language
pathologists, and fieldwork students. To improve its inter-rater
reliability, clarifying the definition for some terms, modifying the
format, and revising items with low reliability, are needed. Other
psychometric properties, such as retest reliability, need to be
investigated.
Introduction
Developmental disabilities (DD) are chronic disabilities
that may persist indefinitely, are manifested before the age
of 22 years, and can be attributed to impairment in mental
and/or physical functions (US Department of Health and
Human Services, 2000). DD result from conditions like
mental retardation, cerebral palsy, genetic and
chromosomal abnormalities, autism, learning impairments
or disorders, serious emotional disturbances, and traumatic
brain injury (Liptak, 1995 as cited in Balloueff, 1998).
Functional limitations in self-care, receptive and expressive
language, learning, mobility, self-direction, capacity for
independent living, and economic independence are
common and often indicate a need for carefully-planned
and highly-individualized multidisciplinary programs.
DD are known to impact negatively on the
acquisition of prevocational skills that are a foundation for
occupations. Normally, prevocational skills emerge at
around three years of age (Lynch, 1982 as cited in
Hemphill, 1997) and develop throughout childhood,
adolescence, and adulthood (Hemphill, 1997). Adolescents
with DD, however, characteristically lag in the acquisition
of such skills. The repercussions of this are far-reaching
since adolescence is the time when people begin
considering work possibilities for independent living
(Llorens, 1991). Expectedly, adolescents with DD
experience greater difficulty in transitioning into work
settings than their typically developing peers because of
their inadequate prerequisite skills for occupations such as
work (Partridge, 1997). Cabatan (2004) reported that
Filipino adolescents with DD are unable to find
employment due to the functional implications of their
disability, such as deficits in cognitive and social skills.
Familial and social attitudes towards the disability,
including those of the employer, further limit their
opportunities for work participation.
Occupational therapy can aid persons with DD
acquire and develop the skills they need for successful
transitioning. To do this, occupational therapists rely on
prevocational and vocational screening and assessment
tools that can provide information on what skill areas
require attention. These tools typically fall under one of
the following categories: work samples, psychometric tests,
situational assessment and job site evaluation (Jacob, 1991;
Hemphill, 1997).
Ocheda and Cabatan (2004) found that most of
the prevocational assessment tools used in institutions in
Manila included psychometric tests (i.e., inventories and
checklists like Interest checklist, Role checklist,
Neuropsychiatric Institute Checklist, Adolescent and Adult
Psychiatric Profile, Reading-free Vocational and Interest
Inventory); and work samples, like the Bay Area
Functional Performance Evaluation, the Jacobs
Prevocational Skills Assessment, and VALPAR Work
Sample. The most commonly used were those that were
psychometric, in which results are compared to
established norms. Several authors, however, have argued
against the use of highly standardized activities, tools,
materials, and instructions on adolescents with DD since
they do not respond well to the nature and characteristics
of standardized tests (Hursh & Kerns, 1988; Polgar, 2003;
Shroeder & Dalrymple, 1993). Further, most, if not all,
standardized tests require a person to have certain degree
of literacy, cognitive skills, self-control, motivation, and
physical capacity for results to be valid (Clark & Kolstoe,
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
5
1995; Peterson, 1997 as cited in Hemphill), which not all
persons with DD possess. There appears to be a need for a
functional instrument in assessing adolescents with DD, as
few prevocational screening tools are functional or
situational in nature.
A situational assessment has distinct advantages
because it evaluates culture-expected skills and involves
observation of actual performance in natural or altered
environments (Ocheda, & Cabatan, 2004). Of the tools
frequently utilized by Filipino occupational therapists, the
Jacobs Prevocational Skills Assessment was the only one
that made use of situational assessment. However, this
instrument was developed abroad and, as such, might have
culture-based biases when applied in the Filipino context.
The Prevocational Screening Test (PST) was
developed specifically in response to the need for a
functional, relevant, and accessible screening tool in the
local setting (Ocheda & Cabatan, 2004). Preliminary
development of the tool was conducted in two phases:
Planning and Construction. In the Planning Phase, a semi-
structured interview was conducted with employers to
acquire information about the prevocational and
vocational assessment instruments they used. Twelve
employers of Metro Manila-based industries responded to
a survey on what employability skills they deemed as
important for hiring employees. Most of the employers
were unfamiliar with the type of work-related instruments
their companies used when hiring employees. Findings
also showed that there was a need to clarify the definition
of prevocation to Metro Manila employers. Results from
the planning phase revealed that work behaviors, such as
punctuality and attention, were viewed as the most crucial
for employment.
The design of the PST took place during the
Construction Phase. Content of the PST was based on a
literature review and survey on the classification of
prevocational skills. The participants in this phase were the
same occupational therapists who took part in the earlier
survey of prevocational and vocational instruments. They
classified items of the PST according to the following
categories: Physical Capacities; Work Behaviors; and Cognitive
Skills. The answers provided by the participants for this
phase contributed to the establishment of the validity of
the items in the PST.
Prevocational skills, as described by the PST
manual, are antecedents in the development of work skills,
behaviors and attitudes that are developed throughout the
lifespan. They support the participation of adolescents in life
activities. The Model of Human Occupation (MOHO)
framework of occupational therapy defines participation as
engagement in school, play and daily living activities that
are part of ones social and cultural context and necessary
for well being (Kielhofner, 2008).
The performance of prevocational activities (and,
likewise, occupational performance in any area,) is the
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
6
result of a dynamic interaction among individuals volition
(personal causation, values and interests) habituation
(habits and roles) and performance capacities (skills), as
well as the environment (Kielhofner, 2008). Prevocational
skills develop based on earlier behaviors and habits, which
progress or are later refined through observation of role
models in environments where productivity is rewarded
(Ocheda and Cabatan, 2004).
Activities identified in the PST that can be used
for screening include daily living tasks, such as, sandwich
assembly, sorting clothing, making a bed, sweeping or
mopping the floor, and emptying the garbage; and task-
oriented activities, such as, packaging, labeling, collating
colored cards, sealing, stacking and assembly, using a
puncher, stapler, and rubber stamp. The tasks were
selected based on functionality and applicability to the
target population that comprises children and adolescents
with DD. Other considerations included: high relation to
work, high interest level for children and adolescents,
practicality of tasks, limited task duration, and adaptability
of the task including instructions. It was also important
that the tasks used inexpensive and locally available tools
and materials to make them readily accessible to
practitioners.
Skills (or areas) assessed include work behaviors,
physical capacities, and cognitive skills. Work behaviors
include attention span, respect for authority, and initiative.
Physical capacities involve motor coordination, the ability
to move quickly, strength, and mobility. Cognitive skills
include the ability to follow instructions, reading, and
math, among others. The results of the test, according to
the authors, would be appropriate for identifying
individuals who might benefit from prevocational and/or
vocational training. To date, however, the PSTs
measurement properties and clinical utility have not been
established. Further, while the intended users include a
range of professionals from different disciplines and the
tool does not require specialized training, there is no
evidence to indicate that ratings made by such users would
be similar regardless of their professional background or
level of experience.
Predictive tools, such as the PST, should show
evidence of acceptable measurement properties, like
reliability and validity, and practicality of application, as
their results are used in making decisions (Law, 1987;
Payton, 1979 as cited in Minichiello, Sullivan, Greenwood,
& Axford 1999; Polgar, 2003). Sarmiento et al. (2007)
explored the PSTs measurement properties by
investigating its inter-rater reliability with a small sample of
participants. Reliability was determined for scores acquired
by two occupational therapists and two fieldwork students
who directly observed a group of adolescents with DD.
Reliability estimates were moderate (Spearmans rho =
.542) between the two occupational therapists and poor
(Spearmans rho= .068) between the fieldwork students.
Acceptable reliability was estimated for the ratings of each
of the occupational therapists and those of the first
fieldwork student (Spearmans rho = .616 and .554). The
second fieldwork students ratings disagreed strongly with
those of the occupational therapists (Spearmans rho =
.093 and .058). Looking closely into this disagreement, the
authors concluded that measurement was threatened by
two potential sources of error. First, the fact that the
second fieldwork student facilitated the group activity and
then rated the clients on the PST may have exposed the
ratings to recall bias. Second, the participants
accomplished the PST using real-time observation that
could have been open to random errors and allowed little
opportunity for verification of item-level ratings. The
results of the study indicated further exploration of the
measurement properties of the PST. In particular,
alternative set-ups to real-time group testing was
suggested.
Aside from establishing its measurement
properties, it would be important to determine if the PST
is cost efficient and easy to use. Clinical utility refers to
whether or not an instrument, particularly the details and
information it provides, is useful in the management and
service delivery for a particular setting. In assessing clinical
utility, the following criteria are considered: availability and
ease of use; administration time; learnability and
clinicians qualifications; format; scoring and information
derived; and meaningful and relevant information obtained
(Polgar & Barlow, 2004).
This study aimed to determine the inter-rater
reliability and clinical utility of the PST in the evaluation of
a video-taped client engaged in a supervised activity.
Specifically, the study aimed to answer the following
questions:
1. What is the inter-rater reliability of the PST when used
by occupational therapists, physical therapists, and
speech-language pathologists?
2. What is the inter-rater reliability of the PST when used
by therapists and fieldwork students?
3. Is the PST clinically useful as administered by
therapists and fieldwork students?
Method
Participants
Two groups of participants (n = 28) who had no prior
experience in using the PST were selected through
convenience sampling. One group was composed of
licensed occupational therapists and physical therapists,
and certified speech-language pathologists. To be included,
the therapists must have had at least two months clinical
experience providing services to adolescents with DD.
Seventeen therapists were recruited: 5 occupational
therapists, 8 physical therapists, and 4 speech-language
pathologists. The other group was composed of fieldwork
students (5 from occupational therapy, 3 from physical
therapy, and 3 from speech-language pathology) who had
completed at least one month of supervised clinical
practice at the Clinic for Therapy Services Adult and
Adolescent Section (CTS-AA). The CTS-AA is an
outpatient clinic that provides interdisciplinary training to
final-year undergraduate physical therapy, occupational
therapy, and speech-language pathology students at the
University of the Philippines Manila. The majority of the
clientele that seek occupational therapy services at the
clinic are adolescents with DD. The study included
therapists and fieldwork students because the PST was
originally developed for use at the CTS-AA.
Data collection instruments
The PST includes a form for documenting the clients
demographic data, a selection of activities, and a checklist
of behaviors. The checklist contains a total of 91 items and
is divided into three categories: Work Behaviors (include
attendance, punctuality, concentration, attention span,
respect for the rights of others, respect for authority, and
others); Physical Capacities (such as work tolerance, fine
motor coordination, and physical coordination); and
Cognitive Skills (such as the ability to recognize and correct
errors, the ability to follow instructions, and decision-
making). Each prevocational skill had to be rated using
symbols (+) for present; (+/-) for emerging; (-) for
absent; or (N/A) meaning not applicable.
With help from one of the authors of the PST
(Cabatan), the investigators determined which of the items
were not applicable to a one-on-one session and not
observable in video-based evaluation. The PST author
watched a video of a client and rated the performance on
the PST. Items were marked as N/A (not applicable) on
the scoring sheet accordingly, before copies were
distributed to the participants. The participants referred to
Cabatans ratings to confirm their own ratings for the same
client when they practiced with the video-based PST
application.
Clinical utility was assessed using a two-part
clinical utility evaluation form designed by the
investigators. It adhered to the criteria suggested by Law
(1987) and Polgar and Barlow (2004). On the basis of their
experiences and perceptions in using the PST, the
participants rated clinical utility first on a Likert scale of 1
(strong disagreement) to 5 (strong agreement), and then
used narrative statements. The clinical utility criteria are
listed in Table 1. Space was provided per criterion for
additional remarks from the participants. On the second
part, the participants were encouraged to provide short
comments and recommendations on how to improve the
tool.
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
7
Table 1
Criteria used to rate the clinical utility of the PST
Criterion
Simplicity and conciseness of format
Completeness and clarity of instructions for administration
Completeness and clarity of instructions to clients
Practicality of time requirement for test administration
Conductibility over more than one session
Practicality of time requirement for learning administration
Availability/accessibility of manual
Ease of understanding and user-friendliness of manual
Accessibility and cost of required materials and tools
Practicality of time requirement to complete screening tool
Having no special training requirements
Ease of scoring and recording results
Interpretability by health practitioners
Having no special knowledge and background
requirements to interpret results
Usefulness of data within clinical setting
Each participant received a packet containing one standard
DVD-R (Kodak DVD-R) that contained the videos of
three adolescents with DD, or clients, as they made and
packaged a sandwich under the supervision of the principal
investigator. Making and packaging a sandwich is an
instrumental activity of daily living (IADL) (AOTA, 2008).
As an IADL, meal preparation requires safety precaution
and awareness, work behaviors, physical and cognitive
skills (Culler in Crepeau, Cohn, & Boyt Schell, 2003; Foti,
2010). The DVD file was divided into three video chapters
for Client P, Client A, and Client B. Each video was
approximately 20 minutes long. The video of Client P was
intended for practice, while the videos of Clients A and B
were to be rated by the participants and included in the
data analysis.
Procedure
Prior to data collection, the protocol for the study was
evaluated and approved by the Ethical Review Committee
(ERC) of the University of the Philippines Manila
College of Allied Medical Professions. The investigators
oriented the participants to the studys aim and protocol,
particularly the administration and scoring of the PST.
Also included in the orientation were: a clarification of
terms used in the PST, discussion of queries about aspects
of the PST, and reiteration of ethical measures to protect
the privacy and rights of the clients in the videos.
The participants signed an agreement form stating
that they would: 1) view each video only once before
rating; 2) neither copy nor reproduce the videos; 3) not
allow other people to view any of the videos; 4) not
communicate the results of their evaluation with anyone
other than the investigators, and; 5) be subjected to legal
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
8
measures should they violate any of the provisions. The
participants were given a packet which included: three
copies of the blank PST checklist for each of the three
videotaped performances they would evaluate; one copy of
prevocational activity proposal in which the selected
activities, sandwich making and packaging, were broken
down into a series of steps; one clinical utility form; and
the coded DVD containing three videos of clients (Clients
P, A, and B).
The participants watched the practice video
(Client P) and then rated it using the PST. Practice allowed
the participants to familiarize themselves with the tool.
After rating the client in the practice video on their own,
the participants compared their ratings with the PST
authors ratings. The comparison provided the participants
with feedback on whether their evaluation of client
performance using the PST paralleled that of the standard.
After practicing on the video of Client P, the
participants then watched and rated the 2 remaining videos
(Clients A and B). After rating Clients A and B, the
participants accomplished the clinical utility form. The
investigators personally collected all the PST packets with
the completed forms, approximately one to two weeks
after the packets were distributed.
Data analysis
To analyze the data for inter-rater reliability, the symbols
used in rating the prevocational skills (i.e., +, -, +/-) were
converted to numerical values (i.e., + = 3, +/- = 2, and -
= 1). The investigators also itemized the PST items, from
1 to 91, according to the corresponding behavior being
evaluated. To determine inter-rater reliability, the average
deviation (AD) for each item was computed comparing
the ratings made by the following: occupational therapists,
physical therapists, and speech-language pathologists;
fieldwork students from occupational therapy, physical
therapy, and speech-language; and therapists and fieldwork
students. Inter-rater reliability of the PST was computed
per item and per subscale (Work Behaviors, Physical
Capacities, and Cognitive Skills). Average deviation (Kelley,
1947) is a measure of dispersion or variation and refers to
the average of the deviations of each participants score,
on a single item, from the median of the score of all the
participants. When an item has no variation (i.e., AD = 0),
then it has perfect reliability (Amstadter, 1971). For this
study, only those items having 0 to 0.5 AD were
considered reliable. The investigators arbitrarily set the
limit to 0.5 AD because it is at this level that less than half
the participants of a sample could have ratings that were
not far from the rating given by the majority (e.g. if two of
five participants gave a rating of 2 and the rest gave 3,
or vice versa, the average deviation would be 0.48).
To determine the inter-rater reliability of each
subscale of the PST, items belonging to specific subscales,
Work Behaviors, Physical Capacities, and Cognitive Skills, were
grouped. The total percentages of reliable items per
subscale were compared among all therapists, all fieldwork
students, and, finally, all participants. The greater the
variety or heterogeneity of the scores, the greater the
deviations would be detected (Healey, 2001).
For the clinical utility of the PST, the ratings of
the therapist group, student group, and both groups were
tabulated separately. The investigators computed the mean
and standard deviation of the participants rating for each
of the item.
Results
Inter-rater Reliability
Average deviations of the participants ratings
provided estimates of the inter-rater reliability of each item
in the PST. Reliability levels ranging from reliable to
unreliable were obtained after the average deviations were
ranked in increasing order. Tables 2 and 3 present a
summary of the number of items per subscale and the
corresponding percentages of items determined to be
reliable for Clients A and B, respectively.
With Clients A and B, most of the items were
reliable. With Client A, 75% to 90% of the items were
reliable as rated by the occupational therapists, physical
therapists, speech-language pathologists, and fieldwork
students. Similarly, for Client B, 60 to 90% of the items
were reliable as rated by the participants from all the
disciplines. Only a few items fell under the range
interpreted as unreliable, 3 to 9 items (7 % to 22%) for
Client A and 3 to 16 items (7 % to 40%) for Client B.
Certain items were reliable for all groups of
participants and for both clients. Of the items, Remains
at activity site and until given permission to move (28)
and Accepts supervision (29), both under the Work
Behaviors subscale, consistently yielded average deviations
of 0, meaning these were always reliable. Conversely,
Remains at work station (22) and Introduces self (41)
for both Client A and B, and Recognizes work defects
(60) for Client B were consistently unreliable.
Variation in the percentages of reliable items for
both clients was greater for occupational therapists
compared to physical therapists and speech-language
pathologists. The occupational therapy fieldwork students,
on the other hand, had greater inter-rater reliability within
their group, compared to the licensed occupational
therapists.
With Clients A and B, Physical Capacities
obtained the highest agreement for all groups of
participants. With Client A, Work Behaviors obtained the
lowest agreement. On the other hand, Cognitive Skills had
the lowest agreement for Client B, as rated by therapists
and fieldwork students. For fieldwork students, Work
Behaviors had the lowest agreement. The results of
therapists ratings varied the most for Work Behaviors and
Cognitive Skills.
Clinical Utility
Table 4 presents a summary of the means and
standard deviations for the specific questions in the PST
clinical utility form.
The participants generally gave a rating of Agree or
Strongly agree to the clinical utility items pertaining to the
PST. The standard deviation of scores was low, indicating
minimal variation in the ratings given by the participants.
Only the criteria conductibility in one session, no
special training required, and no need for knowledge
background obtained lower means and high standard
deviations, indicating high variability among the ratings of
the participants.

Table 2
Percentages of Reliable Items for Client A
Percentage (%) of Reliable Items
Participants Rating Client A Total Work Physical Cognitive
Items (N=40) Behaviors Capacities Skills (n=12)
(n=24) (n=4)
Occupational Therapists 83 79 100 100
Physical Therapists 80 83 100 67
Speech-Language Pathologists 80 79 100 75
All Therapists 75 71 100 75
Occupational Therapy Fieldwork Students 90 83 100 100
Physical Therapy Fieldwork Students 78 67 100 92
Speech-Language Fieldwork Students 93 88 100 92
All Fieldwork Students 75 63 100 92
All Therapists and Students 75 71 100 75
Note. N = total number of items rated in the PST. n = total number of the items in each subscale.

Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2

9
Table 3
Percentages of Reliable Items for Client B
Percentage (%) of Reliable Items
Total Items Work Physical Cognitive
Participants Rating Client B
(N=40) Behaviors Capacities Skills
(n=24) (n=4) (n=12)
Occupational Therapists 60 58 100 50
Physical Therapists 90 96 100 75
Speech-Language Pathologists 75 79 100 58
All Therapists 73 79 100 50
Occupational Therapy Fieldwork Students 93 92 100 92
Physical Therapy Fieldwork Students 83 79 100 83
Speech-Language Fieldwork Students 93 88 100 100
All Fieldwork Students 80 83 100 67
All Therapists and Students 78 83 100 58
Note. N = total number of items rated in the PST. n = total number of the items in each subscale.

Table 4
Average Ratings for the Clinical Utility of the PST
Therapists and
Fieldwork
Therapists Fieldwork
Criteria Students
Students
Mean (SD) Mean (SD) Mean (SD)
Simplicity and conciseness of format 4.35 (0.49) 4.18 (0.40) 4.29 (0.46)
Completeness and clarity of instructions 4.25 (0.58) 4.36 (0.67) 4.30 (0.61)
Completeness and clarity of instructions to clients 4.06 (0.56) 4.36 (0.67) 4.18 (0.61)
Practicality of time requirement for test
4.50 (0.63) 4.27 (0.79) 4.41 (0.69)
administration
Conductibility over more than one session 2.18 (1.07) 2.55 (1.37) 2.32 (1.19)
Practicality of time requirement for learning
4.56 (0.51) 4.18 (0.98) 4.41 (0.75)
administration
Availability / accessibility of manual 4.18 (0.73) 4.50 (0.53) 4.30 (0.67)
Ease of understanding and user-friendliness of
4.38 (0.62) 4.27 (0.47) 4.33 (0.55)
manual
Accessibility and cost of required materials and tools 4.47 (0.51) 4.45 (0.69) 4.46 (0.58)
Practicality of time requirement to complete
4.38 (0.50) 4.18 (0.98) 4.30 (0.72)
screening tool
Having no special training requirements 3.59 (1.00) 2.64 (1.21) 3.21 (1.17)
Ease of scoring and recording results 4.25 (0.68) 4.27 (0.47) 4.26 (0.59)
Interpretability by health practitioners 4.00 (0.61) 4.20 (0.79) 4.07 (0.68)
Having no special knowledge and background
2.00 (0.89) 1.55 (0.69) 1.81 (0.83)
requirements to interpret results
Usefulness of data within the clinical setting 4.82 (0.39) 4.73 (0.47) 4.79 (0.42)
Note. SD = standard deviation. Rating scale used: 5 = Strongly agree, 4 = Agree, 3 = Undecided, 2 = Disagree, 1 = Strongly disagree.

Discussion Items classified as reliable had consistent average

The study aimed to determine the inter-rater reliability and
clinical utility of the Prevocational Screening Test as
administered by occupational therapists, physical
therapists, speech-language pathologists, and fieldwork
students, using videos of persons with DD engaged in
making and packaging a sandwich.
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
10
deviations of 0, and were associated with being easily
observable. Thus, there were no variations in the
participants ratings. For items consistently categorized as
unreliable among all the participants, such results could
have been due to possible errors from the participants.
According to Minichiello et al. (1999), an error could come
from the participant doing the measurement or the
instrument itself. In this case, the items, remains at work
station (22) and recognizes work defects (60), were Skills. However, comparing the three disciplines ratings
usually missed by the participants, resulting in lost data.
Items with lost data, when statistically analyzed, have
relatively high average deviations, rendering them
unreliable. Similarly, missed items meant that fewer ratings
could be analyzed for the study. Thus, the reliability of the
abovementioned items could not be accurately determined.
This can be considered as an error on the part of the
participants. It was assumed that these items were skipped
in part because of the PSTs format, which does not have
uniform spaces for ratings.
The item introduces self (41) was labeled as
N/A by some of the participants and left out by the
others. In the PST authors ratings, this item was not
predetermined as N/A because it was observed in the
practice session. However, this item was not observed in
the other videos (Client A and Client B) as there was no
opportunity for the client to exhibit the skill. This resulted
in variations in how the participants scored the item.
The reason for the low level of agreement
obtained for Work Behaviors for both clients was reflected
in the qualitative data obtained from the clinical utility
form. Differences in results across disciplines showed that
the participants had varying familiarity with the terms.
Also, qualitative data from the clinical utility form revealed
that the participants felt that the definitions of terms
pertaining to Work Behaviors were not sufficient for them
to understand such abstract concepts. Another factor that
could have contributed to the differences in ratings was
clinical experience. Level of expertise and clinical
experience affects expectations regarding the skills of
clients (Rogers & Holm, 1997). These suggest the need to
provide clear definitions for the terms in the PST and
greater information on the level of performance expected
for each of the scores on the scale.
The occupational therapists had lower inter-rater
reliability for Client B compared to the physical therapists
and speech-language pathologists. Dreyfus and Dreyfuss
(1986) concept of professional expertise has been applied
to occupational therapy. An underlying assumption of this
process is that both experience and reflection about that
experience is necessary for expertise to develop. The
occupational therapists may have had more variation in
their results because they were of different levels of
experience and expertise compared with the physical
therapists and speech-language pathologists (Boyt Schell,
Crepeau & Cohn, 2003). Moreover, occupational
therapists, physical therapists, and speech-language
pathologists undergo training in different ways and make
use of unique perspectives and frames of reference. The
differences are not surprising as the PST uses an
occupational therapy framework. Thus, the occupational
therapists may have been more affected by variations in
their levels of expertise, as they would have had clearer
expectations regarding Work Behaviors and Cognitive
for Client A, the differences in variation of inter-rater
reliability were not significant. Most of the percentages of
reliability of items were still noted to be acceptable for
both clients.
The occupational therapists also had more
variable results compared with the fieldwork students. This
may be because the fieldwork students had more or less
the same level of clinical experience. According to Dreyfus
and Dreyfus (1986), the development of and advancement
of clinical reasoning skills requires experience. To some
degree, these are issues of attention and memory. Because
therapy is a complex process, newcomers to the field have
difficulty concentrating on everything at once. It is not
until some skills become more automatic that practitioners
can concentrate on other parts of the process (Boyt Schell
et al., 2003).
Even with these differences, in general, the PST
had more items that were reliable than those that were not.
This suggests that the PST is promising as a reliable
screening tool when used by occupational therapists,
physical therapists, speech-language pathologists, and
fieldwork students.
The generally high perceptions of clinical utility
can be associated with both the inherent characteristics of
the tool and the measures taken by Ocheda and Cabatan,
the authors of the PST, to facilitate its ease of use. The
PST requires no other materials and tools for its
administration other than the checklist and everyday items
for functional activities, such as materials for sandwich
making and packaging. However, the participants varied
ratings on the tools conductibility in one session may be
due to their uncertainty that this can be done without
compromising its reliability. In their qualitative feedback,
some participants stated that the PST has too many items
that make administration time too long. Potential strategies
to address this issue would be to administer the tool over
two sessions or decrease the number of items on the list.
Applying any of these strategies would require further
testing to determine if and how such strategies would
affect the PSTs reliability.
In the qualitative section of the clinical utility
form, the participants stated that they had difficulty rating
items related to attention span due to difficulty
differentiating items from each other. These difficulties,
however, did not cause variations in the ratings. The items,
attending behavior (8), high rates of attending (9) and
high rates of attention on low and high interest tasks
(10) consistently had high inter-rater reliability among all
participants.
The criteria, need for special training (11), and
need for knowledge and background (14), also obtained
varied results. This was reflected by the participants
comments about lack of background and unfamiliarity
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
11
 with the some parts of the PST, including scoring,
behaviors to be observed, and how they are measured, and
the occupational therapy terms used. In general, the
participants expressed that they did not have a hard time
completing the PST. Although some items were vague,
overall, the participants stated that it is a promising
screening tool.
Discipline-specific differences in the interpretation
of the items, on the other hand, have not been established
as a factor in the overall reliability of the tool. Although
unfamiliarity with certain terms, varying interpretation of
terms, and differences in experience in evaluating skills
were reported in the clinical utility forms, particularly by
the physical therapists and speech-language pathologists, it
was observed that the items in the PST had a generally
acceptable level of inter-rater reliability. The source of
disagreements in ratings stemmed from individual errors,
regardless of the discipline.
Limitations of the study and recommendations
Convenience sampling has its own limitations, as the
participants in the study may not be the representative of
the population who will actually use the PST. Furthermore,
there may have been bias in the selection of participants.
The potential impact would be that high reliability would
be expected for a homogenous sample (Minichiello et al,
1999).
Another limitation of the study was that there
were items that may have been inadvertently missed by
some of the participants because they were located
between items that were previously labeled as N/A. This
resulted in lost data that contributed to higher average
deviation and lower reliability for certain items. This is
consistent with a participants comment that the
highlighted areas or the items labeled as N/A were
confusing, leading to the participants tendency to miss
certain items. Since not all items were rated by the
participants, the investigators suggest that another study be
conducted to explore the inter-rater reliability of items
labeled N/A and those with missing data.
Although the present study included a relatively
large number of participants compared to the study of
Sarmiento et al. in 2007, some groups had less than five
participants. These small samples might have affected
statistical computation. This was different from the
originally targeted number of five participants for each
group of therapists and fieldwork students per discipline.
An even bigger sample size is recommended to investigate
the PSTs reliability and other critical measurement
properties such as validity. Administration of the tool with
clients with varying levels of function may also help
determine the tools applicability across developmental
disabilities. In addition, determining the inter-rater
reliability of the tool per item would be help identify which
items to retain, eliminate, or consolidate.
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
12
With regard to clinical utility, the participants
raised issues about the vagueness of some terms and
confusion due to the PSTs format. It is recommended
that a glossary of terms be included in the PSTs manual.
Similar items may also be combined into a single item to
minimize confusion between two or more related terms.
This would also make the checklist more concise, as
participants opined that having many items prolonged
administration time. For example, items that may be
consolidated include attending behavior (8), higher
rates of attending (9), and high rates of attention on low
and high interest tasks (10). These may be stated
singularly as attends to tasks until completion. The
items, attempts new task without hesitation (18), and
initiates work tasks (19) can be consolidated to initiates
work tasks without hesitation. Lastly, recognizes work
defects (60) and corrects works defects (63) can be
stated as recognizes and corrects work defects. Other
measures to make the PST more concise as well as reliable
include eliminating or clarifying the unreliable items.
For the PST format, enumerating the items may
reduce the likelihood of the participants missing or
skipping any of the items. Another way to make the format
clearer would be to include the rating scale in one column
on the checklist itself, so that the participants will only
have to encircle their rating for an item. Instead of the
symbols -, +/-, and +, respectively, it is suggested that a
simpler numerical scale of 1, 2, and 3, respectively, be used
for rating.
Further studies may investigate the usefulness of
deriving cumulative and subscale scores for the PST. It
recommended that the ratings be based on the level of
independence of the client doing the task and spontaneity
in task performance, aside from the frequency or the
percentage of the time that a certain behavior or skill is
observed throughout the activity. For example, a rating of
3 would mean that the client is able to exhibit a skill or
behavior independently or spontaneously; 2 would mean
that assistance, cues, or prompts are required; and 1, would
mean that the client is still unable to exhibit the skill or
behavior even with assistance, cues, or prompts; and 0,
would mean that the skill is not expected during the
activity. This additional guideline may address issues
related to the subjectivity of the rating scale.
Conclusion
The PST appears to be a clinically useful screening tool for
adolescents with DD. It has a high percentage of reliable
items when used by occupational therapists, physical
therapists, speech-language pathologists, and fieldwork
students. Errors attributable to the participants and the
PST format affected the degree of inter-rater reliability.
Revisions are recommended for both its format and
content in order to make it more concise and prevent
missed items or lost data. Investigation of the tools test-
retest reliability and validity is also warranted.
Acknowledgments
Data from this study were used by Carina Marie L.
Castillo, John Stanley D. Catacutan, and Deanne Mea R.
Esguerra in completing an undergraduate thesis which was
a requirement for the degree of Bachelor of Science in
Occupational Therapy under the Department of
Occupational Therapy, College of Allied Medical
Professions, University of the Philippines-Manila. Sincere
gratitude is extended to the people whose invaluable
contributions helped make the study possible, namely: Ma.
Concepcion Cabatan, MHPEd; Joselito Magadia, PhD;
Catherine Capio, MScErg; Abelardo Apollo David,
MOccThy; the clients and their families for their
cooperation during the videotaping sessions; and the
therapists and students who participated in the study.
References:
American Occupational Therapy Association (2008).
Occupational therapy practice framework:
Domain and process (2nd ed.). American Journal of
Occupational Therapy, 62, 625-683.
Amstadter, B. L. (1971). Reliability mathematics: Fundamentals;
practices; procedures. New York, NY: McGraw Hill
Book Company.
Anderson, D. R., Sweeney, D. J., & Williams, T.A. (1986).
Statistics: Concepts and applications. Eagan, MN: West
Publishing Company.
Bailey, D. (1997). Research for the health professional: A practical
guide (2nd ed.). Philadephia, PA: F. A. Davis.
Baloueff, O. (1998). Introduction to the pediatric
population. In Crepeau, E.B., and Neistadt, M. E.
(Eds.), Willard and Spackmans occupational therapy (9th
ed.) (pp. 569-575). Philadelphia: Lippincott,
Williams & Wilkins.
Boyt Schell, B. A., Crepeau, E. B., and Cohn, E. S., (2003).
Professional development. In Crepeau, E. B.,
Cohn, E. & Schell B. A. B. (Eds.), Willard and
Spackmans occupational therapy (10th ed.) (pp.141-
146). Philadelphia: Lippincott, Williams & Wilkins.
Cabatan, M. C. C. (2004). Status & trends: Adolescents /
Adults with DD. Unpublished manuscript,
University of the Philippines Manila.
Cabatan, M. C. C. (2004). Transition related dilemmas of
persons with developmental disabilities. Asian
Journal of Occupational Therapy, 3, 15-26.
Culler, K. H. (2003). Home management. In Crepeau, E.
B., Cohn, E. & Schell B. A. B. (Eds.), Willard and
Spackmans occupational therapy (10th ed.) (pp. 491-
554). Philadelphia: Lippincott, Williams & Wilkins.
Foti, D. (2001). Activities of daily living. In L. W. Pedretti
& M. B. Early (Eds.), Practical skills for physical
dysfunction (5th ed.) (pp. 124-171). St. Louis,
Missouri: Mosby.
Healey, J. F. (2001). Statistics: A tool for social research (6th ed).
CA: Wadsworth/ Thomson Learning.
Peterson, C. Q. (1997). Prevocational assessment in mental
health. In Hemphill, B. J. (Ed.). Mental health
assessment in OT (pp.179-193). New Jersey:
SLACK.
Jacobs, K. (1991). Occupational therapy work-related programs
and assessments. Boston: Little, Brown and
Company.
Kelley, T. L. (1947). Fundamentals of statistics. Cambridge,
Massachusetts: Harvard University Press.
Kielhofner, G. (2008). Model of human occupation: Theory and
application (4th ed.). Baltimore, MD: Lippincott,
Williams, & Wilkins.
Law, M. (1987). Measurement in occupational therapy:
Scientific criteria for evaluation. Canadian Journal of
Occupational Therapy, 54, 133-138.
Llorens, L. A. (1991). Performance tasks and roles
throughout the life span. In C. Christiansen & C.
Baum (Eds.), Occupational therapy: Overcoming human
performance deficits (pp. 44-66). NJ: Slack
Incorporated.
Merrill, S. C., & Mulligan, S. E. (2003). Neurological
dysfunctions in children. In Crepeau, E. B., Cohn,
E. & Schell B. A. B. (Eds.), Willard and Spackmans
occupational therapy (10th ed.) (pp. 699-720).
Philadelphia: Lippincott, Williams & Wilkins.
Minichiello, V., Sullivan, G., Greenwood, K., & Axford, R.
(Eds.). (1999). Handbook for research methods in health
sciences. Sydney: Addison-Wesley.
Ocheda, T. S. O., & Cabatan, M. C. (2004). Development of a
prevocational screening test for children and adolescents with
developmental disabilities seen at the CTS UP-CAMP
(NIH 089-2000). Unpublished manuscript,
National Institute of Health, Manila, Philippines.
Partridge, T. (1997). Occupational therapy within a
prevocational programme for adults with acquired
brain injuries: A case study. British Journal of
Occupational Therapy, 60, 238-243.
Polgar, J. M. (2003). Critiquing assessments. In Crepeau, E.
B., Cohn, E. & Schell B. A. B. (Eds.), Willard and
Spackmans occupational therapy (10th ed.) (pp.299-
313). Philadelphia: Lippincott, Williams & Wilkins.
Polgar, J. M., & Barlow, I. (2004). Measuring the clinical
utility of an assessment: The example of the
Canadian Occupational Performance Measure.
Retrieved August 28, 2007 from
http://www.seatingandmobility.ca/ISS2002/ToSu
nnyHill2/iss2002html/007_MeasuringtheClinical
UtilityofanAssessment.htm
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
13
 Proposed CTS Adolescent Program (2005). Manila:
University of the Philippines Manila - College of
Allied Medical Professions.
Rogers, J. C. & Holm, M. B. (1997). Diagnostic reasoning:
The process of problem identification. In C.
Christiansen, & C. Baum (Eds.), Occupational
therapy: Enabling function and well being (pp. 138-150).
Thorofare, New Jersey: Slack Inc.
Sarmiento, I. C., Combista, J., Perocho, J., & Sanchez, A.
(2007). Inter-rater reliability study of the Prevocational
Screening Test for the Clinic for Therapy Services
Adolescent and adult group protocol. Unpublished
undergraduate thesis, University of the Philippines
- College of Allied Medical Professions, Manila,
Philippines.
The Developmental Disabilities Assistance and Bill of
Rights Act of 2000. Retrieved March 24, 2010
from
http://www.acf.hhs.gov/programs/add/ddact/D
DA.html.
Philippine Journal of Occupational Therapy 2011. Volume 4, Number 2
14
View publication stats
Irene Carolina A. Sarmiento, OTR, OTRP
Lecturer
Department of Occupational Therapy
College of Allied Medical Professions
University of the Philippines Manila
Consultant
Timothy Habilitation Center in Quezon City,
Skill Builders Therapy Center in Manila
ica.sarmiento@gmail.com
Edward James R. Gorgon, MPhysio, PTRP
Chair and Assistant Professor
Department of Physical Therapy
College of Allied Medical Professions
University of the Philippines Manila
Carina Marie L. Castillo, OTRP
Instructor
Department of Occupational Therapy
College of Allied Medical Professions
University of the Philippines Manila
Consultant
Speciabilities Development Center in Quezon City
Deanne Mea R. Esguerra, OTRP
Consultant
Therabilities in Pasig City, Center for Goals in Caloocan
City, and
Nurturing Early Skills Therapy Center (NEST) in Cainta
John Stanley D. Catacutan, OTRP
Consultant
Therapy Works, Inc. in Paraaque City, Activity Works,
Inc. in Cavite, and Creative Therapies, Inc. in Quezon City