Article

Critique

Internet Self
Reporting in
Social Anxiety
Disorder: A
Psychometric
Evaluation
EDPS 612.02
Alison Lessard and Christina Majcher

Introduction
Why choose an article about internet-based
self-reporting?

What will we cover in this presentation?

Research Purpose and Hypothesis
Sample
Measurement and Procedures
Data Analysis
Results and Conclusions
Discussion

Research Purpose and

Hypothesis
Why this article? Limited studies internet-based
self reporting questionnaires
Why use the internet?
Advantages- rate of completion, scoring,
reduces internal drop-out rate, information
can be quickly exported to statistical
programs
Disadvantages- Are psychometric properties
the same as a paper and pencil based
questionnaire? (Buchanan, 2003)

Aim of the Study

To test the equivalence of paper-andpencil and Internet administered
versions of self-report questionnaires used
in social phobia research

Sample
N= 121 participants
2 separate studies
Study 1- Paper and Pencil
51.56% women
average age 37.33 (SD= 10.22)
Study 2- Internet-Based
64.91% women
average age 32.72 (SD = 9.21)

Measurement
Six questionnaires were completed:
Liebowitz Social Anxiety Scale (LSAS)
two versions similar reliability and validity
two versions strongly correlated (r = 0.85)
high internal consistency (Cronbachs 0.95)
high test reliability over 12 weeks (r = 0.83)
convergent and discriminant validity is strong
Social Phobia Scale (SPS)
high test-retest reliability (r = 0.66- 0.93),
high internal consistency (Cronbachs 0.89)
good discriminant validity
Social Interaction Anxiety Scale (SIAS)

high test-retest reliability, (r = 0.82- 0.93)

high internal consistency (Cronbergs 0.83)
good discriminant validity

Measurement

high test-retest reliability (r = 0.80- 0.94)

high correlation with BDI (r = 0.87), good
convergent validity
Beck Depression Inventory and Beck Anxiety
Inventory (BDI, BAI)
good discriminant and convergent validity
high internal consistency (Cronbachs = 0.92)
good test-retest reliability over 1 week (r = 0.75)
Quality of Life Inventory (QOLI)
high internal consistency (Cronbachs = 0.79- 0.82)
good one month test-retest reliability (r = 0.92)
good discriminant, convergent and nomological
validity
Montgomery Asberg Depression Rating Scale (MADRS-S)

Procedure
Paper-Pencil Group (PP)
Participants were sent questionnaires:
LSAS-SR, SIAS, SPS, BAI, MADRS-S and QOLI
Requested to fill out in order and return
asap
Internet Group (IT)
Password protected website to fill out
questionnaires
Unique codes to identify respondents
Questionnaires completed in same order as
PP group
Option to go back and review answers

Procedure
Paper-Pencil and Internet Version
No experimenters
All measures-Swedish language
(translated, validated version)
Both studies approved by an ethics
committee

Procedure
Across administration formats
researchers compared:
Construct validity
Internal consistencies
Mean scores

Procedure
Construct validity was derived using the following
criteria:
Intercorrelations of LSAS, SIAS and SPS should be
large and differ significantly from zero
Correlations between these measures should be
larger than their correlations between the other
questionnaires because the MADRS-S, BAI and QOLI
measure different constructs (depression, anxiety,
subjective quality of life)
Cronbachs Alpha should be larger than all the other
correlations

Data Analysis
Internal Consistency
Cronbachs Alpha was between 0.77 and 0.94
Measure

Paper-Pencil

Internet

LSAS

0.94

0.94

SIAS

0.93

086

SPS

0.93

0.89

MADRS-S

0.82

0.86

BAI

0.84

0.89

QOLI

0.77

0.81

Data Analysis
Mean Values
(Standard Deviations) between group differences
Measure

Administration Format

Mean(SD)

LSAS

Paper-Pencil
Internet

67.58(21.53)
69.60(22.48)

SIAS

Paper-Pencil
Internet

44.61(14.42)
42.09(12.57)

SPS

Paper-Pencil
Internet

34.13(14.98)
37.00(15.69)

MADRS-S

Paper-Pencil
Internet

12.61(6.59)
13.46(7.26)

BAI

Paper-Pencil
Internet

13.42(6.90)
14.82(8.41)

QOLI

Paper-Pencil
Internet

1.29(1.29)
0.58(1.84)

Data Analysis
Intercorrelations and Construct Validity
IT Group: All measures significantly correlated in
expected direction
Paper-Pencil:
11 out of 15 were statistically significant
non-significant correlations were
between
MADRS-S and SPS
BAI and SIAS
QOLI and SPS
QOLI and BAI

Intercorrelations of
Questionnaires
Measure

1. LSAS

.67

.77

.60

.58

-.46

.47

.40

.36

-.35

.54

.69

-.33

.68

-.57

2. SIAS

.67

3. SPS

.73

.60

4. MADRS-S

.25

.32

.12

5. BAI

.27

.24

.35

.34

6. QOLI

-.36

-.43

-.24

-.39

Upper Panel shows internet sample, lower panel paperpencil sample

-.29
-.00

Results
Results suggest that the psychometric
properties are adequate for the 3 internet
administered social anxiety
questionnaires (LSAS-SR, SIAS, SPS)
There are good psychometric properties for
the assessments related to general anxiety
(BAI) and depression (MADRS-S)
Differences that appeared between PP and IT
group appeared to have been explained by
responses on the QOLI
The 2 groups are very similar

Limitations
The sample size was small n=121
The design makes it difficult to specifically
isolate the effect of the administration
format
Researchers did not measure invariance
between paper and pencil and internet groups
The results may not generalize to real life
settings

Conclusions
Internet administration format for social
anxiety measures used is reliable and
valid
This study did not lend support to the
previously held belief that respondents give
higher severity ratings when answering via
the internet
Further research needs to be conductedin particular having each participant fill out
both types of questionnaires and randomize
the order of administration

Discussion
How would we change the research to make it
better?

Are offline norms appropriate for online

tests?
What implications would the general
internet population have upon results?
Are there limitations to a Swedish-only
study?
What do you feel are appropriate
measures to be completed online?

References
Buchanan, T. (2003). Internet-based questionnaire assessment:
Appropriate
use in clinical contexts. Cognitive Behaviour
Therapy, 32, 100109.
Furr, R. M., & Bacharach, V. R. (2013).
Psychometrics: An Introduction (2nd Ed.). Thousand Oaks, CA: Sage
Publications.
Hedman, E., Ljtsson, B., Rck, C., Furmark, T., Carlbring, P.,
Lindefors, N., & Andersson, G. (2010). Internet administration of
self-report measures
commonly used in research on social
anxiety disorder: A psychometric evaluation. Computers in
Human Behavior, 26(4), 736-740.
Hirai, M., Vernon, L. L., Clum, G. A., & Skidmore, S. T. (2011).
Psychometric properties and administration measurement
invariance of social phobia
symptom measures: paperpencil vs. Internet administrations. Journal of Psychopathology and
Behavioral Assessment, 33(4), 470-479.