P

E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
The impact of questionnaire item
format on ability to fake good
1
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
format on ability to fake good
Anna Brown (SHL Group)
In Exploring the use of ipsative measures in personnel selection,
Presented at the ITC Conference, 15 July 2008, Liverpool.
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Faking research
Jackson, Wroblewski & Ashton (2000) compared single
stimulus and forced choice format for integrity-related
personality items
> Those simulating applying for a job gave 1 SD better scores
when using single stimulus format instrument
> Shift in the mean only one third SD for the forced choice
format
Martin, Bowen & Hunt (2002) show that ipsative OPQ is
more resistant to faking instructions than normative
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
22
Martin, Bowen & Hunt (2002) show that ipsative OPQ is
more resistant to faking instructions than normative
Christiansen et al. (2005)
> Both FC and normative format susceptible to distortion, but
FC more robust with applicants
> High ability individuals tend to be better at distorting FC
format instruments than those of lower ability
> Triad harder to distort than dyad format
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
However
1. Partially ipsative measures were used in some
studies
> What about pure ipsative data?
2. Traditionally derived ipsative scores were used
in all studies
> Variance of the total scale score is 0, so conventional
factor-analytic procedures are inappropriate
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
3
factor-analytic procedures are inappropriate
We cannot get to the true scale correlations using ipsative
data due to the distortion to the covariance structure
> The sum of scores on the test is constant for each
individual
It is impossible to get all high/low scores on the profile
> So how can we compare the impact of faking on
construct validity, score elevation and individual
profiles?
3
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Objectives for this study
To investigate the degree of resistance to faking for the
two formats in a controlled study
Compare pure normative and ipsative scales (not
partially ipsative)
Use appropriate response models
> Two-dimensional IRT model for forced-choice pairs
> One-dimensional IRT model for single-stimulus items
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
4
> One-dimensional IRT model for single-stimulus items
Recover latent traits with multivariate structural models
Compare the latent traits derived from the two formats
regarding:
> Construct validity
> Scores elevation
> Individual profiles
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
5
STUDY
Comparing the impact of faking on single-stimulus and
forced-choice formats
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Instrument and sample
Customer Contact Styles Questionnaire (CCSQ7.2)
Used for assessment for customer service and sales
roles internationally.
16 work-related dimensions cover a wide range of
traits, with a strong emphasis on achievement
motivation (SHL, 1997).
Combines both Likert and forced-choice formats
> 128 items grouped into 32 quads
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
6
> 128 items grouped into 32 quads
> Each item has to be rated on a 5-point Likert scale
> Most like me and least like me items selected for each quad
Sample
Undergraduate students from Nottingham University
N=140 (61 were males, 79 females)
Mean age 24
All participants reported previous employment history
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Manipulated conditions
1. First completion applicant faking condition
1. Participants are unfamiliar with the test
2. Supervised group setting, P&P test
3. Participants are given a job description of an
Advertising Sales Representative
4. The participants are asked to complete the test as
to present themselves in the best possible way for
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
7
to present themselves in the best possible way for
the job
2. Second completion honest condition
1. Within a week of the first completion
2. Feedback report was used as an incentive to
complete
3. The participants completed in their own time
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
8
METHOD
IRT models for scoring SS and FC formats
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
IRT scoring of single-stimulus format
IRT makes some improvement to accuracy of estimation,
particularly for low number of categories
Item responses are treated as categorical data
Latent traits are estimated based on probabilities of a
given response with the Graded Response Model
(Samejima)
* *
( ) ( ) ( ) P P P
+
=
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
99

.0
.2
.4
.6
.8
1.0
-4.5 -3.5 -2.5 -1.5 -0.5 0.5 1.5 2.5 3.5 4.5
Trait ()
()
a
i
= 1.70
b
i1
= -1.85
b
i2
= -0.90
b
i3
= 0.10
b
i4
= 1.65

X = 0 X = 4
X = 3
X = 1
X = 2
( 1)
( ) ( ) ( )
ix ix i x
P P P
+
=
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Two-dimensional IRT model for paired
comparisons (Brown, 2007)
Ranking task is recoded as paired comparisons
Outcome of each comparison:
1 if A preferred to B
0 if B preferred to A
Ordering of items A,B,C,D creates 6 directional pairs:
{A,B} {A,C} {A,D} {B,C} {B,D} {C,D}
A two-dimensional IRT model describes
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
10
A two-dimensional IRT model describes
the probability of preferring A to B
depending on:

i
and
j
- strengths of the relationship
with each of the two underlying dimensions,

21
- the correlation between the two traits,

ij
- a threshold.
1 2
1 2
2 2
21
( 1 , )
1 2
ij i j
ij
i i i j
P y

+ +

= =



P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Structural IRT model for the whole test
Both single-stimulus and forced-choice IRT
models are embedded in SEM framework to
model responses to the whole test
Single-stimulus categories and Paired
comparisons are observed variables
(categorical)
Traits are latent factors
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
11
Traits are latent factors
Traits can be freely correlated or have a
second order factor structure
The whole test is estimated and scored at
once using limited information estimators
General purpose SEM software capable of
handling categorical observed variables
(MPlus by Muthen & Muthen, 2006).
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
IRT estimation of latent traits
A large CCSQ standardisation sample (N=616)
was used to establish all IRT model
parameters for both SS and FC models
> In the standardisation sample SS and FC model
produced very similar factor structures
The current study responses were then scored
using the fixed parameters established in the
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
12
using the fixed parameters established in the
standardisation sample
> SS responses for both conditions were scored with
the fixed SS structural model
> FC responses for both conditions were scored with
the fixed FC structural model
PERFORMANCE
P
E
O
P
L
E

P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
1
3
R
E
S
U
L
T
S
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Construct validity
Honest condition
Single-stimulus format Forced-choice format
1. Driven and resilient 1. Supportive
2. Conscientious 2. Conscientious
3. Supportive 3. Driven and resilient
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
14
Maximum likelihood factoring with oblique rotation
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Faking condition
Single-stimulus format Forced-choice format
Essentially 1 factor 1. Supportive and Resilient
2. Driven
3. Conscientious
4. Open to experience and change
Impact of faking on construct validity
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
15
Loss of construct validity Enhanced construct validity
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Scores elevation
Faking success would be indicated by maximising scores
on scales that recruiters would look for in a hypothetical
selection situation.
Six scales particularly relevant to the job in the present
study (their published validities in brackets):
persuasive (.37), self-control (.46), innovative (.37),
flexible (.26), structured (.53) and competitive (.26)
> Faking SS format: average six scale score z=1.38
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
16
> Faking SS format: average six scale score z=1.38
> Faking FC format: average six scale score z=0.19
SS format: construct validity is not resistant to faking,
so any scale can be inflated easily
FC format: construct validity is resistant to faking, so as
long as selection criteria relies on scales from
DIFFERENT factors, it is hard to maximise scores on all
relevant scales
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Individual profiles
People try to fake the same scales in both formats, so in
Faking condition SS and FC individual profiles are
similar in shape:
> Faking: average cross-format profile correlation 0.62
People are pretty successful in faking SS
format
> All but 2 people managed to increase their scores on
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
17
> All but 2 people managed to increase their scores on
the six target scales
In faking FC format they are not as successful:
> Out of top 10 candidates on the six scales in HONEST
condition by trying to fake 8 dramatically reduced
their scores, and 2 retain them about the same
> Out of top 10 successful fakers (increased score to
about 0.8) 9 already had higher than average scores
on the six scales
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Summary of individual scores
20
25
30
35
40
F
r
e
q
u
e
n
c
y
Composite score on six target scales - Faking
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
18
0
5
10
15
20
F
r
e
q
u
e
n
c
y
Theta score
FC SS
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Conclusions
In this study we instructed people to fake so
this is the worst case scenario and not what
happens in real life
Nevertheless it allows us to estimate how
successful somebody would be if they fake
On SS format a faker could successfully inflate
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
19
On SS format a faker could successfully inflate
all dimensions
On FC format it is very hard if the important
dimensions belong to different factors
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Discussion
The main problem with faking (when it is
going on) is that it changes true ordering of
people on the traits
With self-report tests this happens regardless
of the test format one is free to put on a
persona and change the item responses
accordingly
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
20
accordingly
But, with forced-choice format we can
> be sure that fakers cannot get a considerable
advantage over other candidates
> communicate the almost random faking success
levels to the test takers and warn them of potentially
worsening their results
P
E
R
F
O
R
M
A
N
C
E
PEOPLE PERFORMANCE
Special thanks to Adrian Husbands (MSc graduate from
I-WHO) for recruiting the participants and collecting data
P
E
O
P
L
E
P
E
R
F
O
R
M
A
N
C
E
21
THANK YOU
I-WHO) for recruiting the participants and collecting data
for this study