See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/331978516

Contextualization as a means to improve the predictive validity of personality

models

Article  in  Personality and Individual Differences · July 2019

DOI: 10.1016/j.paid.2019.03.007

CITATIONS READS

0 389

2 authors:

Victor Swift Jordan B Peterson

University of Toronto University of Toronto
5 PUBLICATIONS   0 CITATIONS    130 PUBLICATIONS   7,289 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

The psychological significance of the Biblical stories: http://bit.ly/2rMHp08 View project

All content following this page was uploaded by Victor Swift on 12 July 2019.

The user has requested enhancement of the downloaded file.

 Personality and Individual Differences 144 (2019) 153–163

Contents lists available at ScienceDirect

Personality and Individual Differences

journal homepage: www.elsevier.com/locate/paid

Contextualization as a means to improve the predictive validity of T

personality models
⁎,1
Victor Swift , Jordan B. Peterson
University of Toronto, 27 King's College Circle, Toronto, Ontario M5S 1A1, Canada

A R T I C LE I N FO A B S T R A C T

Keywords: Research investigating performance-related outcomes has established that the predictive validity of Big Five
Contextualization models can be doubled, simply by contextualizing scale items. We herein test whether this contextualization
Big Five effect can be leveraged to universally improve the predictive validity of Big Five models by considering the
Predictive validity breadth, depth, and nature of contextualization effects. Study 1 (N = 320) compared the predictive validity of an
Conscientiousness
uncontextualized measure of Conscientiousness (C), with three measures of C contextualized to organizational,
Frame of Reference
academic, and romantic settings, respectively. Nine outcome variables served as indices of validity (three per
context). We hypothesized that contextualized scales would be superior in predicting all context-congruent
outcomes. Study 2 (N = 680) and Study 3 (N = 378) extended the results to consider alternative trait and
outcome measures (e.g., Implicit Association Task). Across all three studies, predictive validity was enhanced for
contextualized vs. uncontextualized measures only when outcomes were context-congruent. However, this en-
hancement varied greatly, with contextualized scales improving the prediction of some outcomes by as much as
32% (GPA), and as little as 1% (income). Our findings suggest that scale contextualization can be adopted by
psychometricians to easily improve the predictive validity of personality models. Guidelines for contextualiza-
tion are discussed.

The development of reliable Big Five personality measures has
ushered in an era of personality screening. Today, Big Five scores are
used to predict important social, political, organizational, inter-
personal, and individual outcomes (Ozer & Benet-Martínez, 2006). The
magnitude of these predictions, however, remain moderate to small, as
suggested by meta-analyses across disciplines (Barrick & Mount, 1991;
Berry, Ones, & Sackett, 2007; Bogg & Roberts, 2004; Poropat, 2009).
Thus, there is much to be gained from improving the predictive validity
of Big Five measures.
One of the most promising approaches for improving the predictive
validity of personality models is contextualization. Simply by asking
participants to complete traditional personality scales with a particular
context in mind, the predictive validity of the consequent personality
models can be significantly improved (Bing, Davison, & Smothers,
2014). For instance, scales comprised of personality items such as “do
you finish what you start at work” can explain two times more variance
in job performance than scales that do not specify a context (for meta-
analysis see Shaffer & Postlethwaite, 2012). The improvement of pre-
dictive validity by means of contextualization is referred to as the
Frame of Reference (FOR) effect (Holtrop, Born, & de Vries, 2014).
Since its discovery (Schmit, Ryan, Stierwalt, & Powell, 1995) the
FOR effect has been exclusively investigated as a potential means to
improve the prediction of performance outcomes. Indeed, the FOR ef-
fect has been reliably shown to emerge in the prediction of job per-
formance (Bing et al., 2014; Farmer, 1999; Hunthausen, Truxillo,
Bauer, & Hammer, 2003; Pace & Brannick, 2010), and academic per-
formance (Bing, Whanger, Davison, & VanHook, 2004; Holtrop, Born,
de Vries, & de Vries, 2014; Lievens, De Corte, & Schollaert, 2008; Liu,
2014; Roenicke, 2013; Seiler, 2011; Schmit et al., 1995). However, it
remains unclear as to whether FOR effects emerge in the prediction of
non-performance outcomes.
Before contextualization can be considered a viable means to im-
prove the general predictive validity of the Big Five, three features of
the FOR effect must be elucidated. Firstly, we must determine the
breadth of FOR effects by establishing whether these effects emerge in
the prediction of a variety of outcomes, across a variety of contexts.
Contextualization will prove to be most psychometrically useful if FOR
effects emerge broadly. Secondly, we must determine the depth of these

⁎
Corresponding author at: 100 St. George Street, Toronto, Ontario M5S 3G3, Canada.
E-mail address: victor.swift@mail.utoronto.ca (V. Swift).
1
Victor Swift was responsible for design, data collection, data preparation, data analysis, and report writing. Jordan Peterson was responsible for resources and
supervision. This study was not preregistered.

https://doi.org/10.1016/j.paid.2019.03.007
Received 28 February 2018; Received in revised form 5 March 2019; Accepted 7 March 2019
Available online 14 March 2019
0191-8869/ © 2019 Elsevier Ltd. All rights reserved.
V. Swift and J.B. Peterson
effects by establishing whether contextualization results in meaningful
increases of predictive validity – as trivial increases due to technicalities
would disqualify the practical value of contextualization. Lastly, we
must determine the nature of FOR effects by establishing the conditions
that give rise to FOR effects. Only by knowing the necessary and suf-
ficient conditions for FOR effects can we maximize the psychometric
utility of contextualization.
1. Breadth of frame of reference effects
We hypothesize that FOR effects emerge in the prediction of a
variety of outcomes, not just performance-related outcomes. This ex-
pectation is based, in part, on current trends in contextualization re-
search. Our review of the contextualization literature revealed that six
studies have considered non-performance outcomes (Ching et al., 2013;
Heller, Ferris, Brown, & Watson, 2009; Holtrop, Born, de Vries, & de
Vries, 2014; Slatcher & Vazire, 2009; Wood, 2007; Wood & Roberts,
2006). In all of these studies, trait-outcome correlations were greater
for contextualized, than for uncontextualized, trait measures. For in-
stance, the correlations between relationship-specific personality and
relationship satisfaction were consistently larger (ravg = 0.38) than the
correlations between uncontextualized personality and relationship
satisfaction (ravg = 0.15; Slatcher & Vazire, 2009; Wood & Roberts,
2006). Likewise, the correlation between fraternity satisfaction and
fraternity-specific personality (r = 0.44) was found to be larger than
the correlation between fraternity satisfaction and uncontextualized
personality (r = 0.18; Wood, 2007).
Simply comparing the magnitude of correlation coefficients does
not confer whether one predictor is stronger than another, particularly
when the predictors are highly correlated (Steiger, 1980), as is the case
in the existing literature. Moreover, the precise magnitude of correla-
tions is unlikely to generalize to other samples (Vispoel, Morris, &
Kilinc, 2018). As such, it remains unclear as to whether con-
textualization does indeed improve predictions of non-performance
measures. We herein address this issue by utilizing statistical proce-
dures that enable the systematic comparison of correlation coefficients.
2. Depth of frame of reference effects
Predictive validity can be trivially improved by increasing the
equivalence of item content between predictor and outcome variables
(Nicholls, Licht, & Pearl, 1982). This poses a problem for con-
textualization research where items are deliberately transformed to be
more similar to outcome variables. Indeed, it is difficult to determine
whether FOR effects are genuine or caused by increases in item
equivalence in some performance research. For instance, Bing et al.
(2014) found that the substitution of items such as “I am usually able to
convince my friends what we ought to do together next” with “If a potential
buyer says ‘no’ to me it means I need to work harder to get my point across”
improved the prediction of supervisor-rated sales performance on scales
comprised of such items as “This employee uses suggestive selling in a very
skillful fashion”.
Item equivalence does not pose a problem for all FOR findings.
Specifically, when outcome variables are categorically distinct from the
trait being measured, contextualization is unlikely to affect item
equivalence. For instance, contextualizing trait conscientiousness
measures, which are comprised of items such as “I make plans and follow
through with them”, and “I tend to be organized”, improves the prediction
of job satisfaction, which is comprised of items such as “I feel fairly
satisfied with my job”, and “I like my job better than the average worker
does”. However, adding the tag “at work” to conscientiousness items
does not seem to increase the semantic similarity with job satisfaction
items. In these circumstances, instead of increasing item equivalence,
contextualization increases the relevance of the predictor to the out-
come. We suspect that it is precisely this condition of relevance that
underlies, and legitimizes, FOR effects.
154
Personality and Individual Differences 144 (2019) 153–163
3. Nature of frame of reference effects
According to Identity Theory (Stryker, 1986), when a traditional
trait measure is contextualized, the latent variable that is being mod-
elled changes from a broad trait to a role-specific trait (Heller et al.,
2009). This transformation has two consequences which contribute to
FOR effects. Firstly, contextualization reduces the ambiguity of per-
sonality items, as the imposed role serves as a standard reference frame.
By restricting the interpretation of items, this reference frame reduces
within-person inconsistency and between-person variability (Lievens
et al., 2008). In turn, this reduction in error increases predictive va-
lidity. Secondly, contextualization adjusts the specificity of trait scales
to better match the specificity of context-specific outcomes (Ajzen,
1987). By aggregating across contexts, general personality scales have
less predictive accuracy due to the lack of context-specific, but criterion
valid, variance (Paunonen & Ashton, 2001).
Simply specifying a role is not sufficient to produce FOR effects.
Crucially, the role that is specified must be “conceptually relevant” to
the outcome being predicted (Farmer, 1999; Lievens et al., 2008). For
instance, FOR effects emerge when predicting job-performance from
workplace-specific, but not school-specific, trait measures. Likewise,
FOR effects emerge when predicting academic performance from
school-specific, but not workplace-specific, trait measures (Farmer,
1999; Lievens et al., 2008). Thus, FOR effects seem to be contingent on
manipulations of criterion relevance (Bing et al., 2004). It remains
unclear, however, as to what it means for a criterion to be conceptually
relevant to a context, as any notion of relevance is inherently unin-
formative without allusion to some specified guidelines (Davie, 2005).
We herein define criterion relevance as a congruency between the
context of an outcome measure and the context of a trait measure.
4. Hypotheses
In three studies, we investigate the utility of contextualization as a
means to improve the predictive validity of personality models.
Specifically, we seek to determine whether FOR effects are broad and
meaningful enough to justify the instrumental use of contextualization.
Moreover, we seek to determine the necessary and sufficient conditions
for producing FOR effects in order to substantiate guidelines for con-
textualization. Accordingly, we posit three hypotheses:
H1. (Congruency Hypothesis) FOR effects emerge when there is congruency
between the contexts of outcome and trait measures.
H2. (Breadth Hypothesis) FOR effects emerge across a variety of contexts
and for a variety of outcomes.
H3. (Depth Hypothesis) FOR effects are independent of item equivalence.
Our examination focuses on a single Big Five trait:
Conscientiousness (C) – defined as a tendency to be hardworking, re-
strained, orderly, and rule abiding (Roberts, Jackson, Fayard, Edmonds,
& Meints, 2009). Of the Big Five traits, C remains the best predictor of
job performance (Schmitt, 2014) and academic performance (Poropat,
2009), and the most researched trait in the FOR context (Roenicke,
2013). Thus, by focusing on C we are able to simultaneously draw from
and contribute to FOR research that may have broad practical im-
plications for academic and industrial domains.
In Study 1, all three hypotheses are tested using a Big Five measure
contextualized to three contexts: workplace, school, and relationship. In
Study 2, we extend our results by testing whether FOR effects emerge
when predicting non-performance outcomes from a novel con-
textualized measure of C. The results of this second study reflect the
generalizability of non-performance FOR effects. In Study 3, we extend
our results further by considering whether FOR effects emerge for other
Big Five traits, and when predicting non-performance outcomes that are
not based on self-report. Across all studies, outcome variables are
considered that have been unexplored in existing FOR research.
V. Swift and J.B. Peterson
5. Study 1
To test whether FOR effects emerge when there is congruency be-
tween the contexts of outcome and trait measures (H1), we consider
three categories of outcomes across three different contexts: workplace,
school, and relationships. Across contexts, C has been found to predict
three categorical outcomes: performance, virtues, and satisfaction.
Thus, we expect contextualized measures of C to be better predictors of
performance, virtues, and satisfaction only when context congruency is
satisfied. For instance, we expect workplace-C to predict workplace
performance, workplace virtues, and workplace satisfaction, over and
above uncontextualized and other-contextualized measures of C.
Moreover, we expect that FOR effects do not emerge when the context
of a trait measure is not congruent with the context of an outcome
measure. For instance, we expect that workplace-C will not predict
school or relationship performance, virtues, and satisfaction, over and
above uncontextualized and other-contextualized measures of C.
Nine outcome measures are considered which represent perfor-
mance, virtues, and satisfaction, for workplace, school, and relationship
contexts (3 categorical outcomes x 3 contexts). Because we are interested
in changes in predictive validity for measures of C, the outcomes that
were selected are based on previous research into the categorical out-
comes associated with C. In terms of performance, Conscientious people
have been found to be better workers (Barrick & Mount, 1991), students
(Poropat, 2009), and partners (Ahmetoglu, Swami, & Chamorro-
Premuzic, 2010). In part due to their dutifulness, Conscientious people
tend to be more virtuous, as demonstrated by reductions in workplace
deviance (Berry et al., 2007), academic procrastination (Steel, 2007),
and relationship infidelity (Schmitt, 2004). Finally, Conscientious
people tend to be more satisfied with life in general (Hayes & Joseph,
2003), and across domains, including workplace (Judge, Higgins,
Thoresen, & Barrick, 1999), school (Weber & Huebner, 2015), and re-
lationships (Schaffhuser, Allemand, & Martin, 2014).
We expect that all nine of these outcome measures are associated
with FOR effects (H2) – while specific effects depend on context con-
gruency (H1).
5.1. Study 1: Methods
5.1.1. Personality measures
Three contextualized measures of C were derived from the 20-item
Conscientiousness sub-scale of the International Personality Item Pool
Representation of the Revised NEO Personality Inventory (IPIP-NEO;
Goldberg et al., 2006). The IPIP-NEO was used because it demonstrates
the highest internal consistency of any measure of C (Goldberg, 2017).
Contextualization consisted of specifying a unique setting for each
of the 20 items of C. The three contexts that were specified represented
academic, romantic, and organizational settings, respectively. Each of
these three contextualized measures used the same 20 IPIP-NEO items
in order to ensure an equal representation of C's facets. For instance, the
IPIP-NEO item “I need a push to get started” was translated into the
academic item “I need a push to start school projects or homework”, the
romantic item “I need a push to start things with my partner”, and the
organizational item “I need a push to start projects at work” (see
Appendix A).
To assess the other Big Five traits, along with uncontextualized C, an
unaltered version of the IPIP-NEO was administered (100 items total,
20 items for C). On all personality items, participants indicated their
agreement with statements using a 5-point Likert scale (1 = strongly
disagree, 5 = strongly agree).
5.1.2. Performance measures
5.1.2.1. Workplace performance. Workplace performance was assessed
using self-report, which has been shown to reliably represent actual
workplace performance (Heidemeier, 2005). Participants completed
the 12-item Task Performance subscale of the Job Performance scale
155
Personality and Individual Differences 144 (2019) 153–163
(Carlos & Rodrigues, 2016). Agreement with items was indicated using
a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree). While
self-rated performance is somewhat controversial as a substitute for
supervisor-rated performance, for our purposes (i.e., demonstrating
relative predictive validity), this issue is irrelevant. Moreover, the focus
of this study is FOR effects beyond job performance, as FOR effects for
performance are well established (see introduction).
5.1.2.2. Relationship performance. Relationship performance was
assessed with a 9-item version of Sternberg's Triangular Love Scale
(Lemieux & Hale, 1999). The scale consisted of 3 components (3 items
each): intimacy, passion, and commitment. Agreement with items was
indicated on a 5-point scale (1 = not at all, 5 = extremely).
5.1.2.3. Academic performance. To assess academic performance,
participants were asked to report their GPA for each level of
schooling that they completed (High School, Community College,
University, and Graduate School). GPA was measured using a 13-
point scale which was labeled with letter grades and their
corresponding percentages and GPA values [1 = F (< 59%);
13 = A+ (99% to 100%, 4.0 GPA)]. A final academic performance
score was calculated by averaging GPA across levels of schooling.
5.1.3. Satisfaction measures
5.1.3.1. Workplace satisfaction. Workplace satisfaction was assessed
using Hoppock's Job Satisfaction Measure (McNichols, Stahl, &
Manley, 1978), a 4-item scale with variable 7-point Likert responses
(1 = low satisfaction, 7 = high satisfaction). This scale has been used in
previous research to demonstrate the relationship between C and Job
Satisfaction (Judge, Heller, & Mount, 2002).
5.1.3.2. Relationship satisfaction. Relationship satisfaction was assessed
with the 7-item Relationship Assessment Scale (RAS; Hendrick, 1988).
Respondents indicated agreement with items (e.g., “How well does your
partner meet your needs”) on a 5-point scale (1 = low, 5 = high).
5.1.3.3. Academic satisfaction. Academic satisfaction was assessed with
a variation of the 7-item RAS. The word “school” was substituted in
place of the word “partner” (e.g., “How well did your school meet your
needs”). Agreement with items was indicated on a 5-point scale
(1 = low, 5 = high). Participants were asked to respond with their
most recent schooling experience in mind.
5.1.4. Virtue/vice measures
5.1.4.1. Workplace deviance. Organizational Deviance (OD) refers to
harmful or counterproductive behaviors that undermine an
organization, including unlawful or dishonest actions. OD was
assessed by a 7-item Workplace Deviance scale (Bennett & Robinson,
2000) that assessed agreement with items on a 7-point scale (1 = never,
7 = daily).
5.1.4.2. Relationship infidelity. Relationship fidelity was assessed using
the relationship exclusivity sub-scale of the Sexy Seven Measure of
Sexuality (Schmitt & Buss, 2000). Participants indicated agreement
with 8 sexually connotative self-ascriptions (e.g., promiscuous,
adulterous) using a 9-point scale (1 = extremely inaccurate,
9 = extremely accurate).
5.1.4.3. Academic procrastination. Participants completed a short form
(5-item) version of the Academic Procrastination Form (Yockey, 2016).
Agreement with items (e.g., “I knew I should work on schoolwork, but I
just didn't do it”), was indicated on a 4-point scale (1 = strongly agree,
4 = strongly disagree).
5.1.5. Procedure
345 participants recruited through Amazon's Mechanical Turk
V. Swift and J.B. Peterson Personality and Individual Differences 144 (2019) 153–163

(MTurk) completed our survey, which was assembled and hosted on Table 1
Qualtrics. Twenty-five participants (7%) who failed at least one of the Scale parameters of all measures used in Study 1.
nine attention checks were excluded from the dataset before any ana- Alpha Mean SD
lyses were conducted, resulting in a final sample of 320 participants
(51% female; Mage = 40, SDage = 12). Conscientiousness
General 0.95 4.2 0.71
Participation was restricted to users who had completed > 1000
Workplace 0.94 4.5 0.56
surveys and demonstrated acceptable responding at least 99% of the School 0.98 3.9 1.04
time (the same restriction applied to subsequent studies). MTurk users Relationships 0.94 4.3 0.74
are representative of the general U.S population (Buhrmester, Kwang, & Satisfaction
Gosling, 2011) and have been repeatedly shown to produce reliable and Workplace 0.92 4.9 1.22
valid data (Mason & Suri, 2012; Paolacci & Chandler, 2014). School 0.95 3.5 1.08
After providing consent to participate, participants were notified Relationships 0.95 3.9 1.03
that the survey would consist of approximately 230 multiple choice Performance
questions that may appear repetitive, but which ultimately differ in Workplace 0.74 5.5 0.71
School (GPA) N/A 3.0 0.70
terms of context. The survey was divided according to contextual
Relationships 0.89 4.0 0.83
blocks, in order to ensure that participants maintained the appropriate
context in mind. The workplace block consisted of the Job Performance Virtues
Workplace 0.84 1.6 0.57
Scale, Job Satisfaction Scale, Workplace Deviance Scale, and the School 0.96 2.7 1.06
workplace-contextualized measure of C. The academic block consisted Relationships 0.91 2.2 1.02
of the Academic Performance measure, Academic Satisfaction scale,
Academic Procrastination scale, and the measure of C contextualized to
the classroom. The relationship block consisted of the Relationship increase the overlap between C items and outcome items.
Performance scale, Relationship Satisfaction scale, Infidelity scale, and Strong measurement invariance is classically represented by a
the relationship-specific measure of C. A non-contextual block was combination of configural (factor structure), metric (item weight), and
administered, consisting of the entire IPIP-NEO, and demographics scalar (average score) invariance (Beaujean, 2014). Configural in-
survey. These blocks were randomly presented, as were the constituents variance is suggested by good overall model fit and matching sig-
within the blocks. To ensure valid responses, nine unique attention- nificant loadings (Hirschfeld & von Brachel, 2014). Metric and scalar
checks were scattered throughout the survey and embedded within the invariance is suggested by minimal reduction in Comparative Fit Index
multiple-choice surveys (e.g., “To indicate focus choose the third option”). (CFI) between steps (ΔCFI < 0.01). Following the guidelines of Hu and
At the end of the survey, participants were debriefed and compensated Bentler (1998), overall goodness of fit for configural invariance was
with $3.00. assessed using the Standardized Root Mean Square Residual (SRMR)
index with a cutoff at 0.08 (Hu & Bentler, 1999).
Table 2 depicts the results of the MACS analysis. For all models,
5.2. Study 1: Results
SRMR at the configural step suggested good overall fit (SRMR ≤ 0.06).
Likewise, for all models change in CFI was < 0.01 between configural
The sample was predominantly Caucasian (87%) and 95% of the
and metric steps, and between metric and scalar steps. Together, these
respondents were currently employed (5% retired or disabled). One
results demonstrate that there was strong measurement invariance be-
participant who reported never being employed in an organization was
tween contextualized and uncontextualized measures, suggesting that
excluded from analyses involving organizational variables. Thirty-four
contextualization did not increase item overlap.
percent of participants were single or in a causal relationship. Nine
Intercorrelations between measures of C and outcome variables are
participants who reported never having a romantic relationship were
depicted in Table 3. The uncontextualized measure of C was most
excluded from analyses involving relationship variables. All partici-
strongly associated with workplace-C (r = 0.8), and comparably asso-
pants had completed a high-school degree, and 8% were currently
ciated with school-C (r = 0.6), and relationship-C (r = 0.6). All of the
students (3.5% part-time).
outcome measures (regardless of context) were significantly correlated
With the exception of the Job Performance measure, all scales ad-
with all of the measures of C. This pattern confirms prior research de-
ministered herein exhibited excellent internal consistency (see Table 1).
monstrating that C predicts performance, satisfaction, and virtues
Two items from the Job Performance measure were negatively corre-
across contexts. The significance of context-incongruent trait-outcome
lated with the overall scale (r1 = −0.08, r2 = −0.16), thereby under-
correlations indicates that the contextualized measures were re-
mining the reliability of this measure. These two items (#19 and #24 in
presentative of a general factor of C, in addition to context-specific
the original published scale) were removed from the scale before our
factors of C.
hypotheses were tested, increasing the internal consistency from poor
The congruency hypothesis (H1) and the breadth hypothesis (H2)
(α = 0.57) to acceptable (α = 0.74).
were tested using K-fold cross validation regression analysis. Whereas
To determine whether contextualization increased the overlap be-
traditional regression analyses provide an unverified estimate of pre-
tween traits and outcomes (H3: depth), measurement equivalence was
dictive power, K-fold regression analysis verifies the predictive power
tested using eight2 Means and Covariance Structures (MACS) – also
of models by fitting a model on a subset of data and subsequently
known as multi-group CFA (Beaujean, 2014). Each MAC structure re-
quantifying the test error on the remainder of the data (Yarkoni &
presented a two-factor model consisting of a latent C variable and a
Westfall, 2017). Free of overfitting, the resultant K-folds regression
latent C-related outcome variable (e.g., Job Satisfaction) and their
coefficient can be interpreted as a true estimate of prediction, unlike
covariance. Measurement equivalence was determined by comparing
correlation (Yarkoni & Westfall, 2017). As such, we employed K-folds
the goodness of fit for models in which latent C was based on un-
analysis to determine the relative predictive power of contextualized
contextualized items, with models in which latent C was based on
and uncontextualized measures.
context-congruent items. Strong equivalence of the uncontextualized
Three K-folds regressions were conducted for every outcome vari-
and contextualized models would suggest that contextualization did not
able, one for each contextualization condition – i.e., uncontextualized,
context-congruent, and context-incongruent. Uncontextualized regres-
2
One MACS model for each item-based outcome measure (i.e., GPA was sions predicted outcome variables (e.g., scored Job Satisfaction) from
excluded). scored C; context-congruent regressions predicted outcome variables

156
V. Swift and J.B. Peterson Personality and Individual Differences 144 (2019) 153–163

Table 2
Measurement invariance across Study 1 measures.
Steps Satisfaction Vice Performance

Work School Relation Work School Relation Work Relation

Configural 0.829(0.06) 0.883(0.05) 0.910(0.05) 0.815(0.06) 0.886(0.05) 0.871(0.06) 0.829(0.06) 0.837(0.06)

Metric 0.826(0.07) 0.879(0.07) 0.907(0.06) 0.813(0.07) 0.881(0.07) 0.868(0.07) 0.826(0.07) 0.835(0.07)
Scalar 0.823(0.07) 0.871(0.07) 0.894(0.07) 0.810(0.07) 0.873(0.07) 0.854(0.07) 0.823(0.07) 0.821(0.07)

Note. Values represent CFI (SRMR).

Table 3 models. Moreover, none of the context-incongruent measures demon-

Inter-correlations of trait scales and outcome measures for Study 1. strated greater predictive validity than uncontextualized measures.
R-C W-C S-C NEO-C
Although K-fold analysis provides greater confidence in effect size than
regular regression analysis, we opted to additionally test whether effect
S-C .58⁎ sizes were significantly different between groups using Hittner, May,
W-C .45⁎ .80⁎ and Silver's (2003) test for dependent overlapping correlations (Hitt-
R-C .52⁎ .39⁎ .62⁎
School
ner's test). All context-congruent coefficients were found to be sig-
Performance .19⁎ .11t .54⁎ .22⁎ nificantly larger than context-incongruent coefficients (p < .05). Ad-
Virtue .32⁎ .38⁎ .75⁎ .46⁎ ditionally, six of the nine outcome variables were significantly more
Satisfaction .24⁎ .22⁎ .60⁎ .31⁎ strongly predicted by context-congruent measures than un-
Workplace
contextualized measures. The three outstanding outcomes were Job
Performance .44⁎ .68⁎ .35⁎ .62⁎
Virtue .38⁎ .55⁎ .33⁎ .47⁎ Satisfaction (z = 1.30, p = .10), Relationship Performance (z = 1.30,
Satisfaction .27⁎ .35⁎ .25⁎ .31⁎ p = .10) and Relationship Vice (z = 1.32, p = .09).
Relationship
Performance .60⁎ .33⁎ .20⁎ .35⁎
6. Study 2
Virtue .35⁎ .24⁎ .14t .26⁎
Satisfaction .58⁎ .32⁎ .22⁎ .38⁎
Study 1 found support for our hypotheses using measures of C that
Note. R-C: Relationship-C, W-C: Workplace-C, S-C: School-C; NEO-C: Un-con- were contextualized at the item level. These measures simply add
textualized C. contextual information to existing Big Five personality items. In Study
⁎
p < .0001. 2, we test our hypotheses with a contextualized measure of C that was
t
p < .05. derived independently of the Big Five. In addition to substantiating the
generalizability of our FOR hypotheses, positive results in this study
from scored context-congruent C (e.g., workplace-C); and context-in- would suggest trait-related FOR effects are not scale dependent.
congruent regressions predicted the outcome variable from the re- The contextualized measure employed herein represented work-
maining context-incongruent C variables (e.g., School C and place C. Two context-congruent outcome measures were administered
Relationship C). Each K-fold regression split the data into 10 training that have previously been established as workplace-related correlates of
and testing folds. Cross validation was repeated 100 times for each C: Job Satisfaction and Income (Judge et al., 1999). To balance the
model in order to derive more stable parameters (see Yarkoni & design, we administered two context-incongruent outcome measures
Westfall, 2017). that have been reliably associated with C: Proactive Health Behaviors
Results of the K-fold regression analysis are depicted in Table 4. In (Takahashi, Edmonds, Jackson, & Roberts, 2013), and Political Con-
line with H1, we found predictions to be greater for all context-con- servatism (Sibley, Osborne, & Duckitt, 2012). All outcome variables
gruent models, compared to uncontextualized and context-incongruent measured herein differ from those employed in Study 1 – this includes
Job Satisfaction, which is measured with an alternative scale in this
Table 4 study.
K-fold based standardized correlation coefficients for Study 1. We expect workplace C to be better than uncontextualized measures
only in the prediction of the Income and Job Satisfaction (H1
Outcome Congruent Incongruent General
Congruency). Although different contextual features influence these
Work variables, they are nevertheless the only two variables that imply a
Performance .69a,b .50 .65 shared context with the workplace trait measure. Incidentally, finding
Vice .58a,b .46 .53 FOR effects for Income and Job Satisfaction will provide support for H2
Satisfaction .39a .33 .35
(Breadth), as these represent additional outcome variables.
School
Performance .75a,b .44 .48
Vice .75a,b .42 .48
6.1. Study 2: Methods
Satisfaction .60a,b .30 .33
6.1.1. Measures
Relationship
Performance .40a
.33 .35 6.1.1.1. Uncontextualized measure of C. In order to ensure that
Vice .40a .32 .33 uncontexualized C was comprehensively measured, a 50-item
Satisfaction .59a,b .37 .42 measure of C was derived by combining three popular personality
measures: the IPIP-NEO (120-item version; Johnson, 2014), the BFAS
Note. Standardized r values for context-congruent (Congruent), context-incon-
(DeYoung, Quilty, & Peterson, 2007), and the HEXACO (Lee & Ashton,
gruent (Incongruent) and uncontextualized (General) measures of
2004). Responses were made using a 5-point Likert scale (1 = disagree
Conscientiousness.
a
Congruent coefficient is significantly larger than incongruent coefficient strongly, 5 = agree strongly). This composite measure exhibited excellent
(p < .05). internal consistency (α = 0.96, M = 3.8, SD = 0.6).
b
Congruent coefficient is significantly larger than uncontextualized coeffi-
cient (p < .05). 6.1.1.2. Big Five Competencies Grid (CG). Contextualized for the

157
V. Swift and J.B. Peterson
Table 5
Pearson's Correlations between trait and outcome measures in Study 2.
Workplace C Aggregated C
⁎⁎
Aggregated C .63
Job satisfaction .35⁎⁎ .27⁎⁎
Income .13⁎⁎ .12⁎⁎
Conservatism .07 .15⁎⁎
Proactive health .09⁎ .22⁎⁎
⁎
p < .05.
⁎⁎
p < .01.
workplace at the instructional and item level, CG measures the Big Five
for the workplace context (Consiglio, Alessandri, Borgogni, & Piccolo,
2013). Rather than simply adding “at work” tags to traditional
personality items, CG restructures traditional items to apply solely to
the workplace. The following CG item is representative of such
restructuring: “I respect deadlines even if it means working overtime”.
Participants indicated how frequently they participate in a given
workplace behavior using a 7-point Likert scale ranging from 1
(never) to 7 (every time), with specific frequencies in between – for
instance, “occasionally–in about 30% of the chances when I could have”.
Workplace C is comprised of items from two subscales: Accomplishment
and Process Management (11 items total). Workplace C as measured by
CG exhibited good internal consistency in our sample (α = 0.89,
M = 5.29, SD = 0.88).
6.1.1.3. Outcome variables. Job Satisfaction was measured using an 8-
item scale that surveyed satisfaction in multiple domains including job
security, supervision, income, and co-workers. This scale has been used
in previous research to demonstrate the relationship between C and
intrinsic career success (Judge et al., 1999). Participants provided
responses using a 5-point Likert scale (1 = dislike it very much, 5 = like it
very much). This measure exhibited good internal consistency
(α = 0.84, M = 3.5, SD = 0.75).
Annual untaxed income was measured using an 8-point scale ran-
ging from “Less than $20,000” to “More than $106,000”, in incremental
intervals of $15,000. Considering the variable economic situations of
online workers, unemployed participants were given the option to in-
dicate the prior year's income. Accordingly, participants were asked if
their work-related details (i.e., job satisfaction, and income) re-
presented their current, or recent, status.
Political orientation was calculated by averaging two 5-point items
that surveyed identification with right-wing and left-wing (reverse
coded) ideals, respectively (1 = disagree strongly, 5 = agree strongly). A
score below 3 represented a left-wing leaning, a score of 3 represented
political neutrality, and a score above 3 represented a right-wing
leaning. A meta-analysis of 73 studies investigating the role of per-
sonality in political orientation (Sibley et al., 2012) has demonstrated
that C is the best Big Five predictor of Conservatism (r = 0.10). The
internal consistency of this measure was excellent (α = 0.90, M = 2.49,
SD = 1.2).
Proactive health was calculated by averaging two z-scored items
that surveyed the frequency of exercise and alcohol consumption (re-
verse coded), using a 7-point scale (1 = never, 7 = most days).
Conscientious individuals are characterized by a tendency to partake in
preventative health behaviors (Takahashi et al., 2013) – notably, less
frequent alcohol consumption (Cook, Young, Taylor, & Bedford, 1998),
and more frequent exercising (Conner & Abraham, 2001).
6.1.2. Procedure
714 participants recruited through Amazon's Mechanical Turk
(MTurk) completed our survey. Fifty participants (7%) were excluded
from analyses a-priori on the basis of negligent responding, as indicated
by a combination of negligent response checks, unvarying responses,
and survey completion in under 10 min. Thus, the sample used for
158
Personality and Individual Differences 144 (2019) 153–163
Job satisfaction Income Conservatism
.33⁎⁎
.08 .08
.11⁎⁎ .02 .05
analyses was composed of 664 participants (50.1% female; Mage = 36,
SD = 11).
After reading a consent form that broadly disclosed the main com-
ponents of the survey, participants were asked to complete a demo-
graphic questionnaire including SES, political orientation, exercise
frequency, and alcohol consumption questions. Participants then com-
pleted the 50-item measure of C, followed by the workplace measure of
C (CG) and the Job Satisfaction questionnaire. After being debriefed,
participants were compensated with $2.00.
6.2. Study 2: Results
All participants resided in the United States and the majority were
Caucasian (80.4%). With the exception of two participants, all partici-
pants were high school graduates, and 34.1% of participants were
students at the time of the survey (9% part-time students). The average
reported untaxed income for this sample was $40,000 (SD = $26,000).
Approximately thirteen percent of participants were unemployed at the
time of taking the survey and subsequently reported work-related in-
formation for their most-recent job. Fifty-two percent of participants
identified with a clear left-wing political ideology with 25% of parti-
cipants identifying between left-wing and right-wing. Out of a possible
5 points, the Job Satisfaction score followed a normal distribution with
a slight negative skew (M = 3.5, SD = 0.75). Proactive Health score (z-
score based) was fairly normally distributed, with a slight negative
skew (M = −0.01, SD = 0.63).
Intercorrelations between the measures of C and outcome variables
are reported in Table 5. The correlation between uncontextualized C
and the workplace measure of C was substantially smaller than the
analogous correlation in Study 1 (r = 0.63 vs. r = 0.8). This finding is
unsurprising, considering that the workplace measure employed herein
was derived independently from the Big Five. All outcomes significantly
correlated with uncontextualized C, confirming past research asso-
ciating C with Job Satisfaction, Income, Conservatism, and Proactive
Health.
Again, the congruency hypothesis (H1) and the breadth hypothesis
(H2) were tested using K-fold cross validation regression analysis. Two
K-folds regressions were conducted for every outcome variable, one for
each contextualization condition – i.e., uncontextualized C and context-
congruent C. Uncontextualized regressions predicted outcome variables
(e.g., scored Job Satisfaction) from the uncontextualized measure of C;
context-congruent regressions predicted outcome variables from
workplace-C. Each K-fold regression split the data into 10 training and
testing folds and cross validation was repeated 100 times. Results of this
analysis are presented in Table 6.
In line with H1, contextualized predictive coefficients were only
larger for context-congruent outcome measures (i.e., Job Satisfaction
and Income). However, Hittner's test for dependent overlapping cor-
relations revealed only Job Satisfaction to be significantly greater for
the congruent measure (z = 1.94, p = .02) – while Income coefficients
were not significantly different (z = 0.15, p = .44).
V. Swift and J.B. Peterson
Table 6
K-fold based standardized correlation coefficients for Study 2.
Outcome Congruent General
Job satisfaction .37a .32
Income .17 .14
Conservatism .17 .20
Health .17 .26
Note. Standardized r values for context-congruent (Congruent) and un-
contextualized (General) measures of conscientiousness.
a
Congruent coefficient is significantly larger than uncontextualized coeffi-
cient (p < .05).
7. Study 3
In order to extend our findings beyond self-report measures, and
account for their potentially confounding effects, Study 3 considered
whether FOR effects emerge in non-self-report measures. Specifically,
this study tested whether FOR effects emerge in the prediction of trait-
relevant implicit attitudes, as measured by the Implicit Associations
Test (IAT). Additionally, Study 3 was designed to test whether FOR
effect emerge for other personality traits – namely, Extraversion (E) –
and under other contextualization conditions – i.e., instruction-based
contextualization.
The outcome variables for this study were derived from self-report
and IAT of Job Satisfaction. Although FOR effects were modest for Job
Satisfaction in both preceding studies, this outcome variable was
nevertheless favoured because it is a well-known correlate of both C
and E (Judge et al., 2002), that has been reliably measured through
implicit association tests (Boyd, 2010; Boyd, 2017).
7.1. Procedure
397 participants recruited through Amazon's Mechanical Turk
(MTurk) completed our survey. Twenty participants (5%) were ex-
cluded from analyses a priori on the basis of negligent responding, as
indicated by negligent response checks. Thus, the final sample for
analysis was comprised of 378 participants (50.3% female; Mage = 38.8,
SD = 11.2).
Participation was restricted to MTurk users who were using laptop
or desktop computers. Before commencing the study, participants were
asked to minimize distractions and noise and confirm a disturbance-free
environment. Participants who did not provide confirmation were di-
verted from the survey.
After providing consent to complete a variety of questionnaires and
“word-matching” tasks, participants were informed that despite ap-
parent repetition, the survey portions contained important contextual
differences. The survey began with a brief demographics questionnaire
(age, gender, nationality), followed by a block of workplace-con-
textualized personality items and a block of school-contextualized
items. Both contextualized blocks contained subsets of C and E items
which were randomized – the order of the subsets was also randomized.
These blocks were followed by a brief work-history questionnaire, and
an instruction-contextualized measure of workplace-C and E. Next,
participants completed a brief academic history questionnaire, followed
by an uncontextualized measure of the Big Five personality traits. The
uncontextualized measure was followed by a self-report measure of Job
Satisfaction. The intentional position of this block minimized transfer
effects from the workplace measures of personality.
Following the survey portion of the study, participants were in-
formed of the ensuing “word-matching” task and asked to compose a
brief open-ended essay describing their work. This essay exercise has
been used in other research to prime the target object (i.e., workplace)
in implicit association paradigms (Boyd, 2017), as it appears to increase
the relationship between implicit measures and behaviors (Perugini &
Prestwich, 2007). Finally, participants completed the IAT measure of
159
Personality and Individual Differences 144 (2019) 153–163
Job Satisfaction, were debriefed, and compensated with $4.00.
7.2. Measures
7.2.1. Uncontextualized Big Five measure
As in Study 1, the Big Five personality traits were assessed with an
unaltered version of the IPIP-NEO (100 items total, 20 items for C, 20
items for E).
7.2.2. Item-contextualized measures
These measures were identical to the contextualized measures used
in Study 1 with the exception that E items were now also contextualized
(following the Study 1 procedure). For instance, the E item “I make
friends easily”, was translated into the workplace item “I make friends
easily at work”, and the school item “I made friends easily at school”.
There were 20 items per contextualization condition (work or school)
for both E and C (80 items total).
7.2.3. Instruction-contextualized measures
E and C items from the IPIP-NEO Domain questionnaire (Goldberg
et al., 2006) were preceded with instructions that imposed a workplace
frame of reference (see Appendix A for details). This contextualization
approach has been used to successfully produce FOR effects elsewhere
(Bing et al., 2004; Hunthausen et al., 2003). Participants indicated their
agreement using a 5-point Likert scale (1 = strongly disagree,
5 = strongly agree).
7.2.4. Job satisfaction
Self-reported Job Satisfaction was assessed with 9 job dissatisfaction
(reverse coded) items from the Brayfield and Rothe (1951) Index.
Measuring dissatisfaction provided a contrast and extension to the Job
Satisfaction findings from Study's 1 and 2, which were based on scales
using positive items only. The scale included items such as “my job is
pretty uninteresting”, which were rated using a 5-point Likert scale
(1 = strongly disagree, 5 = strongly agree). This measure has been shown
to significantly correlate with implicit measures of Job Satisfaction
(Boyd, 2017).
7.2.5. Implicit job satisfaction (IAT-JS)
A single-target IAT was programmed and administered through
Inquisit (2016). The test was modelled after two existing validated IAT
measures of Job Satisfaction (Boyd, 2010; Boyd, 2017). In order to
maximize the relevance of the test, participants manually entered five
keywords pertaining to their current workplace role. These keywords
served as the targets for association (positive or negative). A detailed
explanation of this measure can be found in Appendix A.
7.3. Study 3: Results
All participants had completed a high-school degree, (7.4%) were
students at the time of the survey (3.4% part time). Thirty-three par-
ticipants (8.7%) were unemployed at the time of the survey, and five of
these participants had no employment experience. These participants
were excluded from any analyses based on workplace experience (i.e.,
Job Satisfaction, IAT, workplace-C, and workplace-E).
Before analyses were conducted, IAT scores were excluded on the
basis of keyword anomalies. Specifically, five participants provided
nonsensical keywords (e.g., “asdf”), and 53 participants provided in-
admissible entries – i.e., missing keywords or providing job evaluations
(e.g., fantastic) rather than job roles. Thus, all analyses involving IAT
scores utilized a restricted sample of 315 participants. IAT d scores –
which range from −1 (negative association) to 1 (positive association)
– were normally distributed around a mean of 0.16 (SD = 0.35), sug-
gesting a slight positivity bias in the sample. All self-report measures in
this study exhibited excellent internal consistency, as indicated by
Cronbach's alpha levels above 0.9 (see Table 7).
V. Swift and J.B. Peterson Personality and Individual Differences 144 (2019) 153–163

Table 7 Table 9
Scale parameters of all measures used in Study 3. K-fold based standardized correlation coefficients for Study 3.
Alpha Mean SD Model Congruent items Congruent Incongruent items General
instructions
Item-based Work C 0.94 4.39 0.58
Item-based School C 0.97 3.86 0.98 JS ~ E .47a,b .45a,b .29 .40
Item-based Work E 0.95 3.33 0.92 JS ~ C .54a,c .45a .35 .53
Item-based School E 0.97 3.06 1.10 IAT ~ E .24 .24 .21 .23
Instruction-based Work C 0.91 4.08 0.62 IAT ~ C .25 .21 .23 .24
Instruction-based Work E 0.94 3.36 0.83
Conscientiousness 0.95 4.10 0.72 Note. Standardized r values for item-level context-congruent (Congruent Items),
Extraversion 0.96 3.17 0.98 instruction-level context-congruent (Congruent Instructions), item-level con-
Job Satisfaction 0.91 3.76 0.96 text-congruent (Incongruent Items) and uncontextualized (General) trait mea-
sures. Models represent predictions of job satisfaction from extraversion
(JS ~ E) and conscientiousness (JS ~ C), and IAT measured job satisfaction
Correlations between all dependent and independent variables fol- from extraversion (IAT ~ E) and conscientiousness (IAT ~ C).
lowed the expected pattern (see Table 8). Uncontextualized C corre- a
Congruent coefficient is significantly larger than incongruent coefficient
lated strongly with item-based workplace-C (r = 0.78) and instruction- (p < .05).
b
based workplace-C (r = 0.79), which in turn correlated highly with one Congruent coefficient is significantly larger than uncontextualized coeffi-
another (r = 0.81). Likewise, uncontextualized E correlated strongly cient (p < .05).
c
with item-based workplace-E (r = 0.87) and instruction-based work- Item-level coefficient is significantly larger than instruction-level coeffi-
place-E (r = 0.85), which in turn correlated highly with one another cient (p < .05).
(r = 0.89). Self-rated Job Satisfaction correlated significantly with IAT
Job Satisfaction (r = 0.34, p < .001). 8. Discussion
Again, the congruency hypothesis (H1) was tested using K-fold cross
validation regression analysis. Sixteen K-folds regressions were con- Across three studies, we sought to determine whether FOR effects
ducted predicting either Job Satisfaction or IAT-JS from E or C for the 4 are broad and meaningful enough to justify the instrumental use of
contextualization conditions – i.e., uncontextualized, context-congruent contextualization. We hypothesized that FOR effects would emerge any
item based (workplace), context-congruent instruction based (work) time there is a congruency between the contexts of outcome and trait
and context incongruent C (school). Each K-fold regression split the measures (H1). This hypothesis found support across all three studies,
data into 10 training and testing folds, with cross validation repeated where trait-outcome correlations were found to be greater for all con-
100 times. text-congruent measures, compared to uncontextualized measures and
In line with H1, predictive coefficients were larger for all item-based context-incongruent measures. Because we considered a variety of
context-congruent measures. However, Hittner's test for dependent contexts (i.e., school, work, and relationships), and a wide variety of
overlapping correlations revealed only Job Satisfaction correlations to outcome measures, we have reason to believe that FOR effects are a
be significantly different (Table 9). Specifically, the correlation coeffi- broadly applicable phenomenon (H2). Moreover, this phenomenon
cient for item-based context-congruent E was significantly larger than appears to be independent of outcome-trait item equivalence (H3), as
the coefficient for uncontextualized E (z = 2.67, p = .004). The same uncontextualized and contextualized scales exhibit measurement in-
pattern was found for instruction-based context-congruent E (z = 2.07, variance (Study 1).
p = .019). While FOR effects can be broadly coerced by means of context-
Instruction-based context congruent measures did not consistently congruence, the utility of contextualization does not appear to be uni-
outperform uncontextualized measures, however, with such workplace- versal. That is to say, increases in effect size by means of con-
C variables being worse predictors of both IAT-SR and Job Satisfaction. textualization may be negligible, as we found in the predictions of four
Item-based context-congruent measures outperformed instruction- of the thirteen context-congruent outcomes we considered: Income,
based context-congruent measures on all predictions. Moreover, Relationship Performance, Relationship Vice, and IAT-JS. Moreover,
Hittner's test revealed this pattern to be significant in predictions of Job the magnitude of FOR effects appeared to be inconsistent for Job
Satisfaction from C (z = 3.27, p = .001). Satisfaction, which was measured across all three studies. This incon-
sistency suggests that FOR effects are not contingent on degree of
congruency alone. A likely source of this inconsistency appears to be

Table 8
Pearson's correlations between trait and outcome measures in Study 3.
CWork CSchool EWork ESchool Instruction CWork Instruction EWork C E JS

CSchool .43
EWork .36 .32
ESchool .25 .42 .60
Instruction CWork .81 .43 .30 .24
Instruction EWork .39 .35 .89 .61 .37
C .78 .52 .40 .32 .80 .47
E .33 .32 .86 .69 .30 .84 .38
JS .52 .32 .44 .23 .42 .43 .50 .37
IAT .19 .15⁎⁎ .16⁎⁎ .11a .11a .17⁎⁎ .18⁎⁎ .14⁎ .35

Note. Conscientiousness (C), Extraversion (E), work contextualized C (CWork), school contextualized C (CSchool), instruction-level work-contextualized C
(Instruction CWork), instruction-level work-contextualized E (Instruction EWork), self-reported job satisfaction (JS), IAT measured job satisfaction (IAT). Unless
otherwise indicated, all correlations are significant at p < .001.
a
ns.
⁎
p < .05.
⁎⁎
p < .01.

160
V. Swift and J.B. Peterson
range restriction, as FOR effects appeared to be largest when context-
congruent trait measures had the largest range. This is best exhibited in
Study 3 where the correlation between Job Satisfaction and personality
was significantly larger for workplace-E (min score = 1.2, max
score = 5), but not workplace-C (min score = 2.7, max score = 5).
Further research is needed to elucidate why FOR effects only emerge
in context-congruent models. In line with Identity Theory (Stryker,
1986), we posit that context-congruence aligns the demands and in-
centives implied in trait and outcome measures, thereby facilitating
overlap between trait and outcome scores. For instance, if role-specific
demands are low enough, and incentives high enough, a person who is
otherwise a slacker may be found to be more Conscientious (role-spe-
cific trait), and subsequently a high-performance worker (outcome).
Likewise, if role-specific demands are too high, or incentives too low, an
otherwise hardworking person may be found to be less Conscientious in
the working environment (role-specific trait), and as an outcome,
deemed a poor worker (Maddox & Markman, 2010). By relying on
uncontextualized measures of C, we can neglect information that is
crucial to role-specific performance – namely demands and incentives –
and thereby undermine predictions of role-related outcomes.
Although it is reasonable to assume that a Conscientious student
will perform well in the workplace, it is not clear that their
Conscientious behaviors will persist under changing demands and in-
centives, compared to a generally Conscientious person. This may ex-
plain why workplace-C is superior to traditional measures of C when
predicting workplace performance, while academic-C exhibits this
benefit only in the prediction of academic performance (Farmer, 1999;
Lievens et al., 2008; Seiler, 2011). Applied to our data, the demand/
incentive interpretation of FOR effects suggests, for instance, that aca-
demic-C exclusively improved the prediction of school satisfaction be-
cause effort (i.e., C) at school and satisfaction at school both depend on
the demands and incentives of the classroom. This explanation can be
tested in future experiments by manipulating demands and incentives
and subsequently testing for FOR effects.
If FOR effects arise when the demands and incentives implied in
trait and outcome measures are aligned, any factors that disturb this
alignment may reduce FOR effects. This idea may explain the negligible
FOR effects we uncovered in the prediction of relationship perfor-
mance/vices. For instance, the social desirability of relationship fidelity
may have caused participants to downplay their relationship vices and
thereby withhold crucial information regarding relationship demands
and incentives that would otherwise be shared with the contextualized
trait measure. Likewise, implicit Job Satisfaction may have been asso-
ciated with negligible FOR effects because implicit attitudes are more
strongly based on affective tendencies and cognitive biases (Rudman,
2004) than they are based on demands and incentives. Moreover, FOR
effects may be stronger following item-level contextualization than
instruction-level contextualization – as found in Study 3, and preceding
research (Lievens et al., 2008) – because contextualized items restrict
item interpretations (Bing et al., 2014) and thereby encode more role-
specific information.
Likewise, our demand/incentive theory can be used to explain the
largest FOR effects, which should arise when demands and incentives
are most tightly aligned for predictor and outcome variables. In other
words, the more role-specific information an outcome variable encodes,
the more beneficial it is to include role-specific information in the
predictor variable. The largest FOR effects across all three studies were
for the prediction of school performance (GPA) and school vice (pro-
crastination). For both of these variables, school-specific C explained
34% more variance than general C. Whereas GPA was based on a
unidimensional self-report of performance evaluation history, procras-
tination scores were based on a multi-dimensional self-report of work
attitudes and tendencies. Nevertheless, both measures encode the same
demands (e.g., parental expectations) and incentives (e.g., academic
success), and appear to do so with minimal error. These same demands
and incentives are clearly implicated in the school-specific measure of
161
Personality and Individual Differences 144 (2019) 153–163
C, which was based on a multi-dimensional self- report of work atti-
tudes and tendencies. Thus, despite being structurally most similar to
the procrastination measure, the school-specific measure of C corre-
lated equally highly with GPA, presumably due to a similar overlap in
demands and incentives.
A major theoretical implication of our demand/incentive theory is
that overlap in demands and incentives contributes to the overall cri-
terion related validity of all uncontextualized measures. That is to say,
uncontextualized measures inherently encode some information about
contextual and cross-contextual demands and incentives, and the de-
gree to which an outcome variable jointly encodes these demands/in-
centives contributes to the magnitude of association. Thus, C may be
reliably found to predict job satisfaction, job performance, and work-
place deviance, because uncontextualized measures of C inherently
encode some information regarding workplace demands and incentives
– while contextualized scales encode this information better.
Although we found broad support for our hypotheses, our findings
were nevertheless subject to some limitations. Firstly, generalizations
from our data may be impeded by our decision to use the online MTurk
platform to conduct this study. However, it is important to note that
data provided by MTurk users does not appear to differ meaningfully
from community samples in terms of validity, or in terms of personality
(McCredie & Morey, 2018), which were the most important sample
characteristics for our purposes. Similarly, although we have found
evidence for broad contextualization effects for a number of diverse
outcome measures, we did not consider task-based or observational
outcome measures. While previous research has demonstrated con-
textualization effects for these kinds of measures – both task-based
(Holtrop, Born, de Vries, & de Vries, 2014) and observational (Bing
et al., 2014) – further research is needed in order to test the applic-
ability of our demand/incentive theory with these various measures in
a variety of contexts.
Finally, although our congruence guideline is purported to apply to
any contextualized measures, herein we have only tested our guideline
using measures of C and E. As such, it is possible that future research
using other traits or constructs may find that FOR effects do not emerge
under alternative conditions. Logically, however, our FOR guideline is
not unique to C or E, as context-congruence is applicable to any con-
struct that can be contextualized.
9. Conclusions
The confirmation of our hypotheses has implications for any do-
mains that make use of personality measurement. In practical domains,
for instance, clinicians who use personality scores to predict therapeutic
outcomes may generate more accurate predictions by contextualizing
trait measures to the clinical setting. Simply by specifying a context for
a personality measure, the predictive validity of context-congruent
outcomes can be significantly and meaningfully increased by up to
30%. The likelihood of enhancing predictive validity may be increased
by utilizing item-level (over instruction-level) contextualization, redu-
cing range restriction and response biases, and maximizing overlap of
demands and incentives encoded by predictor and outcome variables.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect
to the research, authorship, or publication of this article.
Acknowledgments
Preparation of this manuscript was supported by Grant 533803 from
the Social Sciences and Humanities Research Council of Canada.
V. Swift and J.B. Peterson Personality and Individual Differences 144 (2019) 153–163

Appendix A 12. Make plans to finish work and stick to them

13. Am always prepared for projects at work
20 IPIP-NEO Conscientiousness Items Contextualized for 14. Need a push to start projects at work
Relationship, School, and Workplace Settings. 15. Don't pay attention during work hours
The following characteristics describe how people act in relationships. 16. Do just enough work to get by at work
Please indicate how much you personally agree with each statement. 17. Make a mess of things at work
18. Complete workplace tasks successfully
1. Finish what I start for my partner 19. Do things according to a plan at work
2. Leave things unfinished with my partner 20. Don't see things through at work
3. Need a push to start things with my partner
4. Make plans to help my partner and stick to them Instruction-Based Contextualization.
5. Do just enough work to get by with my partner “We would like to ask you some more questions about your experiences
6. Am always prepared for anniversaries in the workplace. The following are traits that may be expressed in the
7. Waste my time with my partner workplace. Please take a moment to think about what you are like in the
8. Am exacting in my relationship efforts workplace, and indicate how well each statement describes your workplace
9. Make a mess of things with my partner personality.”
10. Carry out my plans to help my partner
11. Complete partner mandated chores successfully 1. Carry out my plans
12. Do things according to a plan for my relationship 2. Finish what I start
13. Follow through with my plans for my partner 3. …
14. Don't pay attention when I am with my partner
15. Mess things up in my relationships Implicit Job Satisfaction (IAT-JS).
16. Find it difficult to start things for my partner A single-target IAT was programmed and administered through
17. Get favors for my partner done right away Inquisit (2016). The test was modelled after two existing validated IAT
18. Don't see things through with my partner measures of Job Satisfaction (Boyd, 2010; Boyd, 2017). In order to
19. Pay attention to details when my partner is talking maximize the relevance of the test, participants manually entered five
20. Shirk my relationship duties keywords pertaining to their current workplace role. These keywords
served as the targets for association (positive or negative).
The following characteristics describe how people act at school. Please The task consisted of five trials in which participants were required
indicate how much you personally agree with each statement. to quickly pair their target words (e.g., “data entry”), positive words
(e.g., “great”) or negative words (e.g., “miserable”) with the labels
1. Made a mess of things at school “good or work” or “bad or work” by pressing the “e” or “i” key as
2. Didn't see things through at school quickly as possible. Incorrect associations were demarcated with a red
3. Got my school work done right away “x”. On all trails, the stimuli were presented in the middle of the screen
4. Made plans to finish school work and stuck to them and the sorting labels (“good” or “bad”) were located in the top right
5. Wasted my time at school and left of the screen, respectively. On trial 1, participants were in-
6. Paid attention to details when I was doing school work troduced to the matching procedure using only the positive and nega-
7. Found it difficult to start school projects tive keywords. The subsequent trail pairs (2 and 3, and 4 and 5) were
8. Finished what I started at school counter balanced and began with a practice set (i.e., trials 2 and 3). In
9. Needed a push to start school projects or homework one trial pair, participants were required to correctly match target,
10. Did things according to a plan at school positive, and negative words with the labels “good or work” (left side)
11. Followed through with my plans at school and “bad” (right side). In the remaining trial pair, and participants had
12. Was always prepared for projects at school to match words with the labels “good” (left side) and “bad or work”
13. Did just enough work to get by at school (right side).
14. Completed academic tasks successfully
15. Didn't pay attention during school hours References
16. Shirked my academic duties
17. Left things unfinished at school Ahmetoglu, G., Swami, V., & Chamorro-Premuzic, T. (2010). The relationship between
18. Messed things up at school dimensions of love, personality, and relationship length. Archives of Sexual Behavior,
39, 1181–1190.
19. Carried out my plans at school Ajzen, I. (1987). Attitudes, traits and actions: Dispositional prediction of behavior in
20. Was exacting in my efforts at school personality and social psychology. In L. Berkowitz (Ed.). Advances in experimental and
social psychology (pp. 1–63). New York: Academic Press.
Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job
The following characteristics describe how people act at work. Please performance: A meta-analysis. Personnel Psychology, 44, 1–26.
indicate how much you personally agree with each statement. Beaujean, A. A. (2014). Latent variable modeling using R: A step-by-step guide. New York,
NY: Routledge/Taylor & Francis Group.
Bennett, R. J., & Robinson, S. L. (2000). Development of a measure of workplace de-
1. Waste my time at work viance. Journal of Applied Psychology, 85, 349–360.
2. Am exacting in my job at work Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational
3. Get my work done right away deviance, and their common correlates: A review and meta-analysis. Journal of
Applied Psychology, 92, 410–424.
4. Mess things up at work
Bing, M. N., Davison, H. K., & Smothers, J. (2014). Item-level frame-of-reference effects in
5. Leave things unfinished at work personality testing: An investigation of incremental validity in an organizational
6. Carry out my plans at work setting. International Journal of Selection and Assessment, 22, 165–178.
7. Find it difficult to start projects at work Bing, M. N., Whanger, J. C., Davison, H. K., & VanHook, J. B. (2004). Incremental validity
of the frame-of-reference effect in personality scale scores: A replication and exten-
8. Shirk my duties at work sion. Journal of Applied Psychology, 89, 150–157.
9. Pay attention to details when I am working Bogg, T., & Roberts, B. W. (2004). Conscientiousness and health-related behaviours: A
10. Follow through with my plans at work meta-analysis of the leading behavioural contributors to mortality. Psychological
Bulletin, 130, 887–919.
11. Finish what I start at work

162
V. Swift and J.B. Peterson
Boyd, B. (2010). An implicit measure of job satisfaction (Unpublished Master's Thesis)New
York, NY: Baruch College.
Boyd, B. (2017). Personalizing an implicit measure of job satisfaction (Order No. 10265612).
(Available from ProQuest Dissertations & Theses Global. (1898765668)).
Brayfield, A. H., & Rothe, H. F. (1951). An index of job satisfaction. Journal of Applied
Psychology, 35, 307–311.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new
source of inexpensive, yet high-quality, data. Perspectives on Psychological Science,
6, 3–5.
Carlos, V. S., & Rodrigues, R. G. (2016). Development and validation of a self-reported
measure of job performance. Social Indicators Research, 126, 279–307.
Ching, C. M., Church, A. T., Katigbak, M. S., Locke, K. D., de Vargas-Flores, J., Ibáñez-
Reyes, J., ... Roslan, N. A. (2013). Cross-cultural generalizability of the Personality
and Role Identity Structural Model (PRISM): Implications for trait and cultural psy-
chology. Journal of Research in Personality, 47, 894–907.
Conner, M., & Abraham, C. (2001). Conscientiousness and the theory of planned beha-
vior: Toward a more complete model of the antecedents of intentions and behavior.
Personality and Social Psychology Bulletin, 27, 1547–1561.
Consiglio, C., Alessandri, G., Borgogni, L., & Piccolo, R. F. (2013). Framing work com-
petencies through personality traits: The Big Five competencies grid. European
Journal of Psychological Assessment, 29, 162–170.
Cook, M., Young, A., Taylor, D., & Bedford, A. (1998). Personality correlates of alcohol
consumption. Personality and Individual Differences, 24, 641–647.
Davie, J. W. (2005). The problem of relevance (order no. MR09762). (Available from
ProQuest Dissertations & Theses Global. (304990472)).
DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007). Between facets and domains: 10
aspects of the Big Five. Journal of Personality and Social Psychology, 93, 880–896.
Farmer, S. J. (1999). Enhancing criterion-related validity by assessing context-specific traits:
Implications of conditional dispositions for personnel selection (9947493). (Available
from ProQuest Dissertations & Theses Global. (304597555)).
Goldberg, L. R. (2017, February 22). Index of 274 labels for 463 IPIP scales. Retrieved
from: http://ipip.ori.org/newIndexofScaleLabels.htm.
Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., &
Gough, H. C. (2006). The International Personality Item Pool and the future of public-
domain personality measures. Journal of Research in Personality, 40, 84–96.
Hayes, N., & Joseph, S. (2003). Big 5 correlates of three measures of subjective well-
being. Personality and Individual Differences, 34, 723–727.
Heidemeier, H. (2005). Self and supervisor ratings of job-performance: Meta-analyses and a
process model of rater convergence. Universität Erlangen-Nürnberg. Retrieved from
https://opus4.kobv.de/opus4fau/frontdoor/deliver/index/docId/143/file/
Heidemeier_Diss.
Heller, D., Ferris, D., Brown, D., & Watson, D. (2009). The influence of work personality
on job satisfaction: Incremental validity and mediation effects. Journal of Personality,
77(4), 1051–1084.
Hendrick, S. S. (1988). A generic measure of relationship satisfaction. Journal of Marriage
and the Family, 50, 93–98.
Hirschfeld, G., & von Brachel, R. (2014). Multiple-group confirmatory factor analysis in
R—A tutorial in measurement invariance with continuous and ordinal indicators.
Practical Assessment, Research & Evaluation, 19(7), 12.
Hittner, J. B., May, K., & Silver, N. C. (2003). A Monte Carlo evaluation of tests for
comparing dependent correlations. The Journal of General Psychology, 130, 149–168.
Holtrop, D., Born, M. P., de Vries, A., & de Vries, R. E. (2014). A matter of context: A
comparison of two types of contextualized personality measures. Personality and
Individual Differences, 68, 234–240.
Holtrop, D., Born, M. P., & de Vries, R. E. (2014). Predicting performance with con-
textualized inventories, no frame-of-reference effect? International Journal of Selection
and Assessment, 22, 219–223.
Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to
underparameterized model misspecification. Psychological Methods, 3(4), 424–453.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure
analysis: Conventional criteria versus new alternatives. Structural Equation Modeling:
A Multidisciplinary Journal, 6(1), 1–55.
Hunthausen, J. M., Truxillo, D. M., Bauer, T. B., & Hammer, B. L. (2003). A field study of
frame-of-reference effects on personality test validity. Journal of Applied Psychology,
88, 545–551.
Inquisit 5 [Computer software] (2016). Retrieved from https://www.millisecond.com.
Johnson, J. A. (2014). Measuring thirty facets of the Five-factor model with a 120-item
public domain inventory: Development of the IPIP-NEO-120. Journal of Research in
Personality, 51, 78–89.
Judge, T. A., Heller, D., & Mount, M. K. (2002). Five-factor model of personality and job
satisfaction: A meta-analysis. Journal of Applied Psychology, 87, 530–541.
Judge, T. A., Higgins, C. A., Thoresen, C. J., & Barrick, M. R. (1999). The Big Five per-
sonality traits, general mental ability, and career success across the life span.
Personnel Psychology, 52, 621–652.
Lee, K., & Ashton, M. C. (2004). Psychometric properties of the HEXACO personality
inventory. Multivariate Behavioral Research, 39, 329–358.
Lemieux, R., & Hale, J. L. (1999). Intimacy, passion, and commitment in young romantic
relationships: Successfully measuring the triangular theory of love. Psychological
Reports, 85, 497–503.
Lievens, F., De Corte, W., & Schollaert, E. (2008). A closer look at the frame-of-reference
effect in personality scale scores and validity. Journal of Applied Psychology, 93,
268–279.
163
View publication stats
Personality and Individual Differences 144 (2019) 153–163
Liu, M. (2014). Contextualized extraversion and its change in cross-cultural adjustment
(1564334). (Available from ProQuest Dissertations & Theses Global. (1614532098)).
Maddox, W. T., & Markman, A. B. (2010). The motivation-cognition interface in learning
and decision making. Current Directions in Psychological Science, 19, 106–110.
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical
Turk. Behavior Research Methods, 44, 1–23.
McCredie, M. N., & Morey (2018). Who are the Turkers? A characterization of MTurk
workers using the personality assessment inventory. Assessment.
McNichols, C. W., Stahl, M. J., & Manley, T. R. (1978). A validation of Hoppock's job
satisfaction measure. Academy of Management Journal, 21, 737.
Nicholls, J. G., Licht, B. G., & Pearl, R. A. (1982). Some dangers of using personality
questionnaires to study personality. Psychological Bulletin, 92(3), 572–580.
Ozer, D. J., & Benet-Martínez, V. (2006). Personality and the prediction of consequential
outcomes. Annual Review of Psychology, 57, 401–421.
Pace, V. L., & Brannick, M. T. (2010). Improving prediction of work performance through
frame-of-reference consistency: Empirical evidence using openness to experience.
International Journal of Selection and Assessment, 18, 230–235.
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a
participant pool. Current Directions in Psychological Science, 23, 184–188.
Paunonen, S. V., & Ashton, M. C. (2001). Big five factors and facets and the prediction of
behavior. Journal of Personality and Social Psychology, 81(3), 524–539.
Perugini, M., & Prestwich, A. (2007). The gatekeeper: Individual differences are keys in
the chain from perception to behaviour. European Journal of Personality, 21, 303–317.
Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and aca-
demic performance. Psychological Bulletin, 135, 322–338.
Roberts, B. W., Jackson, J. J., Fayard, J. V., Edmonds, G., & Meints, J. (2009).
Conscientiousness. In M. Leary, & R. Hoyle (Eds.). Handbook of individual differences in
social behavior (pp. 369–381). New York, NY: Guilford Press.
Roenicke, C. C. (2013). Extending the frame of reference effect beyond conscientiousness
(3564859). (Available from ProQuest Dissertations & Theses Global. (1410336008)).
Rudman, L. A. (2004). Sources of implicit attitudes. Current Directions in Psychological
Science, 13(2), 79–82.
Schaffhuser, K., Allemand, M., & Martin, M. (2014). Personality traits and relationship
satisfaction in intimate couples: Three perspectives on personality. European Journal
of Personality, 28, 120–133.
Schmit, M. J., Ryan, A. M., Stierwalt, S. L., & Powell, A. B. (1995). Frame-of-reference
effects on personality scale scores and criterion-related validity. Journal of Applied
Psychology, 80, 607–620.
Schmitt, D. P. (2004). The Big Five related to risky sexual behaviour across 10 world
regions: Differential personality associations of sexual promiscuity and relationship
infidelity. European Journal of Personality, 18, 301–319.
Schmitt, D. P., & Buss, D. M. (2000). Sexual dimensions of person description: Beyond or
subsumed by the Big Five? Journal of Research in Personality, 34, 141–177.
Schmitt, N. (2014). Personality and cognitive ability as predictors of effective perfor-
mance at work. Annual Review of Organizational Psychology and Organizational
Behavior, 1, 45–65.
Seiler, S. N. (2011). The impact of item characteristics on contextualized personality assess-
ment (3503898). (Available from ProQuest Dissertations & Theses Global.
(1010625066)).
Shaffer, J. A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic in-
vestigation of the relative validity of contextualized and uncontextualized personality
measures. Personnel Psychology, 65, 445–494.
Sibley, C. G., Osborne, D., & Duckitt, J. (2012). Personality and political orientation:
Meta-analysis and test of a threat-constraint model. Journal of Research in Personality,
46, 664–677.
Slatcher, R. B., & Vazire, S. (2009). Effects of global and contextualized personality on
relationship satisfaction. Journal of Research in Personality, 43, 624–633.
Steel, P. (2007). The nature of procrastination: A meta-analytic and theoretical review of
quintessential self-regulatory failure. Psychological Bulletin, 133, 65–94.
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological
Bulletin, 87, 245–251.
Stryker, S. (1986). Identity theory: Developments and extensions. In K. Yardley, & T.
Honess (Eds.). Self and identity (pp. 89–104). New York: Wiley.
Takahashi, Y., Edmonds, G. W., Jackson, J. J., & Roberts, B. W. (2013). Longitudinal
correlated changes in conscientiousness, preventative health-related behaviors, and
self-perceived physical health. Journal of Personality, 81, 417–427.
Vispoel, W. P., Morris, C. A., & Kilinc, M. (2018). Using generalizability theory to dis-
attenuate correlation coefficients for multiple sources of measurement error.
Multivariate Behavioral Research, 1–21.
Weber, M., & Huebner, E. S. (2015). Early adolescents' personality and life satisfaction: A
closer look at global vs. domain-specific satisfaction. Personality and Individual
Differences, 83, 31–36.
Wood, D. (2007). Using the PRISM to compare the explanatory value of general and role-
contextualized trait ratings. Journal of Personality, 75, 1103–1126.
Wood, D., & Roberts, B. W. (2006). Cross-sectional and longitudinal tests of the
Personality and Role Identity Structural Model (PRISM). Journal of Personality, 74,
779–809.
Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology:
Lessons from machine learning. Perspectives on Psychological Science, 12(6),
1100–1122.
Yockey, R. D. (2016). Validation of the short form of the academic procrastination scale.
Psychological Reports, 118, 171–179.