Journal of Contextual Behavioral Science 6 (2017) 8090

Contents lists available at ScienceDirect

Journal of Contextual Behavioral Science

journal homepage: www.elsevier.com/locate/jcbs

Empirical research

Examining gender-STEM bias among STEM and non-STEM students using MARK
the Implicit Relational Assessment Procedure (IRAP)
Lynn Farrell, Louise McHugh
School of Psychology, University College Dublin, Beleld, Dublin 4, Ireland

A R T I C L E I N F O A BS T RAC T

Keywords: Women remain under-represented in the elds of Science, Technology, Engineering and Maths (STEM),
Gender-STEM bias constituting only 28% of science researchers worldwide. Research has identied implicit gender-bias as a major
IRAP barrier to women's progression in these elds. Previous research using the Implicit Association Test (IAT)
Implicit cognition suggests that individuals studying or working in STEM elds exhibit dierent levels of implicit male-STEM bias
than those in non-STEM elds. The Implicit Relational Assessment Procedure (IRAP), a non-relative measure,
was compared with the IAT which allowed us to probe this dierence further among STEM and non-STEM
students. The IRAP revealed a more detailed and complex picture of gender-STEM bias. All groups
demonstrated a signicant implicit pro-male-STEM bias. However, there was also evidence of a pro-female-
STEM bias, signicant only among female STEM students. A number of correlations were observed between the
explicit measure and the IRAP. The presence of a pro-female-STEM bias has implications for the development
of interventions. If this relational response can become more normative it may inuence attitudes and behavior
towards women in STEM.

1. Introduction
The number of women employed within the elds of Science,
Technology, Engineering and Maths (STEM) remains low despite
numerous eorts to recruit and retain women in these areas (e.g.
Fisher & Margolis, 2002; see Schiebinger, 2010; see Cheryan, Master,
& Meltzo, 2015). Women constitute < 25% of the STEM workforce in
Ireland (Accenture, 2014) and only 28% of science researchers world-
wide (UNESCO, 2015). Addressing this underrepresentation is a key
concern of a number of initiatives and would increase opportunities for
women and enhance economic growth (Gatta & Trigg, 2001; Global
Gender Gap Report, 2015), promoting increased creativity, innovation
and competiveness within the STEM workforce (Hill, Corbett & St.
Rose, 2010).
The evidence that biological sex dierences lead to women's under-
representation in STEM is inconclusive (Ceci, Williams & Barnett,
2009), with previous research instead identifying systematic gender
bias (often implicit) to be the greatest obstacle to STEM gender equality
(National Academy of Sciences, 2006; see Carnes et al., 2012). Gender,
distinct from biological sex, is a social structure (see Risman, 2004)
with implications for individuals in terms of gender roles (e.g. Eagly &
Karau, 2002) and prescriptive gender stereotypes (e.g. Prentice &
Carranza, 2002). Gender stereotypes suggest that men are naturally
more suited to STEM subjects with stereotypically masculine traits,
such as agency, linked to STEM achievement (see Gatta & Trigg,
2001). Traditionally, Western culture views women as more communal
(Eagly & Karau, 2002). Males and scientists, on the other hand, are
perceived to be highly agentic (Carli, Alawa, Lee, Zhao, & Kim, 2016)
and STEM careers are often stereotyped as being incompatible with the
communal goals commonly valued most by women (Diekman, Clark,
Johnston, Brown, & Steinberg, 2011). There is also the belief that
STEM careers are only for the brainiest students, particularly middle-
class males (ASPIRES, 2013). At university level women experience a
greater loss of condence in their intelligence (Faulkner, 2006; Gatta
& Trigg, 2001) and under-estimate their ability to do well in STEM
subjects (e.g. Ehrlinger & Dunning, 2003). These STEM and gender
stereotypes interact in ways that limit and disadvantage women
interested in STEM (Hill, Corbett, & St Rose, 2010; see ScienceGrrl,
2014). When STEM is portrayed as highly masculine or stereotypical
(e.g. the socially awkward computer nerd; Margolis & Fisher, 2003)
women experience less feelings of belonging and interest, and may also
feel less able to succeed in STEM elds (e.g. Cheryan, Plaut, Davies, &
Steele, 2009; Cheryan, Siy, Vichayapi, Drury & Kim, 2011).
Exposure to gender-STEM stereotypes can lead to attitudinal biases
and can even negatively inuence women's performance on STEM
related tasks, as the substantial literature on stereotype threat demon-
strates (see Nguyen & Ryan, 2008 for a meta-analysis). Research on
stereotype threat has found that negative self-relevant stereotypes can

E-mail address: lynn.farrell@ucdconnect.ie (L. Farrell).

http://dx.doi.org/10.1016/j.jcbs.2017.02.001
Received 27 August 2016; Received in revised form 25 January 2017; Accepted 2 February 2017
2212-1447/ 2017 Association for Contextual Behavioral Science. Published by Elsevier Inc. All rights reserved.
L. Farrell, L. McHugh
cause anxiety or discomfort which may elicit stereotype conrming
behavior such as a woman performing poorly on a maths exam (e.g.
Spencer, Steele, & Quinn, 1999). This gender-STEM bias is measur-
able at the implicit level. Nosek et al. (2009) found evidence of a
pervasive implicit gender-science stereotype across 34 countries asso-
ciating men with Science and women with Arts subjects on the Implicit
Association Test (IAT; Greenwald, McGhee, & Schwartz, 1998).
Implicit bias can predict behavior better than explicit attitudes,
particularly with socially sensitive domains (Power, 2010; Greenwald,
Poehlman, Uhlmann, & Banaji, 2009). Implicit bias favoring males has
previously been demonstrated in hiring-related behavior such as
appraisals of applicants for STEM positions (Moss-Racusin, Dovidio,
Brescoll, Graham, & Handelsman, 2012; Steinpreis, Anders & Ritzke,
1999) or mathematical tasks (Reuben, Sapienza, & Zingales, 2014)
and letters of recommendation (Schmader, Whitehead, & Wysocki,
2007; Trix & Psenka, 2003). Thus to gain a full appreciation of the
nature and prevalence of gender-STEM bias one must examine it both
implicitly and explicitly.
The majority of implicit research examining this bias has involved
indirect measures that are relative, such as the IAT. The IAT involves
categorizing words together in a manner compatible with social
stereotypes (e.g. male and Science words) or incompatible (e.g. female
and Science words). However, the IAT may be unable to capture the full
complexity of the implicit bias being examined. For example, an IAT
demonstrating a weak male-science/female-arts bias cannot probe the
directionality of this bias any further; it cannot tell us whether a pro-
female-science bias is present that may explain this weaker male-
science bias. The IAT also deals with associations only and so it cannot
examine more complex relations which may enhance our understand-
ing of a particular bias. In addition to this, the standard IAT, which is
most commonly used particularly in the domain of gender and STEM,
cannot examine attitudes towards STEM separately, often comparing
STEM subjects with Arts subjects. Finally, there are concerns that a
score of zero on the IAT may not reect a true neutral point (Blanton
& Jaccard, 2006; cf Greenwald, Nosek, & Sriram, 2006) and that it's
predictive validity may be lower than was previously reported (Oswald,
Mitchell, Blanton, Jaccard, & Tetlock, 2013).
An alternative measure is the Implicit Relational Assessment
Procedure (IRAP; Barnes-Holmes et al., 2006) which nds its theore-
tical basis in Relational Frame Theory (RFT; Hayes, Barnes -Holmes,
& Roche, 2001), a contemporary behavioral approach to human
language and cognition. According to RFT stimulus relations are of
key importance. Humans learn to relate stimuli rst on the basis of
non-arbitrary properties of the stimuli (e.g., size) before developing the
crucial ability to relate stimuli in an arbitrarily applicable manner
based on contextual cues, such as for example, the word is (see Hayes
et al., 2001 for a more detailed discussion of RFT). For example, if
someone tells us that a strange object we have not seen before is a
chair then is cues us to relate the chair and name bidirectionally (see
Trneke, 2010 for a detailed discussion). While the cognitive tradition
behind the IAT conceptualizes implicit attitudes as associations
between concepts, RFT conceptualizes implicit and explicit responding
as examples of arbitrarily applicable relational responding that vary
along a continuum (Hughes, Barnes-Holmes, & Vahey, 2012). Implicit
responses represent brief, immediate relational responses (BIRRs) that
occur relatively quickly with lower levels of relational complexity and
derivation (i.e. it has previously been derived often) whereas explicit
responses are extended, elaborated relational responses (EERRs)
involving higher levels of complexity and derivation requiring addi-
tional time to be emitted (see Hughes et al., 2012 for a more detailed
discussion).
One of the main advantages of the IRAP is that it is non-relative and
can explore more complex relations, thus aording researchers greater
detail regarding the nature of verbal relations and bias. A recent IRAP
study (Farrell, Cochrane & McHugh, 2015) found females held a
signicant pro-Male-Science bias while the male group's bias failed to
81
Journal of Contextual Behavioral Science 6 (2017) 8090
reach signicance. However, both groups did not exhibit the anti-
Female-Science bias that one might have expected. Instead, the
directionality of both males and females responding was indicative
of a pro-Female-Science bias (though weak and non-signicant) that
was not observable using the IAT alone. A recent IAT paper by Smyth
and Nosek (2015) examined gender-STEM bias among college-edu-
cated participants, with a particular focus on those majoring or
working in STEM domains. Males in STEM were found to have
stronger a Male-Science bias than STEM females and non-STEM
males, while females in STEM held a weaker Male-Science bias than
non-STEM females and STEM males. The IAT could not probe this
dierence further, as noted within the study (Smyth & Nosek, 2015).
For example would the presence of a positive Female-Science bias
explain the lower Male-Science bias found with the IAT? This ner
grained analysis could inuence how this bias is understood and
approached in terms of potential interventions.
2. The present study
The current study expanded upon the study by Farrell et al. (2015),
further exploring gender-STEM1 bias among key student groups,
namely STEM and non-STEM students. The previous study was the
rst published study to use the IRAP to examine gender-STEM bias
and was exploratory. Further investigation was warranted to enhance
our understanding of the level and prevalence of gender-STEM bias
among more specic populations. University students are at an
important pre-career stage. Therefore, achieving a greater understand-
ing of their bias could aid the development of bias interventions with
the potential to inuence students career choices and gender-STEM
attitudes going forward into the workplace. Previous large-scale studies
suggest there is a dierence in the levels of gender-STEM bias between
STEM and non-STEM students (Nosek & Smyth, 2011; Smeding,
2012; Smyth & Nosek, 2015) and between males and females within
STEM (Smyth & Nosek, 2015). These studies were conducted using
the IAT which as discussed, is a relative measure, therefore it is unclear
what components of this bias are underlying or driving this dierence.
The current study analytically separated the components of a male-
STEM/female-Arts bias by utilizing the 4 trial types of the IRAP (i.e.
men-STEM, men-Arts, women-STEM and women-Arts) to examine
levels of gender-based STEM and Arts bias separately among gender by
STEM status (i.e. STEM or non-STEM) student groups. This allowed us
a clearer picture of what may be driving the dierential levels of bias
found previously among these groups using the IAT. Our use of the IAT
in conjunction with the IRAP allowed us to determine whether we
replicated the ndings of previous IAT studies (e.g. Smeding, 2012)
and how this bias then manifested through our gender-STEM IRAP.
We hypothesized that there would be a dierence in levels of gender-
STEM bias between the participant groups. On the basis of previous
research, we expected that STEM males would have stronger male-
STEM/female-Arts bias than STEM females and non-STEM males
while STEM females would have weaker male-STEM/female-Arts bias
than STEM males and non-STEM-females. It was less clear exactly how
this bias would manifest on the remaining IRAP trial types, however,
due to the results of the previous IRAP study on this topic (Farrell
et al., 2015) we expected the STEM-relevant IRAP trials would be of
particular interest with the potential to nd evidence for a pro-Female-
STEM bias (at least directionally) in addition to a pro-Male-STEM bias.
The aims of the study therefore were: (i) to examine if there was a
signicant gender-STEM bias for each of the participant groups and
observe how this manifested on the IRAP and (ii) to compare levels of
gender-STEM bias between the groups.
1
While gender-science or male-science bias have previously been used to refer to this
bias we have chosen the term gender-STEM bias to acknowledge that our implicit and
explicit stimuli (e.g. maths, physics) target bias related to STEM elds as a whole not
just science.
L. Farrell, L. McHugh
Table 1
Stimuli used for the IAT and IRAP.
Science Arts Male
Science Arts Boy
Maths English His
Physics Drama He
Chemistry Music Him
Computing French Mr
Engineering History Men
The label stimuli of the current gender-STEM IRAP (Men/Women
more suited to) were subtler than in the previous study (see Farrell
et al., 2015). This was in response to concerns that the previous label
stimuli Men/Women better at may have activated a form of stereo-
type threat for the female participants which could have inuenced
their responses, particularly as it has recently been shown that the
IRAP is not a-contextual (Hussey et al., 2016). The choice of label
statement in the IRAP can exert contextual control within the task over
responding towards the contrasting label statement (e.g. Men more
suited to versus Women more suited to). This contextual variable
oers the potential for increasingly detailed analyses of topics by
intentionally targeting specic relations (Hussey et al., 2016). The new
label statement used in the current study (Men/Women more suited to)
tapped into an important element of the bias found within the
literature the suggestion that men are naturally more suited to
science through biology as well as ability (e.g. see Gatta & Trigg,
2001). Due to this change of label statement both the IRAP and IAT
were utilized in order to compare responding on both measures. This
allowed us to ensure that both measures were targeting a similar
concept, namely gender-STEM bias, albeit with diering levels of
specicity due to their procedural dierences. It also allowed us to
examine participants responding on both the IAT and the IRAP giving
us a picture of how this bias manifested on both measures. For
example, how would a lower male-STEM/female-Arts bias manifest
in the 4 IRAP trial types?
3. Method
Ethical approval was obtained from the host institute's Humanities
Research Ethics committee with informed consent obtained from all
individual participants.
3.1. Participants
In an eort to keep each participant group as balanced as possible a
stratied, volunteer sampling method was utilized with the aim to
recruit at least 15 participants per group. Sixty-four participants took
part in the study however only sixty-one participants were eligible for
inclusion in the analysis due to the test requirements of the IRAP and
IAT which will be detailed below. The age range was 1838 years (M
=22.5; SD =5.3). Two participants failed to indicate their age. A
requirement for participation however was to be age 18 or above and
the two students who did not indicate their age were non-mature
undergraduates and therefore would have fallen within the above
range. Participants consisted of STEM (N1 =31) and non-STEM (N2
=30)2 students from the host institute, 30 of which identied their
gender as female and 31 as male. Participants were able to select a non-
binary gender identity in order to accurately represent the gender of
our sample. Participants had uent English and normal or corrected to
normal vision. In terms of nationality 90.2% identied as Irish, 1.6% as
2
Participants self-identied as STEM or non-STEM students based on the course they
were enrolled in. Non-STEM students consisted of those from the Humanities and Social
Sciences while STEM students consisted of those from the elds of Science, Technology,
Engineering and Maths similar to the distinction made by Smyth and Nosek (2015).
Journal of Contextual Behavioral Science 6 (2017) 8090
French, 3.3% as American, 1.6% as Polish, 1.6% as Welsh and 1.6% as
Kuwaiti. This mix of nationalities is representative of a contemporary
Irish university.
Female
Girl 3.2. Materials
Hers
She
Her
The recording of participants responses along with instructional
Mrs and stimulus presentation were conducted with a standard laptop
Women computer.
3.2.1. IAT
The IAT is a response latency task during which participants must
sort words into their corresponding categories. The contrasting pairs of
category labels were Science versus Arts and Male versus Female.
Six male and six female words and six Arts and six STEM subjects acted
as the target words to be sorted into the appropriate categories (see
Table 1).
3.2.2. IRAP
The IRAP is a response latency task in which the participant must
respond to stimuli under time pressure. The label stimulus Men more
suited to or Women more suited to was presented along with one of
the same six Arts and STEM subjects used in the IAT (see Table 1
above). Participants responded to relations between the label stimuli
and target words using the terms True (d key) orFalse (k key).
These response options remained xed throughout the task.3
3.2.3. Rating scale
Participants were to rate on an 11-point scale whether men or
women are more suited to the Arts and STEM subjects presented in the
IRAP and IAT. High scores indicated females were rated more suitable,
low scores indicated males were rated more suitable, while 6 repre-
sented a neutral score.
3.3. Procedure
The study was conducted in a quiet room to minimize distractions.
The measures were counterbalanced across participants in the follow-
ing ways: (i) IRAP or IAT rst; (ii) blocks consistent with social
stereotypes (men-science/women-arts) or inconsistent (women-
science/men-arts) with social stereotypes rst.
3.3.1. IAT
Over the course of seven blocks participants had to categorize target
words into their relevant categories as quickly and accurately as
possible. The categories appeared at the top right and left of the screen
while the target words appeared individually in the center of the screen.
Participants pressed the d key if the target word belonged in the
category to the left of the screen and the k key for the right. A correct
response cleared the screen for a 400 ms inter-trial interval. If the word
was incorrectly categorized, a red X appeared below the target word
until the participant made the correct response. At the end of each
block participants were presented with their percentage of correct
responses and their median response time for that block. Participants
had to achieve an accuracy mean of 80% or above and a mean response
latency of 2 s or less. Blocks 1 and 2 were practice blocks during which
participants had to categorize academic subjects as Arts or Science
then categorize words as Male or Female. These categories were then
combined in the third and fourth test blocks where the d key
corresponded to the categories Male and Science and the k key
corresponded to the categories Female and Arts when the block order
3
While the positioning of the response options may be alternated throughout the task,
unpublished research suggests it has little impact on IRAP performance (see Barnes-
Holmes, Barnes-Holmes, Stewart & Boles, 2010).
82
L. Farrell, L. McHugh
Fig. 1. Participants are presented with categories to the left and right of the screen (e.g. female OR science), and a target word in the center of the screen (e.g. History, Computing).
The response keys are noted - d for the categories to the left and k for the categories to the right. During the test blocks (blocks 3 and 4) the categories male and female are paired up
with either the category science or arts (see left). In the next set of test blocks (blocks 6 and 7) the category pairings are reversed (see right).
presented consistent blocks rst. If the inconsistent blocks were
presented rst Female and Science were paired up and Male and
Arts. Block 5 reversed the positions of the Science and Arts
categories. During test blocks 6 and 7 the categories were again paired
up, now Male and Arts shared the response key d and Female and
Science the k key (see Fig. 1). Blocks 13, 5 and 6 consisted of 24
trials each, while 4 and 7 consisted of 48 trials each.
3.3.2. IRAP
Prior to beginning the IRAP the researcher provided instructions
regarding the task. These instructions were adapted from the experi-
menter script which was previously available through the IRAP website
(this site was unavailable at the time of writing) as it has been noted
that proper instruction may be key in lowering attrition rates with this
measure (Murphy, MacCarthaigh, & Barnes-Holmes, 2014).
Participants were shown visual samples of the IRAP trials in order to
ensure that they understood how to respond (see Fig. 2). The key point
conveyed to participants was that correct responding was governed by a
rule and not their own opinion. At times the rule required them to
respond as though men were more suited to STEM subjects and women
were more suited to Arts subjects (stereotype-consistent blocks) while
the other blocks required the opposite pattern of responding i.e. men
were more suited to Arts subjects and women were more suited to
STEM subjects (stereotype-inconsistent blocks).
Participants had to respond to the relations between the label
stimuli (Men/Women more suited to) and target words (Arts and
STEM subjects from the IAT) using the terms True (d key) or False
(k key). Upon a correct response stimuli were removed from the
screen for a 400 ms inter-trial interval. If the incorrect response was
selected, a red X appeared on the screen until the participant made a
correct response. Each of the 12 target words were presented twice in
quasi-random order (24 trials in each block), once with each label
statement. The rule switched after every block with half of the
participants receiving stereotype-consistent blocks rst and the other
half receiving stereotype-inconsistent blocks rst. They were exposed
to a minimum of 2 and a maximum of 8 practice blocks. At the end of
each block participants were presented with their percentage of correct
responses and their median response time for that block. In order to
progress to the test blocks participants had to achieve a mean accuracy
of 80% or above and a median response latency of 2 s or less. Once
achieved, participants progressed to the six test blocks. An exclamation
mark appeared at the bottom of the screen as a prompt to respond
when the time limit was near. Participants were advised to take short
breaks between blocks both to remind themselves of the current rule
and to take a break from looking at the computer screen. This was done
in order to avoid any fatigue eects which could have led to errors.
3.3.3. Rating scale
Finally, participants completed the explicit measure - a rating scale.
Once completed, participants were thanked and fully debriefed. At this
point gender bias was revealed as the variable of interest as previously
Journal of Contextual Behavioral Science 6 (2017) 8090
the study had been described as an exploration of attitudes towards
STEM and Arts subjects.
3.3.4. Analytic Plan
A number of statistical analyses were conducted in line with
previous research using the IRAP and IAT (e.g. Farrell et al., 2015).
The data collected for this study were: D-scores from both the IRAP
and the IAT which represented levels of bias and scores from the rating
scale which represented explicit levels of bias. An Analysis of Variance
(ANOVA) was conducted for each measure to test for order eects both
of the blocks (consistent or inconsistent rst) and the indirect measures
(IRAP rst or IAT rst). This ensured that none of the D-scores were
moderated by these procedural variables as has been examined in
previous implicit studies (e.g. Cullen, Barnes-Holmes, Barnes-Holmes
& Stewart, 2009).
Mean D-IAT scores were calculated for each participant group to
give an indication of levels of bias. In order to examine whether this
bias was signicant a series of one sample t-tests using Bonferroni
adjusted alpha levels of .0125 were conducted for each of the groups.
To examine whether bias levels were signicantly dierent based on
participant group a one-way ANOVA was conducted.
Mean D-IRAP scores for each participant group were calculated for
each of the 4 IRAP trial types to give an indication of levels of bias for
each component of gender-STEM bias (i.e. Male-STEM, Male-Arts,
Female-STEM, Female-Arts). A series of one sample t-tests using
Bonferroni adjusted alpha levels of .0125 were conducted to examine
whether the mean biases of each separate participant group were
signicantly dierent from a score of 0. A 4 (male or female STEM or
non-STEM student) 4 (IRAP trial types) mixed ANOVA examined
whether scores on the IRAP trials were signicantly dierent based on
the participant group. We expected an interaction between the
participant group and the IRAP trial types. Based on previous research
in the area (Smeding, 2012; Smyth & Nosek, 2015), we anticipated
dierences between STEM males and females, STEM and non-STEM
males and STEM and non-STEM females, particularly on the STEM-
relevant IRAP trials (see Farrell et al., 2015). Further planned
comparisons investigated these group dierences using independent
t-tests. The internal consistency of the IRAP and IAT scores was
assessed using an odd-even split-half procedure using the Spearman-
Brown formula.
Correlations were conducted between the scores from the implicit
and explicit measures on the sample as a whole to examine the
relationship between these biases. Finally, correlations were conducted
between the IAT and the IRAP to examine the relationship (if any)
between these measures.
4. Results
Only data from the test blocks were used in the subsequent analysis
of the indirect measures. In order to be included participants had to
achieve an average accuracy of 80% and latency of 2 s in the IAT
83
L. Farrell, L. McHugh Journal of Contextual Behavioral Science 6 (2017) 8090

Fig. 2. Participants are presented with a label statement (Men/Women more suited to) at the top of the screen, a target word in the center of the screen (e.g. Science, Arts) and two
response options (True, False) at the bottom left and right. In the consistent trials (left) participants are informed that a correct answer relates men to STEM and women to Arts
subjects. In the inconsistent trials (right) a correct answer relates men to Arts and women to STEM subjects. The arrows with superimposed text boxes were not on the sample shown to
participants but indicate which response options were correct and incorrect for each of the trials types.

(e.g. Greenwald & Farnham, 2000) and a median latency of 2 s and
mean accuracy of 80% across the 3 consistent and 3 inconsistent
blocks separately in the IRAP (e.g. Farrell et al., 2015). Three
participants were excluded on the basis of this criteria leaving 61 in
the nal analysis. Participants were divided into groups based on their
gender and subject area resulting in four groups of students: male
STEM, female STEM, male non-STEM and female non-STEM.
4.1. IAT
IAT data consisted of response latencies (dened as the time from
the onset of a trial in milliseconds until a correct response was emitted)
converted into a D-IAT score for each participant using the D-
algorithm suggested by Greenwald, Nosek, and Banaji (2003). A 22
ANOVA revealed no signicant main or interaction eects of the block
order (consistent or inconsistent rst) or the order of indirect measures
(IRAP rst or IAT rst). Positive mean D-IAT scores indicate a Male-
STEM/Female-Arts bias while negative D-IAT scores indicate a
Female-STEM/Male-Arts bias. A series of one sample t-tests using
Bonferroni adjusted alpha levels of .0125 indicated that all groups had
a signicant Male-STEM/Female-Arts bias [male STEM, t(15) =8.4,
p=.000; female STEM, t(14) =3.7, p=.002; male non-STEM, t(14) =3.4,
p=.004 and female non-STEM t(14) =3.3, p=.005]. Male STEM
students had the highest mean bias (M =.50; SD =.239) followed by
female non-STEM students (M =.35; SD =.41). Male non-STEM
students had a mean bias of .33 (SD =.37) while female STEM students
had the lowest mean bias at .26 (SD =.28). However, a one way ANOVA
revealed no signicant eect of participant group on D-IAT scores
[F(3,57) =1.462, p=.234], therefore there was no signicant dierence
in bias between the groups.
4.2. IRAP
Response latencies (time from the onset of a trial to the rst correct
response) transformed into D-IRAP scores (see Barnes-Holmes et al.,
2010) were the primary data for the IRAP. A 22 mixed ANOVA was
conducted to test for order eects on the D-IRAP scores with the order
of indirect measures and test block order as between-participants
variables. There was a signicant main eect for trial type [F(3, 171)
=31.58, p=.000, p2 =.36] and block order [F(1, 57) =5.38, p=.02, p2
=.086]. The main eect for test block order indicated that participants
D-IRAP scores were generally larger on average when exposed to
inconsistent-rst relative to consistent-rst test blocks. Importantly,
this variable did not interact with the order of implicit measures or,
critically, the IRAP trial types. Therefore, there was a signicant main
eect of IRAP trial types which was not signicantly modied by order
of test blocks or order of implicit measures (see Cullen et al., 2009).
This is consistent with other previous IRAP studies that have found no
interaction between block order and the IRAP eect (e.g. McKenna,
Barnes-Holmes, Barnes-Holmes, & Stewart, 2007; Vahey, Barnes-
Holmes, Barnes-Holmes, & Stewart, 2009; see Barbero-Rubio, Lpez-
Lpez, Luciano, & Eisenbeck, 2016). These order variables were
therefore dropped from the subsequent analysis.
The IRAP consisted of four trial types: Male-STEM, Male-Arts,

84
L. Farrell, L. McHugh Journal of Contextual Behavioral Science 6 (2017) 8090

Fig. 3. Mean D-IRAP scores for each participant group across the 4 trial types. Signicant D-IRAP scores are indicated by an asterisk.

Female-STEM and Female-Arts. The mean D-IRAP scores for each of
the four student groups are presented in Fig. 3. To avoid confusion with
the similarly named participant groups the IRAP trials are referred to
as M-S (Male-STEM), M-A (Male-Arts), F-S (Female-STEM) and F-A
(Female Arts) on Fig. 3's X axis. The bars belonging to Male student
groups (i.e. STEM or non-STEM) are marked by dots while Female
student groups (i.e. STEM or non-STEM) are marked by diagonal lines.
Positive scores indicated a pro-Male-STEM/Female-Arts bias while
negative scores indicated a pro-Female-STEM/Male-Arts bias.
A number of one sample t-tests were conducted for each trial type
using Bonferroni adjusted alpha levels of .0125. All groups had a
signicant pro-Male-STEM bias (see Table 2 for statistical results).
Male and female STEM and male non-STEM students also showed a
signicant anti-Male-Arts bias. Female STEM students were the only
participant group for whom the observed pro-Female-STEM bias
reached signicance [t(14) =4.25, p=.001, Cohen's d =1.10].
Though directionally the other student groups seemed to show a pro-
Female-STEM bias (i.e. responding quicker with True when respond-
ing to Women more suited to STEM subjects) this response bias failed
to reach signicance for these remaining groups and only resulted in
small (d's=.2) and very small (d =.1) eect sizes (see Table 2).
A 44 mixed ANOVA was conducted with participant group (i.e.
Male or Female STEM or non-STEM student) as the between-partici-
pants variable and D-IRAP scores as the within-participants variable.
There was a main eect for trial type [F(3, 171) =32.28, p=.000, p2 =.36]
Table 2
Statistical results for one sample t-tests on D-IRAP scores for each trial type.
Participant Group IRAP Trial df t-value Cohen's d
but no other signicant main eects. The interaction between partici-
pant group and trial type was marginally signicant [F(9, 171) =1.85,
p=.06, p2 =.089]. However, as detailed in our analytic plan, and in line
with previous implicit studies (e.g. Smyth & Nosek, 2015), we
predicted a dierence in gender-STEM bias particularly on the
STEM-relevant IRAP trials between STEM males and females, STEM
and non-STEM males and STEM and non-STEM females. Therefore,
we conducted further comparisons of this marginally signicant
interaction, comparing STEM and non-STEM students and STEM
males and females on the two STEM relevant IRAP trial types (i.e.
Male-STEM and Female-STEM trial types).
An examination of the bar graph of participant group means (see
Fig. 3) suggested that on the Female-STEM trial female STEM
students mean bias was higher than the other groups. In order to test
whether these group dierences were signicant independent t-tests
were conducted. Female STEM students pro-female-STEM bias sig-
nicantly diered from that of male STEM students [t(29) =2.106, p
(one-tailed) =.02, Cohen's d =.76], female non-STEM students [t(28)
=2.012, p (one-tailed) =.03, Cohen's d =.74] and male non-STEM
students [t(28) =1.985, p (one-tailed) =.03, Cohen's d =.73]. The
eect sizes of these dierences ranged from medium to large (all
Cohen's d's > .7). Male STEM and non-STEM students did not sig-
nicantly dier on the Female-STEM trial (p=.95). Despite female non-
STEM students appearing to have a lower pro-male-STEM bias than
the other groups on Fig. 3, there were no signicant dierences
between the groups on the Male-STEM trial (all p's > .2) with eect
sizes ranging from very small (Cohen's d =.1 for female STEM versus
male STEM students) to medium (Cohen's d =.5 for female STEM
versus female non-STEM students).
p-value

Male STEM Male-STEM 15 5.94 1.48 .000

Male-Arts 15 3.91 .98 .001
Female-STEM 15 .975 .24 .345 4.3. Internal consistency
Female-Arts 15 .818 .20 .426
The reliability of the IRAP and IAT scores was assessed using an
Female STEM Male-STEM 14 6.102 1.58 .000
odd-even split-half procedure as described by De Houwer and De
Male-Arts 14 3.373 .87 .005
Female-STEM 14 4.245 1.10 .001 Bruycker (2007) using the Spearman-Brown formula. Using this
Female-Arts 14 .038 .01 .970 procedure, a split-half reliability of .58 was obtained for the IRAP
and .86 for the IAT. These are both reasonably strong results, similar to
Male Non-STEM Male-STEM 14 3.810 .98 .002 those obtained in previous studies using these measures (e.g. Remue,
Male-Arts 14 2.806 .72 .014
Female-STEM 14 .739 .19 .472
De Houwer, Barnes-Holmes, Vanderhasselt, & De Raedt, 2013; see
Female-Arts 14 1.227 .32 .240 Nosek, Greenwald, & Banaji, 2006) and compare favorably to other
latency-based measures of implicit bias (see Golijani-Moghaddam,
Female Non-STEM Male-STEM 14 3.022 .78 .009 Hart, & Dawson, 2013). As noted by Remue et al. (2013) further
Male-Arts 14 1.756 .45 .101
conclusions about dierences in reliability between the IRAP and
Female-STEM 14 .541 .14 .597
Female-Arts 14 1.91 .49 .077 measures such as the IAT will become possible when these reliability
measures are more consistently reported.

85
L. Farrell, L. McHugh
Fig. 4. Mean Summed Ratings for the Arts and STEM subjects for each of the 4 participant groups.
4.4. Explicit measure
A total score was calculated for the 6 Arts and 6 STEM subjects
separately, giving a maximum total score of 66. Lower scores indicated
participants rated males as more suited to the subject and higher scores
indicated that females were deemed more suited. The mean summed
scores for each group can be seen in Fig. 4.
Overall all groups rated STEM subjects as relatively more suited to
males and Arts subjects as relatively more suited to females. However,
these ratings were not extreme with means generally in the 30's range.
A series of one sample Wilcoxon signed rank tests revealed that the
Arts subjects rating was signicantly higher than the median of 36 for
all groups (Female non-STEM: p=.007; Female STEM: p=.008; Male
non-STEM: p=.03; Male STEM: p=.01) while the STEM subjects rating
was signicantly lower than the median of 36 for all groups (Female
non-STEM: p=.003; Female STEM: p=.005; Male non-STEM: p=.01;
Male STEM: p=.02). This indicated that the explicit ratings were
signicantly dierent from the neutral point, indicating an explicit
male-STEM/female-Arts bias. This pattern of responding was largely
consistent with the implicit biases found for each group with the
exception of the female STEM group who we might have expected to
rate the STEM subjects more neutrally based on their competing pro-
male and pro-female-STEM biases. There were no statistically signi-
cant dierences between any of the groups in terms of their explicit
ratings of the Arts and STEM subjects (all p's > .7). Spearman's rho was
used for all correlations involving the explicit measure. All groups
exhibited a negative correlation between their STEM and Arts subject
ratings (Male STEM: rs =.71, p=.002; Female STEM: rs =.72,
p=.003; Male non-STEM: rs =.77, p=.001; Female non-STEM: rs
=.82, p=.000), in line with an explicit Male-STEM/Female-Arts bias.
4.5. Implicit-explicit correlations
Due to the relatively small sample size correlations were conducted
on the sample as a whole (N =61) as opposed to the separate
participant groups. There were no signicant correlations between D-
IAT scores and the explicit measures however there were three
signicant correlations between the IRAP and the explicit measures.
There was a positive correlation between the Arts subjects mean score
and the Male-Arts trial type (rs =.351, p=.006). As the explicit ratings
of the Arts subjects increased, indicating that females were perceived as
more suitable for these subjects, the D-IRAP scores for the Male-Arts
trial type also increased indicating an anti-Male-Arts bias. There was
also a negative correlation between the STEM subjects mean score and
86
Journal of Contextual Behavioral Science 6 (2017) 8090
the Male-Arts trial type (rs =.369, p=.003). As the explicit ratings of
the STEM subjects decreased (indicating males were rated more
suitable for these subjects), the D-IRAP scores on the Male-Arts trial
increased demonstrating an anti-Male-Arts bias. Finally, there was a
negative correlation between explicit STEM ratings and the Female-
STEM trial type (rs =.305, p=.017). As the explicit ratings of the
STEM subjects decreased which indicated that males were rated more
suitable for these subjects, D-IRAP scores on the Female-STEM trial
increased which suggested an anti-Female-STEM bias. These correla-
tions were in line with the stereotypically consistent responses on the
gender-STEM IRAP, namely pro-Male-STEM, anti-Male-arts and anti-
Female-STEM.
4.6. IRAP-IAT correlations
Correlations were conducted on the IRAP and IAT scores to
examine the relationship (if any) between scores on these measures.
These correlations were conducted on the whole sample (N =61). There
was a signicant positive correlation between the IAT and the IRAP's
Female-Arts trial (r =.277, p=.03). As the D-IAT scores increased,
representing a Male-STEM/Female-Arts bias, the D-IRAP scores on the
Female-Arts trial also increased indicating a pro-Female-Arts bias.
There were no other signicant correlations between these two
measures of implicit bias. This is perhaps due to the procedural
dierences between the measures. While both target gender-STEM
bias they do so with dierent levels of specicity the IAT examined
associations while the IRAP targeted a more specic relation, namely
whether men or women were more suited to STEM or Arts subjects.
This dierence may therefore result in eects that are unlikely to be
functionally equivalent as noted by Farrell et al. (2015).
5. Discussion
The current study expanded on previous research using the IRAP to
investigate gender-STEM bias (see Farrell et al., 2015), exploring the
gender-STEM bias of STEM and non-STEM male and female students.
As hypothesized, and similar to previous IAT research (e.g. Smeding,
2012; Smyth & Nosek, 2015), all groups in the study showed a
signicant Male-STEM/Female-Arts bias on the IAT, with male STEM
students exhibiting the highest mean bias score relative to the other
groups and female STEM and male non-STEM students exhibiting the
lowest. However, there was no statistically signicant dierence
between the groups on this measure. The IRAP was able to oer a
more detailed analysis. All groups exhibited a signicant pro-Male-
L. Farrell, L. McHugh
STEM bias as expected. Interestingly, however, female STEM students
also exhibited a signicant pro-Female-STEM bias with a large eect
size that signicantly diered from the other participant groups. This
may explain the lower Male-STEM/Female-Arts bias of STEM females
both in this study and previous IAT studies (Nosek & Smyth, 2011;
Smeding, 2012; Smyth & Nosek, 2015), suggesting that it is the STEM
bias that is driving this result. Rather than simply having a lower Male-
STEM bias, female STEM students had a competing Female-STEM
bias. This is an important result as it provides a new insight into how
female STEM students implicitly relate women to STEM. Interestingly,
the 3 other student groups were quicker to conrm rather than deny a
relation between women and STEM also. However, it is very important
to note that the strength of this response bias was non-signicant for
these 3 groups and very weak in terms of eect sizes. This suggests that,
while directionally a pro-Female-STEM bias was present, it is not a
highly probable response for these groups. Finally, there was also a
signicant anti-Male-Arts bias found among the male and female
STEM and male non-STEM groups.
This pattern of bias diered in a number of ways from Farrell et al.
(2015). However, this is unsurprising as the previous study was
exploratory, included a sample of students and non-students and did
not distinguish between STEM and non-STEM students. It is clear from
the current study's results that this is an important distinction at
university level. The current study also used a more subtle IRAP label
phrase in an attempt to lessen the potential for stereotype threat
among female participants. The lack of an overall pro-Male bias among
the female participants may have been inuenced by this change
however as we did not directly compare this IRAP with the previous
study's IRAP this cannot be stated with certainty. Future research
should investigate this comparison to examine the inuence, if any, of
this change further.
The results of the rating scales showed that participants did not
completely disguise their Male-STEM/Female-Arts bias explicitly,
though ratings were not extreme and remained close to the neutral
point. The binary format of the rating scales may account for this
tendency towards the neutral point. Future IRAP research in this area
may consider designing the scales so that male suitability and female
suitability for STEM and Arts subjects are rated separately. This would
more closely resemble the IRAP trial types and may provide a more
informative or nuanced picture of explicit ratings of gendered suit-
ability that may correlate dierently with implicit responses.4 There
were a number of stereotype-consistent correlations between this
explicit measure and the IRAP but not the IAT. However, this may
be partly due to the IRAP having 4 scores (1 for each trial type)
available for correlation in comparison to the IAT's single score.
The potential for an implicit pro-Female-STEM bias, observable
only using the IRAP, may be of importance to those interested in bias
intervention. It is important to note that the aim of intervention is not
necessarily to lessen the belief that men are good at or suited to STEM
subjects, instead the focus is on strengthening a positive relation
between women and STEM subjects, making women in STEM norma-
tive. Inuencing implicit beliefs may impact upon overt behavior and
attitudes towards women in STEM. Studies have found a link between
strong male-science/math bias and weaker science identication,
career aspiration and math ability self-concept among females (e.g.
Cundi, Vescio, Loken, & Lo, 2013; Steens & Jelenec, 2011;
Schmader, Johns, & Barquissau, 2004), while weaker male-science
associations among STEM females coincided with stronger explicit
science identication (Smyth & Nosek, 2015). A pro-Female-STEM
relation, observable in the current study's IRAP and signicant only for
the female STEM group, may be underlying this weaker male-science
association and, therefore, may increase belonging and identication
4 5
We would like to thank an anonymous reviewer of this article for raising this point
regarding the binary format of the explicit measure and its implications.
87
Journal of Contextual Behavioral Science 6 (2017) 8090
with STEM for females; increasing their willingness to pursue a STEM
career. However, it must be noted that as the aforementioned IAT
studies were correlational it is unclear whether lower implicit gender-
STEM bias produced stronger science identication or vice versa. It is
therefore important to explore what variables promote a pro-Female-
STEM relation using the IRAP. There is a suggestion that strengthening
science identity may be important for females in STEM (e.g. see Smyth
& Nosek, 2015). However, this form of intervention, while important,
could arguably be viewed as yet another attempt to x the women (see
for example Schiebinger, 2010). It is also important to explore other
avenues that will allow positive female-STEM relations to be fostered
among males and females more broadly so as to intervene against
gender-STEM bias within our culture.
A further point of interest would be to investigate the contexts in
which this pro-Female-STEM bias inuences female STEM students
behavior. Situations that usually evoke stereotype threat may represent
one such instance. Crisp, Bache, and Maitner (2009) found that when
gender was made salient math performance was negatively aected
among female psychology students while female engineering students
actually performed better. Smeding (2012) also examined female
engineering students and found their weaker implicit gender-STEM
stereotypes were not negatively correlated with their math performance
unlike female humanities students. Perhaps these students possessed a
pro-female-STEM bias that positively inuenced their STEM perfor-
mance. This is of course currently speculatory as the aforementioned
studies did not investigate implicit gender-STEM bias non-relatively. In
relation to this it would be useful to determine when and how a pro-
female-STEM bias develops for female STEM students. For example, do
female STEM students acquire a pro-female-STEM bias through their
engagement with STEM coursework and colleagues? Or does this pro-
female-STEM bias develop prior to their formal entry in to STEM
subjects at third level? If the latter is the case it may be that the presence
of a pro-female-STEM bias inuences their decision to pursue a STEM
career in the rst place. This would again highlight the need for the
development of interventions to strengthen the relation between women
and STEM. Researchers may perhaps use the IRAP to measure implicit
gender-STEM bias among females, particularly those intending to
pursue a STEM course across a number of key transition periods such
as before entry to STEM courses and during the course to examine the
potential development and/or strengthening of a female-STEM bias.
There are some methodological issues to bear in mind when
considering these results. First, the accuracy criteria used for the
IRAP in the current study is not typical of many of the published IRAP
studies (with the exception of Farrell et al., 2015). While there are
studies that use a mean of 80% for each test block as their accuracy
criteria throughout, there are other accepted procedures such as
lowering the accuracy threshold (usually no lower than 70%; e.g.
Vahey et al., 2009) based on factors such as, for example, sample
population and stimuli used. If participants fail to reach this criterion
in one of the test blocks researchers may remove the pair of test blocks
for which this lower accuracy occurred and calculate D-scores based on
the remaining blocks (e.g. Nicholson & Barnes-Holmes, 2012). The
current study stipulated that participants had to reach an average of
80% across the three consistent and inconsistent test blocks separately5
to be included in the analysis as these are the responses we were
comparing (consistent versus inconsistent). This allowed participants
to dip below a mean accuracy of 80% in one or two of the test blocks
but still required them to achieve at least 80% in the remaining tests
blocks. Therefore, the mean accuracy was not lowered completely
across the blocks and we were able to include data from all 6 test
blocks. This approach helped to lower attrition rates particularly as,
anecdotally, the IRAP has been found to be relatively more dicult to
This was not the accuracy criteria conveyed to participants. Participants were told to
aim for at least 80% mean accuracy in each block.
L. Farrell, L. McHugh
complete than, for example, the IAT (see Roddy, Stewart, & Barnes-
Holmes, 2010), assessing more complex relations. Only 12 of the 61
participants included in the analysis dropped below 80% in either a
consistent or inconsistent test block (usually getting a mean of 79% on
one test block). The IRAP continues to be optimized as a measure and
so it remains to be seen which (if any) of these data screening
procedures is the best t for particular samples or for the measure
itself.
Second, implicit bias seems to be malleable to a certain extent, with
research on the reduction of gender stereotypes showing the inuence
of contextual factors (see Lenton, Bruder, & Sedikides, 2009 for a
meta-analysis). The majority of the participants in the current study
were guided through by a female experimenter. Only 6 participants had
a male experimenter. The role of experimenter required competence in
the experimental process and research knowledge to be demonstrated
by both experimenters. Blair, Ma, and Lenton (2001) found that
counter-stereotypical mental imagery, such as imagining a strong
woman, reduced implicit stereotypes on a subsequent IAT. It is
possible that the sex of the experimenter may have some inuence
on an individual's implicit bias. As we did not systematically vary the
sex of the experimenter in the current study we were unable to address
this question, however, future IRAP research on gender-STEM bias
should examine this potential contextual inuence.
Third, a recent publication by Finn, Barnes-Holmes, Hussey &
Graddy (2016) suggests that the type of rule used during the IRAP (i.e.
from the general to the specic) may have an impact upon the strength
of the IRAP eect for certain IRAP trial types. The rst and fourth trials
of the IRAP seemed to produce the strongest eects when more specic
rules were used with the suggestion that responding may have been less
BIRR-like or implicit for the other trials (i.e. Trials 2 and 3). The
IRAP rules used in the current study could be classied as specic in
that participants were shown examples of each of the 4 trial types for
both consistent and inconsistent blocks when rst introduced to the
IRAP and the rule was specied for them before each block (e.g. Please
respond as if women are more suited to science subjects and men are
more suited to arts subjects) as was the standard setting of the IRAP
available to the authors. While the pattern of results in this study did
not exactly reect those found by Finn et al. (2016) as Trial 4 in the
current study (Women-Arts) was not signicant for any group and
there was no interaction between block order and the D-IRAP scores,
this issue is important to consider when conducting implicit research
using latency-based measures. Finn et al. (2016) suggest that IRAP
eects could potentially be stronger with the use of general rules,
however, they note that participants may still generate their own rules
even in these instances and further empirical examination is required
to explore the potential explanations and implications of these ndings.
Finally, while our total sample size (N =61) is in line with previous
IRAP studies and suciently powered to observe an eect size of .80
(see Vahey, Nicholson, & Barnes-Holmes, 2015), the size of our
subgroups were relatively small (at least 15 per group). For this reason,
the correlational analysis was conducted with the larger groups out of
concern for statistical power. This sample size is also small in
comparison to the large scale studies that have been conducted with
the IAT. An online version of the IAT has aided the collection of data
from hundreds of thousands of participants (see Smyth & Nosek,
2015). While the results of our study are, to a certain extent, in line
with these larger scale studies (i.e. revealing lower male-STEM bias
among female STEM students) and also enhance our knowledge of this
bias through the use of a non-relative measure, replication of the
current study with larger sample sizes per condition would be benecial
in ensuring that the results reported here are indeed generalizable. The
availability of an online version of the IRAP would perhaps aid the
collection of larger data sets similar to those achieved by the online
version of the IAT (see https://www.projectimplicit.net). There are
indications that such a version of the IRAP is currently being developed
(see Timmins, Barnes-Holmes, & Cullen, 2016).
88
Journal of Contextual Behavioral Science 6 (2017) 8090
6. Conclusion
In summary the current study was able to further explore gender-
STEM bias among STEM and non-STEM male and female students
using the IRAP. All groups exhibited a pro-Male-STEM bias on the
IRAP. As implicit gender bias is agged as a major obstacle for women
in STEM it is important to develop interventions against this bias.
Perhaps the most interesting result in relation to this was the evidence
for a pro-Female-STEM bias found among the groups. This bias was
strongly signicant for the female STEM students while it appeared
only directionally for the other groups, failing to reaching signicance
or produce even a medium eect size. This relation could only be
observed using the IRAP as it was a non-relative, relation-based
measure as opposed to a relative, association-based measure like the
IAT. Given that previous research has predominantly identied STEM
as more easily associated with men, a relation between women and
STEM (other than perhaps an anti-Female-STEM relation) seemed
unlikely, particularly among male STEM students. Though again it
must be noted that this response bias was very weak and non-
signicant for all groups with the exception of the female STEM
students. However, if it is possible to strengthen the relation between
women and STEM (i.e. that women are suited to STEM) among males
and females irrespective of their eld of study this could improve
attitudes and behavior towards women in STEM elds. Ultimately the
goal is to reduce gender-STEM bias and stereotypes so as to make
women in STEM subjects or careers as normative as men. Subsequent
IRAP studies should explore the malleability of this bias as few such
interventions have targeted or assessed implicit attitude or bias change
and those that have utilized relative measures of implicit bias (e.g.
Jackson, Hillard, & Schneider, 2014). The IRAP is capable of giving us
a clearer picture of implicit bias reduction at the level of individual trial
types and importantly can reveal the impact interventions have on the
strength of a female-STEM relation separately. This may be very
important in helping to determine what elements of gender-STEM
bias various interventions inuence.
Author note
This research was conducted by the rst author as part of her
doctoral studies.
Acknowledgements
The authors would like to thank all of the participants who
volunteered their time for this study. They would also like to thank
Emma Ni Neill and Darragh Power for their help in collecting data for
the current study, Nigel Vahey for consultation on statistical analysis
regarding the IRAP, Andy Cochrane for consultation on the current
study's design and the anonymous reviewers whose feedback strength-
ened this manuscript.
The rst author is supported by a Government of Ireland
Scholarship from the Irish Research Council.
References
Accenture (2014). Powering economic growth; Attracting more young women into
science and technology. Retrieved 8th April 2014 from http://www.accenture.com/
SiteCollectionDocuments/Local_Ireland/PDF/accentureSTEM-powering-economic-
growth.pdf
ASPIRES (2013). Young peoples science and career aspirations, age 1014. Department
of Education & Professional Studies; Kings College London. Retrieved 4th
February 2016 from https://www.kcl.ac.uk/sspp/departments/education/
research/aspires/ASPIRES-nal-report-December-2013.pdf
Barbero-Rubio, A., Lpez-Lpez, J. C., Luciano, C., & Eisenbeck, N. (2016). Perspective-
taking measured by Implicit Relational Assessment Procedure (IRAP). The
Psychological Record, 66(2), 243252.
Barnes-Holmes, D., Barnes-Holmes, Y., Power, P., Hayden, E., Milne, R., & Stewart, I.
(2006). Do you really know what you believe? Developing the Implicit Relational
Assessment Procedure (IRAP) as a direct measure of implicit beliefs. The Irish
L. Farrell, L. McHugh
Psychologist, 32(7), 169177.
Barnes-Holmes, D., Barnes-Holmes, Y., Stewart, I., & Boles, S. (2010). A sketch of the
Implicit Relational Assessment Procedure (IRAP) and the Relational Elaboration and
Coherence (REC) model. The Psychological Record, 60(3), 527.
Blair, I. V., Ma, J. E., & Lenton, A. P. (2001). Imagining stereotypes away: the
moderation of implicit stereotypes through mental imagery. Journal of Personality
and Social Psychology, 81(5), 828.
Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. American
Psychologist, 61, 2741.
Carli, L. L., Alawa, L., Lee, Y., Zhao, B., & Kim, E. (2016). Stereotypes about Gender and
Science: Women Scientists. Psychology of Women Quarterly, 40(2),
244260. http://dx.doi.org/10.1177/0361684315622645.
Carnes, M., Devine, P. G., Isaac, C., Manwell, L. B., Ford, C. E., Byars-Winston, A.
Sheridan, J. (2012). Promoting institutional change through bias literacy. Journal of
Diversity in Higher Education, 5(2), 6377.
Ceci, S. J., Williams, W. M., & Barnett, S. M. (2009). Women's underrepresentation in
science: Sociocultural and biological considerations. Psychological Bulletin, 135(2),
218.
Cheryan, S., Master, A., & Meltzo, A. N. (2015). Cultural stereotypes as gatekeepers:
Increasing girls' interest in computer science and engineering by diversifying
stereotypes. Frontiers in Psychology, 6, 49.
Cheryan, S., Plaut, V. C., Davies, P. G., & Steele, C. M. (2009). Ambient belonging: how
stereotypical cues impact gender participation in computer science. Journal of
personality and Social Psychology, 97(6), 1045.
Cheryan, S., Siy, J. O., Vichayapai, M., Drury, B. J., & Kim, S. (2011). Do female and male
role models who embody STEM stereotypes hinder women's anticipated success in
STEM? Social Psychological and Personality Science, 2(6), 656664.
Crisp, R. J., Bache, L. M., & Maitner, A. T. (2009). Dynamics of social comparison in
counter-stereotypic domains: Stereotype boost, not Stereotype threat, for women
engineering majors. Social Inuence, 4,
171184. http://dx.doi.org/10.1080/15534510802607953.
Cullen, C., Barnes-Holmes, D., Barnes-Holmes, Y., & Stewart, I. (2009). The Implicit
Relational Assessment Procedure (IRAP) and the malleability of ageist attitudes. The
Psychological Record, 59, 591620.
Cundi, J. L., Vescio, T. K., Loken, E., & Lo, L. (2013). Do genderscience stereotypes
predict science identication and science career aspirations among undergraduate
science majors? Social Psychology of Education, 16(4), 541554.
De Houwer, J., & De Bruycker, E. (2007). The Implicit Association Test outperforms the
Extrinsic Aective Simon Task as an implicit measure of inter-individual dierences
in attitudes. British Journal of Social Psychology, 46, 401421.
Diekman, A. B., Clark, E. K., Johnston, A. M., Brown, E. R., & Steinberg, M. (2011).
Malleability in communal goals and beliefs inuences attraction to STEM Careers:
Evidence for a goal congruity perspective. Journal of Personality and Social
Psychology, 101(5), 902918.
Eagly, A. H., & Karau, S. J. (2002). Role congruity theory of prejudice toward female
leaders. Psychological Review, 109, 573598.
Ehrlinger, J., & Dunning, D. (2003). How chronic self-views inuence (and potentially
mislead) estimates of performance. Journal of personality and Social Psychology,
84(1), 5.
Farrell, L., Cochrane, A., & McHugh, L. (2015). Exploring attitudes towards gender and
science: The advantages of an IRAP approach versus the IAT. Journal of Contextual
Behavioral Science, 4(2), 121128.
Faulkner, W. (2006). Genders in/of engineering. University of Edinburgh Economic and
Social Research Council. Retrieved 26th April 2014 from http://www.aog.ed.ac.
uk/__data/assets/pdf_le/0020/4862/FaulknerGendersinEngineeringreport.pdf
Finn, M., Barnes-Holmes, D., Hussey, I., & Graddy, J. (2016). Exploring the behavioral
dynamics of the implicit relational assessment procedure: The impact of three types
of introductory rules. The Psychological Record, 66(2), 309321.
Fisher, A., & Margolis, J. (2002). Unlocking the Clubhouse: The Carnegie Mellon
Experience. ACM SIGCSE Bulletin, 34(2), 7983.
Gatta, M., & Trigg, M. (2001). Bridging the gap: Gender equity in science, engineering
and technology. Rutgers University, Center for Women and Work. Retrieved 13th
February 2014 from http://opas.ous.edu/Committees/Resources/Research_
papers/Gatta_Trigg_Bridging.pdf
Global Gender Gap Report (2015). Retrieved 25th November 2015 from http://reports.
weforum.org/global-gender-gap-report-2015/
Greenwald, A. G., & Farnham, S. D. (2000). Using the Implicit Association Test to
measure self-esteem and self-concept. Journal of personality and Social
Psychology, 79(6), 1022.
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual
dierences in implicit cognition: the implicit association test. Journal of Personality
and Social Psychology, 74(6), 14641480.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the
Implicit Association Test: I. An improved scoring algorithm. Journal of Personality
and Social Psychology, 85(2), 197216.
Greenwald, A. G., Nosek, B. A., & Sriram, N. (2006). Consequential Validity of the
Implicit Association Test: Comment on Blanton and Jaccard. American
Psychologist, 61(1), 5661.
Greenwald, D. E., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009).
Understanding and using the Implicit Association Test: III. Meta-analysis of
predictive validity. Journal of Personality and Social Psychology, 97, 1.
Golijani-Moghaddam, N., Hart, A., & Dawson, D. L. (2013). The implicit relational
assessment procedure: Emerging reliability and validity data. Journal of Contextual
Behavioral Science, 2(3), 105119.
Hayes, S. C., Barnes -Holmes, D., & Roche, B. (2001). Relational Frame Theory: A post-
Skinnerian account of human language and cognition New York: Plenum.
89
Journal of Contextual Behavioral Science 6 (2017) 8090
Hill, C., Corbett, C., & St Rose, A. (2010). Why so few? Women in Science, Technology,
Engineering, and Mathematics Washington: American Association of University
Women. 1111 Sixteenth Street NW (DC20036).
Hughes, S., Barnes-Holmes, D., & Vahey, N. (2012). Holding on to our functional roots
when exploring new intellectual islands: a voyage through implicit cognition
research. Journal of Contextual Behavioral Science, 1(12),
1738. http://dx.doi.org/10.1016/j.jcbs.2012.09.003.
Hussey, I., Mhaoileoin, D. N., Barnes-Holmes, D., Ohtsuki, T., Kishita, N., & Hughes, S.
(2016). The IRAP is non-relative but not a-contextual: changes to the contrast
category inuence men's dehumanization of women. The Psychological Record,
66(2), 291299.
Jackson, S. M., Hillard, A. L., & Schneider, T. R. (2014). Using implicit bias training to
improve attitudes toward women in STEM. Social Psychology of Education, 17(3),
419438.
Lenton, A. P., Bruder, M., & Sedikides, C. (2009). A metaanalysis on the malleability of
automatic gender stereotypes. Psychology of Women Quarterly, 33(2), 183196.
Margolis, J., & Fisher, A. (2003). Unlocking the Clubhouse: Women in Computing. MIT
Press.
McKenna, I. M., Barnes-Holmes, D., Barnes-Holmes, Y., & Stewart, I. (2007). Testing the
fake-ability of the Implicit Relational Assessment Procedure (IRAP): The rst study.
International Journal of Psychology and Psychological Therapy, 7(2), 253268.
Moss-Racusin, C.A., Dovidio, J.F., Brescoll, V.L., Graham, M., & Handelsman, J. (2012).
Science facultys subtle gender biases favor male students. Proceedings of the
National Academy of Sciences of the USA. Retrieved 3rd October 2012 from http://
www.pnas.org/content/109/41/16474.full.pdfhtml
Murphy, C., MacCarthaigh, S., & Barnes-Holmes, D. (2014). Implicit Relational
Assessment Procedure and attractiveness bias: Directionality of bias and inuence of
gender of participants. International Journal of Psychology and Psychological
Therapy, 14(3), 333351.
National Academy of Sciences, National Academy of Engineering, & Institute of Medicine
of the National Academies (2006). Beyond Biases and Barriers: Fullling the
Potential of Women in Academic Science and Engineering Washington, DC:
National Academies Press.
Nguyen, H. H. D., & Ryan, A. M. (2008). Does stereotype threat aect test performance of
minorities and women? A meta-analysis of experimental evidence. Journal of
Applied Psychology, 93(6), 1314.
Nicholson, E., & Barnes-Holmes, D. (2012). The Implicit Relational Assessment
Procedure (IRAP) as a measure of spider fear. The Psychological Record, 62(2), 263.
Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2006). The Implicit Association Test at
age 7: A methodological and conceptual review. , in: Bargh, J. A. (Ed.). (2006). Social
Psychology and the Unconscious: The Automaticity of Higher Mental Processes . .
Psychology Press, 265292.
Nosek, B. A., & Smyth, F. L. (2011). Implicit Social Cognitions Predict Sex Dierences in
Math Engagement and Achievement. American Educational Research Journal,
48(5), 11251156.
Nosek, B. A., Smyth, F. L., Sriram, N., Lindner, N. M., Devos, T., Ayala, A., & Greenwald,
A. G. (2009). National dierences in genderscience stereotypes predict national sex
dierences in science and math achievement. Proceedings of the National Academy
of Sciences, 106(26), 1059310597.
Oswald, F. L., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P. E. (2013). Predicting
ethnic and racial discrimination: a meta-analysis of IAT criterion studies. Journal of
Personality and Social Psychology, 105(2), 171.
Power, Patricia M. (2010). Developing the Implicit Relational Assessment Procedure
(IRAP) as a measure of Implicit Racial bias. Unpublished doctoral dissertation,
National University of Ireland Maynooth.
Prentice, D. A., & Carranza, E. (2002). What women and men should be, should not be,
are allowed to be, and do not have to be: the contents of prescriptive gender
stereotypes. Psychology of Women Quarterly, 26(4), 269281.
Remue, J., De Houwer, J., Barnes-Holmes, D., Vanderhasselt, M. A., & De Raedt, R.
(2013). Self-esteem revisited: Performance on the implicit relational assessment
procedure as a measure of self-versus ideal self-related cognitions in dysphoria.
Cognition & Emotion, 27(8), 14411449.
Reuben, E., Sapienza, P., & Zingales, L. (2014). How stereotypes impair women's careers
in science. Proceedings of the National Academy of Sciences, 111(12), 44034408.
Risman, B. J. (2004). Gender as a social structure theory wrestling with activism. Gender
& Society, 18(4), 429450.
Roddy, S., Stewart, I., & Barnes-Holmes, D. (2010). Anti-fat, pro-slim, or both? Using
two reaction-time based measures to assess implicit attitudes to the slim and
overweight. Journal of Health Psychology, 15(3), 416425.
Schiebinger, L. (2010, September). Gender, science and technology. In Background
paper for the Expert Group meeting on Gender, Science and Technology, United
Nations Division for the Advancement of Women (DAW, part of UN Women) and
United Nations Educational, Scientic and Cultural Organization (UNESCO),
Paris:, France (Vol. 28)
Schmader, T., Johns, M., & Barquissau, M. (2004). The costs of Accepting Gender
dierences: The role of Stereotype endorsement in women's experience in the math
domain. Sex Roles, 50(1112), 835850.
Schmader, T., Whitehead, J., & Wysocki, V. H. (2007). A linguistic Comparison of Letters
of recommendation for male and female chemistry and Biochemistry job Applicants.
Sex Roles, 57(78), 509514.
ScienceGrrl (2014, March). Through both eyes: The case for a gender lens in STEM.
Retrieved 9th April 2014 from http://sciencegrrl.co.uk/assets/SCIENCE-GRRL-
Stem-Report_FINAL_WEBLINKS-1.pdf
Smyth, F. L., & Nosek, B. A. (2015). On the genderscience stereotypes held by scientists:
explicit accord with gender-ratios, implicit accord with scientic identity. Frontiers
in Psychology, 6, 415. http://dx.doi.org/10.3389/fpsyg.2015.00415.
L. Farrell, L. McHugh
Smeding, A. (2012). Women in Science, Technology, Engineering, and Mathematics
(STEM): An Investigation of Their Implicit Gender Stereotypes and Stereotypes
Connectedness to Math Performance. Sex Roles, 67(11), 617629.
Spencer, S. J., Steele, C. M., & Quinn, D. M. (1999). Stereotype threat and women's Math
performance. Journal of Experimental Social Psychology, 35(1), 428.
Steens, M. C., & Jelenec, P. (2011). Separating implicit gender stereotypes regarding
math and language: implicit ability stereotypes are self-serving for boys and men, but
not for girls and women. Sex Roles, 64(56), 324335.
Steinpreis, R. E., Anders, K. A., & Ritzke, D. (1999). The impact of gender on the review
of the curricula Vitae of job applicants and tenure candidates: a national empirical
study. Sex roles, 41(78), 509528.
Timmins, L., Barnes-Holmes, D., & Cullen, C. (2016). Measuring implicit sexual response
biases to nude male and female pictures in androphilic and gynephilic men. Archives
of Sexual Behavior, 45(4), 829841.
90
Journal of Contextual Behavioral Science 6 (2017) 8090
Trneke, N. (2010). Learning RFT: an introduction to Relational Frame Theory and its
Clinical application. New Harbinger Publications.
Trix, F., & Psenka, C. (2003). Exploring the color of glass: letters of recommendation for
female and male medical faculty. Discourse and Society, 14, 191220.
UNESCO (2015). Women in Science. Retrieved 25th November 2015 from http://www.
uis.unesco.org/ScienceTechnology/Pages/women-in-science-leakypipeline-data-viz.
aspx
Vahey, N. A., Barnes-Holmes, D., Barnes-Holmes, Y., & Stewart, I. (2009). A rst test of
the Implicit Relational Assessment Procedure as a measure of self-esteem: Irish
prisoner groups and university students. The Psychological Record, 59(3), 371.
Vahey, N. A., Nicholson, E., & Barnes-Holmes, D. (2015). A meta-analysis of criterion
eects for the Implicit Relational Assessment Procedure (IRAP) in the clinical
domain. Journal of Behavior Therapy and Experimental Psychiatry, 48, 5965.