The Clinical Neuropsychologist

ISSN: 1385-4046 (Print) 1744-4144 (Online) Journal homepage: http://www.tandfonline.com/loi/ntcn20

Current limitations of neuropsychological tests

and assessment procedures

Diane Howieson

To cite this article: Diane Howieson (2019): Current limitations of neuropsychological tests and
assessment procedures, The Clinical Neuropsychologist, DOI: 10.1080/13854046.2018.1552762

To link to this article: https://doi.org/10.1080/13854046.2018.1552762

Published online: 04 Jan 2019.

Submit your article to this journal

Article views: 25

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=ntcn20
THE CLINICAL NEUROPSYCHOLOGIST
https://doi.org/10.1080/13854046.2018.1552762

ARTICLE

Current limitations of neuropsychological tests and

assessment procedures
Diane Howieson
Department of Neurology, Oregon Health & Science University, Portland, OR, USA

ABSTRACT ARTICLE HISTORY

Objective: Neuropsychological tests are used for a wide variety of Received 26 March 2018
purposes: assessing basic cognitive abilities or disabilities; deter- Accepted 18 November 2018
mining cognitive patterns associated with brain disorders or
injury; exploring brain-behavior relationships; and, in some cases, KEYWORDS
addressing psycho-legal issues. This paper will focus on the limita- Neuropsychological tests;
neuropsychology; cognitive
tions of current neuropsychological tests for these purposes. neuroscience; functional
Method: Current limits of neuropsychological tests and assess- neuroimaging
ment procedures are reviewed and recommendations for
improvements are made. The relationship between the concep-
tual basis of neuropsychological tests and modern theories of
brain functional organization is discussed. In addition, psycho-
legal concerns are briefly outlined.
Conclusions: Existing tests and techniques can be improved.
Normative data for some neuropsychological tests are based on
small samples or have limited validity or reliability data. A variety
of neuropsychological tests are available but particular cognitive
domains are underrepresented, particularly those involving high-
level cognitive skills and social skills. Progress is being made in
the availability of tests in multiple languages, but the restricted
range of languages and cultural influences demands more atten-
tion. Standard test procedures often deviate from real-world activ-
ities. In this modern age, some tests have become too familiar
because of the Internet. Understanding of complex neural net-
works activated by cognitive tasks is only beginning to be appre-
ciated. Procedures for determining psycho-legal issues need
improvements in some areas.

Assessing basic cognitive abilities or disabilities

Neuropsychological assessment procedures continue to evolve in scope and effective-
ness. Evaluations have many uses ranging from assessing basic cognitive abilities or
disabilities, predicting prognosis from brain injury, exploring brain-behavior relation-
ships, and, in some cases, determining psycho-legal issues. Many tests have been
developed specifically for evaluating people with known or suspected brain

CONTACT Diane Howieson howiesod@ohsu.edu Department of Neurology (CR131), Oregon Health & Science
University, 3181 SW Sam Jackson Park Road, Portland, OR 97239 USA
ß 2018 Informa UK Limited, trading as Taylor & Francis Group
2 D. HOWIESON

dysfunction (Lezak, Howieson, Bigler, & Tranel, 2012) and this paper will focus on areas
of needed improvement.

Limitations of frequently used tests

Although assessment of memory is one of the most frequent concerns, many memory
tests do not have alternate versions equated for difficulty with the original that can be
used for re-test purposes and this includes the widely used Wechsler Memory Scale-IV.
As another example, data are limited on the equivalency of alternate forms of the
often-used Rey Auditory Verbal Learning Test.
Certain tests can be failed for any one of several reasons, making interpretation
unclear. For example, Animal Fluency might be failed because lack of initiation or per-
sistence of an effective strategy, random responding, or cultural differences. Many
tests take too long to administer. A prime example is the frequently used Wisconsin
Card Sorting Test (WCST; Heaton, Chelune, Talley, Kay, & Curtiss, 2003), which can take
30 min. In this era of pressure to reduce the cost of examinations, more streamline
procedures for obtaining data efficiently are needed.

Need to expand the range of neuropsychological tests

During standard examination procedures emotional influences often are minimized as
much as possible because emotional activation can interfere with cognitive control
processes. However important information is lost under such controlled circumstances,
particularly when assessing individuals with a history of poor judgment or aberrant
behavior. A variety of rating scales focus on regulation of behavior and emotions
(Tate, 2010), but few tests are designed to address these questions. Some problem-
solving tests such as the WCST and The Iowa Gambling Test (Bechara, 2007) are likely
to elicit emotional reactions because of uncertainty of correct responses. Likewise,
timed tests that place a major emphasize on speed may cause arousal. Tests exploring
individual differences in emotional regulation should help identify patients who are
vulnerable to disruption of cognitive processes, although these procedures must be
justified and within ethical standards to do no harm.
Another less studied area is social cognition, which relies on a combination of cog-
nitive and emotional factors. In 2002, Carol Gregory and her colleagues studied the
ability of patients with frontotemporal dementia to recognize another person’s mental
state (theory of mind), to recognize if stories contain a faux pas, and to choose from
two possibilities what a person is feeling or thinking by looking at photographs of the
eye region (Gregory et al., 2002). Only a few tests of social cognition with normative
data are available such as: The Awareness of Social Interference Test—TASIT
(McDonald, Flanagan, Rollins, & Kinch, 2003), the EMOTICOM (Bland et al., 2016), and
the Social Cognition and Emotional Assessment-SEA (Funkiewiez, Bertoux, de Souza,
Levy, & Dubois, 2012). More tests are needed to evaluate social cognition associated
with conditions such as traumatic brain injury, Huntington’s disease, Parkinson’s dis-
ease, and psychiatric conditions.
 THE CLINICAL NEUROPSYCHOLOGIST 3

Language and cultural considerations

Another goal has been to improve the application of tests for diverse languages and
cultures. Diversity around many parts of the globe is increasing at a steady pace.
Cross-cultural progress is being made in the availability of tests in languages other
than English and from different cultures, although not as much as needed. An
example is the Montreal Cognitive Assessment (MoCA), a brief screening examination
available in 16 languages (Nasreddine et al., 2005).
Translating a test from one language to another does not eliminate the need to
assess cultural influences. Familiarity of items will vary from culture to culture and
tests need to be adapted as well as translated. When scores on the WAIS-III were com-
pared across similar cultures, people in France, Germany, and the UK performed better
on Block Design and Matrix Reasoning than people in the US. Using the US norms,
the average Scaled Scores for people in these countries were greater than the US
average of 10 (Roivainen, 2010). The reason for these differences warrants further
study. The American Academy of Clinical Neuropsychology is launching a project to
develop an interactive platform for multi-cultural/multi-lingual norms for open access
cognitive tests (American Academy of Clinical Neuropsychology, 2017).

Tests’ reliability and validity

An important factor that allowed for the refinement of neuropsychological assessment
techniques was the development by educational psychologists of examination techni-
ques based on large-scale, statistic dependent testing procedures (Stigler, 1992). As
examples, the Wechsler intelligence and memory tests (WAIS and WMS versions) as
well as other achievement tests were originally designed for educational assessment
purposes. The adoption by neuropsychologists of standard tests and procedures is
used to increase the accuracy of interpretation of results when comparing an individu-
al’s performance with data obtained from a normative group or a diagnostic group
using the same procedures.
Test characteristics such as reliability, construct and convergence validity, sensitivity
and specificity, and positive predictive value are used to determine tests’ usefulness.
Large batteries typically provide good validity data although more up-to-date
approaches might be incorporated. A systematic review of 139 articles reporting valid-
ity data of neuropsychological tests found that few studies used modern validity pro-
cedures or emphasized item development (Pawlowski, Segabinzai, Wagner, & Bandeira,
2013). Other than large batteries like the Wechsler tests, tests in use often have been
derived from research projects and the normative data come from small samples,
which likely results in limited reliability and reliable change indices. For example, there
are numerous versions of the Stroop Test with either small samples sizes or incom-
plete demographic information. See Bilder and Reis (this issue) and Miller (this issue)
for suggestions on how neuropsychology data across centers or practitioners might be
combined to achieve better normative data. Furthermore, some tests suffer from prob-
lems with inter-rater reliability, such as scoring of the Rey-Osterrieth Complex
Figure copy.
4 D. HOWIESON

Ecological validity
Standard procedures provide advantages but they have limitations. In an attempt to
control the examination, procedures often deviate from real-world demands. Tasks
tend to be administered one at a time and in isolation from other activities. In add-
ition, the examiner’s structure of the testing environment eliminates distractions out-
side of the test demands, or nearly so. Most real-world activities occur while
experiencing significant environmental distractions. Ecological validity has become an
increasingly important consideration.
Everyday activities involve the interaction of complex variables and tests rarely
probe complex situations despite the need to understand the context in which a per-
son’s cognitive problems occur. For example, structured tests may not require the indi-
vidual to conceptualize a problem and plan a solution. Copying a complex figure or
deciding a strategy for generating words within a category requires the individual to
select an approach. Other more classic examples are the various tower tasks and maze
tasks that require problem solving and thinking ahead for success. These less struc-
tured tests may be necessary to show an individual’s problem with the initiation, plan-
ning or persistence required in everyday activities. With these exceptions few tests
require breaking down complex actions into manageable units and prioritize them in
the right order or address a person’s ability to analyze and evaluate an argument or
situation. The use of unstructured problem-solving tests like Twenty Questions from
the Delis-Kaplan Executive Function System (Delis, Kaplan, & Kramer, 2001) addresses
some of these shortcomings.
Only a small number of tests, such as the Rivermead Behavioural Memory Test—
Third Edition (RBMT-3; Wilson et al., 2008), are designed to assess the ability to suc-
ceed at everyday tasks such as remembering name-face associations, where an article
of clothing has been placed, or to perform a task at a future time. A meta-analysis of
the ability of neuropsychological tests to predict ability to work found that measures
of executive functioning, intellectual functioning, and memory were the best predic-
tors of employment status (Kalechstein, Newton, & van Gorp, 2003). Other ecologically
oriented approaches for assessing everyday functioning are reviewed in Marcotte and
Grant (2009).

Test familiarity
Most tests have been in use since the 1990s or before, although some have updated
version. In this era of easily available computer searches some tests have become too
familiar through Internet searches and publicity. The Anna Thompson story and the
Rey-Osterrieth figure test are easily found online. How many people looked up the
MoCA after the US President Trump reported he passed?

Computer administered tests

Many current tests are standard paper-and-pencil or question-and-answer tests. Most
require giving a fixed set of items. However, neuropsychological assessment is on the
verge of a number of changes in procedures as the variety of computer-based
 THE CLINICAL NEUROPSYCHOLOGIST 5

assessment programs and advanced biomedical informatics become available (Rabin

et al., 2014: see Bilder & Reise, this issue; Parsons & Duffield, this issue). As examples,
the Cognitive Toolbox (www.nihtoolbox.org) and NIH EXAMINER (Kramer et al., 2014)
were developed by the US National Institute of Health as brief computer/administrator
assessment tools to provide common data across research projects. Computerized ver-
sions of tests have advantages for measuring speed and response consistency. On the
other hand, they have disadvantages as well. Some aspects of performance cannot be
captured, such as free recall of memory material. Observations of patient’s effort may
be lacking or the absence of a personal interaction with an examiner might decrease
motivation to perform well (Yantz & McCaffrey, 2007).

Assessing specific brain disorders

If known, test selection depends on the region(s) of the brain injury. For diseases that
produce a predictable cognitive profile, some test batteries have been developed:
Alzheimer’s disease (Verma et al., 2015; Weintraub et al., 2009), Huntington disease
(Stout et al., 2014), multiple sclerosis (Benedict et al., 2002, 2012), frontal injuries
(Dubois, Slachevsky, Litvan, & Pillon, 2000), and schizophrenia (Gold et al., 2012). Other
common neurological diseases such as Parkinson’s disease, Lewy Body disease, and
vascular dementia including CADASIL lack an established neuropsychological battery
and tend to have data based on small studies.

Neuroscientific basis of neuropsychological assessment

Foundation of examination procedures
Modern roots of clinical neuropsychological assessment extend to the mid-twentieth
century and can be traced largely to the work of A.R. Luria, who developed a theory
of brain-behavior relationships and instruments for studying higher cortical functions
from his study of World War II soldiers that sustained gunshot wounds to the brain
(Luria, 1966). During the following decades many people with discrete brain lesions
were studied, leading to a fuller understanding of the varied roles of specific brain
areas in cognition and behavior.
We are learning that assumptions about what brain areas are critical for particular
cognitive tasks based on lesion studies are wrong or incomplete in some cases.
Advances in functional neuroimaging, particularly functional magnetic resonance
imaging (fMRI), have made it possible to image networks of brain activity during the
performance of short cognitive tasks by measuring localized changes in blood oxygen
use when areas are activated. The response mode typically is a finger action.
Seemingly simple cognitive and motor tasks engage large interconnected
brain regions.
As an example, the WCST is frequently interpreted as a test of executive function
based on the requirement that success depends on concept attainment, working
memory, maintaining behavior based on rewards, and shifting behavior when rewards
shifts to a new concept, all generally regarded as frontal lobe functions. Lesion studies
have found impaired performance by patients with various lesions within the frontal
6 D. HOWIESON

lobe and, in some cases, outside of the frontal lobe (Lezak et al., 2012). An fMRI study
adapted the WCST to examine neural networks activated during the various cognitive
requirements on successively simpler variants of the task (Lie, Specht, Marchall, & Fink,
2006). The observed pattern of neural activation supported previous findings of the
important role of the right dorsolateral prefrontal cortex for the standard version of
the task. An additional central finding was that activation of the rostral anterior cingu-
late cortex was associated with activation of the bilateral temporoparietal junction
compatible with an attentional network for error detection while activation of the cau-
dal cingulate cortex correlated with activation of the right dorsolateral prefrontal cor-
tex, consistent with increased attentional control in the context of increased working
memory demands. Studies such as these are expected to make a major contribution
to our understanding of brain functions during performance of neuropsychological
tests (Cooper & Shallice, 2011).
Several limitations to fMRI studies exist. Because of expense and limits in task
administration, fMRI is mostly a research rather than clinical tool. Also, many standard
neuropsychological tests are not suitable for fMRI studies because they are too lengthy
or require an action by the patient that causes the head to move. However, neuro-
psychological tests are being cleverly adapted for use during fMRI (e.g. Allen, Owens,
Fong, & Richards, 2011) and they are challenging established interpretations of fre-
quently used neuropsychological tests. As Price (2018) points out in her review of how
neuroimaging can be used to refine models of human brain functions, understanding
neural pathways for a task in normal and damaged brains should help predict, explain,
and lay the foundation for improving cognitive function after brain damage.

Psycho-legal issues
Neuropsychological evaluations for legal purposes are prepared for nonclinicians, often
the courts, which require clear explanations of procedures and findings. The more
common reasons for evaluations include the assessment of disabilities, determination
of capacity to make personal decisions, issues related to civil and criminal competen-
cies, and criminal responsibilities. Because of the potential gravity of opinions from
forensic evaluations, forensic neuropsychological evaluations demand the highest level
of expertise (Larrabee, 2017). Yet not all neuropsychologists who practice in this field
have the necessary training (Hirst et al., 2017).
Cultural factors such as race, acculturation, ethnicity, and education play are signifi-
cant factors when planning and interpreting data from forensic evaluations.
In most cases existing normative data appear unlikely to represent justice-involved
individuals due to significant demographic and clinical factors specific to this popula-
tion (LaDuke, DeMatteo, Heilbrun, Gallo, & Swirsky-Sacchetti, 2017). In addition, anxiety
and psychological arousal may impair self-regulatory efforts or cause justice-involved
individual to display nonverbal behaviors associated with deception.
Evaluation of response bias (malingering) is central to many evaluations and is
inherently fraught with complexities. It is easier to establish that a disability exist than
it does not exist. It is even possible that a person with an established brain injury may
exert low effort on tests or exaggerate problems for misguided reasons. Cognitive
 THE CLINICAL NEUROPSYCHOLOGIST 7

biases by forensic examiners also can easily lead to error even by people who are
motivated to avoid bias (Zapf, Kukucka, Kassin, & Dror, 2018). Standards have been
proposed for assessing response bias (Slick, Sherman, & Iverson, 1999) but assessment
variability exists from clinician to clinician. Better understanding of what is meant by
invalid performance is needed (Miller, Azelrod, Schutte, & Davis, 2017).

Summary
An impressive number and variety of neuropsychological tests are in use but some
have one or more deficiencies: problems with reliability or validity, inadequate norma-
tive data, lack of alternative forms, take too long to administer, lack ecological validity,
or have become too familiar to the public. Particular areas are underrepresented, espe-
cially those involving high-level cognitive skills and social skills. More and more tests
are available in multiple languages, but the range of languages and cultural influences
need more examination. The current advances in electronic technology and brain
imaging are making possible new assessment procedures and increasing the
understanding of brain-behavior relationships. Procedures for determining
psycho-legal matters require special expertise.

Disclosure statement
No potential conflict of interest was reported by the author.

References
Allen, M. D., Owens, T. E., Fong, A. K., & Richards, D. R. (2011). A functional neuroimaging ana-
lysis of the Trail Making Test-B: Implications for clinical application. Behavioural Neurology,
24(2), 159–171. doi:10.1155/2011/476893
American Academy of Clinical Neuropsychology. (2017). The Relevance 2050 Initiative. Retrieved
from https://theaacn.org/relevance-2050-initiative/#gsc.tab¼0
Bechara, A. (2007). Iowa Gambling Test (IGT) professional manual. Lutz, FL: Psychological
Assessment Resources.
Benedict, H. B., Amato, M. P., Boringa, J., Brochet, B., Foley, F., Fredrikson, S., … Langdon, D.
(2012). Brief International Cognitive Assessment for MS (BICAMS): International standards for
validation. BMC Neurology, 12(1), 55. doi:10.1186/1471-2377-12-55
Benedict, R. H. B., Fischer, J. S., Archibald, C. J., Arnett, P. A., Beatty, W. W., Bobholz,
J., … Munschauer, F. (2002). Minimal neuropsychological assessment of MS patients: A con-
sensus approach. Clinical Neuropsychologist, 16(3), 381–397. doi:10.1076/clin.16.3.381.13859
Bilder, R., & Reise, S. (this issue). Neuropsychological test of the future: How do we get there
from here? Clinical Neuropsychologist, 33(2). doi:10.1080/13854046.2018.1521993
Bland, A. R., Roiser, J. P., Mehta, M. A., Schei, T., Boland, H., Campbell-Meiklejohn, D. K., … Elliott,
R. (2016). EMOTICOM: A neuropsychological test battery to evaluate emotion, motivation,
impulsivity and social cognition. Frontiers in Behavioral Neuroscience, doi:10.3389/
fnbeh.2016.00025
Cooper, R. P., & Shallice, T. (2011). The roles of functional neuroimaging and cognitive neuro-
psychology in the development of cognitive theory: A reply to Coltheart. Cognitive
Neuropsychology, 28(6), 403–413. doi:10.1080/02643294.2012.679919
Delis, D. C., Kaplan, F., & Kramer, J. A. (2001). Delis-Kaplan executive function system. Toronto, CA:
PsychCorp.
8 D. HOWIESON

Dubois, B., Slachevsky, A., Litvan, I., & Pillon, B. (2000). The FAB: A frontal assessment battery at
bedside. Neurology, 55(11), 1621–1626. doi:10.1212/WNL.55.11.1621
Funkiewiez, A., Bertoux, M., de Souza, L. C., Levy, R., & Dubois, B. (2012). The SEA (Social cogni-
tion and Emotional Assessment): A clinical neuropsychological tool for early diagnosis of
frontal variant of frontotemporal lobar degeneration. Neuropsychology, 26(1), 81–90. doi:
10.1037/a0025318
Gold, J. M., Barch, D. M., Carter, C. S., Dakin, S., Luck, S. J., MacDonald, A. W., … Strauss, M.
(2012). Clinical, functional, and intertask correlations of measures developed by the cognitive
neuroscience test reliability and clinical applications for schizophrenia consortium.
Schizophrenia Bulletin, 38(1), 144–152. doi:10.1093/schbul/sbr142
Gregory, C., Lough, S., Stone, V., Erzinclioglu, S., Martin, L., Baron-Cohen, S., & Hodges, J. R.
(2002). Theory of mind in patients with frontal variant frontotemporal dementia and
Alzheimer’s disease: Theoretical and practical implications. Brain, 125(4), 752–764. doi:10.1093/
brain/awf079
Heaton, R. K., Chelune, G. J., Talley, J. L., Kay, G. G., & Curtiss, G. (2003). Wisconsin card sorting
test. Lutz, FL: Psychological Assessment Resources (PAR).
Hirst, R. B., Han, C. S., Teague, A. M., Rosen, A. S., Gretler, J., & Quittner, Z. (2017). Adherence to
validity testing recommendations in neuropsychological assessment: A survey of INS and NAN
members. Archives of Clinical Neuropsychology, 32, 1–16. doi:10.1093/arclin/acx009
Kalechstein, A. D., Newton, T. F., & van Gorp, W. G. (2003). Neurocognitive functioning is associ-
ated with employment status: A quantitative review. Journal of Clinical and Experimental
Neuropsychology, 25(8), 1186–1191. doi:10.1076/jcen.25.8.1186.16723
Kramer, J. H., Mungas, D., Possin, K. L., Rankin, K. P., Boxer, A. L., Rosen, H. J., … Widmeyer, M.
(2014). NIH EXAMINER: Conceptualization and development of an executive function battery.
Journal of the International Neuropsychological Society, 20(01), 11–19. doi:10.1017/
S1355617713001094
LaDuke, C., DeMatteo, D., Heilbrun, K., Gallo, J., & Swirsky-Sacchetti, T. (2017). The neuropsycho-
logical assessment of justice-involved men: Descriptive analysis, preliminary data, and a case
for group-specific norms. Archives of Clinical Neuropsychology, 32(8), 929–942. doi:10.1093/
arclin/acx042
Larrabee. (2017). Selection of tests and batteries for forensic neuropsychological evaluations. In
S. S. Bush, G. J. Demakis, & M. L. Rohling (Eds.), APA handbook of forensic neuropsychology.
Washington, DC: American Psychological Association.
Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012). Neuropsychological assessment
(5th ed.). New York, NY: Oxford University Press.
Lie, C.-H., Specht, K., Marshall, J. C., & Fink, G. R. (2006). Using fMRI to decompose the neural
processes underlying the Wisconsin Card Sorting Test. Neuroimage, 30(3), 1038–1049. doi:10.
1016/j.neuroimage.2005.10.031
Luria, A. R. (1966). Higher cortical functions in man. New York, NY: Basic Books.
Marcotte, T. D., & Grant, I. (2009). Neuropsychology of everyday functioning (The science and prac-
tice of neuropsychology). New York, NY: Guilford Press.
McDonald, S., Flanagan, S., Rollins, J., & Kinch, J. (2003). TASIT: A new clinical tool for assessing
social perception after traumatic brain injury. Journal of Head Trauma Rehabilitation, 18(3),
219–238.
Miller, J. B. (this issue). Big data and biomedical informatics: Preparing for the modernization of
clinical neuropsychology. Clinical Neuropsychologist, 33(2).
Miller, J. B., Azelrod, B. N., Schutte, C., & Davis, J. J. (2017). Symptom and performance validity
assessment in forensic neuropsychology. In S. S. Bush, G. J. Demakis, & M. L. Rohling (Eds.),
APA handbook of forensic neuropsychology. Washington, DC: American Psychological
Association.
Nasreddine, Z. S., Phillips, N. A., Bedirian, V., Charbonneau, S., Whitehead, V., Collin, I., …
Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: A brief screening tool for
mild cognitive impairment. Journal of the American Geriatrics Society, 53(4), 695–699. doi:
10.1111/j.1532-5415.2005.53221.x
 THE CLINICAL NEUROPSYCHOLOGIST 9

Parsons, T. D., & Duffield, T. (this issue). NIH’s transformative opportunities for clinical neuropsy-
chologists. Clinical Neuropsychologist, 33(2).
Pawlowski, J., Segabinzai, J. D., Wagner, F., & Bandeira, D. R. (2013). A systematic review of valid-
ity procedures used in neuropsychological batteries. Psychology & Neuroscience, 6, 311–329.
doi:10.3922/j.psns.2013.3.09
Price, C. J. (2018). The evolution of cognitive models: From neuropsychology to neuroimaging
and back. Cortex: Journal Devoted to the Study of the Nervous System and Behavior, 107, 37–49.
doi:10.1016/j.cortex.2017.12.020
Rabin, L. A., Spadaccini, A. T., Brodale, D. L., Grant, K. S., Elbulok-Charcape, M. M., & Barr, W. B.
(2014). Utilization rates of computerized tests and test batteries among clinical neuropsychol-
ogists in the United States and Canada. Professional Psychology: Research and Practice, 45(5),
368–377. doi:10.1037/a0037987
Roivainen, E. (2010). European and American WAIS-III norms: Cross-national differences in per-
formance subtest scores. Intelligence, 38(1), 187–192. doi:10.1016/j.intell.2009.10.001
Slick, D. J., Sherman, E. M. S., & Iverson, G. L. (1999). Diagnostic criteria for malingered neurocog-
nitive dysfunction: Proposed standards for clinical practice and research. Clinical
Neuropsychologist, 13(4), 545–561. doi:10.1076/1385-4046(199911)13:04;1-Y;FT545
Stigler, S. M. (1992). A historical view of statistical concepts in psychology and educational
research. American Journal of Education, 101(1), 60–70. doi:10.1086/444032
Stout, J. C., Queller, S., Baker, K. N., Cowlishaw, S., Sampaio, C., Fitzer-Attas, C., & Borowsky, B.
(2014). HD-CAB: A cognitive assessment battery for clinical trials in Huntington’s disease 1, 2,
3. Movement Disorders, 29(10), 1281–1288. doi:10.1002/mds.25964
Tate, R. L. (2010). A compendium of tests, scales and questionnaires: The practitioner’s guide to
measuring outcomes after acquired brain impairment. Hove, UK: Psychology Press.
Verma, N., Beretvas, S. N., Pascual, B., Masdeu, J. C., Markey, M. K., & The Alzheimer’s Disease
Neuroimaging Initiative. (2015). New scoring methodology improves the sensitivity of the
Alzheimer’s Disease Assessment Scale-Cognitive subscale (ADAS-Cog) in clinical trials.
Alzheimer’s Research & Therapy, 7(1), 64. doi:10.1186/s13195-015-0151-0
Weintraub, S., Salmon, D., Mercaldo, N., Ferris, S., Graff-Radford, N. R., Chui, H., … Morris, J. C.
(2009). The Alzheimer’s Disease Centers’ Uniform Data Set (UDS): The neuropsychology test
battery. Alzheimer Disease and Association Disorders, 23(2), 91–101. doi:
10.10097WAD.0b013e318191c7dd
Wilson, B. A., Greenfield, E., Clare, L., Baddeley, A., Cockburn, J., Watson, P., … Nannery, R.
(2008). Rivermead behavioural memory test-third edition (RBMT-3). Retrieved from www.person-
clinical.co.uk
Yantz, C. J., & McCaffrey, R. J. (2007). Social facilitation effect of examiner attention or inattention
to computer-administered neuropsychological tests: First sign that the examiner may affect
results. Clinical Neuropsychologist, 21(4), 663–671. doi:10.1080/13854040600788158
Zapf, P. A., Kukucka, J., Kassin, S. M., & Dror, I. E. (2018). Cognitive bias in forensic mental health
assessment: Evaluator beliefs about its nature and scope. Psychology, Public Policy, and Law,
24(1), 1–10. doi:10.1037/law0000153