Toxicology

Letters
ELSEVIER Toxicology Letters 82/83 (1995) 203-209

Neurobehavioural test batteries: current status, critical

evaluation and new directions

Ann M. Williamson
National Inmtute of Occupational Health and Safety (Worksafe Australra). PO Box 58. Syciney. Austraha. 2001

Abstract

Neurobehavioural techniques are currently making a contribution to knowledge in toxicology which is being used
to set safety limits of exposure to hazards. This, however, puts pressure on researchers in the field to ensure that the
methods used are the best and most informative available. A review of the neurobehavioural tests currently in use
shows that there are a number of issues that need to be addressed to strengthen and increase their utility. Issues
include test selection and the validity, reliability and sensitivity of tests. bias and confounding of measurement in
the testing session and problems in interpreting the results.

Keywords: Neurobehavioral methods; Behavior

1. Introduction 2. Problems of making measures of behaviour

In the assessment of effects of toxic exposure,

2.1. Informed consent
neurobehavioural testing is beginning to come of
age. Increasingly, studies are using neuro-
behavioural techniques to look at the effects of Increasingly, due to ethical considerations,
exposure to potential toxicants for an increasing neurobehavioural toxicity testing requires in-
range of exposures. Despite the very healthy formed consent from study participants. This can
growth, the area is in danger of slowing and bias behaviour measurement through selecting
stagnating because of an apparent tendency to out only volunteers. To avoid such bias, where
avoid facing some critical issues mainly relating possible testing should be mandatory even
to problems of confounding of test measurement. though this often contravenes other ethical prac-
Now that the field is starting to mature, it is tices. Alternatively, we need to establish that
imperative that we tackle some of the issues in a non-participants do not differ from partici-
systematic fashion. This paper focuses on the pants on any important variables. For example,
reasons why we may not be getting accurate only workers who have relatively low exposure
measures of behaviour change in the face of toxic to a hazard may volunteer, so biasing assess-
exposure and why our interpretations of behav- ments of the neurobehavioural function of the
iour change may sometimes be misleading. group.

037%4274/95/$09.50 @ 1995 Elsevier Science Ireland Ltd. All rights reserved

SSDI 0378-4274(95)03556-Z
204 A.M. Williamson I Toxcology
2.2. Effects of motivation to perform
Motivational effects can take 2 opposite forms,
subjects trying to do very well (faking good) and
trying to do poorly (faking bad). Faking good
may be less of a problem in testing for toxic
exposure effects where the potential for com-
pensation payments is a possible reason for
changing motivational state. Even so, in many
situations some workers will try harder than
others, with consequent effects on test perform-
ance. It must be remembered, however, that all
behaviour requires a certain level of motivation
or incentive to generate it. This is particularly so
for tests involving memory or attention.
Research on motivation suggests, however,
that raising motivational level may not actually
produce large changes in performance. For ex-
ample, monetary incentives improved informa-
tion processing performance to a greater extent
than did practice on the tests but this effect was
only seen for a composite measure of perform-
ance [l]. Incentives did not change the relation-
ship between performance on information pro-
cessing tests and intelligence tests, indicating that
manipulating the emotional state of the indi-
vidual did not result in changes in mental capaci-
ty. For some tests, trying harder will not improve
performance much and can even make perform-
ance worse. Trying harder on a relatively easy
task is simply not possible [2]. The effects of
trying hard can sometimes be seen, however, in
tasks in which a speed-accuracy trade-off can
occur [3].
The opposite end of the motivation spectrum,
faking bad or malingering, is often raised as a
serious difficulty for the validity of neuro-
behavioural testing, because there is often con-
siderable payoff for occupationally or environ-
mentally exposed individuals to make the most
of, or exaggerate, any effects of a neurotoxicant.
In an investigation of the relationship between
monetary incentives and poor performance [4],
subjects were given varying levels of information
about how to fake poor performance on a
forced-choice memory test and studied test per-
formance under circumstances where motivation
to fake was manipulated by monetary reward if
Letters 82183 (1995) 203-209
their faking could not be detected. Faking sub-
jects produced poor performance consistent with
the nature of instructions on how to fake but
monetary reward by contrast did not produce
any additional negative effects on performance.
A few techniques have been used to assess the
possible confounding effects of motivation level.
The overall approach is conservative in attempt-
ing to detect individual subjects who may be
judged as ‘faking’. Techniques include methods
such as forced-choice tests [5], examining the
pattern of test performance such as variations in
the speed-accuracy trade-off [3] and using dis-
criminant function analysis [6]. A method which
has not been used in the assessment of neuro-
toxic action involves Signal detection methodolo-
gy [7]. This t ec h nique differentiates sensitivity to
detect the signal from the subject’s bias or
criterion which is their expectation about the
likelihood of the correct response occurring.
Measures of sensitivity are likely to remain
constant over situations where a subject’s criter-
ion setting might be affected by such factors as
external motivation. Consequently the method
would be extremely useful for both detecting the
existence and the extent of the motivation ef-
fects.
2.3. Classical confounders
2.3.1. Test situation
A large number of influences in the test
situation have been shown to affect performance.
All relate fundamentally to the general state of
the individual. Where the factor makes significant
changes to the individual’s level of arousal,
resulting in either too little or too much, per-
formance would be expected to suffer [8]. Evi-
dence is available that factors such as how hot or
cold it is, the effects of background noise, the
lighting level and the time of day can effect
performance [8]. While most studies attempt to
control for these factors by holding them con-
stant or by counterbalancing, test situations such
as those based in industry settings may make
such actions difficult or impossible. Clearly varia-
tions in the test situation must not be overlooked
 A.M. Williamson I Tox~olog~
as they can have relatively large effects on
performance.
23.2. Testing procedures
Factors due to the test procedure may also be
important. For example, the importance of
procedural rigour was demonstrated in a study of
the covariates of peripheral nerve function using
vibrotactile and thermal thresholds and nerve
conduction velocity and amplitude [9,10]. For all
measures, the examiner emerged as a major
covariate, with the difference between examiners
being as much as one standard deviation in
magnitude for some measures. It is not clear why
the examiner exerted such a large influence on
these measures as they differed in the degree of
subjectivity they involve, however these studies
strongly suggest that measures should be as
objective as possible. Methods such as comput-
erised testing, standardised protocols and interra-
ter reliability need to be brought into play here
to avoid these sorts of problem.
2.3.3. Effects of repeated exposure to the test
It is well known that repeated exposure to a
test can influence performance through either the
effects of practice or fatigue. For many tests the
effects of practice or of fatigue are simply not
known. Test performance can be changed sig-
nificantly due to both practice and fatigue. Evi-
dence shows that a single repetition of a test can
produce a practice effect which is sufficient to
counter the effects of 20 or more years of ageing
[ll]. For all tests. asymptotic levels of perform-
ance are reached before measures are taken to
represent stable performance [12], and taking
care to avoid long periods of testing. However
the time taken to benefit from practice and to
show fatigue-related deterioration in perform-
ance can be used as a useful measure of toxic
effect. The signal detection method described
above would also be useful for establishing the
effects of fatigue on performance.
2.3.4. Characteristics of individuals
Differences between individuals are most often
raised as potential confounding factors. These
include the person’s age, their level of education,
Letters 82lS.Z (199.f) X-209 205
and the extent to which they use alcohol. While
most studies attempt to control for these factors,
they often do so without establishing either the
extent of the influence on test performance or
that the factors actually exert unwanted influ-
ences on the tests in question over the age-range
of the study group. Not all functions nor all
aspects of functions are affected by age, unless
the age differences are very great. For example,
it appears that motor performance slows and
becomes more variable with age [13], but mem-
ory capacity may not [14]. Factors like practice
and physical training have also been shown to
minimise the age-related differences between old
and young adults [13]. Studies showing differ-
ences in motor performance due to the age-
related decline compare very great differences in
age, with the older group typically in the 60 plus
age group. Most studies of neurobehavioural
effects involve subjects who are considerably
younger, particularly if focusing on exposures at
the workplace. Consequently, age-related differ-
ences in function may not be a significant prob-
lem in many studies.
These issues are also relevant to other as-
sumed confounding or nuisance variables. Taking
action to remove the effects of these variables
may not simply be a waste of time, particularly if
post-hoc statistical methods are used. If the
assumed confounding variable is significantly
correlated with exposure to a neurotoxicant, as is
often the case with factors like age, education
level, and alcohol use, attempts to reduce the
effect of the variable can also reduce the power
to detect a difference between exposed and non-
exposed subjects [15]. It is important to analyse
the relationships between confounding variables
and both dependent and independent variables
before designing a study.
3. Problems of making particular types of
behavioural measures
Particular types of measures may have par-
ticular problems or confounders associated with
them. Standard neuropsychological tests are
often used in studies because there is information
206 A.M. Williamson I Toxicology
on their validity and reliability [16,17]. The
detection of neurotoxic behavioural effects, how-
ever, places a number of constraints on the use
of both standard and novel tests.
3.1. Expectations of a test
In many assessment situations the perceived
aim or face validity of a test may have an implicit
influence on test performance for some indi-
viduals. For example, the motivation to perform
the test may depend on the subject’s expectations
from his/her experience of the test. Factors like
the title of the test and the level of complexity of
the test (perceived as too easy or too difficult),
may cause subjects to change their performance
in response to their expectations about any
consequences of particular performance on a
test. The extent that these features influence
performance is not understood. Clearly situation-
al factors may also play a role, but at the present
time we do not know how big a role. There is a
need for some careful research on the influence
of these implicit features on performance includ-
ing the face validity of tests.
3.2. Test representation and uniqueness
Two essential features of good measurement
are the extent to which the measures generated
are reliable and valid. Good test reliability is
particularly important because errors of mea-
surement can obscure the relationship between
dependent (test performance) and independent
(exposure) variables. Test developers and users
in this area have used tests regardless of the
reliability of tests and measures. For example,
one study [18] demonstrated that test-retest
reliabilities were only moderate for a large num-
ber of measures from the Neurobehavioral
Evaluation System (NES) [19], one of the most
commonly used test batteries. This has implica-
tions for the utility of this test system in many
settings. In contrast, the approach used in the
development of the automated performance test
system (APTS) [12] selected tests on the basis of
their stability over practice for means, variances
Letters 82183 (1995) 203-209
and intertrial correlations. The measures used in
these batteries have the strength of having
known, high reliabilities available before the
measures are used to any great extent. Kantowitz
[20] makes the point, however, that this approach
does not deal with the issue of validity or
uniqueness of measures. He questions the com-
mon practice of classifying measures into per-
ceptual, motor or cognitive domains, in absence
of any theory to guide test or measure selection.
He argues that functional groupings are some-
what arbitrary, with little other than expert
judgement to support them, and they often
overlook the range of functions required to
perform each test and consquently hamper inter-
pretation of test results. For example, most tests
in neurobehavioural toxicology rely on visual
stimulation, a fact that is usually overlooked in
interpreting study results. If the toxicant does
have effects on the visual system, these could be
fallaciously interpreted as deficits in another
functional domain such as motor performance or
memory. It has been demonstrated that visual
impairments can indeed adversely affect per-
formance on tests which use vision as a vehicle to
deliver test stimuli to other parts of the cortex
WI.
A number of authors have put forward an
alternative approach [l&20,22,23] maintaining
that the field needs tests that relate to an overall
model or theoretical framework. Often the
choice of tests is defended on the grounds that
they are a pragmatic representation of what
functions might be affected based on previous
research. Kantowitz [20] in contrast maintains
that in selecting human performance tests ‘theory
is the best practical tool’.
3.3. Sensitivity of tests
Tests need to be sufficiently sensitive to detect
neurotoxic effects where they exist. Tests with
high reliability and good validity will help to
reduce the error variance so as to reveal effects
due to the independent variable, exposure. Tests
from traditional batteries may appear to be
useful as they usually have good psychometric
 A.M. Williamson I Toxicology Letters 82183 (1995) 203-209
qualities. They are, however, designed to assess
clinically affected individuals, and are not chal-
lenging enough or do not have a large enough
range of measurement to pick up deficits before
they become either irreversible or too limiting to
the individual’s function. Alternative test meth-
ods for performance testing should be considered
including more complex tests such as dual tasks
and grammatical reasoning, self-paced testing
[24], adaptive testing [25] and signal detection
theory [ll].
The issue of sensitivity in this area is rather
like walking a tightrope. From the view point of
interpretation, there is a limit to increasing
sensitivity of tests. Tests can become too sensi-
tive when they detect differences between in-
dividuals due to irrelevant factors such that the
effects of the independent variable can be over-
shadowed. The real issue here is the acceptable
level of concern. When is the size of the exposure
effect large enough to require action? Simply
showing that exposed groups are different from
non-exposed may not be sufficient in practical
terms if the size of the difference is not taken
into account. Choosing a criterion for judgement
is not easy. Some authors suggested that the
effects of aging constitute a good criterion
[26,27]. Other alternatives include effects on
daily functioning and quality of life [15] and
methods to assess delayed neurotoxic effects
P31.
In a few instances researchers have tried to
anchor the size of performance deficits in terms
of the effects of an agent which is known and
accepted to be neurotoxic. For example, 2 groups
have developed dose-response curves for the
effects of alcohol on performance in their test
batteries [12,28]. In both cases the aim was to
create a form of calibration or criterion validity
on which effect sizes of various exposures could
be judged. It does, however, still beg the ques-
tion of when we should be concerned. Should it
be at the equivalent of 0.05 blood alcohol, or one
night without sleep? The issue of practical conse-
quences of exposure effects which are picked up
by sensitive tests is an important one. We need to
pay attention to it if neurobehavioural testing is
to be truly useful in toxicology.
207
4. Problems of the interpretation of behaviour
Issues also exist around the interpretation of
the results of neurobehavioural tests. These
issues mainly stem from problems due to the
nature of the study design and the consequent
analysis of test results.
4.1. Purpose of the study
Studies of neurobehavioural effects are con-
ducted with 2 main purposes, to screen or estab-
lish whether particular individuals are effected by
exposure and to diagnose or determine the
nature of the effect. It is argued that these 2
purposes have implications for the inclusion of
different types of tests in a battery [29]. Tests for
screening need to be suitable for detecting the
specific effect of the toxicant. Diagnosis, on the
other hand, requires a much more comprehen-
sive battery in order to look at the range of
possible effects of the substance. The problem is
that the diagnostic purpose really needs to be
done first and done well in order to establish the
best tests for screening individuals. The effective-
ness of screening depends to a great extent on
the effectiveness of diagnosing what are the real
effects of the toxicant. Too often, in this field, the
diagnostic work is largely equivocal or does not
clarify which specific effects are characteristic of
the neurotoxicant. The screening approach is
taken too early. As a consequence it leads to
poor screening and is probably one of the
reasons for the lack of resolution about the
effects of particular neurotoxicants such as or-
ganic solvents.
4.2. Individual differences in basic capacity
This is an issue because studies in neuro-
behavioural toxicology tend to be cross-sectional
in design. Attempts to match exposed and con-
trol groups on confounding variables may not be
very successful. A variety of methods have been
used to control for baseline intellectual capacity,
including pre-existing intelligence or vocational
test results taken before any toxic exposure had
occurred [30], the use of monozygotic twins [31]
208 A.M. Williamson I Toxico1og.v Letters
prospective study designs in which baseline non-
exposed performances are measured and com-
pared to performances at intervals over a period
of exposure years [32] and through use of mea-
sures taken currently but judged to represent
basic intellectual capacity such as the vocabulary
test from the WAIS [33]. The first 2 methods can
be difficult to achieve so have not been used very
frequently in neurobehavioural toxicology
studies. The last method is used much more
often, but presents some particular problems as it
is very likely that the measures of basic capacity
taken after exposure may also be affected by
exposure [34]. In fact it is possible that any
measure of intellectual capacity may be affected
by the neurotoxicant so making it impossible to
use these measures to show prior baseline func-
tioning once exposure has already occurred.
5. Conclusions
Neurobehavioural testing is at an important
place in its development. It is now clearly estab-
lished and accepted as a methodology which can
add to the body of knowledge about how toxic
exposure can affect health and safety. Unfor-
tunately, however, there are many loose ends to
this development. Many issues are unresolved
and are increasingly being ignored in favour of a
pragmatic approach, ‘let’s get measuring’. Now is
the time, however, for us to reassess our tools
and to attempt to build up the strengths and
work on the weaknesses. Issues associated with
the choice of tests, reliability, validity, sensitivity
and the role of confounders must be addressed
before the field can move on.
Acknowledgements
I am grateful to Deborah Biancotti for her
assistance in producing this paper. The views
expressed in this paper are those of the author
and do not necessarily reflect those of the Na-
tional Occupational Health and Safety Commis-
sion.
82183 (1995) 203-209
References
[l] Larson, G.E., Saccuzzo. D.P. and Brown, J. (1994)
Motivation: cause or confound in information process-
ing/intelligence correlations? Acta Psychol. 85. 25-37.
[2] Kahneman. D. (1973) Attention and Effort. Prentice-
Hall, Englewood Cliffs, New Jersey.
[3] Williamson, A.M., Clarke, B. and Edmonds, C.W. (1989)
The influence of diving variables on perceptual and
cognitive functions in professional shallow water
(abalone) divers. Environ. Res. 50. 93-102.
[4] Martin, R.C., Bolter, J.F.. Todd, M.E.. Gouvier, W.D.
and Niccolls, R. (1993) Effects of sophistication and
motivation on the detection of malingered memory
performance using a computerized forced-choice task. J.
Clin. Exp. Neuropsychol. 15(6), 867-880.
[5] Pankratz, L. (1983) A new technique for the assessment
and modification of feigned memory deficit. Percept.
Motor Skills 57, 367-372.
[6] Bernard, L.C.. McGrath, M.J. and Houston. W. (1993)
Discriminating between simulated malingering and
closed head injury on the Wechsler Memory Scale-Re-
vised. Arch. Clin. Neuropsychol. 8. 539-551.
[7] Green. D.M. and Swets. J.A. (1966) Signal Detection
Theory and Psychophysics. Wiley, New York.
[8] Hockey, G.R. (1986) Changes in operator efficiency as a
function of environmental stress, fatigue and circadian
rhythms. In: K.R. Boff, L. Kaufman and J.P. Thomas
(Ed.), Handbook of Perception and Human Perform-
ance: Cognitive Processes and Performance,Vol. II, John
Wiley & Sons. New York, pp. 44-49.
[9] Letz. R. and Gerr. F. (1994) Covariates of human
peripheral nerve function. I. Nerve conduction velocity
and amplitude. Neurotoxicol. Teratol. 16(l), 95-104.
[lo] Gerr, F. and Letz. R. (1994) Covariates of human
peripheral nerve function. II. Vibrotactile and thermal
thresholds. Neurotoxicol. Teratol. 16(l). 105-112.
[ll] Welford, A.T. (1992) Psychological studies of aging:
then origins, development and present challenge. Int. J.
Aging Hum. Dev. 34(3), 185-197.
[12] Turnage. J.J. and Kennedy, R.S. (1992) The develop-
ment and use of a computerized human performance
test battery for repeated-measures applications. Hum.
Perf. 5(4), 265-301.
[13] Stelmach, G.E. and Nahom, A. (1992) Cognitive-motor
abilities of the elderly driver. Hum. Fact. 34(l), 53-65.
[14] Salthouse, T.A. (1990) Influence of experience on age
differences in cognitive functioning. Hum. Fact. 32(5),
551-569.
[15] Gullion. C.M. and Eckerman, D.A. (1986) Field testing.
In: Z. Annau (Ed.), Neurobehav. Toxicol., Johns Hop-
kins Press, Baltimore.
[16] Lezak. M.D. (1983) Neuropsychological Assessment.
2nd Ed., Oxford University Press, New York.
[17] Wechsler. D. (1981) Manual for the Wechsler Adult
Intelligence Scale - Revised. The Psychological Corpo-
ration. New York.
 A.M. Williamson I Toxicology Letters 8218.3(1995) 20.3-209 NY

[18] Arcia. E. and Otto, D.A. (1992) Reliability of selected sures of cogmttve functioning. Neurobehav. Toxmol.
tests from the Neurobehavioral Evaluation System. Teratol. 7, 345350.
Neurotoxrcol. Teratol. 14, 103-l IO. [28] Allen, R.W., Stein, A.C. and Mrller, J.C. (1990) Per-
[lY] Baker. E.L. and Letz, R. (1986) Neurobehavioural formance Testmg as a Determmant of Fitness-For-Duty.
testing in monitoring hazardous workplace exposures. J. Paper presented at the Aerospace Technology Confcr-
Occup. Med. 28( 10). 987-990. ence and Exposition. Long Beach. California
[20] Kdntowitz. B.H. (1992) Selecting measures for human [29] Johnson. B L. (1987) Prevention of Neurotoxrc Illness in
factors research. Hum. Fact. 34, 387-398. Working Populations. Wiley, Chrchester.
[21] Kempen. J.H., Krttchevsky. M. and Feldman. S.T. [30] Hanninen. H.. Eskelmen, L.. Husman, K. and Nur-
(1994) Effect of visual impairment on neuropsychologi- minen. M. (1976) Behavioral effects of long-term expo-
cal test performance. J. Clm. Exp. Neuropsychol. 16(2). sure to a mixture of organic solvents Stand J. Work
223-231. Env. Health 4. 240-255.
[22] Smith. P.J. and Langolf. G.D. (1981) The use of Stern- [31] Hannmen. H.. Antti-Polka. M. Juntunen. J. and Kos-
berg’s memory-scanmng paradigm in assessing the ef- kenvuo, M. (1991) Exposure to orgamc solvents and
fects of chemical exposure. Hum. Fact. 23. 701-708. neuropsychologrcal dysfunction: a study on monozygotrc
[23] Wtlliamson. A.M. (1990) The development of a neuro- twms. Br. J. Ind. Med. 48. 18-25.
behavroral test battery for use m hazard evaluatrons m [32] Williamson, A.M. and Winder. C (1993) A prospective
occupational settings. Neurotoxicol. Teratol. 12, SO9- cohort study of the effects of solvent exposure. Env.
51-t. Rcs. 62. 256-371.
[24] Scerbo. M.W., Greenwald. C.Q. and Sawain, D.A. (1993) [33] Hooisma, J.. Hanninen. H , Emmen. H.H. and Kuhg.
The effects of subject-controlled pacing and task type on B.M. (1993) Behavioural effects of exposure to orgamc
sustamed attention and sublective. J. Gen. Psychol. solvents in Dutch painters. Neurotowicol. Teratol 15,
120(3). 2933307. 397-406.
[2S] Werss, D J. and Vale, C.D. (1987) Adaptrve Testmg. [34] Baker, E.L.. White. R.F., Pothrer. L.J., Berkley. CS..
Appl. Psychol. Int. Rev. 36, 249-262. Dmse. G E , Travers. P.H.. Harley. J.P. and Feldman.
[26] Weiss, B. (1983) Behavioral toxicology and environmen- R.G. (1985) Occupatronal lead ncurotoxtctty: rmprove-
tal health science: opportumty and challenge for psy- ment m behavroural effects after reduction of exposure.
chology. Am Psychol. 38, 1174-1187. Br. J. Ind. Med. 42, 5077516.
[27] Smith, P.J. ( 1985) Neurobehavioural toxicology: mea-