ERIC/ECTJ Annual Review Paper
Criteria for Assessing the Trustworthiness
of Naturalistic Inquiries
Egon G, Guba
Egon G. Guba is professor of education, Smith Re-
search Center, Indiana University, Bloomington,
IN 47405.
This is the ninth ERIC/ECTJ Annual Re-
view Paper, preparation of which was
supported by the ERIC [Educational Re-
sources Information Center] Clearing-
house on Information Resources, Syra-
cuse University, Syracuse, New York. The
material in this article was prepared pur-
suant to a contract with the National Insti-
tute of Education, U.S. Department of
Education. Contractors undertaking such
projects under government sponsorship
are encouraged to express freely their
judgment in professional and technical
matters. Points of view or opinions do not
necessarily represent the official view or
opinion of NIE. The author is grateful to
David Clark, Larry Havlicek, Robert
Heinich, John McLaughlin, Cecil Miskel,
and Robert Wolf for their careful critique
of an earlier version of this paper, and to
his wife, Yvonna Lincoln, from whose joint
work with the author in other contexts
many of the ideas expressed in this
paper emanated.
ECTJ, VOL. 29, NO. 2, PAGES75-91
ISSN 0148-5806
This paper deals with the topic of criteria
for judging the trustworthiness of in-
quiries conducted within the naturalistic
inquiry paradigm. That paradigm, also re-
ferred to as the phenomenological, an-
thropological, or ethnographic, has be-
come increasingly popular in recent years;
reports of investigations carried out in this
mode often take the form of case studies or
"portrayals" (Stake, 1975) and bear little
resemblance to the kinds of articles we
have been accustomed to seeing in our
leading professional journals. Both the
novelty of the paradigm and the strange-
ness of the reporting format pose special
problems for editors and referees of jour-
nals, peer review committees or disserta-
tion committees considering, proposals,
and naturalistic investigators themselves
as they attempt to design and monitor
their inquiries. What precisely are the
criteria that ought to be applied to this
class of investigations? 1
1This paper will focus on only a subset of criteria,
those dealing with the trustworthiness of findings.
There are other methodological problems that are spe-
cial to naturalistic inquiry that cannot be dealt with
here. Among these are: b o u n d i n g problems, that is,
issues dealing with the means by which inquirers de-
termine what to include and exclude from considera-
tion; focusing problems, that is, issues dealing with
the means whereby i n q u i r e r s organize and ascribe
meaning to the data they do collect; and the problems
of investigator competence. The last-mentioned is es-
pecially important since, as will be seen, inquirers
often act as instruments (problems of the human as
instrument). Hence it is important to know about the
inquirer's training and experience in reaching a judg-
ment about the trustworthiness of his or her data. This
consideration has been omitted from the present paper
since it has been assumed that the refereeing of journal
76 ECTJ SUMMER1981
W h a t f o l l o w s is a p r i m i t i v e effort to a n -
s w e r t h a t q u e s t i o n . T h e effort is d e s c r i b e d
as " p r i m i t i v e " b e c a u s e , w h i l e p r a c t i t i o n -
ers of n a t u r a l i s t i c a p p r o a c h e s h a v e b e e n
reasonably introspective about what they
d o , t h e y h a v e n o t m a d e s y s t e m a t i c efforts
to c o d i f y t h e s a f e g u a r d s t h a t t h e y i n t u i -
tively build into their inquiries. Hence
t h e r e is little p r i o r c o n c e p t u a l i z a t i o n o n
w h i c h to b u i l d . It is m y h o p e t h a t t h i s arti-
cle, w h i l e it c a n n o t b e d e f i n i t i v e , will at
least open discussion on the matter and
will l e a d , i n d u e c o u r s e , to b e t t e r f o r m u l a -
t i o n s t h a n are p o s s i b l e n o w .
T h e p a p e r c o n s i s t s of five p a r t s . First,
n a t u r a l i s t i c i n q u i r y is d e f i n e d a n d d i f -
f e r e n t i a t e d f r o m t h e r a t i o n a l i s t i c 2 m o d e of
inquiry commonly practiced in the past
a n d still d o m i n a t i n g t o d a y . S e c o n d , t h e
i s s u e of what a s p e c t s of t r u s t w o r t h i n e s s
s h o u l d b e a d d r e s s e d b y t h e c r i t e r i a is
r a i s e d , a n d it is p r o p o s e d t h a t t h e r e are
four: i n t e r n a l v a l i d i t y , e x t e r n a l v a l i d i t y , re-
liability, and objectivity (although these
t e r m s are t r a n s l a t e d i n t o t h e i r n a t u r a l i s t i c
papers will continue to be managed by a "'blind"
method; that is, that the author's name and credentials
will not be among the materials that the referee is
asked to take into account. All three problem areas are
extensively discussed in Guba and Lincoln (in press);
the former two in Guba (1978).
~In previous discussions (Guba, 1978, 1979; Guba &
Lincoln, in press) we have referred to what is called
here the rationalistic paradigm as either the scientistic
or the scientific. The use of even the less pejorative of
these latter two terms now seems to us inappropriate
on two counts. First, readers have tended to view the
naturalistic paradigm as less scientific (or even as
nonscientific) and have, therefore, because of the
enormous legitimation accorded to anything scientific
in our culture, denigrated it as "less valid." Second,
several critics have accused us of setting up a straw
man, on the grounds that vanguard scientific thinkers
have moved beyond the 19th century epistemology of
logical positivism that we describe and ascribe to in-
quirers today. It is undoubtedly true that many scien-
tists now think differently, but such change does not
characterize the large majority of "'scientists" who are
engaged in inquiries today in either the "hard" or the
"soft" sciences. There, the old culture still dominates.
It is to that level of practice that our criticisms are di-
rected, and it is of that moribund culture that our de-
scriptions are apt. However, to avoid the unintended
meanings that some readers have drawn from our
work, we have shifted to the term rationalistic to de-
scribe the paradigm that informs conventional inquiry.
c o u n t e r p a r t s ) . T h i r d , a b r i e f e x p l i c a t i o n of
h o w t h e s e q u e s t i o n s are d e a l t w i t h i n c o n -
v e n t i o n a l i n q u i r y is p r e s e n t e d . F o u r t h , a
parallel or analogous mode for dealing
with them within the naturalistic
p a r a d i g m is o u t l i n e d . F i n a l l y , i m p l i c a t i o n s
of t h e p r o p o s e d c r i t e r i a are d e s c r i b e d . T h e
e x p o s i t i o n of t h e p a p e r w i l l b e n e c e s s a r i l y
b r i e f b e c a u s e of l i m i t a t i o n s b o t h o n s p a c e
a n d i n t h e c u r r e n t s t a t e of t h e art.
WHAT IS NATURALISTIC INQUIRY?
It is b e y o n d t h e s c o p e of t h i s p a p e r to p r o -
v i d e a d e t a i l e d d e s c r i p t i o n of n a t u r a l i s t i c
i n q u i r y ) I n t e r e s t e d r e a d e r s are r e f e r r e d to
G u b a (1978, 1979), G u b a a n d L i n c o l n ( i n
p r e s s ) , W o l f (1979), a n d W o l f a n d T y m i t z
(1977). T h e r e a d e r s h o u l d b e a w a r e , h o w -
e v e r , of t h e f o l l o w i n g :
The term " ' n a t u r a l i s t i c " describes a
paradigm for inquiry, not a method. T h e r e
are many paradigms for arriving at
" t r u t h , " i n c l u d i n g , for e x a m p l e , t h e legal
paradigm that characterizes courtroom
proceedings; the "expert judgment"
paradigm that characterizes accreditation
site v i s i t s , p e e r r e v i e w of p r o p o s a l s , a n d
j u d g m e n t s of a t h l e t i c p e r f o r m a n c e ; a n d t h e
m o d u s o p e r a n d i p a r a d i g m ( S c r i v e n , 1976;
t r a c i n g of c h a r a c t e r i s t i c causal c h a i n s ) t h a t
c h a r a c t e r i z e s f o r e n s i c p a t h o l o g y a n d tele-
vision repair.
Chief among the paradigms that have
b e e n u t i l i z e d i n s u p p o r t of d i s c i p l i n e d i n -
q u i r y are t h e r a t i o n a l i s t i c a n d t h e n a t u -
ralistic. T h e r e is n o b a s i s for c h o o s i n g o n e
of t h e s e p a r a d i g m s o v e r o t h e r s i n e a c h a n d
every inquiry situation. Rather, each rests
on certain assumptions that must be tested
i n t h e c o n t e x t of a p p l i c a t i o n . J u s t as it is
proper to select that analytic statistic
w h o s e a s s u m p t i o n s are b e s t m e t b y a set of
d a t a , so is it p r o p e r to select t h a t p a r a d i g m
w h o s e a s s u m p t i o n s are b e s t m e t b y t h e
phenomenon being investigated. Particu-
3Indeed, it would not be possible to provide such a
description at this time. Especially lacking are good
examples of naturalistic studies that have been com-
pleted and reported. ERIC can provide a valuable and
needed service by establishing a data bank of such
studies.

lar methods can be used in support of
either (or any) paradigm (see comments on
postures, below).
The naturalistic and rationalistic paradigms
differ on certain key assumptions. These in-
clude: 4
9 The nature of reality. The rationalistic
paradigm rests on the assumption that
there is a single reality upon which in-
quiry can converge, and that that reality is
separable or fragmentable into indepefiL-
dently manipulatable parts (commonly
called variables). Thus, certain variables
can be singled out for study (or control)
without essentially influencing others. The
naturalistic paradigm rests on the assump-
tion that there are multiple realities, that
inquiry will diverge rather than converge
as more and more is known, and that all
"parts" of reality are interrelated so that
the study of any one part necessarily influ-
ences all other parts.
9 The nature of the inquirer~object rela-
tionship. The rationalistic paradigm rests
on the assumption that the inquirer can
maintain a discrete (and discreet) distance
from the objects of the inquiry; that is, that
the relationship between the inquirer and
the object is essentially one of indepen-
dence. The naturalistic paradigm asserts,
instead, that the inquirer and the respon-
dent (note the shift in terminology from
"object") are interrelated, with each in-
fluencing the other. Of course naturalistic
inquirers make every effort to maintain an
optimal distance between themselves and
the phenomenon, but never for a moment
do they consider that the "optimal" dis-
tance is i m p e r v i o u s to inquirer-
respondent interchanges.
9 The nature of "truth statements." The
rationalistic paradigm rests on the as-
sumption that generalizations--enduring
truth statements that are context-free-- are
possible; indeed, it is frequently asserted
that inquiry would have no point if this
were not true. The rationalistic approach
~The three a s s u m p t i o n s p r e s e n t e d here are those
most salient for the analysis to follow. O t h e r a s s u m p -
tions also differentiate the p a r a d i g m s . The m o s t inter-
esting a m o n g these others is the extraordinary value
claim of rationalists that their i n q u i r y is value free.
CRITERIA FOR NATURALISTICINQUIRIES 77
aims at developing nomothetic knowledge
and hence focuses on the similarities be-
tween objects of i n q u i r y (similarities
being the stuff out of w h i c h gener-
alizations are made). The naturalistic
paradigm rests on the assumption that
generalizations are not possible, that at
best what one can hope for are "working
hypotheses" that relate to a particular con-
text. The naturalistic approach aims at de-
veloping idiographic knowledge, focusing
on differences between objects as fre-
quently and with as much interest as on
similarities, s
The issue here is not which assumptions
are "true" but which offer the best fit to
the phenomenon under study. 6 More and
more investigators have become con-
vinced of the relative utility of the natu-
ralistic paradigm for studying that class of
p h e n o m e n a that often is called social/
behavioral. Social/behavioral phenomena
exist chiefly in the minds of people, and
there are as many realities as persons. Re-
ality in this sense, moreover, gives the ap-
pearance of "whole cloth"; if one attempts
sit is of interest that while most i n q u i r y develops
from a n o m o t h e t i c posture, applications m u s t always
be m a d e in some i d i o g r a p h i c setting. There is thus a
m a j o r value m i s m a t c h in fields like counseling a n d
g u i d a n c e , school a d m i n i s t r a t i o n , a n d r e a d i n g in w h i c h
the research, done in the rationalistic m o d e , finds no
application in i n d i v i d u a l cases (clients, schools, non-
readers, a n d so on). The situation is similar to that
confronted b y particle physicists as formalized in the
H e i s e n b e r g Uncertainty Principle. Statistical laws h a v e
no i n d i v i d u a l application.
O A s s u m p t i o n s are d e f i n i t i v e l y n o t " s e l f - e v i d e n t
t r u t h s " as the axioms of Euclidian g e o m e t r y were rep-
resented to us in h i g h school m a t h e m a t i c s , b u t are,
instead, statements taken for g r a n t e d b y a g r e e m e n t (or
for the sake of the game). W h e t h e r the findings that
result from the application of a particular i n q u i r y have
a n y t h i n g to do with the p h e n o m e n o l o g i c a l world is a
matter for empirical test, not a r g u m e n t a t i o n . Thus,
while Euclidian g e o m e t r y is s p l e n d i d for dealing w i t h
earth-sized phenomena, Lobachevskian geometry,
b a s e d on different axioms, w o r k s v e r y m u c h better
w i t h astronomical p h e n o m e n a . The question of w h i c h
a s s u m p t i o n s are " r i g h t " is irrelevant; rather one uses
that g e o m e t r y w h o s e t h e o r e m s are f o u n d to be valid for
the space b e i n g dealt with. Similarly, we should not
ask w h e t h e r t h e a s s u m p t i o n s of the r a t i o n a l i s t i c
p a r a d i g m are more or less " r i g h t " than those of the
naturalistic, but rather w h i c h p a r a d i g m provides a bet-
ter " f i t " to the p h e n o m e n o n we seek to u n d e r s t a n d .
78 ECTJ SUMMER1981
to focus attention on certain portions of re-
ality, the whole falls apart as though the
cloth had been cut with scissors. Further,
while investigators may be able to main-
tain a neutral posture with respect to phys-
ical or chemical p h e n o m e n a (although
even that possibility is called into question
by the Heisenberg Uncertainty Principle),
it is impossible to do so when the objects
of investigation are people. Indeed, not
only do respondents exhibit reactivity but
so also does the investigator? Finally,
human behavior is rarely if ever context-
free; hence knowledge of human behavior
individually or in social groups is neces-
sarily idiographic, and differences are at
least as important as similarities to an un-
derstanding of what is happening. To the
extent that the conditions described are so,
the naturalistic paradigm becomes the
paradigm of choice.
The rationalistic and naturalistic paradigms
differ in terms of certain postures which
characterize their practitioners. While not
compelled to do so by the logic of the
paradigms they follow, practitioners of the
naturalistic and rationalistic paradigms
have displayed a propensity to take oppos-
ing positions along certain key dimen-
sions. While other variations in posture
might be described, the following afford
some insight into the characteristic dif-
ferences in approach taken:
9 Methods. Rationalistic practitioners
have preferred quantitative methods while
naturalistic practitioners have preferred
qualitative methods. This predisposition
is so intense that the conflict between the
two paradigms has frequently been mis-
taken for a conflict between quantitative
and qualitative methods, a mistake in logic
that has led to the generation of a great
deal more heat than light. But of course
these two dimensions are orthogonal;
there is no inherent reason why either
paradigm cannot accommodate, and be
contributed to, by either methodology.
9 Quality criterion. Proponents of the ra-
tionalistic approach have insisted that the
single most important criterion for asses-
sing the quality of an inquiry is its rigor,
while proponents of the naturalistic ap-
proach argue for relevance. The distinction
has sometimes been characterized as the
difference between saying, "It doesn't
matter what you do so long as you do it
well," versus saying, "Anything not worth
doing at all is certainly not worth doing
well!" These criteria are both worthy of at-
tention, but unfortunately they are in a
trade-off situation: The more one insists
on rigor (internal validity), and assures it
by control of the sort possible in a labora-
tory, the less relevance (external validity)
one can expect, for the results will apply
only in another laboratory.
9 Source of theory. Adherents of the ra-
tionalistic paradigm prefer a priori theory,
usually of the hypothetico-deductive type;
indeed, such theory is indispensable since
the rationalistic approach requires the
statement of hypotheses to be tested or
questions to be answered in advance. Such
hypotheses or questions can be generated
only from theory existing before the fact.
Adherents of the naturalistic paradigm
prefer to have the theory emerge from the
data themselves, that is, they wish the
theory to be grounded (Glaser & Strauss,
1967) and typically prefer their theory to be
of the pattern type (Kaplan, 1964).
9 Knowledge types used. Polanyi (1958)
d i s t i n g u i s h e s p r o p o s i t i o n a l knowl-
edge-knowledge that can be cast
into language form--from tacit
k n o w l e d g e - - k n o w l e d g e such as intui-
tions, apprehensions, or feelings that can-
not be stated in the form of language but
are s o m e h o w " k n o w n . " E v e r y o n e
"knows" more than he can communicate,
even to himself. Rationalistic inquirers
operate solely at the level of propositional
knowledge, primarily because everything
they investigate is cast into the form of
hypotheses or questions based on a priori
theory. Naturalistic inquirers insist on the
opportunity to build upon and expand
their tacit knowledge as well.
9 Instruments. Rationalistic practitioners
typically interpolate a "layer of instrumen-
t a t i o n " b e t w e e n t h e m s e l v e s and the
phenomena to be studied, partly because
it is believed that by thus removing them-
selves from direct contact they will im-
prove the reliability and objectivity of the
study, and partly because it is felt that
such "objective" instruments can be sharp-
ened and refined to a greater level of sen-

sitivity than can a human observer (both
assumptions are questionable; see Guba &
Lincoln, in press). Naturalistic inquirers,
on the other hand, are inclined to use
themselves as the instruments, willingly
trading off some objectivity and reliability
(in the rationalistic sense) in order to gain
greater flexibility and the opportunity to
build upon tacit knowledge (a feature that
paper-and-pencil or physical instruments
can never have).
9 Design. Rationalistic investigators in-
sist on a preordinate design (Stake, 1975),
that is, they require that every step from
problem specification through data collec-
tion and analysis to reporting be described
in advance. Of course their dependence on
a priori theory, propositional knowledge,
rigor as a quality criterion, and nonhuman
instrumentation (which must be devel-
oped) more or less requires that such a
posture be taken. Moreover, the typical
experimental design used by rationalists
prohibits any change in treatment once the
study is under way lest the variances be
confounded, disallowing interpretation of
the results. Naturalists, on the other hand,
believing in unfolding multiple realities,
in interactions with respondents that will
change both the investigators and the re-
spondents over time, and in grounded
theory, will insist on an emergent (unfold-
ing, rolling, cascading) design, which is
never complete until the inquiry is arbi-
trarily terminated as time, resources, or
other logistical considerations may dictate.
9 Setting. The adherent of the rationalis-
tic paradigm prefers to conduct inquiries
in the laboratory, since the laboratory af-
fords the epitome of control. The natu-
ralist, on the other hand, prefers to con-
duct inquiries in nature, inviting whatever
interference the real world can provide. It
is as though the rationalist is interested in
what happens in the best of all possible
worlds, while the naturalist is concerned
with what happens in the worst.
While adherence to one or another of
these sets of postures is not required nor
logically compelled by the underlying
axioms, followers of the paradigms seem
to be strongly inclined to do so, probably
because they were so trained. The assump-
tion of one of these postures has become
CRITERIA FOR NATURALISTICINQUIRIES 79
identified with the proper way to do
research--a kind of orthodoxy. But such
intransigence is unfortunate. Once the de-
cision to use a particular paradigm has
been made on grounds of "best fit" of as-
sumptions, compromises on postures are
not only possible but well advised, regard-
less of which paradigm has been selected.
Thus, both quantitative and qualitative
methods should be used as the situation
warrants. To seek an appropriate balance
between rigor and relevance seems sensi-
ble. A priori theory can be g r o u n d e d
through earlier inquiry, and to insist that
there should have been grounding at some
prior stage does not seem unreasonable.
Both tacit and propositional knowledge are
useful, and, indeed, it is probably the
hallmark of competent investigators that
they translate tacit knowledge into propo-
sitional knowledge as quickly as possible.
Astute investigators will utilize both
themselves and other instruments, de-
pending on circumstances. Some elements
of design can always be specified in ad-
vance, and the wise inquirer will specify
all such possible elements while retaining
a flexible posture that permits changes and
emendations as the situation may dictate.
Finally, information from both the labora-
tory and the real world has utility in
achieving understanding. Nevertheless, it
seems clear that both camps have evolved
orthodoxies, and that is a fact of life that
must be taken into account in specifying
criteria for judging trustworthiness that
will be found generally acceptable.
WHAT QUESTIONS OF TRUSTWORTHINESS
SHOULD BE ADDRESSED?
Guba and Lincoln (in press) suggest that
four major concerns relating to trustwor-
thiness have evolved, and it is to these
concerns that the criteria must speak. The
concerns as described by these authors are
these:
1. Truth value. How can one establish
confidence in the "truth" of the findings of
a particular inquiry for the subjects (re-
spondents) with which and the context in
which the inquiry was carried out?
2. Applicability. How can one determine
the degree to which the findings of a par-
80 ECTJ SUMMER198]

ticular inquiry may have applicability in s h o w i n g that a plausible alternative

other contexts or with other subjects (re-
spondents)?
3. Consistency. How can one determine
whether the findings of an inquiry would
be consistently repeated if the inquiry were
replicated with the same (or similar) sub-
jects (respondents) in the same (or similar)
context?
4. Neutrality. How can one establish the
degree to which the findings of an inquiry
are a function solely of subjects (respon-
dents) and conditions of the inquiry and
not of the biases, motivations, interests,
perspectives, and so on of the inquirer?
These four terms typically have been
labeled within the rationalistic paradigm,
as shown in Table 1; the parallel naturalis-
tic terms are also shown. The "translation"
requires some justification.
Truth Value. Within the rationalistic
paradigm, internal validity is logically de-
terminable by demonstrating isomorph-
ism or verisimilitude between the data of
an inquiry and the phenomena those data
represent--not an unreasonable expecta-
tion when one begins with an assumption
of a single reality upon which inquiry can
converge. But it is not possible to test
isomorphism directly--to do so would re-
quire absolute knowledge of what the real
world is like. Instead, rationalists fall back
on the strategy of ruling out all plausible
alternative explanations. Thus, one conse-
quence of the rationalistic approach is that
hypotheses can never be directly con-
firmed (since a test of isomorphism is not
possible) but they can be disconfirmed (by
hypothesis has a high probability of being
correct). Nevertheless, the fundamental
idea of isomorphism is useful, for within
the naturalist's framework, the analog of
isomorphism to reality must be isomorph-
ism to respondents' perceptions (multiple
realities existing in the minds of people).
In establishing truth value, then, naturalis-
tic inquirers are most concerned with test-
ing the credibility of their findings and in-
terpretations with the various sources (au-
diences or groups) from which data were
drawn. The testing of credibility is often
referred to as doing "member checks,"
that is, testing the data with members of
the relevant human data source groups.
Applicability. Within the rationalistic
paradigm, applicability--external validity
or generalizability--requires that the in-
quiry be conducted in ways that make
chronological and situational variations ir-
relevant to the findings. If that condition
can be met, the findings obviously will
have relevance in any context. Gener-
alizations are taken as enduring; that is,
unchanging over time, truth statements
that are context-free--that hold in any
context. But Cronbach (1975) has argued
that all generalizations " d e c a y " like
radioactive substances, having half-lives,
so that after a time every generalization is
"more history than science." This judg-
ment underscores the validity of the natu-
ralist's assumption that generalizations of
the rationalistic variety are not possible
because phenomena are intimately tied to
the times and the contexts in which they
are found.

TABLE 1
Scientific and Naturalistic Terms Appropriate to the Four Aspects of Trustworthiness

Aspect Scientific Term Naturalistic Term

Truth Value Internal Validity Credibility

Applicability External Validity Transferability

Generalizability

Consistency Reliability Dependability

Neutrality Objectivity Confirmability


Yet these facts do not obviate the possi-
bility that some transferability between
two contexts may occur because of certain
essential similarities between them. To de-
termine the extent to which transferability
is probable, one needs to know a great deal
about both the transferring and receiving
contexts, to have what Geertz (1973) has
dubbed "thick description" about each. 7 If
the thick descriptions demonstrate an es-
sential similarity between two contexts,
then it is reasonable to suppose that tenta-
tive findings of Context A are also likely to
hold in Context B (although, to be safe, an
empirical test of that presumption should
be made). For the naturalist, then, the con-
cept analogous to generalizability (or ex-
ternal validity) is transferability, which is
itself dependent u p o n the degree of simi-
larity (fittingness) between two contexts.
The naturalist does not attempt to form
generalizations that will hold in all times
and in all places, but to form working
hypotheses that may be transferred from
one context to another depending u p o n
the degree of "fit" between the contexts.
Consistency. W i t h i n the r a t i o n a l i s t i c
paradigm concern over consistency stems
from the fact that instruments must pro-
duce stable results if those results are to be
meaningful. Validity is a direct function of
reliability; so, for example, it is easy to
show that the validity of an instrument
cannot exceed the square root of its relia-
bility (Gulliksen, 1950). Reliability is thus
not so much essential in its own right as it
is a precondition for validity. The natu-
ralist is also concerned with consistency,
and for the same reasons; naturalistic in-
struments no more than rationalistic ones
are likely to yield credible (the analog of
valid) results if they do not exhibit consis-
tency. But consistency is a trickier concept
for the naturalist than for the rationalist.
The latter, believing in a single reality
u p o n which inquiry converges, can treat
all instrumental shifts as error, but the
7The concept of "thick description" is not yet well
explicated. For one effort to outline what such a de-
scription should contain, at least in an evaluation situ-
ation, see Guba and Lincoln (in press).
CRITERIA FOR NATURALISTICINQUIRIES 81
naturalist, believing in a multiple reality
a n d u s i n g h u m a n s as i n s t r u m e n t s - -
instruments that change not only because
of "error" (e.g., fatigue) but because of
evolving insights and sensitivities--must
entertain the possibility that some portion
of observed instability is "real." Thus, for
the naturalist, the concept of consistency
implies not invariance (except by chance)
but trackable variance--variance that can
be ascribed to sources: so much for error,
so much for reality shifts, so much for in-
creased instrumental proficiency (better
insights), and so on. The naturalist thus
interprets consistency as dependability, a
concept that embraces elements both of the
stability implied by the rationalistic term
reliable and of the trackability required by
explainable changes in instrumentation.
Neutrality. Neutrality is commonly termed
objectivity w i t h i n the r a t i o n a l i s t i c
p a r a d i g m . O b j e c t i v i t y is p r e s u m a b l y
g u a r a n t e e d b y m e t h o d o l o g y ; If the
methods are explicated, open to public
scrutiny, replicable, and at least one step
removed from direct investigator-subject
contact, then objectivity is assured (that is,
the biases of the investigator are effec-
tively screened out). But of course meth-
odology inevitably reflects the predisposi-
tions of the investigator. In physics, for
example, it is no longer d i s p u t e d that
whether light is "proved" to be wave-like
or corpuscular in nature depends entirely
on which experiment one chooses to do:
Y o u n g ' s double-slit experiment defini-
tively shows light to be a wave, while
Einstein's work with the photoelectric ef-
fect established its corpuscular n a t u r e
without doubt (Zukav, 1979). In the social
sciences, the cultural and ethnic biases
that can be built into, for example, IQ
measuring instruments, are well known.
Naturalists are especially aware of this
problem because they understand the mul-
tiple realities that one encounters (includ-
ing multiple value systems) and the role
that their o w n predispositions can play
when they use themselves as instruments.
Following the reasoning of Scriven (1972),
naturalists shift the burden of neutrality
from the investigator to the data, requiring
evidence not of the certifiability of the in-
82 ECTJ SUMMER1981

TABLE 2
The Rationalistic Treatment of Trustworthiness

Inquiry can To guard In hope this And produce

be affected Which produce against which action will findings that
by: effects of: we: lead to: are:

Masking or Confounding Control and/ Internal Contamination-

competing or randomize validity proof
factors

Situational Atypicality Require prob- External Context-proof

variations ability validity
sampling

Instrumental Instability Replicate Reliability Inconsistency-

drift or proof
decay

Investigator Bias Insulate the Objectivity Investigator-

predilections investigator proof

vestigator or his or her m e t h o d s but of the
confirmability of the data p r o d u c e d .
WHAT IS THE RATIONALISTIC MODE FOR
DEALING WITH QUESTIONS OF
TRUSTWORTHINESS?
The modes for dealing with the four areas
of t r u s t w o r t h i n e s s t h a t h a v e e v o l v e d
w i t h i n t h e r a t i o n a l i s t i c p a r a d i g m are
s u m m a r i z e d in Table 2. The four rows of
the table c o r r e s p o n d to the four areas:
internal validity, external validity, reliabil-
ity, a n d o b j e c t i v i t y (see e n t r i e s in the
fourth column of the table). The columns
of the table have h e a d i n g s that are in-
t e n d e d to guide the reader through the cell
entries. Begin b y reading the h e a d i n g of
the first column, follow that b y reading the
entry in Cell 1, Row 1, read the h e a d i n g of
the second column and follow that b y read-
ing the entry in Cell 2, Row 1, and so on for
the r e m a i n i n g columns and cells of Row 1.
R e p e a t the process for each of the four
rows. In effect, the table p r o d u c e s four
s e n t e n c e s that recapitulate the c o n v e n -
t i o n a l r a t i o n a l i s t i c r e s p o n s e to t h e
t r u s t w o r t h i n e s s concerns. For example,
the cells in Row I p r o d u c e these sentences:
Inquiry can be affected by masking or compet-
ing factors,
which produce effects of confounding,
to guard against which we control and/or ran-
domize,
in the hope that this action will lead to internal
validity,
and produce findings that are contamination-
proof.
Several observations m a y be m a d e about
Table 2:
1. The entries in the first column of the
t a b l e - - t h e detractors from " p e r f e c t " in-
q u i r y - are perceived as sources of error,
elements extraneous to the thrust of the
i n q u i r y that represent intrusions or bar-
riers to its orderly development. Method-
ology is constructed not so much to take
account of these factors as to guard against
the threats posed by their existence
(Campbell & Stanley, 1963).
2. The aim of the counteractions noted
in the third column of the table is to r e n d e r
the study proof against these threats.
Findings must be c e r t i f i a b l e as
contamination-proof, context-proof,
inconsistency-proof, and investigator-
p r o o f ( C o l u m n 5). This a w e s o m e t a s k
is e n t r u s t e d , w i t h i n t h e r a t i o n a l i s t i c
p a r a d i g m , to m e t h o d o l o g y - - n o t to the ex-
pertise and insight of the investigator.
3. The a p p r o p r i a t e methodologies that
will produce such foolproof data are listed
in the third column of the table. The ac-
tions noted there are the ideal or
 CRITERIA FOR NATURALISTICINQUIRIES 83

" t e x t b o o k " answers to the questions,
"What have you done to guarantee inter-
nal validity? External validity? Reliability?
Objectivity?" Investigators who can doc-
ument that they have controlled or ran-
domized all (noninformation) variables,
selected and assigned subjects to treat-
ments at random, replicated (established
the reliability of) the instrumentation
(even if only in the split-half sense), and
insulated themselves from the phenomena
by a layer of "objective" instrumentation
have provided unassailable responses to
those questions.
4. When a research proposal or report is
examined for trustworthiness, or when the
implementation of a design is monitored
for procedures that will guarantee
trustworthiness, it is the entries of Column
3 that are used as criteria. The examiner or
monitor asks, "Did (or will) the inves-
tigators control or randomize? Did (or will)
they sample by probability methods? Did
(or will) they replicate? Did (or will) they
interpolate a layer of instrumentation be-
tween themselves and the phenomena?" If
the answer (or putative answer) to those
questions is "Yes," the proposal-process-
report is deemed to be satisfactory insofar
as trustworthiness is concerned.
WHAT IS THE NATURALISTICMODE FOR
DEALING WITH QUESTIONS OF
TRUSTWORTHINESS?
The naturalistic mode of dealing with the
four trustworthiness questions is sum-
marized in Table 3, which follows the
same format as Table 2. The reader will
note that the middle column of Table 2 is
represented in Table 3 by two columns,
one detailing steps that can be taken dur-
ing the inquiry, and the second, steps that
can be taken after the inquiry is complete.

TABLE 3
The Naturalistic Treatment of Trustworthiness

To take account of which we:

In the hope And produce

Inquiry can be Which produce these actions findings that
affected by: effects of: During. After: will lead to." are:

Factor Noninterpret- Use prolonged engage- Establish structural Credibility Plausible

patternings ability ment corroboration (co-
Use persistent obser- herence)
vation
Use peer debriefing Establish referen-
Do triangulation tial adequacy
Collect referential Do member checks
adequacy materials
Do member checks

Situational Noncompara- Collect thick de- Develop thick Transfer- Context-

uniquenesses bility scriptive data description ability relevant
Do theoretical/pur-
posive sampling

Instrumental Instability Use overlap methods Do dependability Dependability Stable

changes Use stepwise replica- audit (process)
tion
Leave audit trait

Investigator Bias Do triangulation Do confirmability Confirmability Investigator-

predilections Practice reflexivity audit (product) free
(audit trail)
84 ECTJ SUMMER1981
Following the logic of Table 2, Table 3 also
can be interpreted to produce four sen-
tences. For example, the cells in Row 1
produce these sentences:
Inquiry can be affected by factor patternings,
which produce effects of noninterpretability,
to take account of which we,
while doing (during) the study, use prolonged
engagement, persistent observation, and
peer debriefing, do triangulation, collect ref-
erential adequacy materials, and do
member checks,
and after completing the study, establish
structural corroboration or coherence, estab-
lish referential adequacy, and do member
checks,
in the hope that these actions will lead to credi-
bility,
and produce findings that are plausible.
The sentences in Table 3 are not self-
interpreting; the following discussion will
define and clarify their major terms and
processes.
Credibility. Whereas rationalists are con-
cerned with guarding against masking or
competing factors (sources of error) that
are said to confound the inquiry, natu-
ralists wish to take account of the bewil-
dering array of interlocking factor patterns
that confront them and pose formidable
problems of interpretation. The ration-
alists' solution to the problem is to abstract
several variables of special interest, re-
m a n d i n g the rest to the status of controlled
or randomized variables. The naturalists'
solution is to deal with the patterns in
their entirety but to take certain actions
that take account of the complexities.
The situation is much as depicted in
Figure 1A. The "reality" of the situation is
that m a n y factors, bearing a variety of rela-
FIGURE 1
"'Reality" Versus Scientific Abstraction: Tying and Untying Variables
A. "Reatity"
F A
tionships (correlations, rationalists would
say) to one another, form a "whole" that
cannot be understood if dismembered. Ra-
tionalists do just that, h o w e v e r : t h e y
single out several variables, and cast them
into a design which by definition treats
them as orthogonal (Figure 1B). To use
Brunswik's (1955) terminology, rationalists
tie certain variables in the design (that is,
place them into a specific relationship de-
termined not by nature but by the design)
and untie certain others (that is, treat vari-
ables related in nature as though they were
independent). This tying/untying is the ul-
timate effect of the rationalists' solution of
controlling and/or randomizing.
Naturalists eschew this approach be-
cause they feel it does violence to the
phenomena they seek to understand. In-
stead, they adopt certain other procedures
which, while not as theoretically unassail-
able, nevertheless preserve the holistic
situation. Methods that may be used dur-
ing the study include:
9 Prolonged engagement at a site, to over-
come, so far as possible, distortions pro-
duced by the presence of researchers and
to provide researchers the opportunity to
test their own biases and perceptions, as
well as those of their respondents. Spend-
ing an extended period at a site allows lo-
cals to adjust to the presence of researchers
and to satisfy themselves that they do not
constitute a threat. Researchers are given
time to check their own developing per-
ceptions; for example, by keeping jour-
nals. If, six months into the site engage-
ment, the journals continue to refer to lo-
cals and to characterize their life styles in
the same w a y , it is likely that the re-
searchers are continuing to view the situa-
B. Scientific Abstraction
A
t ~B

tion in terms of their early p r e d i s p o s i t i o n s
and have learned nothing from their pres-
ence at the site. Researchers m u s t exercise
caution, however, to avoid b e c o m i n g over-
involved w i t h the r e s p o n d e n t s - - w h a t the
anthropologists call " g o i n g native.'"
9 Persistent observation, in o r d e r to i d e n -
tify pervasive qualities as well as atypical
characteristics. Extended interaction w i t h
a situation or a milieu leads inquirers to an
u n d e r s t a n d i n g of w h a t is essential or char-
acteristic of it. At the same time they learn
to e l i m i n a t e aspects that are i r r e l e v a n t
while continuing to attend to those that,
w h i l e atypical, are n e v e r t h e l e s s critical
(Eisner, 1979). Inquirers ought to be able
to s h o w that sufficient time was spent at
the site to justify their characterization of
it; t h e i r j o u r n a l s o u g h t to reflect t h e i r
wrestling with the question of w h a t the
pervasive qualities are.
9 Peer debriefing, to p r o v i d e i n q u i r e r s
the o p p o r t u n i t y to test their g r o w i n g in-
sights and to expose themselves to search-
ing questions. Inquirers ought regularly to
detach t h e m s e l v e s from the site a n d to
seek out and interact w i t h other profes-
sionals w h o are able and willing to per-
form the debriefing function; for example,
faculty colleagues or m e m b e r s of a disser-
tation committee. Inquirers ought to ex-
pose their thinking to this " j u r y " of peers
and to deal with whatever questions they
m a y pose. T h e i r j o u r n a l s a n d field ac-
tivities ought to indicate timely redirection
of the i n q u i r y consistent w i t h the critiques
o b t a i n e d d u r i n g these debriefings.
9 Triangulation, w h e r e b y a v a r i e t y of
data sources, different investigators, dif-
ferent p e r s p e c t i v e s (theories), a n d dif-
f e r e n t m e t h o d s are p i t t e d a g a i n s t o n e
another in order to cross-check data a n d
interpretations (Denzin, 1978). For exam-
ple, no item of information ought to be
accepted that cannot be v e r i f i e d from at
least two sources. W h e n possible the re-
search team should be d i v i d e d so that the
perceptions of several investigators can be
compared. Different theories ought to be
b r o u g h t to bear on.data to yield alternative
explanations that can be tested. Different
methods, for example, questionnaires, in-
terviews, and documentary analyses,
CRITERIA FOR NATURALISTICINQUIRIES 85
should be used w h e n possible. In the field,
techniques of cross-examination should be
used w h e n reports from different infor-
mants are in conflict.
9 Collection of referential adequacy mate-
rials, w h e r e b y d o c u m e n t s , f i l m s , v i d -
eotapes, audio recordings, and other
" r a w " or "slice-of-life" data items are col-
lected against which findings and in-
terpretations can later be tested (Eisner,
1979). So, for example, if the i n q u i r y deals
with the b e h a v i o r of classroom teachers,
v i d e o t a p e s of actual c l a s s r o o m s can b e
m a d e and stored. Later, w h e n it is asserted
that teachers exhibit such and such behav-
ior, that assertion can be tested by refer-
ence to the archives.
9 Member checks, w h e r e b y data and in-
terpretations are c o n t i n u o u s l y tested as
they are d e r i v e d w i t h m e m b e r s of the var-
ious a u d i e n c e s a n d g r o u p s from w h i c h
data are solicited. The process of m e m b e r
checks is the single most important action
inquirers can take, for it goes to the heart
of the credibility criterion. Inquirers ought
to be able to d o c u m e n t b o t h having m a d e
such checks as well as t h e ways in which
the i n q u i r y was altered (emerged or un-
folded) as a result of m e m b e r feedback. 8
Methods that can be used after the s t u d y
has b e e n completed include:
9 Establishing structural corroboration or
coherence, that is, testing every d a t u m and
interpretation against all others to b e cer-
tain that there are no internal conflicts or
contradictions. Of course, particular data
items m a y be in conflict because they come
from different sources, represent different
p e r s p e c t i v e s , a n d so on, b u t i n q u i r e r s
ought to be able to make an interpretation
that explains t h e s e a p p a r e n t c o n t r a d i c -
tions. Interpretations should also take ac-
count of possible rival explanations and
negative or deviant cases (Patton, 1980), a
matter of special interest to critics w h o op-
erate from the rationalistic p o i n t of vie~
8An inquirer must be aware, of course, that audience
members may choose to label the data and interpreta-
tions as noncredible for reasons of their own. A discus-
sion of the ways in which audiences may dupe either
themselves or an inquirer is beyond the scope of this
paper. For a fuller discussion see Guba and Lincoln (in
press) and Douglas (1976).
86 ECTJ SUMMER1981
since the elimination of plausible rival al-
ternatives constitutes their chief means for
establishing internal validity. Finally, the
overall report or case study should demon-
strably exhibit coherence; that is, consis-
tency, synchronism, logic, and being "all
of a piece."
9 Establishing referential adequacy, that
is, testing analyses and interpretations
made after completion of the field portion
of the study against documents, record-
ings, films, and the like that were collected
or especially produced for this purpose
while the s t u d y was u n d e r way. Re-
searchers should, within the limits of time
and fiscal constraints, have collected ref-
erential adequacy materials during the
study. These materials can be used to con-
duct empirical tests; for example, by con-
stituting panels or juries of "experts" to
test conclusions against these raw mate-
rials.
9 Member checks, that is, testing the
overall report or case study with source
groups before casting it into final form.
These checks are like those already de-
scribed, but are carried out after comple-
tion of the study rather than during it.
Transferability. Naturalists eschew gener-
alizations on the grounds that virtually all
social/behavioral phenomena are context-
bound. It is not possible, they believe, to
develop "truth" statements that have gen-
eral applicability; rather, one must be con-
tent with statements descriptive or inter-
pretative of a given context--idiographic
or context-relevant statements. During a
study naturalists will:
9 Do theoretical]purposive sampling, that
is, sampling that is not intended to be rep-
resentative or typical (such a purpose fo-
cuses the investigator on similarities and
makes sense only when one is trying to
generalize) but that is i n t e n d e d to
maximize the range of information uncov-
ered. The nature of the sampling process is
governed by emergent insights about what
is important and relevant. Naturalistic in-
vestigators ought to be able to demonstrate
how the samples they selected met this
c r i t e r i o n - - f o r example, that successive
interview subjects were selected by asking
each respondent to nominate someone
whose point of view is as different as pos-
sible from his or her own. 9
9 Collect "thick" descriptive data that will
permit comparison of this context to other
possible contexts to which transfer might
be contemplated (Geertz, 1973). If trans-
ferability depends upon a match of charac-
teristics it is incumbent on investigators to
provide the information necessary to test
the degree of fittingness.
After the study is completed naturalists
will:
9 Develop thick description of the context
in order to make judgments about fitting-
ness with other contexts possible. Inquir-
ers ought to make available, as an appen-
dix to their reports or in a supplement
available to interested parties, a full de-
scription of all contextual factors imping-
ing on the inquiry.
Dependability. Naturalists are concerned
with the stability of data, but must make
allowance for apparent instabilities arising
either because different realities are being
tapped or because of instrumental shifts
stemming from developing insights on the
part of the investivator-as-instrument.
Two steps that can be taken during a study
that parallel the replication steps typically
advocated by the rationalists are:
9 Overlap methods, one kind of triangu-
lation process, whereby different methods
are used in tandem (Campbell & Stanley,
1963; Webb, Campbell, Schwartz, & Se-
chrest, 1966). This approach is usually ad-
vocated to overcome invalidities in indi-
vidual methods; two or more methods are
teamed in such a way that the weakness of
one is compensated by the strengths of
another. But it is clear that if similar results
are found using different methods the case
for stability is also strengthened (Guba,
1978; Guba & Lincoln, in press). This so-
called "multiple-operations" inquiry si-
multaneously undergirds the case for cred-
ibility and stability. Inquirers ought to be
able to report on the use of multiple
m e t h o d s and d e m o n s t r a t e that these
~Auseful discussion of theoretical samplingmay be
found in Glaser and Strauss (1967).
 CRITERIA FOR NATURALISTICINQUIRIES 87

methods were selected because they were 9 Arrange for a "'dependability" audit to
complementary (among other possible rea- be done by an external auditor--someone
sons). competent to examine the audit trail and to
comment on the degree to which proce-
9 Stepwise replication, analogous to the
dures used fall within generally accepted
"split-half" reliability of tests, in which
practice. Such a dependability audit, it
two separate research teams (the original
should be noted, deals primarily with the
team split into halves) deal separately with
processes of the inquiry.
data sources that have also been divided
into halves. Because of the unfolding na-
Confirmability. We have noted the shift by
ture of naturalistic designs, however, the
naturalists away from the concept of inves-
two teams cannot be permitted to pursue tigator objectivity toward the concept of
the inquiry to its end before results are
data (and interpretational) confirmability.
compared. Provision must be made for
In the interest of confirmability, two of the
communication between the teams at im-
steps naturalists can take during a study
portant milestone points, perhaps even on
are:
a daily basis, in order to cross-check de-
9 Triangulation, as already noted in rela-
veloping insights and to decide on appro-
tion to credibility--that is, collecting data
priate next steps. These communication
from a variety of perspectives, using a
sessions should be a d e q u a t e l y docu-
variety of methods, and drawing upon a
mented.
variety of sources so that an inquirer's
A quite different approach to the more
predilections are tested as strenuously as
conventional steps outlined above is built
possible. So far as it is feasible to do so,
on the metaphor of the fiscal auditor
other investigators should also be em-
(Guba, 1978; Guba & Lincoln, in press).
ployed. Research teams can be constituted
When, for example, an auditor from Price,
so as to balance out predispositions; for
Waterhouse is called in to audit the books
example, by seeing to it that both ration-
of the General Electric Company, he or she
alistic and naturalistic perspectives are
takes on two responsibilities: (1) to
represented (Guba & Lincoln, in press).
examine the method of accounting by
An inquirer should provide documenta-
which the GE books are kept, to verify that
tion for every claim from at least two
the method used is among those generally
sources; alternative possibilities and nega-
accepted by the accounting profession
tive instances should be ruled out, and so
(that is, to make sure that no "creative ac-
on.
counting" has taken place), and (2) to cer-
9 Practicing reflexivity, that is, to "inten-
tify that the "bottom line" is correct; that
tionally reveal to his [or her] audience the
is, that supportive documents (data) exist
underlying epistemological assumptions
to support every entry and the addition
which cause him {or her] to formulate a set
(interpretation) is correct. In relation to
of questions in a particular way, and fi-
dependability, it is the first of these appli-
nally to present his [or her] findings in a
cations that is relevant; that is, the exami-
particular way" (Ruby, 1980). One indis-
nation of method. Thus, naturalists will,
pensable technique in support of practic-
during a study:
ing reflexivity is to keep a continuing
9 Establish an "audit trail" that will make journal in which introspections are re-
it possible for an external a u d i t o r to corded on a daily basis (Spradley, 1979);
examine the processes whereby data were these introspections can also be tested dur-
collected and analyzed, and interpreta- ing the peer debriefings already men-
tions were made. The audit trail takes the tioned. Reinharz (1979) calls attention to
form of documentation (the actual inter- the fact that reports of research typically
view notes taken, for example) and a run- include discussion only of the problem
ning account of the process (as in the form and the method; she suggests that it is
of an investigator's daily journal). equally important to discuss the inquirer
After completion of a study naturalists and to document shifts and changes in his
will: or her orientation.
88 ECTJ SUMMER1981
After completion of the study naturalists
will:
9 Arrange for a confirmability audit that
u n d e r t a k e s the second of the two a u d i t i n g
tasks d e s c r i b e d a b o v e - - t h a t is, an audit
c e r t i f y i n g that d a t a exist in s u p p o r t of
every interpretation and that the interpre-
tations have b e e n m a d e in ways consistent
w i t h the available data. This type of a u d i t
is concerned p r i m a r i l y w i t h the products
of the i n q u i r y , a n d r e q u i r e s e x t e n s i v e
d o c u m e n t a t i o n . The a u d i t can be p e r -
formed at the same time b y the same ex-
ternal agent c o m m i s s i o n e d to perform the
d e p e n d a b i l i t y audit, as is usually the case
in the parallel fiscal audit.
Several observations m a y be m a d e about
Table 3:
1. The entries in the first column of the
table, unlike the case of Table 2 entries, are
not v i e w e d as "error" but as the "natural
state of t h i n g s , " as reflections of reality in
the m i n d s and lives of respondents. They
are therefore not to be g u a r d e d against b u t
to be taken account of. Naturalists' strate-
gies for establishing trustworthiness are
a i m e d in that direction.
2. The entries in the m i d d l e two col-
u m n s contains a naturalist's "textbook an-
swers" to questions of trustworthiness just
as d i d Column 3, Table 2 entries for the
rationalistic p a r a d i g m . If a naturalist is
challenged on the issue of credibility, for
example, a p r o p e r defense is to r e s p o n d , "I
used p r o l o n g e d engagement, I used per-
sistent observation ... I did member
checks." But we m u s t note one major dif-
ference b e t w e e n Table 2 and Table 3. Table
2 entries are not only "textbook a n s w e r s "
but in fact constitute an unassailable de-
fense. Given the a s s u m p t i o n s of the ra-
tionalistic p a r a d i g m , the steps outlined in
Column 3 of Table 2 are theoretically com-
plete steps that guarantee the study
a g a i n s t t h r e a t s to t r u s t w o r t h i n e s s . The
s a m e claim c a n n o t b e m a d e for T a b l e
3 - - a b o u t all one can say is that to take
these steps increases the p r o b a b i l i t y of the
s t u d y ' s trustworthiness, lo In that sense the
l~ is an interesting parallel between the deter-
minacy of classical physics and the rationalistic
paradigm on the one hand, and the indeterminacy and
probabilistic nature of quantum physics and the natu-
ralistic paradigm on the other hand.
naturalistic theory of trustworthiness is an
incomplete o n e - - o n e cannot m u s t e r evi-
dence that will compel another to accept
the trustworthiness of the s t u d y b u t only
evidence that will p e r s u a d e the other of its
relative t r u s t w o r t h i n e s s . But this situa-
tion neither surprises nor dismays
n a t u r a l i s t s - - t h a t kind of i n d e t e r m i n a c y is
what they expect of the "real" world. Their
response to someone w h o cannot tolerate
that degree of a m b i g u i t y is s i m p l y to say,
" W h o e v e r p r o m i s e d you a rose g a r d e n ? "
3. The responses outlined in the m i d d l e
c o l u m n s of Table 3 are n o t all e q u a l l y
w e i g h t y in p e r s u a d i n g a critic of t h e
t r u s t w o r t h i n e s s of n a t u r a l i s t i c i n q u i r y .
Some are sine qua nons, some merely de-
siderata. For example, it is inconceivable
t h a t o n e w o u l d b e p e r s u a d e d of t h e
trustworthiness of a s t u d y that involved no
triangulation and no m e m b e r checks. Pro-
longed engagement, persistent observa-
tion, peer debriefing, and collecting ref-
erential adequacy materials m i g h t a p p e a r
less necessary (however desirable). It is
likely that triangulation and member
checks (for credibility), thick description
(for transferability), leaving an a u d i t trail
(for dependability), and triangulation and
practicing reflexivity (for confirmability)
are the m i n i m u m s that should be r e q u i r e d
of naturalistic investigators.
4. Despite the i n d e t e r m i n a c y that con-
tinues to s u r r o u n d the naturalistic study
w h e n q u e s t i o n s of t r u s t w o r t h i n e s s are
raised, it is still the case that the entries in
the m i d d l e columns of Table 3 constitute,
at this point in time and thought, the best
available formulation of criteria for judg-
ing naturalistic inquiries. Thus, w h e n a
naturalistic s t u d y is to be j u d g e d , it is
these criteria that ought to be b r o u g h t to
bear.
IMPLICATIONS
If the preceding analysis has m e a n i n g , a
n u m b e r of implications can be d r a w n from
it:
1. Naturalistic inquiry has its own set of
criteria for adequacy. It is i n a p p r o p r i a t e to
a p p l y the rationalistic criteria of Table 2
u n d e r any circumstances. To suggest, for
example, that a naturalistic s t u d y is unac-

ceptable because controls were not insti-
tuted, subjects were not randomly
selected, instrumental results were not rep-
licated, or the investigator was not p r o p -
erly objective is s i m p l y unjustified. But it
is also i n a p p r o p r i a t e to a p p l y , w i t h o u t
thought, other, b r o a d e r criteria; for exam-
ple, those p r o p o s e d b y Schwen (1977) in
an earlier ERIC/AVCR (now ECTJ) Annual
Review Paper. While some of Schwen's
criteria might be acceptable to naturalists,
others are not; each m u s t be considered
independently. 11
2. The proposed criteria, like scientific
criteria, have utility at several stages in the
inquiry process:
9 For m a k i n g a priori j u d g m e n t s as in
the case of proposals or designs (in-
sofar as that term is appropriate). The
p r o p o s a l or d e s i g n s h o u l d i n d i c a t e
w h a t an i n q u i r e r p r o p o s e s to do to
satisfy each of the criteria suggested in
Table 3.
9 For m o n i t o r i n g i n q u i r y procedures.
Investigators can utilize the criteria of
Table 3 to g u i d e their field activities
and to i m p o s e checks on themselves
to be certain that they are m e e t i n g
criterial requirements.
9 For making ex post facto judgments as
in the case of reports or case studies.
The report or case study should in-
clude statements about w h a t an in-
quirer actually d i d to satisfy each of
the criteria suggested in Table 3. An
i m p o r t a n t part of such reports or case
studies m u s t be the results of d e p e n d -
ability and confirmability audits.
~Schwen did not limit himself to the trustworthi-
ness criteria but took up the question of how to judge
inquiry as professional scholarship. In the context of
the present paper, some of Schwen's criteria would be
readily accepted by naturalists: Inquiry should be pub-
licly verifiable; it should be directed toward an exten~
sion of knowledge; it should demonstrate creative ex-
ploration; and it should make intuitive good sense. But
others of Schwen's criteria would be less acceptable:
Inquiry should be disciplined insofar as that term im-
plies a preordinate design; it should be directed to~
ward generalizability; it should be incorporated within
a conceptual structure or theoretical framework (unless
that structure of framework is grounded in the inquiry
itself); and the instrumental problem should be a rep-
resentation of some larger unresolved logical problem
(which responds to the rationalist's need for gener-
alizability).
CRITERIA FOR NATURALISTICINQUIRIES 89
3. The statement of criteria is not equiva-
lent to the statement of decision rules for ap-
plying these criteria. It is one t h i n g to
suggest that triangulation is n e e d e d , for
example, and quite s o m e t h i n g else to say
h o w much, or what type, of triangulation
will suffice to establish a m i n i m a l level of
trustworthiness. It is one thing to specify
that a d e p e n d a b i l i t y a u d i t be d o n e and
quite another to establish the precise pro-
cesses that constitute an adequate audit. It
seems likely that the d e v e l o p m e n t of deci-
sion rules will b e an empirical matter; only
t h r o u g h efforts to a p p l y the criteria of
Table 3 will the field come to some under-
s t a n d i n g of w h a t d e c i s i o n rules m a k e
sense. What w e have here is a situation
parallel to asking, "Is a reliability of .65
sufficient to establish the adequacy of a
p a p e r - a n d - p e n c i l test?" or, "Is a question-
naire return rate of 46 percent adequate?"
These q u e s t i o n s can be a n s w e r e d o n l y
through experience.
4. The use of the naturalistic paradigm is
fraught with special risks for an investigator.
B e c a u s e of t h e o p e n - e n d e d , i n i t i a l l y
design-less, emergent quality of naturalis-
tic inquiry, it cannot be w a r r a n t e d in ad-
vance to the same extent as can rationalis-
tic inquiry. Rationalists can p r o p o s e to do
r a n d o m sampling, for example, and can
detail just h o w they will go about doing it.
A sponsor or client, a departmental chair-
m a n or college research committee, or a
dissertation committee will have no diffi-
culty in d e c i d i n g w h e t h e r the p r o p o s e d
p r o c e d u r e is r i g o r o u s l y c o r r e c t a n d
w h e t h e r it will or will not p r o d u c e the de-
sired outcomes. W h e n naturalists, b y con-
trast, propose to do theoretical or p u r p o -
sive sampling, the issue is very much in
doubt. Persons w h o m u s t make j u d g m e n t s
a b o u t the l i k e l i h o o d that the p r o p o s e d
procedure will work necessarily feel less
certain about their j u d g m e n t s ; they m a y
feel that they are b e i n g asked to extend
carte blanche to the researcher.
This a m b i g u i t y poses m a n y problems. A
f u n d i n g source having to choose b e t w e e n
a naturalistic i n q u i r y a n d a rationalistic
one is more likely to go w i t h the latter be-
cause the o u t c o m e s seem more certain.
D i s s e r t a t i o n c o m m i t t e e s , a l r e a d y uncer-
tain a b o u t the legitimacy of naturalistic
90 ECTJ SUMMER1981
inquiry, are likely to reject a naturalistic
dissertation because they are unwilling to
risk their status as k n o w l e d g e a b l e critics of
research. In all instances the naturalistic
approach is likely to be tarred with the
b r u s h of " s l o p p y r e s e a r c h " - - r e s e a r c h that
cannot be better specified than that m u s t
perforce be inadequate. There are thus a
good m a n y barriers and constraints to in-
h i b i t an investigator from u n d e r t a k i n g a
naturalistic s t u d y in the first place.
There is little that a naturalistic i n q u i r e r
can do about such attitudes at the m o m e n t
other than to accept them as part of the res
gestae. It is the author's hope that the spec-
ification of criteria in this p a p e r , h o w e v e r
i n a d e q u a t e at this p r i m i t i v e stage, will
serve to stimulate discussion about these
p r o b l e m s a n d to increase the tolerance of
the professional c o m m u n i t y for those of
their colleagues w h o elect to conduct their
studies b y these n e w e r rules.
5. The proposals of Table 3 should not be
reconstituted into an orthodoxy. O n e of the
m a j o r d i f f i c u l t i e s in p r o p o s i n g a n e w
p a r a d i g m is t h a t t h e o l d is so e n -
t r e n c h e d - i t is no longer a w a y to do in-
q u i r y b u t the way. Kaplan (1964) has called
the conceptualizations inquirers p r o d u c e
a b o u t the w a y s t h e y do i n q u i r y recon-
structed logic, At best reconstructed logics
are afterth6ughts that describe w h a t the
i n q u i r e r believes he did; most often they
do not a d e q u a t e l y describe w h a t the in-
quirer actually d i d (what Kaplan terms the
l o g i c - i n - u s e ) . N o w r e c o n s t r u c t e d logics
have m a n y uses: to train the novice, to
facilitate communication among prac-
titioners; to p r o v i d e checkpoints against
w h i c h i n q u i r e r s can t e s t t h e m s e l v e s .
B u t - - a n d m o s t e m p h a t i c a l l y - - t h e y are
not prescriptions of h o w i n q u i r y must be
done. W h e n reconstructed logics are al-
l o w e d to b e c o m e orthodoxies, i n q u i r e r s
are r e d u c e d to b e c o m i n g true believers, a
posture h a r d l y consonant w i t h the open
p o s i t i o n they typically espouse. The level
of t h i s p a p e r , it s h o u l d be recalled, is
primitive; I h o p e n e w and better criteria
will quickly replace those suggested here.
It is d u b i o u s w h e t h e r the "perfect criteria"
will ever emerge; until then, h u m i l i t y in
asserting that a " n e w and truer p a t h to
k n o w l e d g e " has b e e n found will be wise.
REFERENCES
Brunswik, E. Representative design and prob-
ablistic theory in a functional psychology.
Psychological Review, 1955, 62, 193-217.
Campbell, D. T., & Stanley, J. C. Experimental
and quasi-experimental designs for research
on teaching. In N. L. Gage (Ed.), Handbook of
research on teaching. Chicago, Ill.: Rand Mc-
Nally, 1963.
Cronbach, L. J. Beyond the two disciplines of
scientific psychology. American Psychologist,
1975, 30, 116-27.
Denzin, N. K. The logic of naturalistic inquiry.
In Norman K. Denzin (Ed.), Sociological
methods: A sourcebook. New York: McGraw-
Hill, 1978.
Douglas, J. D. Investigative social research. Bev-
erly Hills, Calif.: Sage Publications, 1976.
Eisner, E. W. The educational imagination. New
York: Basic Books, 1979.
Geertz, C. The interpretation of culture. New
York: Basic Books, 1973.
Glaser, B. G., & Strauss, A. L. The discovery of
grounded theory. Chicago, II1.: Aldine Publish-
ing, 1967.
Guba, E. G. Toward a methodology of naturalis-
tic inquiry in educational evaluation. CSE
monograph series in evaluation, No. 8. Los
Angeles, Calif.: Center for the Study of Evalu-
ation, UCLA, 1978.
Guba, E. G. Naturalistic inquiry. Improving
Human Performance Quarterly, 1979, 8, 268-
76.
Guba, E. G., & Lincoln, Y. S. Effective evalua-
tion. San Francisco, Calif.: Jossey-Bass, in
press.
Gulliksen, H. Theory of mental tests. New York:
Wiley, 1950.
Kaplan, A. The conduct of inquiry. San Francisco,
Calif.: Chandler Publishing, 1964.
Patton, M. Q. Qualitative evaluation methods.
Beverly Hills, Calif.: Sage Publications, 1980.
Polanyi, M. Personal knowledge. New York:
Harper & Row, 1958.
Reinharz, S. On becoming a social scientist. San
Francisco, Calif.: Jossey-Bass, 1979.
Ruby, J. Exposing yourself: Reflexivity, an-
thropology, and film. Semiotica, 1980, 30,
153-79.
Schwen, T. M. Professional scholarship in edu-
cational technology: Criteria for judging in-
quiry. AV Communication Review, 1977, 25,
5-24.
Scriven, M. Objectivity and subjectivity in edu-
cational research. In Lawrence G. Thomas
(Ed.), Philosophical redirection of educational
research, 71st Yearbook of the National Soci-

ety for the Study of Education, Part 1.
Chicago, Ill.: University of Chicago Press,
1972.
Scriven, M. Maximizing the power of causal in-
vestigations: The modus operandi method. In
G. V. Glass (Ed.), Evaluation studies review
annual, vol. 1. Beverly Hills, Calif.: Sage Pub-
lications, 1976.
Spradley, J. P. The ethnographic interview. New
York: Holt, Rinehart, & Winston, 1979.
Stake, R. E. Evaluating the arts in education. Co-
lumbus, Ohio: Charles E. Merrill, 1975.
Webb, E. J., Campbell, D. T., Schwartz, R. D.,
CALL FOR CONVENTION PAPERS
This Research and Theory Division
(AECT) announcement is an official call
for papers for presentation at the 1982
AECT Convention, to be held in Dallas.
Types of Papers
The following types of papers are being
sought:
9 Research Papers
9 Conceptual Papers
9 Symposia Papers
9 PapersDescribing Research in Progress
Presentations
Presentations will be 15 minutes in
length, with three to five minutes allowed
for questions. Discussants previously iden-
tified and familiar with the paper will be
present to raise questions and to stimulate
dialogue and discussion after the presenta-
CRITERIAFORNATURALISTICINQUIRIES 91
& Sechrest, L. Unobtrusive measures. Chicago,
Ill.: Rand McNally, 1966.
Wolf, R. L. An overview of conceptual and
methodological issues in naturalistic evaluation.
Paper presented at the annual meeting of the
American Educational Research Association,
San Francisco, Calif.: April 1979.
Wolf, R. L., & Tymitz, B. Toward more natural
inquiry in education. CEDR Quarterly, 1977,
10, 7-9.
Zukav, G. The dancing Wu Li masters: An over-
view of the new physics. New York: Bantam,
1979.
tion. (Authors of accepted papers must
provide discussants with two copies of
their completed papers by April 1, 1982.)
Proposals
Proposals for papers must be submitted
by October 30, 1981. Proposals should be
approximately 1,000 words long and
should summarize the major elements of
the paper.
How to Apply
A special cover sheet must accompany
the proposal. For more information and a
copy of the cover sheet, write to:
James Sucy, Manager
Educational Market Services
Eastman Kodak Company
343 State Street
Rochester, NY 14650