The Conceptualization of Educational Evaluation: An Analytical Review of the Literature

Author(s): David Nevo

Reviewed work(s):
Source: Review of Educational Research, Vol. 53, No. 1 (Spring, 1983), pp. 117-128
Published by: American Educational Research Association
Stable URL: http://www.jstor.org/stable/1170329 .
Accessed: 18/04/2012 19:46

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Educational Research Association is collaborating with JSTOR to digitize, preserve and extend
access to Review of Educational Research.

http://www.jstor.org
 Review of Educational Research
Spring 1983, Vol. 53, No. 1, Pp. 117-128

The Conceptualizationof EducationalEvaluation:

An Analytical Review of the Literature
David Nevo
Tel-Aviv University

Recent decades have been productivein the conceptualizationof educa-

tional evaluation,tryingto clarifyits meaningand exposingthe distinction
betweenevaluationand other related concepts. This article reviewsthe
evaluationliteraturethroughan analyticalframeworkrepresentingissues
addressedby major evaluationapproachesin education. The analytical
frameworkis comprisedof 10 dimensionsreferringto: (1) the definitionof
evaluation,(2) itsfunctions,(3) the objectsof evaluation,(4) the variables
that should be investigated,(5) criteria that should be used, (6) the
audiencesthatshouldbe served,(7) theprocessof doing an evaluation,(8)
its methodsof inquiry,(9) the characteristicsof the evaluator,and (10) the
standardsthatshouldbe usedtojudge the worthand meritofan evaluation.
Some implicationsfor the advancementof evaluationtheoryand practice
concludethis reviewof the literature.

Many attempts have been made in recent years to clarify the meaning of
evaluationand exposethe distinctionbetweenevaluationand otherrelatedconcepts
such as measurementor research.The literaturecontainsmany approachesregard-
ing the conceptualizationof evaluationand the determinationof its countenance
in education.Many of those approacheshave been undulyreferredto as "models"
(e.g., the CIPPModel,the DiscrepancyModel,the ResponsiveModel,or the Goal-
Free Model) in spite of the fact that none of them includesa sufficientdegreeof
complexityand completenessthat might be suggestedby the term "model."Stake
(1981) rightlysuggestedthat they be referredto as persuasionsratherthan models.
For the benefitof those of us who lost their way betweenthe variousevaluation
models,approaches,and persuasions,severalattemptshavebeen madeto put some
order into the growingevaluation literaturethroughclassificationsof evaluation
approaches.Such classifications(e.g., Guba & Lincoln, 1981; House, 1980; Pop-
ham, 1975;Stake, 1976;Stufflebeam& Webster,1980;Worthen& Sanders,1973)
made a significantcontributionthrough their critical reviews of the evaluation
literaturedenotingsimilaritiesand differencesamongthe variousapproaches.Those
classificationswere based on a somewhatholistic approachby placingeach evalu-
ation model as a whole in one of the labeledcategorieswith some other models.
Tryingto do justice to each evaluationmodel as a whole they sometimesignored
the major issues underlyingthe agreementsand disagreementsamong the various
evaluationapproaches.
Stufflebeam(1974) suggestedeight questionsto be addressedin any attemptto
conceptualizeevaluation.Nevo (1981) revisedStufflebeam'slist of questionsand
extendedit to 10 majordimensionsin a conceptualizationof evaluation.These 10

117
DAVIDNEVO

dimensionsrepresentthe majorissuesaddressedby the most prominentevaluation

approachesin education.They will be used here as an organizerfor an analytical
reviewof the literatureon educationalevaluation.
The 10 dimensionsfor our analysisare expressedby the followingquestions:
1. How is evaluationdefined?
2. Whatare the functionsof evaluation?
3. Whatare the objectsof evaluation?
4. Whatkinds of informationshould be collectedregardingeach object?
5. What criteriashould be used to judge the merit and worth of an evaluated
object?
6. Who should be servedby an evaluation?
7. What is the processof doing an evaluation?
8. What methodsof inquiryshould be used in evaluation?
9. Who shoulddo evaluation?
10. By what standardsshould evaluationbe judged?
We shall review the literatureseeking the various answersto those questions
provided by the various evaluation models, approaches,and persuasions.The
significanceof such a review for evaluation practitionersas well as evaluation
theoreticiansand researcherswill be pointedout at the conclusionof the article.
1. How is evaluationdefined?Many definitionsof evaluationcan be found in
the literature.The well-known definition originatedby Ralph Tyler perceives
evaluationas "Theprocessof determiningto whatextentthe educationalobjectives
areactuallybeingrealized"(Tyler, 1950,p. 69). Anotherwidelyaccepteddefinition
of evaluationhas been that of providinginformationfor decisionmakingsuggested
by variousleadingevaluatorssuch as Cronbach(1963), Stufflebeam(Stufflebeam
et al., 1971), and Alkin (1969). In recent years considerableconsensushas been
reachedamong evaluatorsregardingthe definitionof evaluationas the assessment
of merit or worth (Eisner, 1979;Glass, 1969; House, 1980;Scriven, 1967;Stuffle-
beam, 1974), or as an activitycomprisedof both descriptionand judgment(Guba
& Lincoln, 1981; Stake, 1967). A joint committee on standardsfor evaluation,
comprisedof 17 membersrepresenting12 organizationsassociatedwith educational
evaluation, recently published their definition of evaluation as "the systematic
investigationof the worthor meritof some object"(JointCommittee,1981, p. 12).
A major exception to that consensus regardingthe judgmental definition of
evaluation is representedby the Stanford Evaluation Consortium group who
defined evaluation as "[a] systematic examination of events occurringin and
consequentof a contemporaryprogram-an examinationconductedto assist in
improvingthis programand other programshaving the same general purpose"
(Cronbachet al., 1980, p. 14). Cronbachand his associates(1980) clearlyrejectthe
judgmentalnatureof evaluationadvocatingan approachthat perceivesthe evalu-
ator as "an educator[whose]successis to be judged by what otherslearn"(p. 11)
ratherthan a "referee[for] a basketballgame" (p. 18) who is hiredto decide who
is "right"or "wrong".
A definitionthat points to the judgmentalcharacterof evaluationmight create
considerableanxiety among potential evalueesand raise resistanceamong oppo-
nents of evaluation.Obviously,a nonjudgmentaldefinitionof evaluation,such as
"providinginformationfor decisionmaking,"mightbe acceptedmore favorablyby

118
 THE CONCEPTUALIZATION OF EDUCATIONAL EVALUATION

evaluees and clients. However, it may be unrealisticto create positive attitudes

towardevaluationby ignoringone of its majorfeatures.Anotherapproachintended
to develop positive attitudestoward evaluationmight be to demonstrateits con-
structivefunctionswithin the variousdomains of education.
2. Whatare thefunctionsof evaluation?Scriven(1967) was the firstto suggest
the distinction between "formative evaluation" and "summative evaluation,"
referringto two major roles or functions of evaluation,althoughhe was not the
first to realizethe importanceof such a distinction. Later,referringto the same
two functions, Stufflebeam(1972) suggestedthe distinction between proactive
evaluationintended to serve decisionmakingand retroactiveevaluationto serve
accountability.Thus, evaluationcan servetwo functions,the "formative"and the
"summative."In its formativefunction evaluationis used for the improvement
and developmentof an ongoing activity(or program,person,product,etc.). In its
summativefunctionevaluationis used for accountability,certification,or selection.
A thirdfunctionof evaluation,the psychologicalor sociopoliticalfunction,which
has been less often treatedby evaluationliterature(Cronbachet al., 1980;House,
1974; Patton, 1978), should also be considered.In many cases it is apparentthat
evaluationis not servingany formativepurposesnor is it being used for accounta-
bility or othersummativepurposes.However,it is beingusedto increaseawareness
of special activities, motivate desired behavior of evaluees, or promote public
relations.Regardlessof our personalfeelingsaboutthe use (or misuse)of evaluation
for this purpose,we cannot ignoreit.
Anothersomewhat"unpopular"functionof evaluationis its use for the exercise
of authority(Dornbusch& Scott, 1975). In formalorganizationsit is the privilege
of the superiorto evaluate his or her subordinatesand not vice versa. In many
cases a personin a managementposition might evaluatesomeone to demonstrate
his authorityoverthatperson.We may referto this as the "administrative" function
of evaluation.
To summarize,evaluationcan servemany functions:(a) the formativefunction
for improvement,(b) the summative function for selection, for certification,for
accountability,(c) the psychologicalor sociopoliticalfunction for motivationand
to increaseawareness,and (d) the administrativefunctionto exerciseauthority.
Some evaluators(Alkin, Daillak,& White, 1979;Cronbachet al., 1980)express
a clearpreferencefor the formativefunctionof evaluation,but the generalpercep-
tion seems to be that thereare no "right"or "wrong"roles of evaluation,and that
it can servedeliberatelymore than one function.However,differentfunctionscan
be served in various ways and by different evaluation methods. It is therefore
important to realize the existence of the various evaluation functions and to
determinethe specific function(s)of a concreteevaluationat an early stage of its
planning.
3. What are the objectsof evaluation?Studentsand teachershave alwaysbeen
popular objects of evaluation in education. Almost all the measurementand
evaluationliteraturein educationup to the mid-sixtiesdealt with the evaluationof
students' learning. Up to that time one could hardly find in the educational
literatureany substantialguidanceregardingthe evaluationof other objectssuch
as educationalprojectsor programs,curricularmaterials,or educationalinstitu-
tions. Variousdevelopmentsin the educationalsystem of the United States(e.g.,

119
DAVIDNEVO

the Elementaryand SecondaryEducationAct of 1965) led to a significantshift of

focus regardingthe objects of educationalevaluation from students to projects,
programs,and instructionalmaterials,which have been since then most common
in the writingsof the majorauthorsin the evaluationliteraturein education(Alkin,
1969;Provus, 1971;Scriven, 1967;Stake, 1967;Stufflebeam,1969;Stufflebeamet
al., 1971).
Two majorconclusionscan be drawnfrom the reviewof contemporaryevalua-
tion literature:(a) Almosteverythingcan be an objectof evaluation,and evaluation
shouldnot be limitedto the evaluationof studentsor school personnel;and (b) the
clearidentificationof the evaluationobjectis an importantpartof the development
of any evaluationdesign.
In planningan evaluationit seems to be importantto determinewhat is "the
thing" (or "the evaluand,"to use Scriven's,1980, term) that has to be evaluated.
It helps to determinewhat kind of informationshould be collected and how it
shouldbe analysed.A clearobjectidentificationhelps keep an evaluationfocused.
It also helps to clarify and resolve value conflicts and potential threat among
stakeholdersand others likely to be affectedby the evaluation(Guba & Lincoln,
1981).
4. Whatkindsof informationshouldbe collectedregardingeach object?Afteran
evaluationobject has been chosen, a decision must be made regardingthe various
aspectsand dimensionsof the object that should be evaluated.Informationperti-
nent to such aspects must be collected. Earlierapproachesto evaluationfocused
mainlyon resultsor outcomes.Thus, to evaluatean educationalobject(e.g., a new
curriculum)would mean to evaluate the quality of the resultsof its functioning
(e.g., students'achievements).In recentyearssome interestingattemptshave been
made to extend the scope of evaluation variablesin various evaluation models
(Alkin, 1969;Provus, 1971; Stake, 1967; Stufflebeam,1969, 1974; Stufflebeamet
al., 1971). Stufflebeam'sCIPP Model suggests that evaluation focus on four
variablesfor each evaluation object;(a) its goals, (b) its design, (c) its processof
implementation,and (d) its outcomes. Accordingto this approachan evaluation
of an educationalproject,for example,would be an assessmentof (a) the merit of
its goals, (b) the quality of its plans, (c) the extent to which those plans are being
carriedout, and (d) the worthof its outcomes.
Stake (1967) in his CountenanceModel suggestedthat two sets of information
be collected regardingthe evaluatedobject:descriptiveand judgmental.The de-
scriptiveset should focus on intentsand observationsregardingantecedents(prior
conditionsthat may affectoutcomes),transactions(the processof implementation),
and outcomes. The judgmentalset of informationis comprisedof standardsand
judgmentsregardingthe same antecedents,transactionsand outcomes.
Guba and Lincoln (1981), expanding Stake's Responsive Education Model
(Stake, 1975) and applyingthe naturalisticparadigm,suggestthat the evaluator
generatefive kinds of information:(a) descriptiveinformationregardingthe eval-
uationobject,its setting,and its surroundingconditions,(b) informationresponsive
to concernsof relevantaudiences,(c) informationabout relevantissues, (d) infor-
mation about values, and (e) informationabout standardsrelevantto worth and
meritassessments.
Thus, the evaluationliteratureseems to suggestthat a wide rangeof information

120
 THE CONCEPTUALIZATION
OF EDUCATIONALEVALUATION

shouldbe collectedby evaluationregardingthe evaluatedobject.It shouldnot limit

itself to the narrowscope of evaluationregardingoutcomes or results.This does
not mean that each single evaluation must always collect all possible kinds of
information;it may focus on some of them accordingto identified evaluation
prioritiesor practicalconstraints.
5. Whatcriteriashould be used to judge the merit and worthof an evaluation
object?To choose the criteriato be used to judge the meritof an evaluationobject
is one of the most difficulttasks in educationalevaluation.Those who think that
evaluationshouldattemptto determinewhethergoalshave been achieved(Provus,
1971;Tyler, 1950)makethis taskeasy for themselvesby partiallyignoringthe issue
of evaluation criteria.What they actually do is use "goal achievement"as the
evaluation criterion without having justified its being an appropriatecriterion.
What about trivial goals or all kinds of "stated objectives"that aren't worth
achieving?Shouldthey be used as evaluationcriteria?
Anotherway to avoid the issue of evaluationcriteriais to ignorethe judgmental
natureof evaluation.Those who defined evaluationas an informationcollection
activityto serve decisionmakingor other purposes(Alkin, 1969;Cronbach,1963;
Stufflebeam,1969) did not have to deal with the problemof choosing evaluation
criteria.
Apparently,the achievement of (important)goals is one possible basis for
evaluationcriteria.Alternativebases for evaluationcriteriasuggestedby the litera-
ture might be: identifiedneeds of actual and potential clients (Joint Committee,
1981;Patton, 1978;Scriven,1972b),idealsor socialvalues(Guba& Lincoln, 1981;
House, 1980), known standardsset by expertsor other relevantgroups (Eisner,
1979; Guba & Lincoln, 1981; Stake, 1967), or the quality of alternativeobjects
(House, 1980;Scriven, 1967).
Most evaluationexpertsseem to agreethat the criterion(or criteria)to be used
for the assessmentof a specific object must be determinedwithin the specific
context of the object and the function of its evaluation.While in many cases the
evaluatordoes not have the authorityto choose among the various alternative
criteria,it is the evaluator'sresponsibilitythat such a choice be made and that he
be able to providea soundjustificationfor the choice, whetherit is made by him
or by somebodyelse.
6. Who should be served by an evaluation?Those who define evaluation as
providinginformationfor decisionmaking(Alkin, 1969;Cronbach,1963; Stuffle-
beam et al., 1971) seem to have a clear opinion as to who has to be served by
evaluation.They identify the relevantdecisionmakersand attempt to determine
their informationneeds. Others (Cronbachet al., 1980; House, 1980) reject the
notion of serving"decisionmakers" becauseof the threatof co-optationor oversim-
plification of social and organizationalprocesses. Cronbach and his associates
(1980) are inclined to serve the "policy-shapingcommunity" ratherthan some
kind of managerialdecisionmaker.Many authorsreferto "evaluationclients"or
"evaluationaudiences"as those who have to be servedby evaluation.Guba and
Lincoln (1981) suggestedthe term "stakeholders"or "stakeholdingaudience"for
the whole groupof personshavingsome stake in the performanceof the evaluand
and thereforeshould be servedby the evaluation.
If evaluationis to be useful at all, it has to be useful to some specificclient or

121
DAVIDNEVO

audience.The evaluationliteraturedoes not suggestwhichis the "mostappropriate"

audiencefor evaluation,but threeimportantpropositionscan be found in writings
regardingthis issue. They are:(a) An evaluationcan have more than one client or
audience;(b) differentevaluationaudiencesmight have differentevaluationneeds;
and (c) the specificaudiencesfor an evaluationand theirevaluationneedsmust be
clearlyidentifiedat the earlystagesof planningan evaluation.
Differencesin evaluationneeds might be reflectedin many ways: for example,
the kind of informationto be collected,the level of data analysisto be used, or the
form of reportingthe evaluation results.Sometimes it is impossibleto serve all
identifiedevaluation needs at the same time, and some prioritieshave to be set
regardingthe specificevaluationneeds to which the evaluationwill respond.
7. Whatis theprocessof doingan evaluation?The processof doingan evaluation
might differ according to the theoreticalperception guiding the evaluation. A
theoreticalapproachperceivingevaluation as an activity intended to determine
whethergoals have been achieved(Tyler, 1950) might recommendthe following
evaluationprocess:(a) statinggoals in behavioralterms, (b) developingmeasure-
ment instruments,(c) collecting data, (d) interpretingfindings, and (e) making
recommendations.
Accordingto Stake'sCountenanceModel (Stake, 1967) the evaluationprocess
should include (a) describinga program,(b) reportingthe descriptionto relevant
audiences, (c) obtaining and analyzing their judgments, and (d) reportingthe
analyzedjudgmentsback to the audiences.Lateron, in his ResponsiveEvaluation
Model Stake(1975) suggesteda continuing"conversation"betweenthe evaluator
and all otherpartiesassociatedwith the evaluand.He specified12 stepsof dynamic
interaction between the evaluator and his audiences in the process of conducting
an evaluation.
Provus(1971) proposeda five step evaluationprocessincluding(a) clarification
of the programdesign,(b) assessingthe implementationof the program,(c) assessing
its in-termresults,(d) assessingits long-termresults,and (e) assessingits costs and
benefits.
The Phi Delta KappaStudyCommitteeon evaluation(Stufflebeamet al., 1971)
presenteda three-stepevaluationprocess.It included(a) delineatinginformation
requirementsthroughinteractionwith the decisionmakingaudiences,(b) obtaining
the needed informationthrough formal data collection and analysisprocedures,
and (c) providing the information to decisionmakers in a communicable format.
Scriven (1972a) has suggested nine steps in his Pathway Comparison Model.
Gubaand Lincoln(1981) suggestin theirrecentlypublishedbook thata naturalistic-
responsive evaluation be implemented through a process including the following
four stages:(a) initiatingand organizingthe evaluation,(b) identifyingkey issues
and concerns,(c) gatheringusefulinformation,and (d) reportingresultsand making
recommendations.
While there seems to be no agreementamong evaluationexpertsregardingthe
"best"processto followwhen conductingan evaluation,most of them would agree
that all evaluations should include a certain amount of interaction between evalu-
ators and their audiences at the outset of the evaluation to identify evaluation
needs, and at its conclusion to communicate its findings. Evaluation cannot be
limited to the technical activities of data collection and analysis.

122
 THE CONCEPTUALIZATION
OF EDUCATIONALEVALUATION

8. Whatmethodsof inquiryshouldbe used in evaluation?While challengingthe

usefulnessof variousresearchmethodsfor evaluationstudies(Guba, 1969;Stuffle-
beam et al., 1971), recent years have also introducedvariousmethods of inquiry
into the field of educationalevaluation.In additionto traditionalexperimentaland
quasi-experimentaldesigns (Campbell, 1969; Stanley, 1972; Cook & Cambell,
1976), naturalisticmethods (Guba & Lincoln, 1981; Patton, 1980), jury trials
(Wolf, 1979),case studies(Stake, 1978),art criticism(Eisner,1977, 1979),journal-
istic methods (Guba, 1978), the modus operandi method (Scriven, 1974), and
many others became legitimate methods for the conduct of evaluation. Some
methodologistsstill advocatethe superiorityof certainmethodssuch as experimen-
tal design (Boruch & Cordray, 1980; Rossi, Freeman, & Wright, 1979) at one
extreme, or naturalisticmethods (Guba & Lincoln, 1981; House, 1980; Patton,
1980) on the otherextreme,but overallthereseemsto be more supportfor a more
eclectic approachto evaluation methodology.At the present state of the art in
evaluationit looks like "theevaluatorwill be wise not to declareallegianceto either
a quantitative-scientific-summative methodologyor a qualitative-naturalistic-de-
scriptivemethodology"(Cronbachet al., 1980, p. 7). It mightbe also true that for
a complicatedtask such as the conduct of evaluationan approachis neededthat
seeks the best method or set of methods for answeringa particularevaluation
question,ratherthan assumingthat one method is best for all purposes.
9. Whoshoulddo evaluation?Becominga professionalgroup,evaluatorsdevoted
much attentionto identifyingthe characteristicsof "good"evaluatorsand appro-
priateways to train them (Boruch& Cordray,1980;Cronbachet al., 1980;Guba
& Lincoln, 1981;Stufflebeamet al., 1971;Worthen,1975).To be a competentand
trustworthyevaluator one needs to have a combination of a wide variety of
characteristics.These include technical competence in the area of measurement
and researchmethods, understandingthe social context and the substanceof the
evaluationobject,humanrelationsskills,personalintegrity,and objectivity,as well
as characteristicsrelatedto organizationalauthorityand responsibility.Becauseit
is difficultto find one person possessingall these qualifications,it often becomes
necessaryto have a team conduct an evaluationor to choose the personwith the
most appropriatecharacteristicsfor a specificevaluationtask.
The evaluationliteraturealso suggeststwo importantdistinctionsthat shouldbe
taken into account when deciding who should do an evaluation.The first is the
distinctionbetweenan internalevaluatorand an externalevaluator(Scriven,1967,
1975; Stake & Gjerde, 1974; Stufflebeamet al., 1971). An internalevaluatorof a
projectis usually one who is employed by the projectand reportsdirectlyto its
management.Obviously, the internal evaluator'sobjectivityas well as external
credibilitymight be lower than those of an externalevaluator,who is not directly
employedby the projectand/or enjoys a higherdegreeof independence.
The second distinction is between a professionalevaluator and an amateur
evaluator.This distinction,suggestedby Scriven(1967), refersto two differentfoci
of trainingand expertiseratherthan to a valuejudgmentregardingthe qualityof
an evaluation. An amateur evaluator is usually one whose major professional
trainingis not in evaluation,and involvement in evaluationrepresentsonly part
of his or herjob description(e.g., the associatedirectorof a new math curriculum
developmentprojectconductingthe formativeevaluationof the project,who has

123
DAVIDNEVO

an M.A. in math education and some on-the-job training in evaluation). A

professionalevaluatoris one with extensivetrainingin evaluationand whosemajor
(or even only) responsibilityis conductingevaluation(e.g., the internalevaluator
of a special education project,who has an M.A. in measurementand evaluation
and 5 yearsexperienceevaluatingspecialeducation projects).While the amateur
evaluator'stechnicalevaluationskills might be lower than those of a professional
evaluator,he or she might have a better understandingof the project'sunique
evaluationneeds and be able to develop betterrapportwith the membersof the
evaluatedproject.
These two distinctions are independent;there may be an internal-amateur
evaluator,an external-amateurevaluator,an internal-professional evaluator,and
so forth.
10. By whatstandardsshouldevaluationbejudged?Severalattemptshave been
made in recentyearsto developstandardsfor evaluationsof educationaland social
programs(EvaluationResearchSociety, 1980;Joint Committee,1981;Stuffiebeam
et al., 1971;Tallmadge,1977; U.S. GeneralAccountingOffice, 1978). In spite of
the fact that some writers(Cronbachet al., 1980; Stake, 1981) have criticizedthe
rationalefor the whole standard-settingeffort as being prematureat the present
stateof the art in evaluation,thereseems to be a greatdeal of agreementregarding
their scope and content.
Boruchand Cordray(1980) analyzedsix sets of such standardsand reachedthe
conclusionthattherehas been a largedegreeof overlapand similarityamongthem.
The most elaborate and comprehensiveset, and the one based on the largest
amount of consensus, is probablythe set developed and publishedby the Joint
Committeeon Standardsfor EducationalEvaluation(1981). These standardshave
been developedby a committeeof 17 members,chairedby Dr. Daniel Stufflebeam,
which represented12 professionalorganizationsassociatedwith educationaleval-
uation. The committee suggested30 standards,which are dividedinto four major
groups:utility standards(to ensure that evaluation serves practicalinformation
needs), feasibilitystandards(to ensure that evaluation is realisticand prudent),
proprietystandards(to ensurethat evaluationis conductedlegally and ethically),
and accuracystandards(to ensurethat evaluationrevealsand conveystechnically
adequateinformation).

Summary
Riskingoversimplification,one could summarizethe reviewof the literaturewith
the followingmost common answersto our 10 questions.This could be one way
to describe briefly the state of the art in the conceptualizationof educational
evaluation.
1. How is evaluationdefined?Educationalevaluationis a systematicdescription
of educationalobjectsand/or an assessmentof their meritor worth.
2. Whatare thefunctionsof evaluation?Educationalevaluationcan serve four
differentfunctions:(a) formative(for improvement);(b) summative(for selection
and accountability);(c) sociopolitical(to motivate and gain public support);and
(d) administrative(to exerciseauthority).
3. Whatare the objectsof evaluation?Any entity can be an evaluationobject.
Typicalevaluationobjectsin educationare students,educationaland administra-

124
 THE CONCEPTUALIZATION
OF EDUCATIONALEVALUATION

tive personnel,curricula,instructionalmaterials,programs,projects,and institu-

tions.
4. What kinds of informationshould be collectedregardingeach object?Four
groupsof variablesshould be consideredregardingeach object. They focus on (a)
the goalsof the object;(b) its strategiesand plans;(c) its processof implementation;
and (d) its outcomes and impacts.
5. Whatcriteriashould be used to judge the merit of an object?The following
criteriashouldbe consideredin judgingthe meritor worthof an educationalobject:
(a) respondingto identified needs of actual and potential clients; (b) achieving
national goals, ideals, or social values; (c) meeting agreed-uponstandardsand
norms;(d) outdoingalternativeobjects;and (e) achieving(important)statedgoals
of the object.Multiplecriteriashould be used for any object.
6. Who should be servedby an evaluation?Evaluationshould serve the infor-
mation needs of all actualand potentialpartiesinterestedin the evaluationobject
("stakeholders").It is the responsibilityof the evaluator(s)to delineatethe stake-
holdersof an evaluationand to identifyor projecttheir informationneeds.
7. What is the process of doing an evaluation?Regardlessof its method of
inquiry, an evaluation process should include the following three activities:(a)
focusingthe evaluationproblem;(b) collectingand analyzingempiricaldata;and
(c) communicating findings to evaluation audiences. There is more than one
appropriatesequencefor implementingthese activities,and any such sequencecan
(and sometimes should) be repeated several times during the life span of an
evaluationstudy.
8. Whatmethodsof inquiryshouldbe used in evaluation?Beinga complextask,
evaluation needs to mobilize many alternative methods of inquiry from the
behavioralsciences and relatedfields of study and utilize them accordingto the
natureof a specificevaluationproblem.At the presentstate of the art, an a priori
preferencefor any specificmethod of inquiryis not warranted.
9. Who should do evaluation?Evaluationshould be conductedby individuals
or teams possessing(a) extensivecompetenciesin researchmethodologyand other
data analysistechniques;(b) understandingof the social context and the unique
substance of the evaluation object; (c) the ability to maintain correct human
relations and to develop rapport with individuals and groups involved in the
evaluation;and (d) a conceptual frameworkto integratethe above-mentioned
capabilities.
10. By whatstandardsshouldevaluationbejudged?Evaluationshouldstrikefor
an optimal balancein meetingstandardsof (a) utility (to be useful and practical);
(b) accuracy(to be technicallyadequate);(c) feasibility(to be realisticand prudent);
and (d) propriety(to be conductedlegallyand ethically).

Conclusion
As stated at the beginning of this article, a critical analysis of the various
theoreticalapproachesto educationalevaluationmighthaveimportantimplications
for practitionersof evaluationas well as for theoreticiansand researcherswho are
concernedwith developingnew concepts and better methods. All of them could
benefitfrom the analyticalscheme of the 10 questions,which guidedour analysis,
as well as from the reviewof the answerscontainedin the evaluationliterature.

125
DAVIDNEVO

Evaluators could use the 10 questions to organize their own perceptions of

evaluation using the evaluation literature to develop their own sets of coherent
answers for the 10 questions rather than adopting piously one evaluation model or
another. Understanding what others mean when they refer to evaluation could be
another use of the 10 questions. Evaluators may encounter considerable difficulties
if their perceptions of a concrete evaluation differ from those of their clients and
audiences. It is appropriate before one starts planning an evaluation or even decides
to do it at all to find out what is meant by evaluation by the various parties involved
in the evaluation, what purpose it is intended to serve, what is to be evaluated,
what are some feasible alternatives for doing it, and by what standards the evaluation
is to be judged if it is to be conducted at all. In other words, addressing the 10
questions discussed in this article might help evaluators to develop a clear perception
of evaluation and to get a better grasp of their evaluation problems before they get
themselves into all kinds of dubious evaluation adventures.
Discussions among theoreticians of evaluation can be a fruitful contribution to
the advancement of evaluation theory and practice. It could be even more so if
those discussions focused on issues in disagreement rather than on competing
models and paradigms. The contribution would be even more robust if the various
theoretical propositions were substantiated by some research findings. The 10
questions reviewed here could provide a framework to delineate research variables
for an empirical study of evaluation. Data on the actual relationships among those
variables as well as their relationships with other variables (e.g., evaluation utiliza-
tion or variables reflecting the context of evaluation) would be very much appre-
ciated by the evaluation profession.

References
Alkin, M. C. Evaluationtheorydevelopment.EvaluationComment,1969, 2, 2-7.
Alkin, M. C., Daillak,R., & White,P. Usingevaluations:Does evaluationmakea difference?
BeverlyHills, Calif.:Sage, 1979.
Boruch,F. R., & Cordray,D. S. An appraisalof educationalprogramevaluations:Federal,
state, and local agencies.Evanston,Ill.:NorthwesternUniversity,1980.
Campbell,D. T. Reformsas experiments.AmericanPsychologist,1969, 24, 409-429.
Cook, T. D., & Campbell,D. T. The design and conduct of quasi-experimentsand true
experimentsin field settings. In M. D. Dunnette (Ed.), Handbook of industrialand
organizationalpsychology.Chicago:Rand McNally, 1976.
Cronbach,L. J. Courseimprovementthroughevaluation.TeachersCollegeRecord, 1963,
64, 672-683.
Cronbach,L. J., Ambron,S. R., Dornbusch,S. M., Hess, R. D., Hornik,R. C., Phillips,D.
C., Walker,D. E., & Weiner,S. S. Towardreformof programevaluation.San Francisco:
Jossey-Bass,1980.
Dornbusch,S. M., & Scott, W. R. Evaluationand the exerciseof authority.San Francisco:
Jossey-Bass,1975.
Eisner, E. W. On the uses of educationalconnoisseurshipand educationalcriticism for
evaluatingclassroomlife. TeachersCollegeRecord,1977, 78, 345-358.
Eisner,E. W. Theeducationalimagination.New York:Macmillan,1979.
EvaluationResearchSociety.Standardsfor programevaluation.Author, 1980.
Glass, G. V. The growth of evaluation methodology(ResearchPaper No. 27). Boulder:
Laboratoryof EducationalResearch,Universityof Colorado,1969.(Mimeo)

126
 THE CONCEPTUALIZATION
OF EDUCATIONAL
EVALUATION

Guba, E. G. The failureof educationalevaluation.EducationalTechnology,1969, 9, 29-38.

Guba, E. G. Metaphoradaptationreport:Investigativejournalism.Researchon Evaluation
Project.Portland,Ore.:NorthwestRegionalEducationalLaboratory,1978.(Monograph)
Guba, E. G., & Lincoln,Y. S. Effectiveevaluation.San Francisco:Jossey-Bass,1981.
House, E. R. Thepoliticsof educationalinnovation.Berkeley,Calif.:McCutchan,1974.
House, E. R. Evaluatingwithvalidity.BeverlyHills, Calif.:Sage, 1980.
Joint Committee on Standardsfor EducationalEvaluation.Standardsfor evaluationsof
educationalprograms,projects,and materials.New York:McGraw-Hill,1981.
Nevo, D. The evaluationof a multi-dimensionalproject.In A. Lewy,S. Kugelmass,G. Ben-
Shakar,N. Blass,R. F. Boruch,D. J. Davis,B. Nevo, D. Nevo, P. Tamir,& I. Zak,Decision
orientedevaluationin education:The case of Israel. Philadelphia:InternationalScience
Services,1981.
Patton,M. Q. Utilizationfocused evaluation.BeverlyHills, Calif.:Sage, 1978.
Patton,M. Q. Qualitativeevaluationmethods.BeverlyHills, Calif.:Sage, 1980.
Popham,W. J. Educationalevaluation.EnglewoodCliffs,N. J.: Prentice-Hall,1975.
Provus,M. M. Discrepancyevaluation.Berkeley,Calif.:McCutchan,1971.
Rossi, P. H., Freeman,H. E., & Wright,S. R. Evaluation:A systematicapproach.Beverly
Hills, Calif.:Sage, 1979.
Scriven,M. The methodologyof evaluation.In R. E. Stake(Ed.),AERAmonographseries
on curriculumevaluation,No. 1. Chicago:Rand McNally, 1967.
Scriven,M. Prosand cons aboutgoal-freeevaluation.EvaluationComment,1972,3(4), (b)
Scriven,M. Thepathwaycomparisonmodelof evaluation.Berkeley:Universityof California,
January1972.(a) (mimeo).
Scriven,M. Maximizingthe powerof causalinvestigations:The modus operandimethod.In
W. J. Popham(Ed.),Evaluationin education.Berkeley,Calif.:McCutchan,1974.
Scriven,M. Evaluationbias and its control(OccasionalPaperNo. 4). Kalamazoo:Western
MichiganUniversity,1975.
Scriven,M. Evaluationthesaurus(2nd ed.). Inverness,Calif.:Edgepress,1980.
Stake,R. E. The countenanceof educationalevaluation.TeachersCollegeRecord,1967,68,
523-540.
Stake,R. E. (Ed.). Evaluatingthe arts in education:A responsivenessapproach.Columbus,
Ohio:Merrill,1975.
Stake, R. E. Evaluatingeducationalprogrammes.The need and the response.Washington,
D. C.: OECD PublicationsCenter,1976.
Stake,R. E. The case study methodin social inquiry.EducationalResearcher,1978, 7, 5-8.
Stake,R. E. Settingstandardsfor educationalevaluators.EvaluationNews, 1981,2(2), 148-
152.
Stake, R. E., & Gjerde,C. An evaluationof TCITY, the Twin City Institutefor Talented
Youth. In FredN. Kerlinger(Ed.),AERAmonographseries in curriculumevaluation,No.
7. Chicago:Rand McNally, 1974.
Stanley,J. C. Controlledfield experimentsas a model for evaluation.In P. H. Rossi & W.
Williams(Eds.),Evaluatingsocialprograms.New York:SeminarPress,1972.
Stufflebeam,D. L. Evaluationas enlightenmentfor decisionmaking.In W. H. Beatty(Ed.),
Improvingeducationalassessmentand an inventoryfor measures of affectivebehavior.
Washington,D. C.: NationalEducationAssociation,1969.
Stufflebeam,D. L. The relevanceof the CIPPevaluationmodelforeducationalaccountability.
SRIS Quarterly,1972, 5, 3-6.
Stufflebeam,D. L. Meta-evaluation(OccasionalPaperNo. 3). Kalamazoo:WesternMichigan
University,December1974.
Stufflebeam,D. L., Foley, W. J., Gephart,W. J., Guba, E. G., Hammon, R. L., Merriman,
H. O., & Provus,M. M. Educationalevaluationand decision-making.Itasca,Ill.:Peacock,
1971.

127
DAVID NEVO

Stufflebeam,D. L., & Webster,W. J. An analysisof alternativeapproachesto evaluation.

EducationalEvaluationand PolicyAnalysis, 1980, 2(3), 5-20.
Tallmadge,G. K. Joint DisseminationReview Panel ideabook.Washington,D. C.: U. S.
GovernmentPrintingOffice, 1977.
Tyler, R. W. Basic principlesof curriculumand instruction.Chicago, Ill.: University of
ChicagoPress, 1950.
U. S. GeneralAccountingOffice.Assessingsocial programimpactevaluations:A checklist
approach.Washington,D. C.: U. S. GeneralAccountingOffice, 1978.
Wolf, R. L. The use of judicial evaluationmethodsin the formationof educationalpolicy.
EducationalEvaluationand PolicyAnalysis, 1979, 1, 19-28.
Worthen, B. R. Competenciesfor educationalresearchand evaluation.EducationalRe-
searcher,1975,4(1), 13-16.
Worthen,B. R., & Sanders,J. R. Educationalevaluation:Theoryand practice.Belmont,
Calif.:Wadsworth,1973.

AUTHOR
DAVID NEVO, Senior Lecturer, School of Education, Tel-Aviv University, Tel-
Aviv 69978, Israel. Specialization: Evaluation theory, measurement and research
methodology.

128