Вы находитесь на странице: 1из 12

Complex Information Processing Paper #106

June 26. 1967

CHAPTER VII THE ANALYSIS OP PROBLEM-SOLVING PROTOCOLS

As we turn from the examination of the structure of task

environments to the psychological aspects of human behavior in

those environments, we are confronted with novel problems of data

\

analysis. In this chapter we will discuss a method of analysing

thinking-aloud protocols of human subjects.

prove to be among the most informative data that can be brought

Since such protocols

to bear in testing theories of human thinking, it is important

that we have objective and systematic ways of handling them.

To

make the discussion concrete, we will use as an example of such

data the protocol of a subject doing the cryptarithmetic task that

was introduced in Chapter IV.

Testing Informat ion-Process ing Theories

Building and testing psychological theories in information--

processing terms calls for somewhat different approaches than are

First, a great deal of

effort must go, and has gone., into tool building developing a

technology for constructing IPS*s having desired properties. As

typical today in experimental psychology.

we have already seen, much of this tool building has been directed

toward psychological theorizing. The tools themselves often take

the form of computer programming languages.

A second kind of research activity may be called "sufficiency

analysis. 11 It is aimed at constructing programs that can perform

difficult cognitive tasks of the kinds people can perform, with-

CXP #106

2

out detailed concern for performing them in the same way, but with concern that no gross abilities are assumed that lie beyond

known human abilities.

stage of transition to building theories that explain or predict

This style of analysis represents a first

specific human performances.

Those among the problem-solving

programs already described that do not call for large amounts of

search would fit this category of sufficiency analysis$

The Logic

Theorist, perhaps; and the NS8 program and MATER among the chess-

playing programs.

The Greenblatt chess program, on the other hand,

though a strong player,

sometimes examines thousands of positions,

hence would have to be put on the artificial intelligence side of

the line.

ft third kind of information-processing research is directed

at constructing and testing generalized models, which do not pre-

dict the behavior of individual human beings, but, at a more gen-

eral level, the behavior of populations. This line of inquiry parallels traditional research in experimental pyschology more

closely than the other lines, for this is the kind of generalized

prediction to which psychology has usually aspired, and to which

most experimental procedures in the field have been adapted.

In

some later chapters of this book, we shall have occasion to dis-

cuss briefly some generalized models £PAM, a theory of human

verbal learning is an example -that have been tested by comparing

their behavior with the average behavior of samples of human sub-

jects.

GIF #106

3

The fourth, and final, approach has been to construct

IPS 93

to explain in detail a segment of a single individual's behavior.

Theories of this kind can be subjected to severe testing by com-

paring the predicted behavior with the moment-to-moment

the subject's behavior.

The Paradigm for protocol ftnalys.is

record of

The use of thinking-aloud protocols as data in psychology is

not at all new.

Their connection with the introspective method,

especially of the Wurzburgers (

), goes back to the first decades

of this century. They served Duncker well in his classic contri-

but ion to the psychology of problem so3,ving in 1935 (

).

They

formed the primary material in the forties for an intensive

of thinking in chess by the Dutch psychologist De Groot

revised and translated into English ( )

report fell into relative disuse within the mainstream

study

(recently

)» However, free verbal

of bebavior-

ist psychology, especially in the United States.

Andnot until

the advent of the computer« with the corresponding conceptual

elopment in programming, has it been possible to couple protocols

dev-

with precise models of process.,

Spelling out a little more the over-all strategy of this

approach, we cen divide the analysis into four stages;

1. The subject is given a problem to solva.

He is instructed

to say aloud whatever occurs to him while he is ^forking on

the problem.

2. The tape recording of his verbal behavior (along with a

record of non-verbal behaviors,

such as expressions that

CIP #106

4

were written down) becomes the raw record of the ex-

periment. This is the protocol*

The protocol includes

the experimenter's behavior also, although his partici-

pation is usually minimized intentionally.

3. After intensive analysis of the protocol (hours for

every minute of subject behavior), a proposal emerges

for a scheme of information processing that will simu-

late the subject's behavior.

This proposal may* for

example, take the form of a flow diagram.

4. A computer program is coded and debugged that outputs a

record, called the trace, which purports to correspond

to the behavior indicated in the protocol.

The process may sometimes be abridged in certain ways. In

particular, if the flow diagram is not too complicated,

it may be

used directly, without actually coding a computer program and

producing the trace, to predict the trace or important features

of it by hind simulation.

theory with behavior in this book will take this abridged form.

In the case of the General Problem Solver,

Most of the detailed comparisons of

several versions, as

we have seen, have been run on computers, but in examining the

similarities and differences between these and human protocols,

we will also frequently have recourse to hand-simulated

tions. Th@ paradigm just presented has three salient features.

First,

behavior that

with the content of the tasko*

varia-

it deals with the dynamics of an individual episode of

re precise and highly specific.

Third,

it deals

Involvement with content is also

CIP #106

More precisely, a simple and completely specified interpreter

is sufficient to translate the statement of the theory into ade-

quate behavior.

reflected in the use of freely-produced linguistic utterances 33

the primary source of data.

In this respect the protocol is a

natural data form for this type of theory.

also,

It is appropriate,

in providing a large amount of information about the subject

per unit of time.

The necessity for this becomes apparent on con-

sidering how to identify a system as complex as a problem-solving

human.

Th& major problems in protocol analysis arise from these same

dominant features. We will mention two prominent problems before

turning to a third that is the main concern of this chapter.

The Assessment of Fit

In assessing the adequacy of the program to describe or ex-

plain the subject's behavior, several things are missing to which

psychologists have become accustomed.

First, there is no accept-

able way to quantify the degree of correspondence between the trace

of the program and the protocol.

the inference definite or public

side by side, as is done in Figure 1, where a fragment

This is not a problem of making

Trace and protocol can be laid

of a sub-

ject's behavior is compared with a portion of a trace produced

one version of GPS.

However* an elaborate output statement and

free linguistic utterance must still be compared.

CIP

#106

PI0URE

X

CIP #106

7

Numerical measures are not easily adequate to such compari-

sons.

In chess* for example* both trace and protocol identify

not only moves made or considered, but also features of the board

that are noticed and evaluated* decisions points in the analysis,

and so on. Although a human can assess each instance qualitati-

vely* there are no techniques available for quantifying

parison, or summarizing the results of a large set of compari-

the com-

sons.*

This difficulty stems directly from the richness of the

*

A close analogy is the problem of interpreting microphotographs

of biological specimens.

Historically* biologists have always

been satisfied with more rough-and-ready methods of confronting

We will refrain

theory with data than have bean psychologists.

from speculation as to whether this has had anything to do with

the slower rate of advance of psychology.

theory the kinds of things it can assert about human behavior.

Still* it makes the apparatus of statistical testing largely in-

applicable »

In addition f the program has been created partly with the

subject's protocol in view.

Writing a program that will repro-

duce known data is not a trivial task;

it amounts to finding a

simple pattern hidden in a set of inter-connected time series.

In one sense* it is analogous to fitting curves to data* where a ce t.ain number of free parameters are available to be estimated

from the data themselves.

But programs are not parameterized in

any simple way, and no analytic framework exists for counting de~

CIP #106

8

grees of freedom, and thus assessing

bility is available.

What is the Theory?

quantitatively how much flexi-

Programs are symbolic structures that specify the behavior

through time of a system in the same manner that a set of diff-

erence equations does (

). Thus, one has the same justification

for calling a program a theory as for giving that name to a set

of difference or differential equations (e.g.* Newton's laws of

motion)o

In both cases program and equations--the

theory ex-

plains the behavior of the system by giving a rule that specifies

how it is changing at any given moment as a function of its curr-

ent state at that moment.

Still a certain discomfort is felt in the idea of programs

as theories (Reitman, chapter 2).

The discomfort stems partly

from the specificity of the theory, which seems to be limited

to explaining the behavior of a single person during a single

While in other areas, such as the earth's

episode.

geological

development, we are content to construct a theory of the history

of an individual system, clearly little scientific interest atta-

ches per se to the history of a particular

college student in our

laboratory on a particular day.

Thus, we must view these indi-

vidualized theories microtheories would be an appropriate term*

Even though this term is currently

sense to denote a theory applying to

used in a somewhat broader

a miniature domain of be-

havior e.g. , a theory of rat behavior in

the T-maze.

simply as a technical means for bringing a more general theory

CIP #106

9

into contact with its data in an objective

and systematic way.

The more general theory, of course* is neither so formalized

nor precise as the microtheories to which it gives rise.

In part

it includes the basic possibility of viewing a human in a precise

way as an information processing system. But it also includes a

theory of how problem solving is accomplished what mechanisms

and under what

conditions they are evoked? and so on*

cribed in the last chapter is such a general theory* insofar as

common to all humans; what methods are possible

A program like GPS des-

are

it incorporates such common mechanisms, contains a set of methods,

and evokes them under particular conditions.

But we would expect

that GPS would have to undergo some modification

the behavioral data from any particular subject, for the program

in order to fit

certainly makes no explicit provision for individual differences.

we will

gradually learn how to "parameterize" programs to admit individual

differences* and how to distinguish the "essential" structure of

As our experience grows with this kind of theorizing*

the program--*i.e. * the theory-- from details (including boundary

conditions) that reflect such individual differences. At

present,

we are able to do this only in an informal, commonsensical way.

Of a similar nature is the problem of differentiating those

parts of the program that have psychological import hence, that

are to be regarded as parts of the theory from those that are only included to enable the program to run on a digital computer.

It is not only that not everything in the trace can be expected

to appear in the verbal protocol for a subject may fail to ver-

balize things of which he is conscious* and must fail to verbalise

r

GIF #106

10

those that occur in his processing but of which he is unconscious.

There is the further consideration that as one moves from gross

behavior to the details of machine code, both the organization

of

the program and its instructions reflect increasingly the struc-

ture of existing programming languages

(and ultimately, of com-

puters) , instead of the structure of the central nervous system

and its processes.

only down to the level of pyachologically meaningful elementary

As theory, the program must be taken seriously

information processes; and at the present time it is not known

with precision just where this level is, and where the boundary

should be drawn. Such problems have confronted sciences before, notably in

fields like chemistry and genetics where entities (e.g., planetary

electrons, genes) were postulated and assigned properties on in- direct evidence before there was any way of detecting and examin-

ing them by relatively direct methods.

Under such conditions,

scientists have sometimes been guilty of the fallacy of misplaced

concreteness for example, of inventing an ether so that electro- magnetic waves would have something to ruffIs but there does not

appear to be any simple way, whose costs are not greater

gain, for avoiding such errors.

tude of appropriate tentativeness toward the entities the thec/y

than the

We must simply maintain an atti-

postulates, our scepticism increasing as we move downward irco

successive layers of detail, but decreasing over time to the ex- tent that the detail leads to successful predictions of behavior.

CIP #106

11

The Procedure to be Followed

The point of view toward data analysis taken in this book

reflects these considerations.

to a single segment of human behavior will be treated,

as a single data point. That is, the program that accomplishes

the fit is viewed as a completely particularizec"! version (with specific parameter values and particular boundary conditions) of

The fit of an individual program

so to speak,

a more general theory that has not yet been formulated precisely.

The common structure of this more general theory and the nature

of the individual differences are to be inducted by cumulating

the number of pieces of data treated in this way. The present chapter carries out this ki.Md of analysis for a

single protocol of a subject doing the cryptarithmetic task.

Other

protocols, for this and other tasks, will be analysed in later

Our initial emphasis will be upon the extraction of

chapters.

information from the protocol.

The final step, writing an indi-

vidualized computer program (see step 4 in the previous section) c will not be taken, but most of the main features that would have

to be incorporated in the program will become evident from the

analysis. We start with an analysis of the task,

introducing the

technical apparatus needed to describe the subject's behavior. Then we give a gross description of the protocol, followed by

the detailed analysis.

CIP #106

129

is by no means a trivial task to use Whatever degrees of freedom are available to construct a program any program that will come even close to fitting the data in detail.