American Economic Association

Artificial Adaptive Agents in Economic Theory

Author(s): John H. Holland and John H. Miller
Source: The American Economic Review, Vol. 81, No. 2, Papers and Proceedings of the Hundred
and Third Annual Meeting of the American Economic Association (May, 1991), pp. 365-370
Published by: American Economic Association
Stable URL: http://www.jstor.org/stable/2006886
Accessed: 23-06-2015 12:45 UTC

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://www.jstor.org/page/
info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact support@jstor.org.

American Economic Association is collaborating with JSTOR to digitize, preserve and extend access to The American Economic
Review.

http://www.jstor.org

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions

ArtificialAdaptiveAgents in Economic Theory

By JOHN H.

AND JOHN H. MILLER*

HOLLAND

Economic analysis has largely avoided

questionsabout the way in which economic
agents make choices when confrontedby a
perpetuallynovel and evolvingworld. As a
result, there are outstanding questions of
great interestto economicsin areas ranging
from technological innovation to strategic
learning in games. This is so, despite the
importanceof the questions, because standard tools and formal models are ill-tuned
for answeringsuch questions.However,recent advancesin computer-basedmodeling
techniques,and in the subdisciplineof artificial intelligence called machine learning,
offer new possibilities. Artificial adaptive
agents (AAA) can be defined and can be
tested in a wide variety of artificialworlds
that evolve over extended periods of time.
be examinedboth computationallyand analytically,offeringnew waysof experimenting
with and theorizing about adaptive economicagents.
Many economic systemscan be classified
as complexadaptivesystems.Such a system
is complexin a special sense: (i) It consists
of a networkof interactingagents (processes, elements);(ii) it exhibitsa dynamic,aggregate behaviorthat emerges from the individualactivitiesof the agents;and (iii) its
aggregatebehavior can be describedwithout a detailed knowledgeof the behaviorof
the individualagents. An agent in such a
system is adaptive if it satisfies an additional pair of criteria: the actions of the
agent in its environmentcan be assigned a
value (performance,utility, payoff, fitness,
or the like); and the agent behaves so as to
increase this value over time. A complex
adaptivesystem, then, is a complex system

containing adaptive agents, networked so

that the environmentof each adaptiveagent
includesother agents in the system.
Complexadaptivesystemsusuallyoperate
far froma globaloptimumor attractor.Such
systems exhibit many levels of aggregation,
organization, and interaction, each level
havingits own time scale and characteristic
behavior. Any given level can usually be
describedin terms of local niches that can
be exploited by particularadaptations.The
niches are various, so it is rare that any
given agent can exploit all of them, as rare
as findinga universalcompetitorin a tropical forest. Moreover,niches are continually
createdby new adaptations.It is because of
this ongoingevolutionof the niches, and the
perpetualnovelty that results, that the system operates far from any global attractor.
Improvementsare alwayspossible and, indeed, occur regularly.The everexpanding
range of technologies and products in an
economy,or the everimprovingstrategiesin
a game like chess, provide familiar examples. Adaptive systems may settle down
temporarilyat a local optimum,where performance is good in a comparativesense,
but they are usually uninterestingif they
remain at that optimum for an extended
period.
A theory of complex adaptive systems
based on AAA makes possiblethe development of well-defined, yet flexible, models
that exhibit emergent behavior.Such models can capture a wide range of economic
phenomena precisely, even though the developmentof a generalmathematicaltheory
of complex adaptive systems is still in its
early stages.' The AAA models complement currenttheoreticaldirections;they are

*Professor of Psychology and Computer Science and

Engineering, University of Michigan, Ann Arbor, MI
48109; and Assistant Professor of Economics and Decision Sciences, Carnegie Mellon University, Pittsburgh,
PA 15213, respectively; both are External Professors,
Santa Fe Institute, Santa Fe, NM 87501.

'It is important in this research to determine just

where the potential for general solutions exists. There
are simple models of cellular automata, for example,
wherein the solutions to particular questions are computationally irreducible-the shortest way to analyze
the system is to run the complete computation.

The resulting complex adaptive systems can

365

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions

366

AEA PAPERS AND PROCEEDINGS

not intended as a substitute. Many of the

most interestingquestionsconcernpoints of
overlapbetween AAA models and classical
theory.As a minimalrequirement,wherever
the new approachoverlapsclassicaltheory,
it must include verified results of that theory in a way reminiscentof the way in which
the formalismof general relativityincludes
the powerfulresultsof classicalphysics.

MAY1991

The AAA models have severalcharacteristics that are not available in traditional
modelingtechniques.Models based on pure
linguisticdescriptions,while infinitelyflexible, often fail to be logically consistent.
Mathematical models lose flexibility, but
gain a consistentstructureand generalsolution techniques.The AAA models, specified
in a computerlanguage,retain much of the
flexibilityof pure linguistic models, while
having precision and consistency enforced
by the language.The resultingmodels are
dynamicand are "executable"in the sense
that the unfolding behavior of the model
can be observedstep by step. This makes it
possible to check the plausibilityof the behavior implied by the assumptionsof the
model. The precisionof the definitionsalso
opens AAA models to mathematicalanalysis. The ability to explore a wide range of
phenomena involvinglearning and adaptation, linked with the rigor imposed by a
computer language, provides a powerful
modelingtechnique.2
The AAA models offer a way of approaching one of the major questions of
present theory. Current theoretical constructs, based on optimization principles,
often requiretechnicallydemandingderivations. It is an obvious criticism of these
constructsthat real agents lack the behavioral sophisticationnecessaryto derive the
proposed solutions. This dilemma is resolved if it is postulatedthat adaptivemechanisms, driven by market forces, lead the

agentsto act as if they were optimizing(see,

for example,Milton Friedman,1953).AAA
explicitly model this link between adaptation and market forces, and can thus be
used to analyzethe conditionsunder which
optimizationbehaviorwill (not) occur.
Insofar as human behavior is driven by
adaption, an understandingof AAA may
prove to be a useful benchmarkfor, and
provideinsightsinto, existinghumanexperiments (see, for example,J. A. Andreoniand
Miller, 1990; Brian Arthur, 1990).3An experiment consisting of artificial agents allows the utility, risk aversion,information,
knowledge, expectations, and learning of
each subject to be carefully controlled.
Moreover,at any point in the experiment,
the knowledgeand learningof the artificial
agents can be "reset" to any desired previous state, and subtle variationsof the environment can be analyzed.The strategy(as
well as the behavior)of the AAA can always be explicitlyanalyzed, something not
usually possible with human subjects. Finally, the infinite patience and low motivational needs of AAA "subjects"impliesthat
large-scaleexperimentscan be conductedat
a relativelylow cost.
A majorfeature of AAA models is their
ability to produce emergent behavior. A
wide varietyof behaviorscan arise endogenously,even thoughthese behaviors,as with
any model, are constrained by the initial
structure.The possibilitiesare so rich that it
is often difficult to predict on a priori
groundswhat behaviorsand structureswill
emerge. It thus becomes possibleto explore
realms that were unanticipatedwhen the
model was defined. Analysisof these emergent phenomenashould offer both insights
and suggestionsfor new theoremsabout the
effects of adaptive agents in economic systems.
The AAA models may also prove useful
in studyingeconomic systems that have either an absence or a plethoraof theoretical
solutions. Many importanteconomic prob-

2Programming even a simple market is instructive

on the limitations of both the pure linguistic and mathematical approaches.

3Artificialagents could also be used as "subjects" in

pilot studies to identify potentially interesting new human experiments.

I. Why Study Artificial Adaptive Agents?

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions

VOL. 81 NO. 2

LEARNING AND ADAPTIVE ECONOMIC BEHAVIOR

lems, such as double-auction strategies,

multisectoral general equilibrium models,
and the like, have no easily derivedanalytic
solutions. Several AAA techniques were
originallydesignedas optimizationmethods
for environmentsthat are nonlinear,noisy,
discontinuous,or involve enormous search
spaces.As a result,they offer useful numerical techniques for such problems in economics. At the opposite extreme are systems with multiple solutions. For example,
in repeated games, the Folk theorem often
admitsa vast numberof potentialsolutions.
In these cases, the interactionof the adaptive systemswith the economicenvironment
may narrow the set of potential solutions.
Different equilibriamay have different degrees of adaptive complexity.

Beyond complementingcurrent theoretical and empiricalwork, AAA offer the potential for unique extensionsof currenttheory. The mechanismsgeneratingthe global
behaviorof a complex adaptivesystem can
be directlyobservedwhen the computeris
an integral part of the theory. For such
theories, the computerplays a role similar
to the role the microscopeplaysfor biology:
It opens up new classes of questions and
phenomenafor investigation.Problemsthat
prove difficultfor traditionalmathematical
approachesare often easily implementedas
an AAA system. In that form, they can be
dissected and modifiedwith ease, providing
new opportunitiesfor theorygenerationand
testing. More generally, the potential for
the development of a general calculus of
"adaptivemechanics"exists. A calculus of
these systems would combine the advantages of analytic perspicacity with computer-drivenhypothesistesting.
II. Some Current Artificial Adaptive
Agent Techniques

A wide rangeof computer-basedadaptive

algorithmsexist for exploringAAA systems,
including classifier systems, genetic algorithms,neural networks,and reinforcement
learning mechanisms. The multiplicity of
techniquespresents a problemfor analysis.
How sensitiveare the resultsto a particular
incarnation of the adaptive agent? This

367

problem, of course, confronts any attempt

to lessen the rationalitypostulatestraditionally used in economictheory.Usually,there
is only one way to be fully rational, but
there are manywaysto be less rational.It is
important in building a theory based on
AAA to constructagents that exhibitrobust
behavior across algorithmicchoices. Current economic studies of adaptive agents
rely on genetic algorithms(R. M. Axelrod,
1987; Miller, 1989; Andreoni-Miller)and
classifiersystems(R. Marimonet al., 1990;
Arthur).
Genetic algorithms (GAs) were developed
by Holland (1975) as a way of studying
adaptation,optimization,and learning.They
are modeled on the processes of evolutionary genetics. A basic GA manipulatesa set
of structures, called a population. Structures are usuallycoded as stringsof characters drawnfrom some finite alphabet(often
binary).For example, in a game context, a
stringmight be interpretedeither as a simple strategy(a rule table) or as a computer
programfor playingthe game (a finite automaton). Depending upon the model, an
agent maybe representedby a single string,
or it may consist of a set of strings correspondingto a range of potential behaviors.
For example, a string that determines an
oligopolist's production decision could either represent a single firm operatingin a
populationof other firms,or it could represent one of manypossible decision rules for
a given firm. Whatever the interpretation,
each stringis assigneda measureof performance, called its fitness, based on the performanceof the correspondingstructurein
its environment.The GA manipulatesthis
populationin order to producea new population that is better adaptedto the environment.
In execution, a GA first makes copies of
strings in the population in proportionto
their observed performance, fitter strings
being more likely to produce copies. As a
result, fitter strings are more likely to contribute to the new population. After the
copies are produced, they are modified by
the application of genetic operators. The
genetic operatorsprovide for the introduction of new strings (structures) that still

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions

368

AEA PAPERS AND PROCEEDINGS

retain some of the characteristicsof the

fitter stringsin the parent population.
The primarygenetic operatorfor a GA is
the crossoveroperator.The crossoveroperator is executed in three steps: 1) a pair of
stringsis chosen from the set of copies; 2)
the strings are placed side by side and a
point is randomlychosen somewherealong
the length of the strings;3) the segmentsto
the left of the point are exchangedbetween
the strings.For example,crossoverof 111000
and 010101 after the second position produces the offspring strings 011000 and
110101. Crossover,workingwith reproduction accordingto performance,turns out to
be a powerful way of biasing the system
towardcertainpatterns,buildingblocks,that
are consistentlyassociatedwith above-average performance.
It can be proved(see Holland, 1975) that
GAs are a powerfultechnique for locating
improvementsin complicated high-dimensional spaces.They exploitthe mutualinformation inherent in the population, rather
than simply tryingto exploit the best individualin the population.We can liken each
of the potential buildingblocks to one arm
of an n-armedbandit. Under this interpretation, each successive generation samples
the building blocks in a way that closely
correspondsto the optimal solution of an
n-armedbanditproblem.The GA learns by
biasing the search toward combinationsof
above-averagebuilding blocks. Reproduction and crossover are very simple operations that impose low-information and
processingrequirementson the agents employingthem.
A classifier system (CS) (Holland et al.,

1986) is an adaptiverule-basedsystem that

models its environmentby activatingappropriate clusters of rules. It uses a GA to
revise its rules. Each rule is in condition/
action form, and many rules can be active
simultaneously.The action part of a rule
specifies a message that is to be posted
when the rule is activated. The condition
part of a rule specifies messages that must
be present for it to be activated.Thus, each
rule is a simple message-processingdevice
that emits a specific message when certain
other messages are present. Overt actions
affectingthe environmentare the result of

MAY1991

messages directed to the system's output

devices (effectors),while informationfrom
the environmentis received via messages
generated by its input devices (detectors).
The overall system is computationallycomplete in the sense that any programwritten
in a programminglanguage,such as FORTRAN, can also be implementedby a CS.
A CS-ruledoes not automaticallypost its
messagewhen its conditionpart is satisfied.
Rather, it enters a competitionwith other
rules having satisfied conditions. The outcome of this competition is based on a
quantity,called strength,assigned to each
rule. A rule's strength measures its past
usefulness, and it is modified over time by
one of the system'slearningalgorithms(see
below). There may be more than one winner of the competition at any given
time-hence a cluster of rules can react to
externalsituations.A CS operates on large
numbers of rules, with a small number of
simple,domain-independentmechanisms.It
provides emergent, learned capabilitiesfor
reactingto its environment.
A CS adapts or learns through the application of two well-defined machinelearning algorithms. The first algorithm,
called a bucket-brigade algorithm, adjusts

rule strengths. Each rule is treated as an

intermediateproducer in a complex economy, buyinginput messagesand selling output messages.When a satisfiedrule R succeeds in the competition to post its own
message, it pays the rule(s) that supplied
the messages satisfying its condition part.
This amountis subtractedfromR's strength.
On the next time-step, if other rules are
satisfiedby R's message, and win the competition in turn, then R receives the rules'
payment. R's strength is increased accordingly.The net effect of the two transactions
is R's profit (loss). Some rules also act directly on the environment in a way that
produces direct payoff from the environment to the system. Their strength is increasedin proportionto that payoff.A rule's
strength will increase over time only if it
earns a profit,on average,in these transactions. Generally this happens only if the
rule directly produces payoff, or else belongs to one or more causal chains leading
to payoff.Under appropriateconditions,the

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions

VOL. 81 NO. 2

LEARNING AND ADAPTIVE ECONOMIC BEHAVIOR

369

strengthsassignedby the bucket-brigadealfixed points, and convergence,provideonly

gorithmdo convergeto a useful measureof
an entering wedge. In addition we need a
the rule's contributionsto system perfor- mathematicsthat worksin close conjunction
mance (Hollandet al.).
with computer modeling techniques-one
In order to generate and test new apthat puts more emphasis on combinatorics
proachesto the environment,the CS needs
and algorithms.We requiretechniquesthat
a second learningalgorithm,a rulediscovery emphasizethe emergenceof structure,paralgorithm.A GA can be used for this pur- ticularlyinternalmodels,throughthe generation, combination,and interactionof buildpose, because the rules of a CS can be
represented by strings in an appropriate ing blocks.The presentsituationseems quite
alphabet,and a rule'sstrengthamountsto a
similarto that of evolutionarytheory prior
to the developmentof a mathematicalthemeasure of its performance.The GA, by
formingnew rules in termsof tested, above- ory of genetic selection(R. A. Fisher, 1930).
Though there is nothing like an overall
average building blocks, transfers experience fromthe past to new situations.Plausi- theory, there are some extant pieces of
mathematicsthat are relevant.The schema
ble new rules result-rules to be tested
theorems for genetic algorithms(Holland,
and retained or discarded on the basis of
1975) offer some insight into processes that
their abilityto enhance the performanceof
the CS.
discover and recombinebuilding blocks. It
Under the combined effects of the
appears that schema theorems are special
bucket-brigadeand genetic algorithms,rules cases of a much more general formulation
become coupled in complexnetworks.Clus- of the effects of recombinationin evolution.
ters and hierarchiesof rules emerge. Over This formulationshould bring some of the
time, these substructuresserve as building more sophisticated tools of mathematical
blocks for still more complexsubstructures. genetics to bear on adaptiveagent models.
A CS agent can: 1) generate broad cateMathematicalwork aimed at understanding
the evolution of CS may also be useful.
gories for describing its environment (so
that experiencecan be broughtto bear on
The progressivedevelopmentof hierarchinovel situations);2) progressivelyrefine and
cal organizationcan be treated as the adelaborate the relation between categories dition of levels to a quasi homomorphism
(using experience to make distinctionsand
(Hollandet al.).
associationsnot previouslypossible);3) use
Perpetual novelty can be modeled by a
these categories to build internal models
regularMarkovprocessin whicheach of the
that supply the agent with expectations states has a recurrencetime that is large
about the world;4) treat all internalmodels
withrespectto anyfeasibleobservationtime.
as provisional (subject to confirmationor
Equivalence classes can be imposed and
refutation as experience accumulates);and
used as the states of a derived Markov
5) generate new hypothesesthat are plausi- process (Holland, 1986). Work by Miller
ble in terms of accumulated experience. and S. Forrest (1989), based on S. A.
Moreover, because of the bucket-brigade Kauffman's(1984)studiesof randomgraphs,
algorithm,these activitiescan proceed in an
provides additionalinsights into the emerenvironmentwhere payoffis intermittentor
gent structuresof CSs.
rare.Such capacitiesenable a CS agent that
is not omniscientto act with increasingraIV. Conclusions
tionality.
The AAA researchcomplementsongoing
III. Towards a Mathematics of Complex
theoreticaland empiricalwork,allowingexAdaptive Systems
plorationand analysisof previouslyinaccessible phenomena. What are the future
A mathematicalcalculus appropriateto
prospectsfor this line of inquiry?Earlywork
the studyof complexadaptivesystemsmust
with AAA in economicshas shownthat they
meet distinctive requirements.The usual
can acquire sophisticated behavioral patmathematical tools, exploiting linearity, terns. Observationof the course of learning

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions

370

AEA PAPERS AND PROCEEDINGS

in these AAA has already increased our

understanding of some economic issues.
Even limited AAA open up new avenues
for analyzing decentralized, adaptive, and
emergentsystems.Steady advancesin computationand AAA modelingofferever more
powerful tools for programmingartificial
worlds. By executing these models on a
computer we gain a double advantage:
(i) An experimentalformat allowing free
explorationof system dynamics,with complete control of all conditions;and (ii) an
opportunityto check the various unfolding
behaviorsfor plausibility,a kind of "reality
check." Whether or not agents in such
worlds behave in an optimal manner, the
very act of contemplatingsuch systemswill
lead to importantquestionsand answers.
REFERENCES
Andreoni,J. A. and Miller, J. H., "Auctions
with Adaptive Artificially Intelligent
Agents," Santa Fe Institute Working Paper, No. 90-01-004, 1990.
Arthur,W. B., "A Learning Algorithm that
Replicates Human Learning," Santa Fe
Institute Working Paper, No. 90-026,
1990.
Axelrod,R. M., "The Evolution of Strategies
in the Iterated Prisoner's Dilemma," in
L. D. Davis, ed., Genetic Algorithms and
Simulated Annealing, Los Altos: MorganKaufmann, 1987, 32-41.

MAY1991

Fisher,R. A., The Genetical Theoryof Natural

Selection,Oxford:ClarendonPress, 1930.
Friedman,M., Essays in Positive Economics,

Chicago: University of Chicago Press,

1953.
Holland, J. H., Adaptation in Natural and
Artificial Systems, Ann Arbor: University

of MichiganPress, 1975.
, "A Mathematical Framework for

StudyingLearningin ClassifierSystems,"
in J. D. Farmer et al., eds. Evolution,
Games and Leaming, Amsterdam: North-

Holland, 1986, 307-17.

et al., Induction: Processes of Inference, Leaming, and Discovery, Cam-

bridge:MIT Press, 1986.

Kauffman, S. A., "Emergent Properties in

Randomly Complex Automata,"Physica

D, 1984, 10, 145-56.
Marimon,R., McGratten,E. and Sargent,T. J.,

"Money as a Mediumof Exchangein an

Economy with Artificially Intelligent
Agents," Journal of Economic Dynamics
and Control, May 1990, 14, 329-73.

Miller,J. H., "The Coevolutionof Automata

in the Repeated Prisoner's Dilemma,"
Santa Fe Institute Working Paper, No.

89-003, 1989.
and Forrest, S, "The Dynamical Behavior of Classifier Systems," in J. D.
Schaffer, ed., Proceedings of the Third Intemational Conference on Genetic Algo-

______

rithms. San Mateo: Morgan-Kaufmann,

1989, 304-10.

This content downloaded from 139.80.2.185 on Tue, 23 Jun 2015 12:45:31 UTC
All use subject to JSTOR Terms and Conditions